SpiritCode.blog

How to fix “permission denied” when a Kubernetes initContainer tries to chmod /var/run/secrets/kubernetes.io/serviceaccount on EKS pods with runAsUser 1000

How to fix “permission denied” when a Kubernetes initContainer tries to chmod /var/run/secrets/kubernetes.io/serviceaccount on EKS pods with runAsUser 1000

Learn how to resolve permission denied errors when an initContainer on EKS tries to chmod the serviceaccount token file while using runAsUser 1000.

I was debugging a flaky CI/CD job that spun up an EKS pod, ran an initContainer that needed to tweak the service‑account token file permissions, and then crashed with a classic permission denied error:

chmod: cannot access '/var/run/secrets/kubernetes.io/serviceaccount/token': Permission denied

The pod spec used a securityContext.runAsUser: 1000 because the main container ran as a non‑root application user. The initContainer inherited the same UID, so it couldn’t touch the secret mount, which is owned by root:root with mode 0400. In this post I’ll walk through the exact root cause, why the default Kubernetes behavior trips us up on EKS, and three production‑ready ways to fix it without compromising the security posture of the workload.

Why does my initContainer get “permission denied” on /var/run/secrets/kubernetes.io/serviceaccount?

When you ask Kubernetes to mount a service account token into a pod, the kubelet creates the directory /var/run/secrets/kubernetes.io/serviceaccount on the host, copies the token file, and then bind‑mounts it into each container. By default the mount is read‑only (ro) and the files are owned by root:root with 0400 permissions. This is intentional – the token is a credential and should be protected.

If the pod’s securityContext.runAsUser is set to a non‑root UID (e.g., 1000), every container in the pod, including initContainers, runs as that UID unless you explicitly override it. The initContainer therefore lacks the capability to change file ownership or mode, leading to the chmod failure.

On vanilla Kubernetes you could get away with chmod because many clusters mount the secret as writable (rw) and the default file mode is 0644. EKS, however, follows the stricter defaults (ro + 0400) for compliance reasons, so the problem surfaces quickly.

What are the safe ways to let an initContainer modify the service‑account token file?

There are three patterns I regularly use in production. All of them keep the main application container non‑root while giving the initContainer just enough privilege to do its job.

1️⃣ Run the initContainer as root, but keep the main container as non‑root

The simplest approach is to override the runAsUser only for the initContainer. You do this by adding a securityContext block inside the initContainer spec:

apiVersion: v1
kind: Pod
metadata:
  name: my‑app
spec:
  securityContext:
    runAsUser: 1000        # default for all containers
  initContainers:
  - name: token‑fixer
    image: busybox:latest
    command: ["sh", "-c", "chmod 644 /var/run/secrets/kubernetes.io/serviceaccount/token"]
    securityContext:
      runAsUser: 0          # run as root **only** for this initContainer
    volumeMounts:
    - name: kube‑api‑token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  containers:
  - name: app
    image: my‑app:1.2.3
    # runs as UID 1000 automatically
    volumeMounts:
    - name: kube‑api‑token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  volumes:
  - name: kube‑api‑token
    projected:
      sources:
      - serviceAccountToken: {}

Pro Tip: Keep the runAsUser: 0 scoped to the initContainer only. If you forget to reset it for the main container you’ll unintentionally run your app as root, which defeats the purpose of the securityContext.

Why this works: The initContainer runs as root, can chmod the token file, and then exits. The main container continues to run as UID 1000, preserving the least‑privilege model.

Edge cases: If your cluster uses a PodSecurityPolicy (or the newer Pod Security Standards) that disallows privileged containers, you may need to add the allowPrivilegeEscalation: false flag to the initContainer’s securityContext to stay compliant.

2️⃣ Use an initContainer that mounts the secret as a writable volume via emptyDir

Sometimes you don’t actually need to touch the original token file – you just need a copy with relaxed permissions. You can copy the token into an emptyDir and let the initContainer work on that copy.

apiVersion: v1
kind: Pod
metadata:
  name: my‑app‑copy‑token
spec:
  securityContext:
    runAsUser: 1000
  initContainers:
  - name: copy‑token
    image: busybox
    command: ["sh", "-c", "cp /src/token /dst/token && chmod 644 /dst/token"]
    volumeMounts:
    - name: src-token
      mountPath: /src
    - name: dst-token
      mountPath: /dst
  containers:
  - name: app
    image: my‑app:latest
    env:
    - name: KUBE_TOKEN_PATH
      value: /var/run/secrets/kubernetes.io/serviceaccount/token
    volumeMounts:
    - name: dst-token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  volumes:
  - name: src-token
    projected:
      sources:
      - serviceAccountToken: {}
  - name: dst-token
    emptyDir: {}

Here the original secret remains read‑only (mounted into src-token). The initContainer copies it into an emptyDir (dst-token), changes the mode, and the main container consumes the writable copy.

Pro Tip: When you copy the token, also copy the ca.crt and namespace files if your app expects the full service‑account directory structure.

Why this works: emptyDir is writable by any UID that the pod runs as, so the non‑root initContainer can chmod without needing root privileges.

Edge cases: If your app validates the token’s file mode (rare, but some security‑hardened images do), you’ll need to ensure the copied file matches expectations.

3️⃣ Leverage fsGroup to give the pod’s group write access to the secret mount

Kubernetes offers a securityContext.fsGroup field that changes the group ownership of all volume mounts (including projected service‑account tokens). By setting fsGroup: 1000 (or any GID that matches the app’s supplementary groups), the token file becomes 0640 and is writable by the group.

apiVersion: v1
kind: Pod
metadata:
  name: my‑app‑fsgroup
spec:
  securityContext:
    runAsUser: 1000
    fsGroup: 1000          # apply group ownership to all volumes
  initContainers:
  - name: token‑chmod
    image: busybox
    command: ["sh", "-c", "chmod 664 /var/run/secrets/kubernetes.io/serviceaccount/token"]
    volumeMounts:
    - name: kube‑api‑token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  containers:
  - name: app
    image: my‑app:latest
    volumeMounts:
    - name: kube‑api‑token
      mountPath: /var/run/secrets/kubernetes.io/serviceaccount
  volumes:
  - name: kube‑api‑token
    projected:
      sources:
      - serviceAccountToken: {}

Kubelet will chown the secret files to root:1000 and set the mode to 0640. The initContainer (still UID 1000) can now chmod or even write to the file without needing root.

Pro Tip: If you’re using a Pod Security Standard that requires runAsNonRoot: true, setting fsGroup does not make the container run as root; it only changes the group ownership.

Why this works: fsGroup is applied after the volume is mounted, so the secret’s group is changed before any container starts. Both initContainers and the main container see the updated permissions.

Edge cases: Some compliance frameworks (e.g., PCI DSS) forbid widening secret file permissions beyond 0400. In those environments you may need to stick with the root‑only initContainer approach and audit the change.

How to diagnose the problem locally before pushing a fix

When a CI pipeline fails with the permission error, I always run a quick kubectl exec into the failing pod (if it gets past init) or into the initContainer itself:

kubectl get pod my-app-abcde -n dev -o jsonpath='{.status.initContainerStatuses[0].state.waiting.reason}'
# if it says "CrashLoopBackOff" you can inspect logs
kubectl logs my-app-abcde -c token-fixer -n dev

If the logs show chmod: cannot access … Permission denied, inspect the mount:

kubectl exec -it my-app-abcde -c token-fixer -- ls -l /var/run/secrets/kubernetes.io/serviceaccount

You’ll see something like:

-r-------- 1 root root 1234 Jan 01 00:00 token
-rw-r--r-- 1 root root  123 Jan 01 00:00 ca.crt

The -r-------- indicates the restrictive mode. From there you can decide which of the three patterns above fits your security policy.

Real‑world considerations when applying these fixes in production

  1. Pod Security Policies / PSP replacements – If your cluster enforces runAsNonRoot and disallows runAsUser: 0, the root initContainer pattern will be rejected. Use the fsGroup or emptyDir copy method instead.
  2. IAM role for service accounts (IRSA) – Changing the token file mode does not affect the underlying IAM role, but some AWS SDKs perform extra checks on file ownership. Test the SDK version you ship with after applying the fix.
  3. Immutable Secrets – EKS can mount secrets as immutable (immutable: true). When immutable, you cannot chmod the file after mount. The fsGroup approach still works because the group change happens at mount time, not after.
  4. Auditing – Enable auditLogs for the kube‑apiserver and watch for Patch events on pods that modify securityContext. This helps you verify that only intended pods get the elevated initContainer privileges.
  5. Helm chart upgrades – If you manage the pod via Helm, make sure you add the new securityContext fields to the chart’s values.yaml and bump the chart version. A common mistake is to forget to propagate fsGroup into the podSecurityContext block.

FAQ

How can I change the service‑account token permission without running the initContainer as root?

Set securityContext.fsGroup to a group that the initContainer’s UID belongs to. The kubelet will chown the token file to that group and set it to 0640, allowing a non‑root initContainer to modify it.

Why does EKS mount the service‑account token as read‑only by default?

EKS follows the Kubernetes security best practice of exposing credentials with the least privilege. A read‑only mount (ro) and mode 0400 ensure only the owning process (root) can read the token, protecting it from accidental leaks.

Can I avoid modifying the token file altogether?

Yes. Instead of chmod, copy the token into an emptyDir volume, adjust permissions there, and point your application to the copied path via an environment variable.

Will using fsGroup violate the Pod Security Standards “restricted” level?

No. fsGroup only changes the group ownership of volume files; it does not grant root privileges. It remains compliant with the “restricted” level as long as runAsNonRoot is true.

Does changing the token file mode affect IAM role assumptions with IRSA?

No. The IAM role is bound to the service account object, not the file mode. However, some SDKs may reject a token file they consider insecure, so test after the change.

Conclusion

The permission denied error in an initContainer on EKS is almost always a mismatch between a non‑root runAsUser and the default read‑only, root‑owned service‑account token mount. By either running the initContainer as root, copying the token into a writable emptyDir, or leveraging fsGroup to adjust group ownership, you can resolve the issue while keeping the main application container non‑root.

In my own production clusters, I favor the fsGroup method because it requires the least deviation from the original pod spec and works with most Pod Security Standards out of the box. The root‑only initContainer is a quick fix for legacy workloads, and the copy‑to‑emptyDir pattern is ideal when you need a completely isolated token file.

Pro Tip: After applying any of these changes, run a smoke test that reads the token file (e.g., cat /var/run/secrets/kubernetes.io/serviceaccount/token) from both the initContainer and the main container to verify the permissions are exactly what you expect.

That’s it – a practical, battle‑tested way to stop those “permission denied” crashes and keep your EKS workloads secure.

Stay tuned to SpiritCode for more deep‑dive troubleshooting stories like this.