kubectl debug
Created · Updated
kubectl debug starts a container with its own image and tools to inspect a running podpod The smallest unit Kubernetes runs: one or more containers that run on the same node and share some resources, such as a network namespace. or nodenode A machine, physical or virtual, that runs pods in a Kubernetes cluster. by hand. A pod’s containers are fixed when it’s created, and kubectl exec can only run programs that already exist in a container’s image. Many production images have no shell or tools at all: a distroless image holds just the application and its libraries, so kubectl exec app -- sh fails with executable file not found. The alternative is rebuilding the image with tools and redeploying, which restarts the very pod that showed the problem. Instead, kubectl debug can add an ephemeral container to the running pod: a new container from another image, which brings its tools into the pod as it is.
kubectl debug works in three ways:
- Ephemeral container: adds a container to the running pod. It shares the pod’s network, and with
--target, the processes of one container. - Copy of the pod: creates a new pod from the old one’s spec with changes, such as another command or image.
- Node debugging: creates a pod on a node that runs in the node’s own namespaces, with the node’s root filesystem mounted.
Ephemeral containers
An ephemeral container is declared like an ordinary container but with fewer guarantees. It is never restarted. It can’t have ports, so it can’t have probes either, and it can’t set resources, because a pod’s resource allocations can’t change this way. Once added, it can’t be changed or removed; it stays in the pod’s spec until the pod is deleted. Ephemeral containers are stable since Kubernetes 1.25, and static pods don’t support them.
They are added through their own API endpoint, the pod’s ephemeralcontainers subresource, not by editing the pod’s spec. Permission to update a pod doesn’t include it, and the default admin and edit roles leave it out, so adding one needs an explicit grant on pods/ephemeralcontainers.
Adding one to a pod
A pod app ran the pause image, which holds one binary and no shell, so kubectl exec app -- sh failed. An ephemeral container brought busybox into it:
kubectl debug app -it --image=busybox:1.36 --target=app --profile=generalkubectl debug appAdd a debugging container to pod app.-itAttach to the new container with input and a terminal, for an interactive shell.--image=busybox:1.36The debugging container’s image, which has a shell and tools.--target=appJoin the process namespace of container app.--profile=generalDefault, can be left out. The security settings to apply. general is the default in recent kubectl.
The new container shares the pod’s network namespacenetwork namespace A Linux namespace that gives a group of processes their own network interfaces, IP addresses, routes, and ports. The containers in a pod share one., like every container in a pod: ip addr showed the pod’s IP, 10.244.2.10, and hostname printed app. Processes are separate by default. Without --target, ps listed only the debugging shell. With it, ps listed /pause as process 1, running as user 65535. The target’s filesystem shows through that process: ls /proc/1/root listed the app’s own files, including the pause binary. The general profile gave this targeted container the SYS_PTRACE capability, which lets tools such as strace trace other processes. The container runtime must support --target; without it, the container may not start, or may start with its own process namespace.
After the shell exits, the container stays in the pod with state Terminated. kubectl get pod app still showed 1/1 ready and no restarts: ephemeral containers don’t count toward readiness, and adding one doesn’t restart the pod. A second kubectl debug with the same container name failed: existing ephemeral containers "dbg4" may not be changed. Each session adds a new one.
Copying a pod
Some changes can’t be made to a running pod, such as replacing a crashing container’s command. A pod crash printed config missing and exited with code 1 on each start, so it kept restarting. A copy kept the same image and replaced its command with a shell that stays up:
kubectl debug crash --copy-to=crash-debug --container=crash -- sh -c 'sleep 3600'kubectl debug crashDebug pod crash.--copy-to=crash-debugCreate a new pod named crash-debug from crash’s spec, and leave crash as it is.--container=crashChange this container. Without it, kubectl adds a new container instead.--Ends kubectl’s options; the rest is the new command.sh -c 'sleep 3600'The new command, which keeps the container running to inspect it.
The copy ran on the same image and started cleanly. A copy differs from the original in a few ways:
- It has none of the original’s labels, so a Service or ReplicaSet that selects the original doesn’t pick it up.
--keep-labelskeeps them. - Its liveness, readiness, and startup probes are removed, so a failing probe doesn’t kill it.
--keep-livenessand the matching flags keep them. - Its containers share one process namespace, unless
--share-processes=false. --set-image=crash=busybox:1.37swaps a container’s image, and*=swaps all of them.--same-nodeplaces the copy on the original’s node.
The copy is an ordinary pod that nothing manages, so it stays until it’s deleted.
Debugging a node
kubectl debug can also target a node instead of a pod:
kubectl debug node/kind-worker -it --image=busybox:1.36kubectl debug node/kind-workerCreate a debugging pod on node kind-worker.-itAttach with input and a terminal.--image=busybox:1.36The image the debugging pod runs.
It created pod node-debugger-kind-worker-t5b55, pinned to that node, with hostNetwork, hostPID, and hostIPC set. The node’s root filesystem is mounted at /host, through a hostPath volume. Inside it, ip addr showed the node’s address, 172.21.0.4, and ps showed the node’s processes, including the kubeletkubelet The agent that runs on each node in a Kubernetes cluster and makes sure the containers in its pods are running.. /host/etc/os-release named the node’s own system, Debian 13. The pod isn’t privileged by default, so reading some process information or running chroot /host may fail; --profile=sysadmin makes it privileged. After the shell exits, the pod stays as Completed until it’s deleted.
Profiles
A profile sets the debugging container’s security settings, so it fits the pod security level of its namespace or gets the privileges a task needs:
general: a reasonable set for each kind of debugging, such asSYS_PTRACEwhen targeting a container. The default in recent kubectl.baselineandrestricted: settings that pass the Pod Security Standards’ baseline and restricted policies.netadmin: adds network administrator privileges.sysadmin: a privileged container, with system administrator (root) privileges.legacy: the behavior of kubectl 1.22, which older versions used by default.
--custom takes a file with a partial container spec, such as extra environment variables or a security context, and applies it on top of the profile.
Each mode reaches further than the one before: an ephemeral container sees one pod, a copy runs a changed pod, and a node pod sees everything on the node. Reaching a pod while testing covers connecting to a pod from outside it.
References: Ephemeral containers (Kubernetes docs), Debug running pods (Kubernetes docs), kubectl debug (Kubernetes docs), Share process namespace between containers in a pod (Kubernetes docs), distroless