crictl is a command-line client for inspecting and debugging containers through the Kubernetes Container Runtime Interface (CRI). CRI gives kubelet a consistent way to start, stop, pull images, and manage containers across runtimes such as containerd. section 2.1.4 section 2.3.1
Use crictl directly on the node being investigated to:
- List running containers with crictl ps, or include stopped and exited containers with crictl ps -a. section 2.3.1
- Inspect a container's state with crictl inspect, especially when a pod is stuck in CrashLoopBackOff. section 2.3.1
- Read container output with crictl logs <containerid> when kubectl logs is unavailable. section 2.3.1
- Check control-plane containers, including kube-apiserver, kube-controller-manager, and kube-scheduler. section 8.2.5
Diagnose an exited kube-apiserver
Run crictl ps -a | grep kube-apiserver, identify the exited container ID, and run crictl logs <containerid>. An error showing connection refusal to 127.0.0.2:2379 indicates an etcd connectivity problem; correcting the kube-apiserver manifest and restarting kubelet brings the container back to Running. section 8.2.5
root@kind-control-plane:~# crictl ps -a | grep api
0f5f860a6aa86 73afaf82c9cc3 3 minutes ago Exited kube-
apiserver 6 fb22d8abc62ab kube-apiserver-kind-control-
plane kube-system
root@kind-control-plane:~# crictl logs 0f5f860a6aa86
(...)
2:2379: connect: connection refused"
W0311 00:17:23.692945 1 logging.go:55] [core] [Channel #2 SubChannel
#3]grpc: addrConn.createTransport failed to connect to {Addr:
"127.0.0.2:2379", ServerName: "127.0.0.2:2379", }. Err: connection
error:
desc = "transport: Error while dialing: dial tcp 127.0.0.2:2379:
connect: connection refused"
F0311 00:17:26.759139 1 instance.go:226] Error creating leases: error
creating storage factory: context deadline exceeded
Inspect the exited kube-apiserver and its logs to connect container state with the underlying etcd connectivity error. section 8.2.5
