Listen to this Post
Introduction:
Production alerts blazing, pods crash-looping, services unreachable, and a Slack channel exploding with “Is it down?”—every DevOps engineer knows this moment. When Kubernetes breaks, panic is the enemy. The difference between a five-minute fix and a five-hour firefight often comes down to the first few commands you run. This article delivers a systematic, battle-tested troubleshooting workflow built around essential `kubectl` commands that let you observe, understand, and act with surgical precision.
Learning Objectives:
- Master the “Observe → Understand → Act” troubleshooting framework for production Kubernetes incidents
- Execute essential `kubectl` commands to diagnose cluster health, node failures, and pod crashes
- Debug networking, storage, and rollout issues using systematic command sequences
- Recover workloads safely with rollbacks and ephemeral debugging containers
You Should Know:
- Observe – Cluster Health and Node Status (The First 10 Seconds)
When the pager wakes you up at 3 AM, resist the urge to dig into application logs immediately. Start at the infrastructure layer. Your first command should be:
kubectl get nodes
This lists all nodes and their statuses. Every node should display Ready. If you see NotReady, Unknown, or SchedulingDisabled, you’ve found your problem—or at least the starting point. For deeper diagnostics:
kubectl describe node <node-1ame> kubectl get node <node-1ame> -o yaml
These reveal taints, conditions (MemoryPressure, DiskPressure, PIDPressure), and recent events that explain why a node is unhealthy. To check overall cluster component health:
kubectl cluster-info kubectl get componentstatuses legacy, may be deprecated in some versions
Pro tip: Run `kubectl get nodes -w` to watch node status changes in real time during an ongoing incident.
2. Observe – Find Failing Pods Fast
Once nodes are confirmed healthy, shift focus to workloads. The fastest way to spot trouble:
kubectl get pods -A | grep -Ev 'Running|Completed'
This filters out healthy pods and shows only those in CrashLoopBackOff, ImagePullBackOff, Pending, Evicted, or `Error` states. For a namespace-specific view:
kubectl get pods -1 <namespace> -o wide
The `-o wide` flag adds node IP and pod IP, essential for cross-referencing with node issues.
- Understand – The Golden Trio: Describe, Logs, Events
Once you’ve identified a problematic pod, apply the systematic “describe → logs → events” workflow. This trio resolves approximately 90% of Kubernetes issues.
Step 1 – Describe the Pod:
kubectl describe pod <pod-1ame> -1 <namespace>
This reveals the pod’s current state, container statuses, restart counts, and—most critically—the Events section at the bottom. Events often pinpoint the exact failure: Insufficient CPU, FailedMount, FailedScheduling, or Liveness probe failed.
Step 2 – Inspect Logs:
kubectl logs <pod-1ame> -1 <namespace>
For multi-container pods, specify the container:
kubectl logs <pod-1ame> -c <container-1ame> -1 <namespace>
Critical flag: When a pod is in CrashLoopBackOff, the current container may not produce logs. Use `–previous` (or -p) to view logs from the last crashed instance:
kubectl logs <pod-1ame> --previous -1 <namespace>
This often reveals the exact error—missing environment variables, configuration errors, or application exceptions—that caused the crash. For real-time tailing during an ongoing issue:
kubectl logs -f <pod-1ame> --since=5m -1 <namespace>
Step 3 – Review Cluster Events:
kubectl get events --all-1amespaces --sort-by='.metadata.creationTimestamp'
Events provide a chronological timeline of cluster activity. Filter for warnings and errors:
kubectl get events --field-selector type!=Normal --all-1amespaces
4. Understand – Debugging CrashLoopBackOff and ImagePullBackOff
CrashLoopBackOff occurs when a container crashes repeatedly, and Kubernetes keeps trying to restart it. The systematic approach:
1. `kubectl get pods` – identify the crashing pod
2. `kubectl describe pod
3. `kubectl logs
Common causes: missing environment variables, incorrect entrypoint commands, failing liveness/readiness probes, or out-of-memory errors. If probes are the culprit, temporarily disable them or adjust `initialDelaySeconds` to give the application more startup time.
ImagePullBackOff / ErrImagePull means Kubernetes can’t pull the container image. Diagnose with:
kubectl describe pod <pod-1ame>
Check the Events section for the exact error—wrong image tag, private registry without credentials, or network issues. For private registries, create a pull secret:
kubectl create secret docker-registry regcred \ --docker-server=your-registry.com \ --docker-username=<user> \ --docker-password=<pass>
Then reference it in your Pod spec’s `imagePullSecrets` field.
5. Understand – Pods Stuck in Pending State
A `Pending` pod means the scheduler can’t place it on a node. Run:
kubectl describe pod <pod-1ame>
Look for events like Insufficient CPU, Insufficient memory, or 0/3 nodes are available. Common causes:
- Resource constraints: Nodes don’t have enough CPU/memory. Check with
kubectl top nodes. - Taints and tolerations: Nodes have taints that the pod doesn’t tolerate.
- Node selector/affinity mismatches: Pod specifications don’t match any node labels.
Solutions: reduce resource requests, add tolerations, adjust node selectors, or scale the cluster.
- Act – Fixing Broken Rollouts and Safe Rollbacks
When a deployment update goes wrong—new pods failing readiness probes, or the rollout stuck—start with:
kubectl rollout status deployment/<deployment-1ame> -1 <namespace>
This shows real-time rollout progress. If stuck, describe the deployment:
kubectl describe deployment <deployment-1ame> -1 <namespace>
Rollback to the previous working revision (zero-downtime, production-safe):
kubectl rollout undo deployment/<deployment-1ame> -1 <namespace>
To roll back to a specific revision:
kubectl rollout history deployment/<deployment-1ame> -1 <namespace> kubectl rollout undo deployment/<deployment-1ame> --to-revision=<N> -1 <namespace>
If you need to restart a deployment without changing its configuration (e.g., to pick up new secrets or config maps):
kubectl rollout restart deployment/<deployment-1ame> -1 <namespace>
This triggers a rolling restart with zero downtime.
7. Act – Debugging Service and Networking Issues
When a service is unreachable, follow this checklist:
Step 1 – Verify the service exists and check its configuration:
kubectl get svc <service-1ame> -1 <namespace> -o wide kubectl describe svc <service-1ame> -1 <namespace>
Step 2 – Check endpoints (this is the critical test):
kubectl get endpoints <service-1ame> -1 <namespace>
If endpoints show <none>, the service selector isn’t matching any ready pods. Verify the selector matches pod labels:
kubectl get pods --show-labels -1 <namespace> kubectl get svc <service-1ame> -o yaml -1 <namespace> | grep selector -A 5
Step 3 – Test DNS resolution from inside a pod:
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default
If DNS fails, check CoreDNS pods:
kubectl get pods -1 kube-system -l k8s-app=kube-dns kubectl logs -1 kube-system -l k8s-app=kube-dns --tail=50
Step 4 – For network policy issues, check if any NetworkPolicies are blocking traffic:
kubectl get networkpolicies -A
8. Act – Troubleshooting Storage (PVC Pending)
When a pod stays in `ContainerCreating` due to volume mount issues, or a PVC is stuck in Pending:
kubectl get pv,pvc --all-1amespaces
Describe the PVC to see why it’s not binding:
kubectl describe pvc <pvc-1ame> -1 <namespace>
Check the Events section for errors like `storageclass.storage.k8s.io “xxx” not found` or no persistent volumes available for this claim. Verify available storage classes:
kubectl get storageclass kubectl describe storageclass <storageclass-1ame>
If using a cloud provider, ensure dynamic provisioning is enabled and the storage class exists.
9. Advanced – Ephemeral Debugging Containers
For deep inspection without modifying production manifests, use `kubectl debug` (available since Kubernetes 1.18):
Inject a debug container into a running pod:
kubectl debug -it <pod-1ame> --image=busybox --target=<container-1ame>
Create a copy of the pod with a debug image:
kubectl debug <pod-1ame> -it --copy-to=<pod-1ame>-debug --container=debug --image=nicolaka/netshoot
The `nicolaka/netshoot` image includes tcpdump, curl, dig, nslookup, netstat, and other network troubleshooting tools.
For a pod that won’t start, create a copy with a modified entry point to get a shell:
kubectl debug <pod-1ame> -it --copy-to=<pod-1ame>-debug --container=app --image=<image> -- /bin/sh
10. The Systematic Workflow Summary
┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ OBSERVE │ ──▶ │ UNDERSTAND │ ──▶ │ ACT │ ├─────────────┤ ├─────────────┤ ├─────────────┤ │ get nodes │ │ describe │ │ rollout undo│ │ get pods │ │ logs │ │ rollout │ │ cluster-info│ │ logs --prev │ │ restart │ │ get events │ │ get events │ │ debug │ └─────────────┘ └─────────────┘ └─────────────┘
What Undercode Say:
- Master the Trinity First: `kubectl describe pod` → `kubectl logs –previous` → `kubectl get events` resolves the vast majority of production issues. These three commands should be muscle memory for every on-call engineer. According to industry data, IT teams spend an average of 34 working days per year resolving Kubernetes issues—systematic debugging slashes that waste.
-
Network Is the Hidden Time Sink: 60% of cluster management time goes to network troubleshooting. Always start with
kubectl get endpoints—an empty endpoints list immediately tells you the service selector is wrong, saving hours of blind debugging. DNS is the leading cause of network problems; test with `nslookup kubernetes.default` before anything else. -
Safe Production Moves: `kubectl rollout restart` gives zero-downtime pod recycling. `kubectl rollout undo` is your emergency escape hatch—use it without hesitation when a deployment goes sideways. Never delete pods manually as a first response; use the rollout commands to let Kubernetes handle the orchestration safely.
Prediction:
+1 The adoption of ephemeral debugging containers (kubectl debug) will become standard practice in 2026–2027, reducing the need for production pod modifications and improving security postures across cloud-1ative environments.
+1 AI-assisted Kubernetes troubleshooting tools will integrate these command sequences, offering natural-language-to-kubectl translation that accelerates mean-time-to-resolution (MTTR) for junior engineers.
-1 As Kubernetes clusters grow more complex with service meshes, eBPF, and multi-cluster networking, the troubleshooting skill gap will widen—organizations that don’t invest in systematic training will face longer outages and higher operational costs.
+1 The “Observe → Understand → Act” framework will evolve into standardized runbooks integrated with observability platforms, turning reactive firefighting into proactive incident prevention through automated health checks and pre-flight validation.
▶️ Related Video (82% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Subhasmita Das – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
