Runbook: Pod Not Ready
Alert name: PodNotReady
A Kubernetes pod has been in a not-ready state for 5+ minutes.
Diagnostic Steps
-
Identify the pod from the alert labels (
pod,namespace):kubectl describe pod <pod> -n <namespace> --context=k3s-ringtail -
Check events — look for scheduling failures, image pull errors, or probe failures:
kubectl get events -n <namespace> --context=k3s-ringtail --sort-by='.lastTimestamp' | tail -20 -
Check logs:
kubectl logs <pod> -n <namespace> --context=k3s-ringtail --tail=50 -
Check whether the pod can be scheduled.
kubectl topdoes not work here — ringtail’s k3s runs with--disable=metrics-server, so the Metrics API does not exist andkubectl toperrors out.No loss for this alert.
kubectl topreports live utilization, but aPendingpod is a question about requests versus allocatable — the scheduler does not care what is currently in use. Ask Prometheus instead (Grafana → Explore → Prometheus):# What the node has to give kube_node_status_allocatable{cluster="ringtail", resource=~"cpu|memory"} # What is already claimed on it sum by (resource) (kube_pod_container_resource_requests{cluster="ringtail"}) # Free memory on the box, for the OOM-pressure case node_memory_MemAvailable_bytes{cluster="ringtail"} # Which pods are not Running, and what the alert itself sees kube_pod_status_phase{phase!="Running"} > 0 kube_pod_status_ready{condition="false"} > 0There is no cAdvisor scrape either, so per-container live usage (
container_memory_working_set_bytes) is unavailable. If you genuinely need live numbers,kubectl describe nodeprints the allocated-resources table without touching the Metrics API.
Common Causes
- CrashLoopBackOff — app is crashing on startup, check logs
- ImagePullBackOff — container image not found or registry unreachable
- Pending — insufficient resources (CPU/memory), or PVC not bound
- Readiness probe failing — service is running but not healthy
- NFS mount issue — services depending on sifaka (kiwix, transmission, navidrome, jellyfin) will fail if NFS is down
Silencing
- Grafana → Alerting → Silences → Create Silence
- Match
alertname = PodNotReady - Optionally match
namespace = <namespace>to silence a specific service
Related
- deploy-infra-alerting — Alerting pipeline overview
- ringtail — the k3s node, including which built-in components are disabled
- observability — where the Prometheus queries above come from