BlumeOps Docs

Runbook: Service Probe Failure

3 min read

Runbook: Service Probe Failure

Alert name: ServiceProbeFailure

A blackbox HTTP health check has failed for 2+ minutes, meaning a service is not responding to its health endpoint.

Affected Services

This alert covers the in-cluster HTTP services probed by the Alloy blackbox exporter on ringtail’s k3s cluster. The probe targets are defined in argocd/manifests/alloy-ringtail/config.alloy (prometheus.exporter.blackbox):

ServiceHealth Endpoint
argocd/healthz
authentik/-/health/live/
frigate/api/version
grafana/api/health
homepage/
immich/api/server/ping
kiwix/
loki/ready
mealie/api/app/about
miniflux/healthcheck
navidrome/ping
ntfy/v1/health
paperless/accounts/login/
prometheus/-/healthy
shower/ (sent with Host: shower.ops.eblu.me)
teslamate/
tempo/ready
transmission/transmission/web/

ollama is not probed: it is scaled to zero unless explicitly needed, so a probe would fire this alert permanently.

Indri-native and public services (forgejo, zot, devpi, jellyfin, cv) are not in this cluster and are checked directly by mise run services-check, not by this alert.

The failing service is identified by the service label in the alert, extracted from the job label (e.g. integrations/blackbox/immich → immich). To add a service to this alert, add a target block to the blackbox exporter in the Alloy config — no new alert rule is needed, the single label_replace rule covers any integrations/blackbox/* job.

Diagnostic Steps

  1. Check which service is down — the alert label service tells you. You can also run:

    kubectl get pods -n <namespace> --context=k3s-ringtail

  2. Check pod status — look for CrashLoopBackOff, OOMKilled, or pending pods:

    kubectl describe pod -n <namespace> <pod-name> --context=k3s-ringtail

  3. Check pod logs:

    kubectl logs -n <namespace> <pod-name> --context=k3s-ringtail --tail=50

  4. Check if the cluster itself is healthy:

    kubectl get nodes --context=k3s-ringtail
ssh ringtail 'systemctl status k3s'

  5. Check NFS mounts (frigate, immich, kiwix, navidrome, paperless, shower, and transmission depend on sifaka NFS, mounted into pods via NFS PVs — see sifaka-nfs-from-ringtail). A lost mount shows up as pod mount errors:

    kubectl describe pod -n <namespace> <pod-name> --context=k3s-ringtail

Common Causes

  • Pod crashed — check logs, restart with kubectl delete pod
  • NFS mount lost — sifaka offline or its NFS export unreachable. Check pod events for mount errors; see sifaka-nfs-from-ringtail
  • Resource exhaustion — look for OOMKilled in the pod events (step 2). kubectl top does not work here: ringtail’s k3s runs with --disable=metrics-server, so the Metrics API does not exist. For node-level numbers see runbook-pod-not-ready (Prometheus queries, kubectl describe node)
  • k3s down — ssh ringtail 'systemctl status k3s', restart if needed

Silencing

For planned maintenance, silence this alert in Grafana:

  1. Go to Alerting → Silences → Create Silence
  2. Match label alertname = ServiceProbeFailure
  3. Optionally match service = <specific-service> to silence only one
  4. Set duration for your maintenance window

Graph View

Backlinks