How to safely shut down, work on, and restart ringtail — the NixOS host
that runs the entire k3s cluster and doubles as the gaming PC.
Ringtail is a single node with no failover, so a shutdown is a full
Kubernetes outage. The procedure itself is short (kubelet drains the
pods, systemd does the rest); the value of this page is knowing what
goes dark, what to check first, and what to verify afterwards.
What goes dark
Everything in cluster — see argocd/apps/ for the full list. The ones
that change how you operate during the outage:
SSO is down: Grafana, ArgoCD, heph hub login, anything OIDC-fronted. Already-issued sessions on indri services keep working.
CloudNativePG databases
Immich, Paperless, Mealie, Miniflux, TeslaMate etc. lose their Postgres. CNPG shuts Postgres down cleanly on SIGTERM; no data loss from a normal poweroff.
In-flight container builds fail; the job shows as failed on forge and needs a re-run.
Heph spokes
Both spokes stop syncing; the hub on indri is unaffected. They catch up on boot.
Factorio
Guests are disconnected.
Indri, sifaka, and the Fly proxy are unaffected. Forge, Zot, Caddy, Jellyfin
and the heph hub stay up. Public *.eblu.me routing stays up but every
service that is backed by ringtail returns errors.
Prerequisites
SSH access to ringtail, or a seat at the console.
Tailscale connected (if working remotely).
Shutdown procedure
1. Pre-checks
Nothing is required here — systemd stops everything gracefully — but each of
these turns a surprise into a non-event:
mise run runner-logs # no build in progress on the ringtail runnermise run verify-runs # no horkos-dispatched run in flightargocd app list | grep -v Synced # nothing mid-sync (pin/unpin later if so)ssh ringtail 'sudo k3s kubectl get pods -A | grep -v -E "Running|Completed"'
If the argocd CLI token has expired (invalid session), the same answer
without an SSO round-trip:
Expect a few Error / UnexpectedAdmissionError pods with 0 restarts and an
old age — GPU husks left behind by earlier hard reboots (kubelet readmission
racing the device plugin; see ringtail > Maintenance Notes). A reaper
CronJob in the nvidia-device-plugin app deletes them automatically ~10
minutes after each boot; a pod that is restarting is the thing to look at.
If an agent session is active (a heph task in progress, a PR being built),
let it finish or accept that it dies.
If you are doing this for a kernel update (see manage-lockfile), the
pending generation is already activated and a plain reboot is all that is
needed.
2. Drain transmission (recommended)
Kubelet Graceful Node Shutdown (eblume/blumeops#906) now terminates the pod
on a host reboot — 10 s cap and preStop included — for a normal
systemctl reboot/poweroff; a hard power cut still kills it mid-write. The
drain is the fastest path and leaves the NFS download mount with nothing in
flight.
torrent-ringtail runs syncPolicy.automated: {} with no selfHeal: a bare
kubectl scale is undone by ArgoCD within seconds, so suspend the sync first.
Restore the sync policy on boot — Startup, step 3.
3. Power off
ssh ringtail 'sudo systemctl poweroff'
Or sudo reboot if there is no hardware work to do. From the console, the
same commands in a terminal — Sway has no power menu configured.
This is the whole graceful shutdown: the kubelet detects the shutdown through
a logind delay lock and terminates the pods (grace periods, preStop included)
while the network is up, then systemd stops k3s.service and the heph spokes
and runner, and the zram swap is discarded (it is RAM-backed; nothing to
persist).
Shutdown takes about a minute, plus the kubelet’s drain window (up to the
60 s shutdownGracePeriod) before service teardown. The last two clean
shutdowns (April and May 2026) were each down ~5 minutes wall-clock including
the boot (pre-Graceful-Node-Shutdown).
Observed 2026-09-06: an undrained reboot was held ~10 minutes while the
transmission pod terminated against its NFS download mount. The drained reboot
the same night was down ~2.5 minutes to SSH-back.
Observed 2026-09-08 (journal analysis, eblume/blumeops#906): every
retained reboot since May, timed — the ~10-min hold lives afterJournal stopped, inside systemd-shutdown, where nothing is logged. k3s itself
stopped in 2–3 s and all seven NFS mounts unmounted in under a second in
every boot — the only slow one was the 09-02 reboot, started while sifaka
was already unreachable. And the kubelet never terminated any pod on a host
reboot: k3s sets no shutdownGracePeriod, so the kubelet ran its built-in
0 s default, and the terminationGracePeriodSeconds: 10 cap and the preStop
never ran on the reboot path at all. The Graceful Node Shutdown
setting added to nixos/ringtail/configuration.nix changes that: the next
undrained reboot exercises kubelet’s drain path, and kernel-to-kernel should
stay ~2 min like the 09-03 and 09-06 reboots.
Observed 2026-09-02 (plain sudo reboot after 113 days up, gen 100,
kernel 6.12.87 → 6.12.93, done as a dry run before the RAM swap):
Mark
Time after reboot
What
SSH drops
~0:40
systemd has stopped k3s and sshd
Boot menu
Generation list plus Memtest86+ entry, confirmed on the console. (The pause to look at it is the only reason the next row is not ~1:30.)
SSH back, greetd up
~5:00
k3s.service starts within ~10 s of the graphical target
GPU pods rescheduled
k3s + ~1:00
nvidia-device-plugin restarts twice, then Frigate and Immich ML land fresh Running pods; the old ones stay behind as UnexpectedAdmissionError husks until the reaper CronJob deletes them (~10 min)
Everything settled
k3s + ~3:30
All workloads Running, every ArgoCD app Synced/Healthy, CNPG clusters healthy, systemctl --failed empty
Alerts quiet
k3s + ~5–10 min
LogStreamSilent* and ServiceProbeFailure sit Pending while Prometheus and Loki refill their windows. Pending is expected; Firing after ten minutes is not.
Swap was 11 GB of zram before the reboot and starts empty after; the memory
comparison in the RAM-swap checklist is against the pre-reboot figure.
4. Hardware work (if any)
Unplug the PSU mains cable and wait for the motherboard’s standby LED to go
out before touching anything. Ground yourself. See the RAM-swap checklist
below for the memory case specifically.
Startup procedure
1. Boot
Power on. The systemd-boot menu shows the NixOS generations and a
Memtest86+ entry (see below). The default boots after a short timeout.
There is no TPM and no disk encryption prompt; boot is unattended. The wired
address is static, so ringtail is on the LAN and the tailnet within about a
minute of POST.
2. Log in at the console
Log in through greetd — Sway starts. This matters even if you are working
remotely because two things live in the user systemd manager and need
the eblume session:
eblume-heph-spoke (Erich’s heph sync)
myeve-heph-sync.timer
Linger is enabled, so they start without a graphical login too, but the
1Password desktop app (needed for op on this host) does not. Start it from
the launcher if you will need op.
To check the user units from another host (a plain systemctl --user over
SSH has no session bus):
Note that op on gilbert is also unavailable while nobody is at gilbert
to approve the biometric prompt, and mise run agent-health needs op. If
you walk away from both machines mid-verification, the alert check waits
until you are back.
3. Verify
k3s comes up on its own and ArgoCD reconciles the auto-sync apps. Give it a
few minutes, then:
ssh ringtail 'systemctl --failed' # expect: 0 loaded unitsssh ringtail 'sudo k3s kubectl get pods -A | grep -v -E "Running|Completed"'mise run agent-health # Grafana alert statemise run services-check # fuller check (gilbert)
If the last rebuild bumped the heph pin: both spokes’ hephd was compiled into
the toplevel by the hephaestus flake input (the ringtail-flake-check runner
builds it on ringtail’s own nix store, so the switch reuses the store path —
no background install), and the switch restarts both spokes on unit-file
change. Then confirm:
Both should print active; each spoke’s version is the flake input pin by
construction.
If you drained transmission before the power-off, restore the suspended
sync policy once k3s is up; ArgoCD then brings the pod back on its own. The
drain nulled the wholesyncPolicy, so the restore has to put back
everything argocd/apps/torrent-ringtail.yaml declares under it — the
automated flag, the CreateNamespace option and the managed namespace
labels. Restoring only automated leaves the live Application differing
from its manifest, and the apps app (which owns it) sits OutOfSync with the
argocd-sync alert firing until someone notices (2026-09-22):
ssh ringtail 'sudo k3s kubectl -n argocd patch application torrent-ringtail --type merge -p "{\"spec\":{\"syncPolicy\":{\"automated\":{},\"syncOptions\":[\"CreateNamespace=true\"],\"managedNamespaceMetadata\":{\"labels\":{\"pod-security.kubernetes.io/warn\":\"restricted\",\"pod-security.kubernetes.io/audit\":\"restricted\"}}}}}"'argocd app sync torrent-ringtail # re-applies the namespace labels; auto-sync alone won't re-run on an unchanged revisionargocd app get torrent-ringtail # expect Synced/Healthy — and `argocd app get apps` should be Synced too
Things that commonly need a nudge:
Pods stuck ContainerCreating on the GPU node — the NVIDIA device
plugin registers after the driver loads; wait, then check
nvidia-device-plugin.
GPU UnexpectedAdmissionError husks persisting — the reaper CronJob in
the nvidia-device-plugin app clears them ~10 min after boot; if they are
still there past that, check the plugin DaemonSet and the reaper’s Job logs
(kubectl -n nvidia-device-plugin get jobs).
Frigate or Immich CrashLoopBackOff — usually waiting on its CNPG
cluster to finish recovery; watch kubectl get cluster -A.
ArgoCD Unknown health on everything — ArgoCD itself is still starting.
Not an error.
Manual apps (apps, argocd, cloudnative-pg-ringtail,
external-secrets-crds-ringtail) do not sync on boot — they were
not meant to, and a reboot does not need them to. Only sync if their
manifests changed while ringtail was down.
After an unclean reboot
If ringtail came back on its own — nobody ran reboot, and the boot log
shows fsck journal recovery on every disk plus journald “was not properly
shut down” — read the pstore first.kernel.panic_on_oops=1 and
kernel.panic=10 mean a kernel oops reboots the host ten seconds later
instead of hanging, and the panic’s dmesg tail is written to EFI variables
and archived by systemd-pstore on the next boot:
ssh ringtail 'journalctl --list-boots | tail -3' # when each boot endedssh ringtail 'sudo ls -la /var/lib/systemd/pstore/' # one numbered dir per panicssh ringtail 'sudo grep -h -A12 "BUG:\|Kernel panic" /var/lib/systemd/pstore/*/*/dmesg*'
The pstore record names the panicking task (Comm:), the faulting function
(RIP:) and the call trace — that is the cause, and it is usually the whole
investigation. An empty /var/lib/systemd/pstore/ (and /sys/fs/pstore/)
after an unclean boot points the other way: power loss or a hardware reset,
which no log will explain — check the UPS (power) and the PSU.
Only then look at the previous boot’s journal tail for what was running
(journalctl -b -1 -n 200), OOM-killer lines (journalctl -b -1 -k | grep -i oom), and whether a ringtail-rebuild was mid-switch (mise run verify-runs).
A red or canceled ringtail-rebuild run does not mean the apply failed — every
nixpkgs update rewrites the priv runner’s unit file, so such an apply
restarts the runner and kills its own job after the profile has applied.
Before re-dispatching, check whether the switch finished:
ssh ringtail 'nixos-rebuild list-generations | head -3' # newest generationssh ringtail 'stat -c %y /etc/nixos' # when the profile link movedssh ringtail 'sudo tail -25 /var/log/ringtail-apply/<sha>.log' # the apply's own verdict
Genuinely interrupted switches (power loss, kernel panic) still need a
re-run — the run’s warrant will show failed.
This is the procedure that closed the 2026-09-10 double reset in ten
minutes: both pstore records showed the same beyla BPF uprobe fault
(obi_protocol_tcp, once on a fresh tailscaled sidecar, once on a postgres
backend). The record is in ringtail > Maintenance Notes; the beyla
re-enable and its canary are eblume/blumeops#983.
RAM-swap checklist
Reboot once with the old RAM first. A host that has been up for months
has several generations and probably a kernel bump queued behind the next
boot. Doing a plain sudo reboot before opening the case separates “does the
current generation boot and bring k3s up” from “does the new kit train”, and
proves the Memtest86+ entry actually appears in the menu before you depend on
it overnight. It costs ~5 minutes; the 2026-09-02 dry run above is what that
looks like when it goes well.
Written for the 2026-09 upgrade from 4x8 GB Corsair CMK16GX4M2B3200C16
(XMP/DOCP 3200 CL16 1.35 V) to 2x32 GB Crucial CT2K32G4DFD832A (JEDEC 3200
CL22 1.2 V, dual-rank). Adjust for whatever the next kit is.
Before shutdown
Make sure the Memtest86+ boot entry is present (sudo ls /boot/EFI —
/boot is root-only — or check the boot menu on the next reboot). It is declared in
nixos/ringtail/configuration.nix; if it is missing, do the one-time
bootstrap first — clone blumeops into /etc/blumeops, mise run seed-secrets ringtail, then the first apply (see
docs/reference/operations/disaster-recovery.md) — you cannot add a boot
entry once the box is
down.
(dmidecode is not installed system-wide; the nix-build path is the
least-effort way to run it as root.)
Physical
Two sticks go in DIMM_A2 and DIMM_B2 — the grey pair, second and
fourth slots counting from the CPU. That is ASUS’s dual-channel
recommendation for this board and the slot names match what the BIOS
reports in DMI.
The slots are ASUS Q-DIMM: one moving latch at the top, a fixed hook
at the bottom (the end away from the latch). A stick is only seated when
it has snapped into the fixed hook too — there is a distinct second
detent click at the bottom end, separate from the latch closing at the
top. Press straight down on both ends. On the 2026-09-02 swap, A2
(the slot nearer the CPU cooler) went in with the top latch closed but the
bottom end unseated, and the BIOS reported 32768 MB; Tool → ASUS SPD
Information in the BIOS shows which slot is empty, and a reseat fixed
it. Both sticks should sit at identical height with no gold showing.
Old sticks: bag them with the part number. DDR4 resale is worth it.
First boot
Enter the BIOS (Del at POST; F7 for Advanced Mode). On the Extreme
Tweaker tab (ROG’s name for Ai Tweaker), set Ai Overclock Tuner from
D.O.C.P. to Auto — that is the DOCP switch; the D.O.C.P. profile row
disappears when it is off. Check Total Memory reads the full new capacity
(65536 MB for 2x32) before saving; 32768 means a stick is not seated.
The Crucial kit has no XMP profile; its JEDEC table is already 3200 CL22
1.2 V. Leaving a stale DOCP profile selected is the most likely cause of a
failed POST. The Crosshair VI Hero has a MemOK! button and a Q-code
display if it will not train at all.
Save, reboot, pick Memtest86+ in the boot menu, and let it run
overnight (at least 4 full passes). Zero errors or the kit goes back —
do this inside the retailer’s return window. One pass over 64 GB takes
roughly 2–3 hours on the 1700X. If the box is needed the same day, run
it for as long as you can (the first hour covers the tests that catch
dead cells and address faults), Esc out with zero errors, and file the
full overnight run as a heph task dated inside the return window.
If errors appear, look at the address pattern before blaming the kit:
scattered addresses climbing steadily point at the memory controller
running faster than it can hold — drop Memory Frequency one step and
re-test — while a tight cluster points at a stick. Isolate a stick by
testing each alone in A2.
Boot NixOS. Verify capacity, slots and trained speed:
Expect 32 GB in DIMM_A2 and DIMM_B2, No Module Installed in A1/B1,
part number CT32G4DFD832A.M16FF (Micron), Rank 2. Expect 2400 MT/s.
On Auto the BIOS trains this kit to its 3200 JEDEC table and it looks
fine, but the Ryzen 7 1700X memory controller is only rated 2400 for two
dual-rank DIMMs. On 2026-09-02 Memtest86+ at 3200 hit 68 errors within
minutes (test 2, scattered addresses = controller, not cells); at
Extreme Tweaker → Memory Frequency = DDR4-2400 it ran clean through
test 6. Capacity was the goal; 2400 is where this CPU lives with 64 GB.
If a BIOS reset puts the frequency back on Auto, re-run memtest before
trusting the box.
Update the RAM row and the “RAM speed” maintenance note in ringtail.
Run the startup verification above. Then watch swap over the following
days — the whole point was to stop living in zram (12.7 GB in use the day
before the swap):
mise run agent-metrics 'node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes'