Provision Indri
indri runs a nix-darwin system profile (the darwin/indri/ flake) plus
the ansible roles it has not absorbed yet (secret placement, source builds,
most services). mise run provision-indri is the single apply path, shaped
like ringtail’s: two guards, then the play.
What the task does
nix flake lock— it only resolves inputs (never evaluates darwinConfigurations), so it works from any controller: native nix where present, thenixos/nixcontainer on gilbert, which has no nix. Ifdarwin/indri/flake.lockchanges, the task stages it and exits: commit, push, re-run.- Fail if HEAD is not pushed to origin. An unreachable forge fails here too, loudly, with git’s own error.
ansible-playbook playbooks/indri.yml -e "indri_commit=<sha>": controller-sideopreads, then therebuildtasks (below), then the service roles. The rebuild tasks sit at the end ofpre_tasks— ansible runstasks:afterroles:, so that is the only place “before the roles” is true.
Steps 1–2 exist to bind indri_commit for the rebuild checkout, so a
role-only run — --tags <role> without rebuild, or --skip-tags rebuild — skips both guards and says so. The pushed-HEAD guard reaches
the forge through caddy on indri, and the caddy rollback re-write is
itself a role-only run: the PR 6 window (2026-09-18) found the guard
failing silently, with the forge unreachable, exactly when the re-write
was the recovery (blumeops#1125). A rebuild cannot be the caddy recovery
either — indri’s checkout fetches https://forge.ops.eblu.me, also
through caddy — so the guards lose nothing by stepping aside.
The rebuild tag (zero-prompt apply)
The checkout and rebuild tasks are tagged rebuild, so
mise run provision-indri -- --tags rebuild applies a new generation with no
1Password prompts (a full run costs ~20, one per op read). The tasks
checkout the bound SHA into a root-owned /etc/blumeops (HTTPS from the
forge, forced) and run darwin-rebuild switch --flake /etc/blumeops/darwin/indri#indri detached via sudo -H nohup, cwd pinned to / so the runner’s per-job workdir cleanup can’t take it out from under the build with nix: cannot get cwd — log at
/var/log/indri-rebuild/<sha>.log plus a .status sidecar (the exit
code, written when the switch ends) — then wait (bounded, 60 min) and fail
with the log embedded on non-zero exit. -e indri_rebuild_wait=false
skips the wait (the warrant apply uses it, below); human runs keep it,
because the service roles need the new generation before they run. -H
matters: macOS sudo keeps $HOME, and root’s nix build would otherwise
leave root-owned files in ~erichblume/.cache/nix that break the
user-run flake check and the CI job on the indri runner.
The rebuild path also converges /etc/resolver/ts.net before the switch:
its resolver-write pre_task is tagged [rebuild, tailnet-dns] and writes
it as a real file — a store symlink would dangle in the pre-Nix-
Store-mount window and mDNSResponder would never rescan it (see the
Reboot test below).
The play resolves darwin-rebuild from /run/current-system, falling back
to the system profile (/nix/var/nix/profiles/system/sw/bin). Before the
first switch there is no /run/current-system at all (it appears only after
an activation has run); after that it survives reboots (the first reboot
test initially found BTM disallowing the daemon; see below).
If a run dies mid-rebuild (session drop, pod replacement, the wait
timing out), the switch keeps running detached and will have left
/var/log/indri-rebuild/<sha>.status (started <epoch> while in flight, then the switch exit code, or a recorded skip verdict) and .log. The play
only rebuilds when the checkout changes, so re-running the same commit
reports clean and does not retry: read the status file before re-applying,
and fix forward with a new commit if the switch failed. A failed switch
now also ntfys ops-approvals on its own (the
mcquack.eblume.indri-rebuild-watcher launchd agent).
Warrant-gated apply
The rebuild tag is also the apply path of the provision-indri.yaml
workflow, so a generation flip is a warrant approval instead of a window.
mise run request-run provision-indri.yaml <full-sha> -i revision=<full-sha> --why "…" files the request (PR comment, heph task, Horkos queue); the
approval dispatches the workflow on the indri runner, which runs exactly
this path headless — the task guards, the checkout, the detached switch —
with indri_rebuild_wait=false: the run launches the switch and exits.
A green run means the switch launched, not that it succeeded. The job
cannot wait: a switch that changes the forgejo-runner agent reloads it,
and launchd kills the running job ~60 s later (see §Forgejo runner
below). A content-identical SHA creates no new generation at all — nix
dedupes to the existing store path and the profile never switches; the
no-op run is the warrant path’s own drill.
If an applied change does not show up, check the switch’s outcome on indri:
ssh indri cat /var/log/indri-rebuild/<sha>.status # 0 = success
ssh indri tail -100 /var/log/indri-rebuild/<sha>.logNo .status yet means the switch is still running. A non-zero status
leaves indri on its previous generation; fix forward with a new commit.
The warrant applies a generation, not a role: the dispatch is always
--tags rebuild, so the job reads no vault and the full provision (roles,
op read) stays a human window. The task’s pushed-HEAD guard is
detached-aware — on a detached checkout it verifies origin/main ancestry
instead of branch-tip equality — so a non-tip merged SHA passes, and
rolling back a flip’s generation (re-apply the previous SHA, then
re-run the gated role) starts with a window-free step.
Toolchain
The generation owns the global mise config (environment.etc."mise/config.toml"
in darwin/indri/configuration.nix, symlinked into the user home by the
postActivation fragment — see indri §Toolchain). The generation also
owns the mise install itself: mise is in homebrew.brews, and the
activation’s brew bundle is additive (it never uninstalls and never
upgrades). A toolchain-only PR (a
pin bump) therefore applies with the usual mise run provision-indri -- --tags rebuild: no service-role tag, and no plist flips. Rollback is a plain
darwin-rebuild --rollback — the fragment re-links the previous generation’s
config, and tool installs are additive, so rolling back never removes a tool.
See provision §Rolling back a service flip for what --rollback does and
does not do.
Metrics agents
The four *-metrics collectors (borgmatic, forgejo, jellyfin, zot) are
nix-managed user agents (PR 4 of the series) at the same labels and plist
paths the ansible roles used, writing the same .prom files into alloy’s
node_exporter textfile dir and the same /opt/homebrew/var/log logs.
Applying the flip is the usual mise run provision-indri -- --tags rebuild
(no service-role tag): activation swaps each plist in place — one reload,
never dual-loaded. The API key files (~/.forgejo-api-key,
~/.jellyfin-api-key) stay controller-side op placement: the roles’
key-file tasks are not gated. Rollback per §Rolling back a service flip,
with all four roles’ gates flipped.
An unload leaves the .prom files in place and node_exporter keeps
serving them, so Prometheus never sees a gap: for these textfile
collectors the outage signal is node_textfile_mtime_seconds aging
(and the TextfileStale alert once it crosses threshold), not a scrape
gap — a window’s “no gaps in the tails” verification therefore means
no gap and mtime fresh.
Zot registry
The zot registry is the first real daemon the series moves (PR 5): a
long-running process, not a collector. The unit is a nix-managed user
agent (mcquack.eblume.zot) at the same label and plist path the
ansible role used; the source-built binary stays at ~/code/3rd/zot
(out of nix’s scope) and the config.json / oidc-credentials.json
files stay role-rendered — the role’s gate (zot_ansible_managed)
covers only the plist + load tasks. Applying the flip is the usual
mise run provision-indri -- --tags rebuild (no service-role tag):
activation writes the plist in place and reloads the agent once. What
changes the drill: with the unit unloaded, zot is down — the registry
and every image pull/push behind it — until something reloads the
unit, so the rollback drill below is planned around a real outage.
Rollback per §Rolling back a service flip, with the role’s gate
flipped.
Caddy
Caddy is the widest daemon the series moves (PR 6): it fronts every
*.ops.eblu.me endpoint and the L4 routes (forge ssh 2222, postgres
5433/5434, the sifaka exporter ports). The unit is a nix-managed user agent
(mcquack.eblume.caddy) at the same label and plist path the ansible
role used. The binary is a nixpkgs caddy.withPlugins build in the
generation’s environment.systemPackages; the Caddyfile, wrapper script
and Gandi token file stay role-rendered, and the role’s gate
(caddy_ansible_managed) covers only the plist + load tasks.
The binary flip (blumeops#1275) is two steps, in order: (1) the usual
warrant apply mise run provision-indri -- --tags rebuild at the merge
SHA — the package lands in the generation’s closure, but the plist did
not change, so zero unit reloads are expected; (2) one role-only
mise run provision-indri -- --tags caddy from gilbert (it reads the
Gandi PAT from 1Password) — it re-renders the wrapper, which now execs
/nix/var/nix/profiles/system/sw/bin/caddy, and the role’s restart
handler reloads the agent. That restart is the flip. Because the
switch is plist-unchanged, the pre-/nix boot race (blumeops#1225) never
triggers EX_CONFIG here: ProgramArguments[0] is the on-disk wrapper,
not a store path — bash starts at boot, the exec fails 127 during the
window, and KeepAlive respawns until /nix mounts.
The drill’s blast radius is everything caddy fronts: with the unit
unloaded, the forge API and ssh, the registry, and every image pull
from the cluster are down, and indri’s own runner cannot reach
forge.ops.eblu.me — so no CI runs land during the drill either.
Ansible itself is unaffected (it reaches indri over tailscale ssh, not
through caddy) and op is unaffected; the provision-indri guards are
skipped for the role-only re-write (see §What the task does), which is
what lets it run while the forge is unreachable. After the forward
switch, verify every endpoint and the L4 routes, per the plan.
Rollback per §Rolling back a service flip, with the role’s gate flipped.
Forgejo runner
The forgejo runner’s binary comes from nixpkgs: the generation runs the
nixpkgs forgejo-runner (13.1.0, the rev the flake’s nixpkgs input pins
for everything else) at the label
(mcquack.eblume.forgejo-runner). The config.yaml (runner token,
labels, the forge-ci GitHub PAT) and the runner home are generation-owned
since eblume/blumeops#1528: activation renders the config from
forge/runners.json + the seed-secrets identity files, and a content
change kickstarts the agent. The source checkout at
~/code/3rd/forgejo-runner stays on disk only as the build runner’s
rollback re-write staging source; the cache prune/sweep agents are
generation-owned (the flake’s launchd.user.agents."mcquack.eblume.runner-cache-sweep"
and launchd.user.agents."mcquack.eblume.runner-cache-prune"). Applying a
change is the usual mise run provision-indri -- --tags rebuild (no
service-role tag).
The drill’s blast radius is CI dispatch only: with the unit unloaded,
indri-label jobs queue on forge until something reloads the unit, but
forge, the registry and every *.ops.eblu.me endpoint stay up (unlike
caddy). A plist-changing switch reloads the agent and launchd kills the
in-flight forge job within ~60 s of the reload: launchd clamps
ExitTimeOut to 60 s in the per-user gui domain, so the plist’s
declared value is not a real drain window (measured on macOS 26 — the
plist says 10800, launchctl print reports 60, and agents without the
key report the 5 s default). What survives is AbandonProcessGroup:
the detached darwin-rebuild (nohup, same process group) is not killed
with the agent, so the box still converges. The warrant apply is
therefore fire-and-forget (eblume/blumeops#1266): its run ends before the
reload, and the outcome is read from the .status sidecar (§Warrant-gated
apply). A human --tags rebuild run from gilbert waits over SSH and is
unaffected. A config.yaml content change (a label, capacity, or the
identity files) kicks the agent the same way, so the ~60 s in-flight-job
caveat covers those applies too. Rollback per §Rolling back a service
flip.
Forgejo
Forgejo is the last unit the series moves (PR 8), and the most critical
service: the generation runs the source-built binary
(~/code/3rd/forgejo/forgejo — the mirror lineage; nixpkgs’ forgejo is
unavailable on aarch64-darwin, so there is no lineage choice, and it is
revisited at the next forgejo upgrade) at the same label and plist path the
ansible role used (mcquack.eblume.forgejo). The app.ini, the work path
and the version build stay role-rendered — the role still does the
source-build upgrade path (upgrade-forgejo); its gate
(forgejo_ansible_managed) covers only the plist + load tasks. Applying
the flip is the usual mise run provision-indri -- --tags rebuild (no
service-role tag): activation writes the plist in place and reloads the
agent once.
The drill’s blast radius is the whole forge: with the unit unloaded, the forge API and the built-in git ssh (2222) are down, as are every forge-bound git op from a talos session and every CI run; caddy stays up (502s, not connection failures) and the registry stays up (zot is a separate unit). Plan the window as for caddy, and for a quiet moment — no CI run or talos session in flight. After the forward switch, verify forge UI + API, a git push, ssh 2222, a registry push, and a live talos session surviving the flip end-to-end. Rollback per §Rolling back a service flip, with the role’s gate flipped.
Jellyfin
Jellyfin is PR 9 of the series (part of eblume/blumeops#1291), and the
first flip whose Label lacks the .eblume. segment: the role always
rendered mcquack.jellyfin, and the in-place swap keeps it verbatim
(alloy’s log tails, logrotate’s globs and restart-indri key on the
name and the mcquack.jellyfin.{out,err}.log filenames).
The generation runs the DMG-installed binary through the stable symlink
~/opt/jellyfin-current (kept by the role’s ungated tasks — nixpkgs’
jellyfin is a server package, not the app bundle the unit execs).
Installs are versioned and side by side, so a bump is role-only:
mise run provision-indri -- --tags jellyfin installs next to the old
version, repoints the symlink and restarts once via the handler — the
nix unit never changes and the gate never flips. First-flip ordering:
on the live box the generation flip must come only after
~/opt/jellyfin-current exists — that is why the flip’s ## Human steps (issue eblume/blumeops#1291) runs the role first
(mise run provision-indri -- --tags jellyfin) before the rebuild warrant.
Applying the flip is the usual mise run provision-indri -- --tags rebuild (no service-role tag): activation writes the plist in place
and reloads the agent once. The dedicated blackbox probe
(eblume/blumeops#1230) is the canary; a dead agent also shows in the
probe gap, unlike a quiet daemon. Rollback per §Rolling back a service
flip, with the role’s gate flipped.
Devpi
Devpi is PR 10 of the series (part of eblume/blumeops#1291), unit-in-nix,
venv-stays-role-rendered: the generation runs the uv-managed venv’s
devpi-server at /Users/erichblume/devpi/venv (the zot/forgejo
source-path precedent — nixpkgs’ devpi 6.19.2 is older than the role’s
6.20.3 + devpi-web, so a store flip would be a downgrade, the same shape
as the plan’s package eval) at the same label and plist path the role
used (mcquack.eblume.devpi). The venv build (uv), the pip install, the
server-dir and the devpi-init seeding stay role-rendered — the role’s
gate (devpi_ansible_managed) covers only the plist + load tasks.
Applying the flip is the usual mise run provision-indri -- --tags rebuild (no service-role tag): activation writes the plist in place and
reloads the agent once.
The drill’s blast radius is the PyPI proxy: with the unit unloaded,
package installs routed through pypi.ops.eblu.me fail — while forge,
the registry and every other endpoint stay up. After the forward switch,
verify that devpi-server answers on 127.0.0.1:3141 (e.g. curl -s http://127.0.0.1:3141/ or the +simple index). Rollback per §Rolling
back a service flip, with the role’s gate flipped.
Borgmatic
Borgmatic is PR 11 of the series (part of eblume/blumeops#1291): all four backup
LaunchAgents move to the generation at the role’s labels and plist paths —
mcquack.eblume.borgmatic (daily 02:00, archive tier: zk + Documents via
config.yaml plus the talos-data config), mcquack.eblume.borgmatic-ops
(daily 03:00, operational tier), mcquack.eblume.borgmatic-photos (daily
04:00) and mcquack.eblume.borgmatic-verify-photos (weekly Tuesday +
2nd-of-month verification) — the last two being agents upstream added after
this PR’s base. The backup units exec the mise pipx latest symlink
(borgmatic 2.1.7; nixpkgs 26.05 has 2.1.5 — a store flip would be a
downgrade, per the plan-cycle eval), so the binary stays role-side and the
units stay in the #1225 safe class, the same shape as devpi’s venv;
verify-photos execs the pipx venv python, which spawns the role’s verification script as a subprocess, so the live /Volumes/photos read is attributed to the TCC-granted identity rather than /bin/bash. Config
yamls, .pgpass, BorgBase key, k8s dump helpers and the mise install stay
role-rendered — the role’s gate (borgmatic_ansible_managed) covers only the
plist + load tasks. Applying the flip is the usual mise run provision-indri -- --tags rebuild (no service-role tag): activation writes each plist in
place and reloads once.
The drill’s blast radius is the box’s BACKUP windows (sifaka-local + BorgBase
offsite, plus the Immich photo library offsite): a broken flip means missed
backup windows, and nothing in the standing checks watches backup freshness,
so the post-apply witness includes one real or deliberately triggered
borgmatic create completing (e.g. launchctl kickstart gui/$(id -u)/mcquack.eblume.borgmatic and confirm a fresh archive / fresh
borgmatic.prom). The main unit mirrors the role’s plist key-for-key,
including both --config args (config.yaml and the never-pruned
talos-data.yaml tier, per the #1441 tier split); a witness should confirm a
fresh talos-data archive alongside the sifaka-local one. Rollback per
§Rolling back a service flip, with the role’s gate flipped.
Alloy
Alloy is PR 12 of the series (part of eblume/blumeops#1291): the
mcquack.eblume.alloy LaunchAgent and the root
mcquack.eblume.macos-power-metrics LaunchDaemon move to the generation at
the role’s labels and plist paths, key-for-key identical to the role’s
renders. The agent execs the source-built CGO binary at ~/.local/bin/alloy
(nixpkgs’ grafana-alloy builds with the netgo tag — pure-Go DNS, which
breaks Tailscale MagicDNS — and is older, v1.16 vs the box’s v1.19.2; a
store flip is both a downgrade and a resolver regression, so the binary
stays role-side, devpi-venv precedent, 1225-safe). config.alloy, the storage dir, the textfile dir, the log tails,
the retired-collector tombstone and the macos-power-metrics.sh script stay
role-rendered — the gate (alloy_ansible_managed) covers only the plist +
load tasks, for both units. Applying the flip is the usual
mise run provision-indri -- --tags rebuild (no service-role tag):
activation writes each plist in place and reloads once.
Blast radius is the box’s metrics flow itself: a broken flip silently blinds indri (the blackbox probes witness the fronts, not the telemetry), so the post-apply witness is metrics continuing to flow to prometheus (scrape + Loki push + log tails). The daemon has no post-reboot gap: nix-darwin activation writes the plist as a real file into /Library/LaunchDaemons and launchd re-registers it at every boot (a #1225 EX_CONFIG-never-retried failure needs a store argv0, and the script is role-rendered outside /nix), so the plan cycle’s boot-gap concern resolves to nothing. The one lost-samples window is the rollback drill: the rolled-back generation stops declaring the daemon, and the system launchd phase unloads + deletes it until the gated role re-run rewrites and loads the ansible plist. Rollback per §Rolling back a service flip, with the role’s gate flipped.
Pre-apply check: indri-flake-check
mise run indri-flake-check builds .#darwinConfigurations.indri.system on
indri itself, at the local HEAD — the exact commit provision-indri would
deploy. It must run on the box: no off-box host can evaluate aarch64-darwin.
Run it from any blumeops checkout before a window.
It does not trust ssh’s rc: the remote script prints
FLAKE-CHECK-OK <commit> only after the build and cleanup succeed, and
the local side fails the check if that exact line is absent from the
captured output, because Tailscale SSH on indri exits 0 for any remote
failure (indri §Nix).
CI coupling
The Indri Flake Check workflow (.forgejo/workflows/indri-flake-check.yaml)
runs the same build on the indri runner when a PR or a push to main
touches darwin/indri/ (or the workflow file itself), against indri’s own
store — the check lives where the target platform is, because ringtail
cannot evaluate aarch64-darwin. The runner’s PATH carries
/nix/var/nix/profiles/default/bin for this (the flake’s
mcquack.eblume.forgejo-runner unit; the role’s template carries the same
PATH for the rollback re-write); erichblume is not a trusted nix user, which
is fine for a flake build.
There is no dry run of a switch
nix-darwin’s activate script starts with #!/usr/bin/env -i bash, which
discards the environment — including the checkActivation=1 that
darwin-rebuild check sets — so any invocation of activate is a full
activation as root. (On 2026-09-16 a “no-write rehearsal” built on that
flag activated the first generation outside the play; no damage, but see
indri §Nix for the host-key consequence.) Rehearse with
indri-flake-check (eval + build) and by reading the abort conditions in
the built activate script; treat everything else as the real switch.
First switch (one-way)
The first generation changes only the target of /etc/static (equivalent
content, re-owned from the dormant 25.05 generation) and declares no
services or agents. The switch is one-way by necessity: the old generation
(system-14) must never be re-activated — its etc check carries the
pre-Tahoe zsh hash list and aborts on the post-Tahoe /etc/zshrc, and it
declares services.tailscale, which would load a second tailscaled beside
the Homebrew root daemon.
Step 0, before the window (human, at indri’s console — a Terminal on
the box, not over ssh): the Determinate upgrade —
sudo installer -pkg Determinate.pkg -target / from
https://install.determinate.systems/determinate-pkg/stable/Universal —
with a tar -h of /etc/static as insurance, then re-verify
nix store info, a trivial nix build, and
launchctl list | grep mcquack unchanged.
Why the console: macOS Tahoe TCC-protects /etc/fstab, and the
responsible process for a Tailscale SSH session is Homebrew tailscaled,
which has no Full Disk Access — so over ssh the installer’s
create_fstab_entry step dies with Operation not permitted (root is
irrelevant to TCC), after its uninstall phase 1 has already removed the
systems.determinate.nix-store boot-mount plist. If that happens: do not
reboot; park the partial receipt (sudo mv /nix/receipt.json /nix/.partial-receipt.json) and run the same install line from
Terminal.app on indri — sudo /nix/nix-installer install macos --no-confirm --encrypt true --ssl-cert-file /etc/nix/macos-keychain.crt --determinate. Under --no-confirm a failed install never reverts (the
volume is safe), but it copies itself to /nix/nix-installer, truncating
that binary to zero bytes if it was that file — fetch the release binary
again if so.
The window (done 2026-09-16; kept as the record of what was verified):
mise run provision-indri(full run). The first switch is driven by the old generation’sdarwin-rebuild(the fallback path), which still callsactivate-user; 26.05 ships that as a deprecated stub, so the log shows a redactivate-user is deprecatedwarning beforesetting up /etc.... That is expected, not a failure — the.statusfile decides.- Verify:
readlink /etc/staticpoints at the new generation;scutil --dnsstill showsts.net→ 100.100.100.100;launchctl print gui/501unchanged. The stale/Library/LaunchDaemons/com.tailscale.tailscaled.plistmust be gone (activation removed it itself on 2026-09-16;sudo rmit by hand if it survives — it is not loaded). - Forge API responds, a registry push works,
mise run agent-healthpasses. - Safety, immediately after
readlink /etc/statichas moved: deletesystem-3…14(nix-env -p /nix/var/nix/profiles/system --delete-generations old), the per-userhome-manager-*links (/etc/profiles/per-user/{erichblume,root}) and root’s. This removes the hazardous rollback targets — a safety action, not cleanup.
Rollback drill (gen 16 → 15)
The machinery is proved without ever re-activating generation 14:
- Scratch-edit
/etc/blumeops/darwin/indri(one extraenvironment.systemPackagesentry) and runsudo darwin-rebuild switch --flake /etc/blumeops/darwin/indri#indri→ gen 16. sudo darwin-rebuild --rollback(16 → 15), verify as above.- Switch forward (re-apply, scratch edit reverted), leaving the newest generation current.
Reboot test
Done 2026-09-16. The first reboot found org.nixos.activate-system not
loading at boot and /run/current-system deleted: Background Task
Management had recorded the daemon as Disposition: [enabled, disallowed, notified] (sudo sfltool dumpbtm). The verdict was a
stale BTM record, not the plist shape: sudo sfltool resetbtm (from
a Terminal on indri — it needs the Authorization Services prompt and
fails over ssh with errAuthorizationInteractionNotAllowed) followed by
a reboot flipped the disposition to [enabled, allowed, notified]. The
daemon now runs at boot (exit 0) and /run/current-system survives
reboots; every other background item stayed allowed after the reset.
The play’s system-profile fallback stays in place — /run/current-system
is still absent before the first switch, and nix-managed user agents live
in ~/Library/LaunchAgents, which launchd loads at login regardless.
A reboot also pops “Enter a password to unlock the disk Nix Store” at
login. Cancel it — the volume password is a random one in the System
keychain and the systems.determinate.nix-store daemon mounts /nix
with it moments later (verify with mount | grep /nix); the dialog is
loginwindow racing the daemon.
This is the only step that takes indri offline (forge, registry, every
*.ops.eblu.me route and the Fly proxy behind them). Pick a moment with
no talos session in flight and no CI run, and have Screen Sharing to
indri open before starting — nothing in gui/501 serves until someone
logs in (restart-indri):
ssh indri 'sudo fdesetup authrestart'— unlocks FileVault for this one boot; without it the mini waits at the unlock screen for a password typed at the console.- ~2 min later the login window appears. Log in via Screen Sharing;
dismiss the tailscaled dialog the first Tailscale SSH connection pops;
start Amphetamine (the sifaka shares are mounted by the nix-managed
mcquack.eblume.sifaka-mounteragent, no manual step). ssh indri 'readlink /run/current-system; sudo launchctl print system/org.nixos.activate-system | head -1', then the verification list above andmise run services-check.
Since the /etc-static boot-race fix (#1150), the window before the Nix Store
dialog is answered is the proof for the two consumers that do not tolerate
a dangling /etc/static link:
ssh indriconnects (a dangling sshd drop-in makes sshd’sIncludeexit 1 and refuse every connection until the mount; the generation’s only sshd drop-in is the real file its activation wrote, never a store symlink, and it carries key-only auth for erichblume only).scutil --dnsshowsts.net→ 100.100.100.100 before login (the resolver is a real file the play writes, not a store symlink).
The same window is a hard failure for the seven user agents whose
ProgramArguments[0] was a store path (sifaka-mounter, forgejo-runner,
logrotate and the four *-metrics): posix_spawn on a missing store path
returns EX_CONFIG (78), and launchd treats that as a configuration error -
it does not apply KeepAlive or StartInterval, so one failed spawn at login
meant a dead CI runner, no sifaka mounts and no indri-side metrics until a
manual launchctl bootout/bootstrap (#1225). Their ProgramArguments[0]
is now ~/.local/bin/mcquack.nix-wait, a bash script the activation writes
before the user agents reload (so a re-pointed agent re-loading in the
same switch finds it) that blocks until /nix/store exists, then execs the
store path. Rollback restores the store argv0 via darwin-rebuild --rollback; the wrapper is inert without a unit pointing at it.
One-time, right after the first apply of that fix (the switch that drops
the old declarations): ls -l /etc/resolver/ts.net is a regular file,
ls /etc/ssh/sshd_config.d shows only Apple’s 100-macos.conf, and
sudo sshd -t passes.
heph hub
The heph hub is the series’ highest-blast-radius flip (eblume/blumeops#1655):
losing it wedges every spoke’s sync and the PWA’s data path until the unit is
reloaded. The unit is a nix-managed user agent (mcquack.eblume.heph) at the
same label, plist path and log names the ansible role used; the store hephd
(the cli output of the hephaestus flake input) runs behind a static
wrapper that cats the seeded authorized-sub file into argv (hephd accepts
--authorized-sub argv-only — the flake reads no vault). The PWA checkout and
its pwa-version.js stamp stay role-managed and ungated: the store is
read-only, and the web root must stay a mutable git checkout. The role’s
daemon legs (cargo install, version gate, plist render + load, the op-read
leg) sit behind heph_ansible_managed (default false). Applying the flip is
the auto-apply on merge (provision-indri, class auto) or mise run provision-indri -- --tags rebuild. Seed before the merge: mise run seed-secrets indri from the PR branch’s checkout seats
~/.config/heph-hub/authorized-sub — the switch’s checks slot refuses any
switch until the file exists. After the flip lands, mise run provision-indri -- --tags heph re-stamps the PWA checkout and its
pwa-version.js to the new heph_version tag. What changes the drill: with the unit
unloaded, the hub is down — every spoke’s sync and the PWA’s data path —
until the switch reloads it; the PWA shell files themselves keep serving from
the role’s checkout. Pre-flip: confirm the latest borgmatic archive-tier
backup of ~/.local/share/heph/heph.db is fresh (the never-lose dump rides
the daily 02:00 archive run) — the hub DB is the only copy of the
task/context data. Rollback per §Rolling back a service flip, with the
role’s gate flipped: mise run provision-indri -- --tags heph -e heph_ansible_managed=true.
Rolling back a service flip
Each service migration writes its plist at the same path ansible used,
under the same label (logrotate’s globs and alloy’s log tails key on it
— mcquack.eblume.* for every service, and the role’s historical
mcquack.jellyfin for jellyfin, which PR 9 keeps verbatim). Once a service is flipped, its ansible role is skipped by
default (a role variable is off), so only the generation owns the plist
— the plist is generation system content: a plist-only change is a new
store path and therefore a new generation, and applying it reloads the
agent.
When a generation changes a service’s plist, the rollback order is fixed:
sudo darwin-rebuild --rollback first, then re-run the role with its
gate flipped — for logrotate, mise run provision-indri -- --tags logrotate -e logrotate_ansible_managed=true — so ansible writes the plist back and
its restart handler reloads the agent. For the metrics flip (PR 4), re-run
all four roles the same way: mise run provision-indri -- --tags borgmatic_metrics,forgejo_metrics,jellyfin_metrics,zot_metrics -e borgmatic_metrics_ansible_managed=true -e forgejo_metrics_ansible_managed=true -e jellyfin_metrics_ansible_managed=true -e zot_metrics_ansible_managed=true.
For the zot registry flip (PR 5), re-run mise run provision-indri -- --tags zot -e zot_ansible_managed=true. Unlike the textfile
collectors, zot is a real daemon: the registry is down between the
rollback and the ansible re-write. For the caddy flip, re-run
mise run provision-indri -- --tags caddy -e caddy_ansible_managed=true
— the gate re-deploys the rollback plist and the restart handler reloads
the agent; the wrapper’s exec target follows the rolled-back generation’s
profile, so no -e override is needed (the ~/code/3rd/caddy checkout
build was retired post-flip, blumeops#1275). — the widest outage: every *.ops.eblu.me endpoint and the L4 routes
(2222/5433/5434 and the sifaka exporter ports) are down during the
window, and no CI runs land while the forge is unreachable. For
the forgejo runner flip, the main runner’s rollback is a plain
darwin-rebuild --rollback - its unit and config are generation-owned
(eblume/blumeops#1528) and there is no ansible re-write to fall back
to; the source checkout now serves only the build runner’s re-write
(see §indri-build runner). For
the forgejo flip (PR 8), re-run mise run provision-indri -- --tags forgejo -e forgejo_ansible_managed=true — the whole forge is down during the
window: the forge API and git ssh (2222), every forge-bound git op from a
talos session and every CI run, while caddy stays up (502s) and the
registry stays up. For the devpi flip (PR 10), re-run
mise run provision-indri -- --tags devpi -e devpi_ansible_managed=true
— the PyPI proxy (pypi.ops.eblu.me) is down during the window; forge,
the registry and every other endpoint stay up. For the borgmatic flip (PR 11), re-run
mise run provision-indri -- --tags borgmatic -e borgmatic_ansible_managed=true
— the backup windows (sifaka-local + BorgBase offsite + photo library)
are lost during the window; no front-facing service is affected. For the alloy flip (PR 12), re-run
mise run provision-indri -- --tags alloy -e alloy_ansible_managed=true
— the box’s metrics flow (prometheus scrape + Loki push + log tails) is
blind during the window, and the macos-power-metrics daemon goes with it;
every front-facing endpoint stays up. For the jellyfin flip (PR 9), re-run
mise run provision-indri -- --tags jellyfin -e jellyfin_ansible_managed=true
— the media endpoints (jellyfin.ops.eblu.me) are down during the
window; forge, the registry and every other endpoint stay up, and the
dedicated blackbox probe witnesses both the flip and any regression.
For the heph hub flip (eblume/blumeops#1655), re-run mise run provision-indri -- --tags heph -e heph_ansible_managed=true — the hub (every spoke’s sync and the PWA’s data path) is down during the window; the PWA shell files keep serving from the role’s checkout.
Never ansible first — that would leave the old plist loaded under the
new generation.
What the rollback itself does depends on the target generation
(nix-darwin modules/system/launchd.nix, unchanged on master as of
2026-09-17): the user-launchd phase — including the loop that unloads and
deletes agents the target does not declare — is emitted only when the
target declares at least one user agent. Rolling back to a generation
with none (gen 19 → 18, the PR 2 case) leaves the nix plist loaded and
running the store script, which survives only as long as the newer
generation is a GC root; the ansible step is what actually replaces it.
Rolling back to a generation that keeps other agents (every flip after
PR 2) unloads and deletes the dropped ones, and the service is down until
ansible runs. Both cases were drilled on 2026-09-17 (blumeops#1125).
The runner cache agents (eblume/blumeops#1528 PR 3) are the exception: rolling back to a generation that predates them (anything before the #1667 apply, system-66) unloads and deletes
mcquack.eblume.runner-cache-sweepand-prune, and nothing re-writes them now, because the role’s cache tasks are gone. Sweep and prune stay off until the next roll-forward (--tags rebuild), and~/.cache/uvregrows at roughly 2G/day in the meantime.
The gate variable must arrive as a boolean: the role’s conditionals apply
| bool, so the -e key=true form above is fine; without the filter,
ansible-core 2.19 rejects the string with “Conditionals must have a boolean
result”.
Verifying an agent after a flip or a rollback: launchctl print "gui/$(id -u)/<label>" (the launchd Label, not the service name —
mcquack.jellyfin for jellyfin) — program = names the owner (a
/nix/store/… path or ansible’s) and last exit code = 0. runs is
the climbing signal for the collectors; for a daemon, activation’s
reload is an unload+load, so the new instance starts at runs = 1 —
the proof of the reload is the new pid, the rebuild log’s
reloading user service line, and the daemon’s own shutdown log line
at the switch timestamp. The wording is per-service — e.g. zot logs
received signal 15, caddy a shutting down apps, then terminating
line carrying "signal":"SIGTERM", forgejo-runner runner: received shutdown signal, and devpi the server’s own restart log line in
mcquack.devpi.out.log — the new pid and the rebuild log’s reloading user service mcquack.eblume.devpi line complete the set. The
StandardOutPath log is not a signal; a quiet
agent writes nothing.
indri-build runner (eblume/blumeops#1357)
The second forgejo runner (label indri-build) is unprivileged by
construction: the dedicated macOS user indri-build (uid 503 in its own
primary group, no sudo, home 0700) runs the daemon and a colima VM
(~indri-build/.colima/indri-build/, the profile dir) hosts the jobs’
docker. The 503/503 pair is declared in darwin/indri/configuration.nix,
where a pre-activation check refuses the switch if uid/gid 503 is already
taken on the box (501 erichblume, 502 the forgejo account); re-check
dscl . -list /Users UniqueID there before reapplying after any rebase. The user, the two system launchd
daemons (mcquack.eblume.colima-build, mcquack.eblume.forgejo-runner-build),
the build user’s mise config and the home dir layout are nix-managed; the
runner config and the colima profile are role-rendered by the
forgejo_runner role, gated on the runner being registered
(uuid/token in 1Password).
Registration comes first (configure-launchd-runner §indri-build
runner). The warrant provision-indri dispatch runs --tags rebuild only
(the nix switch), so it deploys the user and daemons but not the
role-rendered config: after registration, a full local
mise run provision-indri (no tag restriction) renders the config. Before
registration the role renders nothing and the daemons idle.
Acceptance checks (run on indri as the named users after the deploy window):
- As
indri-build:sudo -n truefails (no sudo). - As
indri-build:ls /Users/erichblumeis permission denied. - As
erichblume:ls /Users/indri-buildis permission denied (ls -ldshowsdrwx------). - Container lockdown: from a container started on the build socket,
/Users/erichblumeis not present/unreadable - e.g.docker run --rm -v /:/host busybox ls /host/Usersshows no erichblume home contents reachable (the profile mounts only/Users/indri-build). - Daemon boot proof (the #1357 acceptance for the colima half): from a
rebooted indri, before anyone opens a GUI session,
sudo -u indri-build colima status --profile indri-buildreportsvzas the driver and the docker socket path - the VM came up as a system daemon with no login session (the pre-/nix window is bridged by/usr/local/libexec/mcquack.nix-wait, the unit argv0).
Window hygiene
A window contains exactly one PR: apply pending role merges by tag
(--tags alloy,caddy,borgmatic,zot) first, then the PR’s
provision-indri. Halting between any two PRs leaves indri serving.
Related
- indri - the box and its services
- restart-indri - shutdown and startup
- manage-lockfile - ringtail’s lockfile flows
- agent-change-process - why agent PRs sit pending