v1.18

Changelog archive — v1.18

Moved verbatim from CHANGELOG.md; never edited.

[v1.18.3] - 2026-07-21

Features

• The ringtail agent workspaces now run a heph spoke synced to the indri hub, so agent sessions can use heph for task/context — heph is a deliberate in-boundary agentic-workflow substrate (distinct from the 1Password blumeops vault, which stays isolated). heph/hephd are cargo-installed at a pinned tag via a mise-resolved Rust toolchain (nixpkgs rustc lags heph’s floor) by a agent-heph-install systemd oneshot; a agent-heph-spoke service runs hephd authenticating as a dedicated, independently-revocable heph-agents Authentik identity, with its OIDC token stored in the agents vault via hephd’s command token store (no plaintext at rest). The indri hub authorizes that identity as a co-owner via hephd --authorized-sub (sourced from the vault). Requires heph ≥ v1.7.0.
• Stood up an “under construction” landing page at the apex eblu.me (and www.eblu.me) — a hi-I’m-Erich splash with the obligatory old-school hazard-barricade GIF and links to docs.eblu.me / forge.eblu.me. Unlike the other public sites, it’s served straight from nginx on the Fly proxy (files baked into the image) rather than tunneled back to Caddy on indri, so the front door stays up even when indri or the tunnel is down. The apex uses A/AAAA records to Fly’s ingress IPs (a CNAME is illegal at the zone apex); www is a normal CNAME like the rest.
• Stood up a private Factorio dedicated server on ringtail (services.factorio, UDP 34197), reachable at factorio.ops.eblu.me. It is BlumeOps’ first externally-shared service: guests are shared onto ringtail (autogroup:shared) rather than invited as members, and the Tailscale ACL hands them exactly the one game port — they inherit none of the member-facing services.

Bug Fixes

• Fixed flake input discovery in the dagger flake-update pipeline: nix cannot readFile /dev/stdin (it canonicalizes to a /proc/.../pipe:[...] path), so discovery had been silently empty since the pipeline was written, degenerating every run into a bare nix flake update of all inputs — the nixpkgs-services pin survived only because its URL is rev-pinned. Metadata now lands in a real file, stderr is no longer suppressed, and run 658’s “refusing bare flake update” guard failure becomes a green no-op again.
• Fixed the Ringtail Flake Update workflow hanging in its summary step (git log spawned a pager on the runner; now --no-pager), and two latent bugs in the dagger flake-update pipeline: skip_inputs was never applied (single-quoted $SKIP_INPUTS never expanded, so a real update would have bumped the pinned nixpkgs-services input), and an empty discovered-input list now fails loudly instead of falling back to a bare nix flake update of everything.
• Changed factorio.ops.eblu.me from a pinned A record to a CNAME → ringtail.tail8d86e.ts.net. Tailscale remaps a shared node’s 100.x address inside each guest’s tailnet, so the hardcoded owner-tailnet IP was unroutable for guests and the name timed out for them; the CNAME lets each client’s own MagicDNS resolve ringtail to the address correct for its view.
• Firefox on ringtail now runs under XWayland (MOZ_ENABLE_WAYLAND=0), working around a hard deadlock that froze the whole browser. NVIDIA’s Wayland EGL explicit-sync path can leave a DRM timeline fence unsignaled inside eglSwapBuffers, so the Renderer thread blocks forever in drmSyncobjTimelineWait; the Compositor thread then blocks in WaitUntilPresentationFlushed, and the main thread blocks behind a synchronous SendFlushRendering IPC issued while painting a popup. Painting any doorhanger, menu, or dropdown could trigger it, leaving Firefox wedged at 0% CPU until killed — the trigger that surfaced it was clicking “enable notifications”, which is a repaint, not a network call. No Firefox pref or wlroots/sway toggle governs this path, and nixpkgs production/latest/beta were all pinned to the affected 580.142 driver, so XWayland is the only lever until a newer driver ships.
• Pinned ringtail’s Factorio headless server to 2.0.77 via a temporary versionsJson overlay so it matches the auto-updated Steam client (nixos-25.11 still ships 2.0.76). Remove the overlay once nixpkgs catches up.
• Restored the nightly indri borgmatic run, which had aborted since 2026-07-10. The multi-user hardening that locked /etc/rancher/k3s/k3s.yaml to 0600 root on ringtail also locked out eblume, the user borgmatic’s before: configuration k8s-dump hooks SSH in as to snapshot mealie/shower/navidrome — a non-zero hook exit aborts the whole run, so nothing landed. The hooks now reach the cluster via sudo k3s kubectl, keeping the kubeconfig locked away from the restricted web-agent user.
• container-build-and-release no longer fails spuriously: the Forgejo dispatch endpoint routinely holds the connection open past the read timeout even though the dispatch lands, which surfaced as an unhandled ReadTimeout. The POST timeout is now non-fatal — the task falls through to run verification (polling for the dispatched run) as the real source of truth, and only errors if no matching run appears.
• eblume-heph-spoke moved from a system unit to a systemd user service (with linger): as a system service hephd bound the ~/.local/share fallback socket while interactive shells looked in /run/user/1000, so the heph CLI could never reach the daemon. Shared install machinery split into mkInstallUnits (heph-common.nix).
• mise run services-check: the k3s node-readiness probe now reads the cluster via sudo k3s kubectl instead of KUBECONFIG=/etc/rancher/k3s/k3s.yaml. That kubeconfig went 0600 root-only when ringtail became a multi-user host, so the probe had been failing on permission (the cluster was healthy — k3s-apiserver (remote) and every pod/HTTP check passed). Same sudo-not-locked-kubeconfig pattern as the borgmatic dump hooks. Also tightened grep -q Ready → grep -qw Ready so a NotReady node can’t false-pass on the substring.

Infrastructure

• Agent workspaces: add a native build toolchain (gcc, binutils, pkg-config, gnumake + CC=gcc) to the session PATH so cargo build no longer dies with linker `cc` not found, letting agents compile and verify Rust they author. Also expose the gamedev Bevy project’s Linux native deps — alsa/udev on PKG_CONFIG_PATH, and the dlopen’d windowing/graphics libs (vulkan-loader, libxkbcommon, wayland, libGL, xorg) on LD_LIBRARY_PATH. Building/verifying works headlessly; running a windowed Bevy app still needs a GPU/display, so playtesting stays a human job.

• Ringtail now hosts agent workspaces — per-repo Claude Code Remote Control servers (hephaestus, blumeops, research, playground) run under an unprivileged agent user, steerable on demand from the Claude mobile app. Agents authenticate to 1Password only as a vault-scoped service account and push to Forgejo as a dedicated bot (never main). See agent-workspaces and the bootstrap-agent-workspaces runbook.

• Agent workspaces can now author blumeops: a read-write, agent-owned clone lands at ~agent/code/personal/blumeops (pool-only, no dedicated server) so sessions can edit it and open PRs as the bot. Deploys stay gated — and to make that real, the k3s admin kubeconfig is locked from world-readable 0644 to 0600 (it previously handed the unprivileged agent user cluster-admin and an ArgoCD-admin deploy path).

• Agent workspaces collapsed from per-repo Remote Control servers (ringtail-{hephaestus,research,playground,parsimony}) to a single home-base session, ringtail-agent, rooted in the new agents repo whose AGENTS.md carries the base instructions (repo map, toolbox, execution environments). All other repos are sibling checkouts the session cds into; the playground workspace is dropped (a home-base worktree is already a scratch space).

  Also: heph for Erich on ringtail — a second hephd spoke (eblume-heph-spoke, token-file store, logging in as Erich himself) alongside the agent’s, with the shared version pin and install machinery extracted to nixos/ringtail/heph-common.nix.

• New parsimony agent workspace on ringtail for timberborn-parsimony, a Timberborn “Least Actions Challenge” mod. The remote agent builds the mod against the local game install’s DLLs; playtesting (launching the game) remains a human-session step via the repo’s mise run playtest.

• Agent workspaces: put the research report toolchain (mise, uv, pandoc, typst, weasyprint) on the session PATH, and expose WeasyPrint’s native libraries via LD_LIBRARY_PATH so mise run compile-report renders a PDF on ringtail. Repo mise configs are pre-trusted to avoid interactive prompts.

• The agents bot now authors blumeops via a fork (agents/blumeops) with only read on the canonical repo, instead of pushing branches to eblume/blumeops with write. This closes a privilege-escalation path: workflow_dispatch is write-gated, so a write-access bot could have dispatched a branch workflow that reads blumeops’ deploy-credentialed Actions secrets (ARGOCD_AUTH_TOKEN, FLY_DEPLOY_TOKEN, ZOT_CI_API_KEY, MAIN_PUSH_TOKEN) — Forgejo has no per-run approval gate for write users, so read-only + fork is the enforceable boundary. The ringtail agent-workspaces clone is repointed automatically (origin = fork, upstream = canonical); agents branch off upstream/main and open cross-repo PRs.

• Closed a borgmatic monitoring blind spot: the metrics collector never scraped the main BorgBase offsite repo (only sifaka-local and the immich photos repo were configured), so a failed offsite run produced no metric and no alert — BorgBase’s own email was the only signal. Added the offsite repo to the collector and a BorgmaticStale Grafana alert (fires when any repo’s newest archive is older than 30h, ~7h after a missed nightly run). The hourly per-repo poll stays metadata-only (borg info/list); the heavy per-source manifest listing is now skipped for remote repos to avoid pulling a 100k-entry file manifest over the internet every hour. Also corrected the metrics script’s stale /opt/homebrew/var/forgejo source-path mapping to ~/forgejo.

• Added retries: 3 / retry_wait: 300 to the borgmatic config so a transient failure on a single repository — e.g. a broken SSH pipe partway through a large offsite upload to BorgBase — is retried (resuming from borg’s checkpoint) instead of losing that night’s backup. Surfaced when PR #407 repointed the forgejo source from the empty /opt/homebrew/var/forgejo husk to the real ~8.8 GB tree: the first offsite run under the new config had ~6.7 GB of new deduplicated data to push, the pipe broke mid-transfer (borg exit 87), and the night’s offsite backup failed with no retry. The dump hooks (before: configuration) are not re-run on retry.

• Agent workspaces can now open PRs and commit. Wired tea (with a login seeded from the agents vault), a FORGEJO_TOKEN for the pr/branch/runner mise tasks, and a git author/committer identity into the workspace launcher. The token is an agents-owned Forgejo PAT (scopes write:repository + write:issue — PRs are issues in Forgejo, so tea needs the issue scope — stored concealed as agents-forgejo-token in the agents vault) — structurally bounded to the repos agents collaborates on, with no blumeops-vault dependency. Also revoked the bot’s leftover write on blumeops/project-template/adelaide-baby-shower-app.

• Disabled hephd’s --self-update on the indri heph hub and put its version fully under IaC: ansible/roles/heph now converges the installed hephd to the pinned heph_version (v1.7.0) on every provision — installing, upgrading, or downgrading as the pin changes — instead of bootstrapping once and letting the daemon self-update. The --self-update/--self-update-interval-secs flags are gone from the launchagent, so no release lands on the hub until a human bumps the pin and re-provisions. The ringtail agent spoke was already pin-only (no self-update); gilbert’s manual spoke gets documented steps to strip its self-update flags and pin its binary. See hephaestus (reference/services) for the release → deploy flow.

• blumeops main is branch-protected — a push + merge whitelist limited to eblume — so no write collaborator can merge a PR or push to main; human review/merge is the gate (the agents bot is separately read-only + fork, see the fork-model change). The release workflows (build-blumeops, cv-deploy) commit a version bump back to main, and because the automatic Forgejo Actions token can’t be push-whitelisted (Forgejo #11159) they now push with MAIN_PUSH_TOKEN — an eblume-owned PAT (write:repository) provisioned as an Actions secret via the forgejo_actions_secrets role. This supersedes the earlier “main is intentionally left unprotected against the bot” stance.

• Ringtail: the stray fish ip function (which curled ipinfo.io / checkip.amazonaws.com) was shadowing iproute2’s real ip, so ip addr, ip route, etc. silently hit the network instead of running the command. Renamed it to myip and brought it under home-manager (xdg.configFile) so it can’t drift back, and removed the shadowing ip.fish plus a broken (prettyping-less) ping.fish from the host.

• Dropped the blumeops agent workspace — remote workers are now hephaestus, research, and playground only. Real blumeops work needs the whole blumeops 1Password vault (its ansible pre_tasks and mise tasks op read broadly), and that vault is deliberately the operational-secret blast-radius boundary with no least-privilege subset to grant a service account — so blumeops stays a local-on-gilbert, biometric-op job. Also documented the deployment’s terms-of-use footing (ordinary, individual usage; native-app OAuth). See agent-workspaces §“Why blumeops is not a workspace”.

• Service review: reviewed the ringtail snowflake-proxy. Upstream is at 2.14.1 (2.12/2.13/2.14 released since), but nixos-25.11 is frozen at 2.11.0 and the nixpkgs-services pin is shared with k3s and forgejo-runner. As a minor anti-censorship service, it stays on the in-channel 2.11.0 build rather than mixing in an unstable channel; the version tracking now records the upstream gap.

• Service review: upgraded the ringtail forgejo-runner (nix-container-builder) from 12.7.2 to 12.11.1 via the nixpkgs-services pin, picking up the Go toolchain, golang.org/x/sys, and go-git security fixes released since 12.7.2. k3s and snowflake (sharing the pin) are unchanged.

• Service review: reviewed k3s on ringtail (most stale, 94d). nixos-25.11 is frozen at the deployed 1.34.5+k3s1 (no k3s_1_35/k3s_1_36 attrs; unstable has 1.35.6, upstream latest 1.36.2), so it stays on the in-channel build rather than channel-mixing; the upstream 1.34.9 patch gap (klipper-helm CVE bump, containerd fixes) is recorded in version tracking. Cluster verified healthy. Also ran the weekly ringtail flake.lock update: nixpkgs 3cac626 → b6018f8 (nixos-25.11); disko/home-manager already current, nixpkgs-services pin held.

• Service review: reviewed tempo (most stale, 96d). Upgraded the home-built container 2.10.3 → 2.10.7 — the 2.10.x patch line brings golang.org/x/net and golang.org/x/crypto security fixes; its one breaking change (OpenCensus receiver removal in 2.10.6) doesn’t apply since the distributor only uses the OTLP receiver. The v2.10.6 Go-1.26 floor required overriding buildGoModule’s Go to go_1_26. Upstream 3.0.x is a major with breaking write-path/config changes and was deferred to a separate task. Also reviewed the weekly Prowler K8s CIS report (2026-07-05): all clear, 0 unmuted FAILs, net-zero week-over-week (1127 PASS / 53 muted).

• Service review (2026-07-13): reviewed the seven stalest ArgoCD services against upstream and actioned the three low-risk, self-contained bumps: grafana 12.4.2 → 12.4.5 (in-minor security CVEs — CVE-2026-9029/-33382/-42127; home-built fetchurl container rebuilt via CI), argo-cd v3.3.6 → v3.3.12 (in-minor, CVE-2026-41240 dompurify), and ollama 0.20.4 → 0.31.2 (upstream image tag). Deferred with heph follow-up tasks rather than stamping them reviewed: miniflux 2.3.2 and navidrome 0.63.2 (both bare-nixpkgs packages pinned to the deployed version — need a package override with recomputed hashes; navidrome also needs EnableSharing=false), authentik 2026.2.5 (from-source Nix build — sources + Go/Py/web vendorHashes; SSO blast radius), and immich v3.0.x (breaking major — pgvecto.rs dropped, DB migration). Their last-reviewed dates were intentionally left un-stamped so they resurface in the stale queue until actually upgraded.

• agent-workspaces: add a general CLI toolbox (cliTools: gawk, jq, curl, python3) to the ringtail agent-session PATH alongside the report toolchain. Plain shell agents constantly reach for awk/jq/curl and quick python3 one-liners; without them sessions hit command not found or fall back to slower workarounds. All nixpkgs builds — declarative, no new /run/current-system/sw/bin exposure. python3 is a bare interpreter (uv is already on PATH for uv run --script). Applied on the next provision-ringtail.

• Upgraded Immich v2.6.3 → v3.0.2 (server + machine-learning CUDA). The v3 breaking change (pgvecto.rs dropped for VectorChord) was already satisfied — ringtail’s immich-pg runs cloudnative-vectorchord:17-0.5.0 (vchord 0.5.0, pgvector 0.8.0), both inside v3.0.2’s accepted ranges — so this was a clean tag bump plus immich’s own startup migrations. Closes the v2.6.3 High-severity panorama-OCR stored XSS and the 2026-07-06 advisory cluster. A pre-upgrade pg_dump was taken as a rollback artifact, and post-upgrade a one-time Metadata Extraction re-run is pending (tracked in heph).

• Service review 2026-07-17 (remote-agent): authentik 2026.2.2 → 2026.2.6 (most-stale service; latest patch on the current train — the 2026.5.x train jump is a separate follow-up). Source bump of the nix-built container via the Build Container CI dispatch with TOFU hash discovery; go vendorHash and the client-go pin carry over (go.mod/go.sum unchanged upstream). Also added a manual-dispatch Ringtail Flake Update workflow so remote-agent sessions can land flake.lock bumps via CI on a PR branch. Added a Service Health Grafana dashboard (uid: service-health, folder “Service Health”): degraded deployments/statefulsets, containers stuck waiting, restart counts, and scrape-target status — the read surface for agent post-deploy health checks (credential design for agent API access under discussion). Reworked mise run runner-logs to fetch job logs over the Forgejo web log route (…/jobs/N/attempt/M/logs) with the existing API token — no ssh to indri needed, so it now works from remote-agent sessions and while logs are still in dbfs. Also taught the Build Container workflow to post nix build failures (including TOFU hash mismatches) as PR comments.

• Service review (flyio-tailscale): reconciled tracking (the Fly proxy binary was bumped to v1.94.2 during the 2026-06-22 review but service-versions.yaml still read v1.94.1) and attempted the long-held jump to the 1.98 train. Deployed v1.98.8 to blumeops-proxy and tested MagicDNS inside the Firecracker container — it still SERVFAILs tailnet names (nslookup indri.tail8d86e.ts.net 100.100.100.100) and blacked out all public routing (eblu.me/forge/cv → HTTP 000), the same failure mode as the original v1.96.5 regression. Rolled back to v1.94.2 in a few minutes; routing and MagicDNS verified restored. Notable: the 1.98 train is clean in k3s pods (PR #390) but NOT in the Fly no-DNS-manager container, so the pin stays on 1.94. Root cause (Fly container has no DNS manager, so tailscaled’s split resolver isn’t consulted) and the unblock path are tracked as a heph task.

• Fly proxy nginx base image bumped 1.30.3-alpine → 1.30.4-alpine (security release: CVE-2026-42533, CVE-2026-60005, CVE-2026-56434). Service review stamp for flyio-nginx; tracking entry corrected from stale 1.29.6.

• Service review: upgraded the Fly proxy’s Grafana Alloy sidecar binary from v1.17.0 to v1.17.1 (digest-pinned), and corrected the stale flyio-alloy tracking entry in service-versions.yaml, which claimed v1.14.1 while the Dockerfile had been on v1.17.0.

• Agent workspace repos on ringtail now clone to ~/code/personal/<repo> (was ~/workspaces/<name>/<repo>), so the paths agents read in every AGENTS.md/CLAUDE.md resolve on the agent box. Remote Control session names (ringtail-<name>) are unchanged.

• Fly proxy deploys now work from any host: flyctl 0.4.71 pinned in mise.toml (was gilbert-only via brew).

Documentation

• Doc review: verified the “Manage Ringtail Lockfile” how-to — dagger flake-lock/flake-update pipelines, the prune-ringtail-generations task, and its wiki-links all resolve against the current tooling. Stamped last-reviewed.
• Doc review: corrected the NVIDIA device plugin reference card — time-slicing is 4 replicas per GPU (not 2) and the deployed image tag (v0.19.2) is now recorded.
• Doc review: verified the QArt Tuner reference card against utils/qart/ — CLI flags, web-UI parameter ranges, keyboard shortcuts, mise serve task, and the rsc.io/qr dependency all match the code; wiki-links resolve. Stamped last-reviewed.
• Doc review: verified the No Helm Policy explanation card against the live manifests — no Helm chart references remain anywhere under argocd/, 1Password Connect is plain rendered manifests, and ArgoCD and the Tailscale operator both pull pinned upstream manifests via resources: (forge mirror / raw GitHub) rather than charts. The migration-history table and “all services use kustomize” claim hold. No content changes; stamped last-reviewed.
• Doc review (2026-07-14): reviewed the never-reviewed Caddy reference card and corrected several stale entries — cv/docs are now served as static files from disk (not proxied to Tailscale endpoints), the retired minikube 5432 → pg.tail8d86e.ts.net L4 route was replaced with the live 5433/5434 Postgres routes, and the proxied-service table was refreshed against ansible/roles/caddy/defaults/main.yml.
• Doc review: corrected the mealie reference card for the ringtail k3s migration — manifests path (mealie/ → mealie-ringtail/) and storage class (minikube-hostpath → local-path). Service review: 1password-connect confirmed at upstream-latest 1.8.2 (Synced/Healthy), stamped reviewed.
• Doc review 2026-07-17: rewrote the stale ArgoCD apps.md registry card (post-minikube -ringtail app names, six missing apps, cv/forgejo-runner departures, manual-sync-everywhere policy). Documented the remote-agent variant of the service-review, doc-review, and flake-update runbooks, and retired C0/C1/C2 references from review-documentation.md to match the new AGENTS.md process.
• Doc review: verified the Zot service reference card. Fixed the stale pull-through-cache description — it referred to [[cluster|minikube]], but the cluster has been k3s on ringtail since the 2026-06 migration. Confirmed the registry is healthy (registry.ops.eblu.me/v2/ → 200) and the namespace convention, security model, and API-key-rotation steps still match the deployment. Stamped last-reviewed.
• Doc review: tutorials/ai-assistance-guide.md — replaced the retired minikube-indri kubectl context guidance with k3s-ringtail, corrected the indri native-service list, and noted Jellyfin’s LaunchAgent management.
• Doc review: reviewed tutorials/exploring-the-docs.md — replaced the stale reference to AGENTS.md’s retired kubectl-context requirement with its current critical rule (public repo, no secrets), and stamped last-reviewed.
• Added the ringtail Factorio headless server to service-versions.yaml (nixos type, 2.0.77), documenting the temporary versionsJson overlay that pins it ahead of nixpkgs to match the Steam client — including how to bump the pin and the note to remove it once nixpkgs catches up.
• Forgejo reference: documented the consumers of the api-token PAT (ansible role, runner-logs, and — new — the tea CLI, which keeps a copy in its own config) so a future rotation updates all three. Context: tea 0.14.2’s httpsign deadlock forced tea off SSH-signature auth and onto this shared PAT.
• run-1password-backup: note that the account-wide export now also covers the new agents vault (and that any newly-added vault is swept in automatically — the export has no per-vault selection). The op-backup script is vault-agnostic, so no code change was needed.

[v1.18.1] - 2026-06-29

Bug Fixes

• Restored the argocd/manifests/1password-connect and argocd/manifests/external-secrets base manifests that the minikube decommission (a520f023) deleted while leaving the 1password-connect-ringtail and external-secrets-ringtail Applications still pointing at them. Both apps had been stuck in ComparisonError (ArgoCD could not generate their target state) ever since — the running pods stayed Healthy under Manual sync, so the breakage went unnoticed. The restored bases render images identical to what is live (1password/connect-*:1.8.2; external-secrets:v2.2.0-13895bb-nix via the ringtail overlay), so syncing is a no-op.
• Fixed all paperless-ngx document ingestion (API upload, consume folder, mail), broken since the Nix-image migration: the image ships no /tmp, so SCRATCH_DIR resolved to a nonexistent /paperless, and in the multi-container pod the scratch dir was not shared between the web container (which writes API uploads there) and the worker (which reads them). PAPERLESS_SCRATCH_DIR now points at a shared emptyDir, and a /tmp emptyDir keeps general temp usage off the container overlay.
• Fixed a silent backup gap: borgmatic was backing up the dead Homebrew-era forgejo path (/opt/homebrew/var/forgejo, frozen at 2026-04-06) while the live forge data at /Users/erichblume/forgejo — the 272MB DB and 8.8GB of git repositories — went uncaptured since the brew→source migration. Backups kept succeeding (the husk still existed), so no alarm fired. Now the live repos are in source_directories and forgejo.db is dumped WAL-safe via the online .backup API (it is WAL-mode); the raw db files are excluded to avoid torn copies. Verified a fresh archive captures the live 272MB DB + current repos.
• Added a person object-filter mask to the gablecam Frigate camera to stop a newly-planted Japanese maple (in the bottom-right planter) from being detected as a person and spamming alerts. The false detections scored 0.57–0.84 — overlapping real-person scores — so min_score couldn’t filter them; the mask covers only the front-deck-stairs corner by the building. Trade-off: a person actually on the stairs no longer alerts, but should trip an alert as soon as they step off into the driveway. Mask region derived from the actual false-positive bounding boxes via the Frigate events API.
• Fixed duplicate homepage entries (DJ/Navidrome, Kiwix, Miniflux, ArgoCD, Grafana, Prometheus, Transmission) after retire-minikube — the static services.yaml entries that covered the formerly-invisible minikube services now collided with k8s Ingress annotation auto-discovery on ringtail. The annotations are the source of truth; the static duplicates are removed.
• Paperless-ngx: raised the Celery worker memory limit from 1Gi to 3Gi. The 1Gi cap introduced in the indri→ringtail migration OOMKilled OCR (consume_file) mid-import, stranding scanned uploads and freezing the auto-incrementing ASN at 46.
• Refreshed all stale ZIM torrent URLs in kiwix-ringtail/torrents.txt. kiwix only keeps the .torrent for the latest build of each ZIM, so the whole _2026-01 devdocs section plus the wikipedia_en_top1m_maxi_2025-09 entry had silently 404’d — the kiwix sidecar could no longer add them to transmission. Bumped wikipedia to 2026-04 and every devdocs package to its current build (mostly 2026-04, some 2026-05); all 43 URLs now resolve 200. Added a note documenting the latest-only behavior so future 404s are an obvious date bump.
• Relaxed the frigate liveness probe (timeoutSeconds: 10, failureThreshold: 5; readiness timeoutSeconds: 5). The default 1s timeout / 3 failures was killing the pod with SIGTERM on transient API stalls — 206 graceful-shutdown restarts over 36d, never an OOM — whenever frigate’s API thread blocked under detector/GPU load or NFS recording I/O. The pod now tolerates ~2.5min of unresponsiveness before a restart.

Infrastructure

• Re-broadened the ringtail Alloy blackbox exporter from a single probe (immich) to all 18 in-cluster HTTP services (argocd, authentik, frigate, grafana, homepage, immich, kiwix, loki, mealie, miniflux, navidrome, ntfy, paperless, prometheus, shower, teslamate, tempo, transmission). Coverage had collapsed to immich-only during the minikube retirement, leaving the other services silently unmonitored — and mise run services-check was reporting a green “OK” for 10 of them because its ServiceProbeFailure check passes when no firing instance exists. Each health path was verified to return 2xx unauthenticated from inside the cluster; shower is probed with its Host: shower.ops.eblu.me header and ollama is intentionally excluded (scaled to zero on demand). The single ServiceProbeFailure alert covers every new target automatically via label_replace on the integrations/blackbox/* job label, so no new alert rules were needed. services-check was updated to mirror the probe list (no more false OKs) and to check indri/public services (forgejo, zot, devpi, cv) directly. Also swept stale post-minikube monitoring-host claims from the docs (runbook-service-probe-failure, port-services-check-alerts, federated-login, the tag:loki/tag:k8s-api rows in tailscale, and the prometheus textfile list).

• Migrated the borgmatic role’s pre-backup database snapshot hooks from the deprecated before_backup: key to borgmatic 2.x’s commands: syntax, clearing the before_backup is deprecated warning logged since borgmatic 2.1.4. Using before: configuration also stages the heph/mealie/shower/navidrome dumps once per run instead of once per repository — previously the two repos (sifaka + BorgBase) each re-ran every dump, harmless but redundant. The abort-on-failure guarantee (a non-zero hook aborts the whole backup, so a failed snapshot is never silent) is preserved.

• retire-minikube tail cleanup (follow-up): simplified the container build tooling to nix-only. container-version-check now validates each container’s default.nix against service-versions.yaml (dropped the dead container.py VERSION / Dockerfile ARG rules), and container-build-and-release no longer carries dead container.py/Dockerfile classification. Stale build-model references purged from docs (forgejo, forgejo-runner, dagger tooling, grafana-image how-to, review-services, zot CI-auth) and the build-container.yaml workflow comment.

• Pruned the remaining dead container.py/Dockerfile build-model references left after retire-minikube. container-list and service-review mise tasks dropped their has_container_py/has_dockerfile classification branches (every container is now containers/<name>/default.nix), and the upgrade-grafana and deploy-prowler how-tos now point at the nix builds instead of the retired Dockerfiles.

  Also reconciled the CI runner/dagger docs with the phase-6 host-mode reality (jobs run directly on indri with the mise toolchain — no job container, no runner-job-image): rewrote upgrade-dagger around the mise-pinned host CLI, fixed the runner-environment section of the update-documentation how-to and the job model in configure-launchd-runner, and deleted the obsolete configure-k8s-runner how-to (superseded by configure-launchd-runner), repointing its inbound links.

• retire-minikube tail cleanup: pruned the dead container-build framework. Removed the build, publish, and container_version Dagger functions (and the now-orphaned containers.py container.py-discovery module) — the Dockerfile/docker_build() and native-container.py build paths were retired when the minikube/arm64 runner went away, leaving nix as the only container build path. Updated dagger and build-container-image to document the nix-only reality.

• deploy-fly CI: provision flyctl on the host-mode runner via mise (forgejo_runner_host_tools) and drop the in-job curl | sh install, which hung indefinitely on an interactive PATH prompt on the macOS host runner (the install assumed an ephemeral minikube container). Workflow now calls flyctl directly, matching how every other host-mode job finds its tools.

• Declared AI crawlers (ClaudeBot, GPTBot, meta-externalagent, Amazonbot, …) now get a bare nginx 403 at the Fly edge before the Anubis proxy hop. Outcome of the 2026-06-11 scraper-surge review: these bots were ~88% of all forge.eblu.me traffic and their Anubis/naughty rejection pages were >99% of remaining proxy egress, enough to saturate the proxy VM under storms. Anubis is unchanged and still walls off UA-spoofing crawlers.

• Closed two backup gaps in the borgmatic role on indri. The heph hub DB (heph.db, the only copy of all task/context data) is now snapshotted by a before-backup sqlite3 .backup hook — WAL-safe and fails loud, unlike borgmatic’s native sqlite_databases hook whose .dump can fail silently on a live WAL database. navidrome’s database (users, play counts, playlists) — a gap predating the ringtail migration — is now captured by enabling navidrome’s own ND_BACKUP_* snapshots (/data/backup, daily 01:00, keep 7) and ferrying the newest snapshot off the PVC via a new borgmatic_k8s_file_dumps hook (ls/cat over kubectl exec); this added coreutils to the navidrome nix image for the in-pod tooling.

• Upgraded authentik-redis (Authentik’s cache/broker) from Redis 8.2.3 to 8.6.3, rebuilt from nixpkgs on the nix-container-builder. The deployment has no persistence (ephemeral cache/broker), so the jump carries no data-migration risk. Also reviewed the refactor-services-check-to-query-alerts doc card and the most-stale service (authentik-redis) as part of the recurring maintenance sweep.

• ringtail: add zram swap (zstd, swappiness=10) as an OOM pressure valve and set k3s --kubelet-arg=fail-swap-on=false. Scale ollama to 0 replicas (on-demand) to drop its GPU contention and large memory tail. Relieves host-level OOM kills of k3s pods when gaming pushes memory over the top.

• Downgraded the security posture: kept the weekly Prowler K8s CIS hygiene scan but retired the kingfisher secret scanner entirely (ArgoCD app, manifests, custom container, prek hook, and the forge spork). TruffleHog remains the prek secret scanner. Also remediated the outstanding app-pod seccomp Prowler findings by adding seccompProfile: RuntimeDefault to the authentik (server/worker/redis) and frigate-notify pods rather than muting them.

• Tailscale operator stack: upgraded v1.94.2 → v1.98.5 (operator, proxy, and the new local k8s-nameserver), rebuilt from the forge mirror with the go_1_26 buildGoModule override (v1.98.5 go.mod floor is >= 1.26.3). The in-cluster MagicDNS nameserver is now a local nix-built image (containers/tailscale-k8s-nameserver/), replacing the floating docker.io/tailscale/k8s-nameserver:stable tag — that mutable tag was the vector behind the v1.96.5 MagicDNS-in-containers regression.

• Upgraded kube-state-metrics v2.18.0 → v2.19.1 (service review) — nix container rebuild from the forge mirror, picking up Go-toolchain and golang.org/x CVE fixes plus the pprof auth-filter hardening from v2.19.0. Added a RuntimeDefault seccomp profile to its deployment, remediating one of the Prowler K8s CIS findings. Compliance triage of the weekly Prowler report (18 unmuted, net-zero week-over-week) reworked the mutelist for the k3s finding profile: alloy node agents, the nvidia device plugin, local-path-provisioner, the k3s cloud-controller, and kube-apiserver node-proxy access are now muted as upstream-managed system/node-agent privileges. Remaining app-pod seccomp findings (authentik, frigate-notify) are tracked for RuntimeDefault remediation.

• Upgraded the homepage dashboard v1.11.0 → v1.13.2 (service review). Rebuilt the custom nix image from the forge mirror (src + pnpm-deps hashes re-resolved, full build verified on ringtail). The range adds the ntfy widget, UniFi Drive widget, qBittorrent v5.2.0 / dispatcharr v24 API compatibility, and the GHSA-rg3r-jprv-xq38 security fix — no breaking config changes for our widget set.

• Service review: upgraded the CloudNativePG operator on ringtail from v1.27.1 to v1.29.1. The 1.27 series reached end-of-life upstream (“no longer supported”); 1.29.1 is the latest stable and carries a metrics-exporter privilege-escalation fix plus Go runtime CVE patches. Operand PostgreSQL stays at 18.3. Also corrected a stale service-versions.yaml entry (tracked 1.28.1 while the app deployed 1.27.1) and removed a dead minikube-sibling comment from the Application manifest.

• Upgraded Forgejo from v14.0.3 to v15.0.3 (major version) and made the forgejo Ansible role version-driven: forgejo_version (+ forgejo_go_version/forgejo_node_version/forgejo_build_tags) in the role defaults now pins the deployed tag, and the role fetches from the mirror, checks out, rebuilds when the running binary differs, and restarts. Bumping forgejo_version in a PR is now the whole upgrade — reproducible and DR-safe, closing the gap where the forge version lived only on indri’s filesystem. Added an upgrade-forgejo runbook (build-from-mirror flow, go-toolchain pin requirement, v15 breaking changes, SQLite DB backup, rollback).

• Upgraded Loki 3.6.7 → 3.7.2 (service review). Breaking changes in 3.7 are confined to the experimental v2 engine/scheduler, which blumeops’ single-binary deployment does not use.

• Monthly tooling dependency refresh: prek hooks (trufflehog v3.95.6, ruff v0.15.18, prettier v3.8.4, taplo SHA correction, ansible-core 2.21.1), Fly proxy images (nginx 1.30.3-alpine, alloy v1.17.0), mise-task typer pin 0.26.7, and actions/checkout v6.0.3 in Forgejo workflows.

• Daily recurring review pass (2026-06-18):

  • Service review of frigate (stalest, 86d): already at the latest upstream stable (0.17.1 — v0.17.1 is the newest GitHub release), pod healthy and serving the API. No upgrade needed; last-reviewed bumped in service-versions.yaml. Noted a high lifetime restart count (206 over 36d, last a graceful SIGTERM 3d ago, not an active crash loop) — filed a heph task to investigate.
  • Doc review of port-services-check-alerts (never reviewed): corrected the stale “blackbox exporter already covers 5 services” claim — the ringtail blackbox probe set is currently only immich (argocd/manifests/alloy-ringtail/config.alloy), so re-broadening it is now the most impactful next step. All wiki-links verified; set last-reviewed.

• Service review: bumped blumeops-pg PostgreSQL operand 18.3 → 18.4 (ghcr.io/cloudnative-pg/postgresql), picking up the May 2026 security release (11 CVEs incl. CVE-2026-6473 buffer overruns and several SQL-injection fixes); 18.x→18.x needs no dump/restore. Updated ringtail flake.lock (nixpkgs d6df3513 → 3cac626e, nixos-25.11; nixpkgs-services pin held) via dagger call flake-update.

• Daily recurring review pass (2026-06-17):

  • ntfy upgraded v2.19.2 → v2.24.0 (service review). Custom nix image rebuilt from the forge mirror; no breaking config changes across v2.20–v2.24 (S3 attachment store, verified-email recipients, ACL access cache, SQLite case-sensitive ACL fix, PWA token auto-extend). Image tag bumped in argocd/manifests/ntfy/kustomization.yaml.
  • Doc review of first-alert-and-runbook: marked the alerting POC as complete/deployed and corrected the stale “5 probed services” claim — blackbox coverage is currently only immich (see port-services-check-alerts to re-expand).

• retire-minikube phase 6: the indri forgejo-runner flips to host-mode jobs (no more runner-job-image; jobs run as erichblume with the mise toolchain, which gains pinned dagger and prek). Scope revision: Docker Desktop stays as the dagger engine host (hephaestus/cv CI also use dagger), right-sized from 6cpu/8GiB to 2cpu/4GiB. All stale arm64 build files deleted and build-container.yaml is nix-only.

• Added uv to ringtail’s systemPackages for fast Python package and project management alongside the existing mise tooling.

• Enabled systemd-oomd swap-kill on ringtail (ManagedOOMSwap=kill on the root slice, SwapUsedLimit=80%). A Crusader Kings 3 spike filled zram swap to ~95% and froze the whole host in a multi-minute memory-pressure stall (PSI memory full avg300≈30%) — sshd and the k3s API included. oomd was running but monitored zero cgroups, so nothing got killed. Now oomd kills the heaviest-swap cgroup when swap crosses 80%; because k3s pods are pinned to no-swap (so kubepods hold ~0 swap vs the gaming session’s ~15G), the victim is always a non-k3s process. Pod OOM remains the job of each pod’s memory limit plus the kernel cgroup OOM-killer. Pressure-based root killing (enableRootSlice) was deliberately avoided — it selects by memory footprint and could reap a pod before the game.

• Pinned sifaka to its LAN IP (192.168.1.203) in ringtail’s /etc/hosts via networking.hosts, so NFS mounts resolve over the LAN instead of tailscale MagicDNS. On 2026-06-26 sifaka’s tailscale node key expired; MagicDNS kept resolving sifaka to the now-dead node and every NFS mount on ringtail hung (kiwix, transmission, immich, paperless, etc.). The LAN path is authoritative (/etc/hosts beats MagicDNS), keeps NFS traffic off the tailnet, and is immune to tailscale node-key churn — implementing the design the immich pv-nfs.yaml comment always described. sifaka’s NFS export already permits 192.168.1.0/24. Also disabled key expiry on sifaka’s tailscale node so it stops expiring.

• Removed librewolf from ringtail’s systemPackages. After the nixpkgs flake bump, nixpkgs marks librewolf-151.0.2 insecure (unmaintained in nixpkgs, not a specific CVE), which blocked nixos-rebuild. Firefox remains the configured default browser; librewolf was a standalone extra.

• Removed the retired minikube entry from service-versions.yaml — minikube on indri was retired 2026-06 (see retire-minikube); the service-review staleness queue no longer lists a non-existent service.

Documentation

• Corrected the ServiceProbeFailure runbook’s Affected Services table: verified against live Prometheus that the Alloy blackbox exporter now probes only immich (job=integrations/blackbox/immich), so the table no longer lists the retired indri-era services (miniflux/kiwix/transmission/devpi/argocd). Added superseded notes to the historical zot version-sync/tagging design cards (pin-container-versions, add-container-version-sync-check, adopt-commit-based-container-tags) pointing at the nix-only reality.
• Condensed the docs/how-to/zot/ cards, which were verbose mikado-chain leftovers. Merged the four overlapping container-versioning cards (pin-container-versions, add-container-version-sync-check, adopt-commit-based-container-tags, add-dagger-nix-build) into a single container-versioning card documenting the current nix-only model, and trimmed the zot-hardening cards (harden-zot-registry, wire-ci-registry-auth, register-zot-oidc-client) down to tight present-tense how-tos — dropping the What Was Done / Verification checklists and file-listing tables that read as project records rather than documentation.
• Doc review: purged retired-minikube references from the four alerting runbooks (service-probe-failure, postgres-unhealthy, pod-not-ready, textfile-stale) — --context=minikube-indri → --context=k3s-ringtail, “indri’s minikube cluster” → “ringtail’s k3s cluster”, and minikube health checks → k3s-on-ringtail equivalents. Also corrected the textfile-stale collector table (dropped the dead minikube.prom, added the live forgejo.prom and macos_power.prom).
• Reframed the security docs away from compliance-program emulation: the security reference card was retitled “Security & Compliance” → “Security”, dropped the PCI DSS / SOC 2 / ISO 27001 framework table, and now frames the Prowler scan as a hygiene check. Stale minikube-indri and argocd/manifests/prowler/ paths corrected to ringtail / prowler-ringtail/. Spork docs note that kingfisher (the worked example) has been retired while the spork machinery stays for future sporks.
• Doc review: deploy-infra-alerting runbook updated — the services-check coverage table no longer references the retired minikube cluster (k3s on ringtail is the only cluster post-retire-minikube).
• Doc review: configure-grafana-alerting-pipeline accuracy pass — corrected the stale “Grafana and ntfy are on different clusters” rationale. Both now run on ringtail’s k3s since the minikube retirement, so the cross-cluster Caddy workaround is no longer load-bearing (cluster-internal ntfy DNS is now a viable future simplification).
• Reviewed runbook-argocd-out-of-sync.md (never reviewed). Corrected the stale “30+ minutes” claim — the ArgoCDAppOutOfSync alert actually fires after a 5m for: window. Verified both wiki-links resolve and the alert is defined in grafana-ringtail/alerting.yaml. Set last-reviewed: 2026-06-21.
• Docs review: verified the snowflake-proxy reference card against the running service (snowflake-proxy.service active on ringtail, Prometheus metrics on :9999, v2.11.0 matching nixpkgs) and stamped it last-reviewed: 2026-06-23.
• Recurring reviews (2026-06-29): doc review of the Frigate service card (corrected stale image tag rc2 → 0.17.1-tensorrt, stamped reviewed); service review surfaced Forgejo (see infra entry); weekly Prowler K8s CIS compliance scan all-clear (0 unmuted FAILs, net-zero week-over-week).
• Doc review: stamped observability reference card as reviewed — all five component and seven alerting/runbook wiki-links resolve, and the Pyroscope (blocked on ringtail kernel sysctls) and Faro RUM (not deployed) future sections remain accurate.
• Doc review: automounter card updated for post-retire-minikube reality — consumers table now reflects NFS-direct ringtail workloads, adds the shower share, and flags vestigial mounts (music, torrents, frigate).
• Documented the flyio-proxy Tailscale node name drift (flyio-proxy → flyio-proxy-1 → -2): caused by ephemeral microVM state with no persisted /var/lib/tailscale, benign because routing/ACLs are tag-based and offline nodes auto-GC. Recorded the Fly-volume fix and the decision not to apply it (volume anchors the otherwise stateless proxy to one host).
• manage-forgejo-mirrors: document that the old GitHub fine-grained PAT shows “Last used: never” even after weeks of active syncing (last-used tracking is unreliable for git-over-HTTPS), and add a rate_limit auth check as the reliable way to verify a rotated mirror PAT.

AI Assistance

• Retired the “migrate every container to a locally built image” campaign. The reachable wins are done — all remaining upstream images (frigate’s TensorRT build, immich’s CUDA ML stack, the CloudNativePG PostgreSQL operands) are impractical to rebuild in Nix and stay on their upstream registries. AGENTS.md now frames local container builds as guidance for new services rather than a goal of universal localization, and the recurring “pick one non-local container and make it local” task has been dropped.

Miscellaneous

• Pruned dead minikube-era entries from the Prowler mutelist (removed apiserver.yaml, control-plane.yaml, manual-node-checks.yaml; trimmed core-pod-security.yaml), leaving only entries that match live k3s resources. No change to scan output (0 unmuted FAILs).

[v1.18.0] - 2026-06-11

Bug Fixes

• Granted the offline_access scope on the Authentik heph OAuth2 provider so hephaestus spokes receive a durable 30-day refresh token. Previously the refresh token was session-bound, so spoke sync would silently fail with a 400 Bad Request on the refresh_token grant once the Authentik session lapsed.
• Fixed the tailscale-operator and tailscale-operator-ringtail ArgoCD apps showing Unknown sync status. Their shared base kustomization fetched the upstream operator manifest from the public forge.eblu.me/mirrors/..., which the AI-scraper mitigation now black-holes (403). Pointed the remote resource at the tailnet host forge.ops.eblu.me instead, which the in-cluster repo-server can reach.
• Upgraded Jellyfin on indri from 10.11.6 to 10.11.11, picking up the security fixes in 10.11.7 (disclosed CVEs/GHSAs, flagged “upgrade immediately”) and 10.11.10 (three further GHSAs). Noted the recurring gotcha in the service-versions tracking: after a brew upgrade --cask jellyfin, the re-quarantined .app makes the launchd-spawned process hang silently until the Gatekeeper first-launch dialog is approved on indri’s GUI console — removing the quarantine xattr over SSH is blocked by macOS TCC.

Infrastructure

• Deployed Anubis v1.25.0 (proof-of-work anti-scraper gateway) on the Fly.io proxy in front of forge.eblu.me, after an AWS-hosted crawler with no declared bot UA DoS’d Forgejo by walking the /eblume/* commit-history surface at ~5.7 req/s. Browsers clear one JS challenge per week; git and API clients pass through untouched; declared AI crawlers are denied. The tailnet path (forge.ops.eblu.me) is unaffected. Tier 2b of the AI scraper mitigation plan.
• Completed the external-secrets localization for the ringtail (amd64) cluster. The indri Dagger build (container.py) only produces an arm64 image; added containers/external-secrets/default.nix to build the amd64 variant on ringtail’s nix-container-builder, and gave external-secrets-ringtail a thin kustomize overlay that reuses the shared manifest and points at the -nix image. Both clusters now run the locally-built external-secrets binary on their native architecture.
• Added the hephaestus (heph) sync hub to indri as a self-updating LaunchAgent managed by Ansible (ansible/roles/heph, tag heph). The hub runs hephd --mode server behind heph.ops.eblu.me (Caddy TLS), with self-update on a 10-minute interval and the heph-pwa mobile shell served from --web-root. Access is gated by a new Authentik device-code (RFC 8628) OIDC application. Indri is now the canonical hub; other devices (e.g. gilbert) attach as offline-capable spokes. The hub’s store was seeded from gilbert via the data-safe Path A bring-up (copy store, reset meta.origin).
• Registered the heph-pwa redirect URIs (https://heph.ops.eblu.me/, plus http://localhost:8787/ for dev) on the Authentik heph OAuth2 provider, enabling the PWA’s new Authorization Code + PKCE “Login with Authentik” flow (and the token-endpoint CORS it needs). Pairs with hephaestus PR #9.
• Localized the external-secrets controller image. It now builds from the forge mirror via a native Dagger container.py (single all_providers static Go binary, faithful to upstream’s make build) and is served from registry.ops.eblu.me/blumeops/external-secrets instead of ghcr.io, bringing another platform component under local supply-chain control.
• Localized the Tailscale operator stack: the k8s-operator image (both clusters) and the ProxyClass proxy image (indri, completing parity with ringtail) are now built from the forge mirror instead of pulled from Docker Hub.
• Phase 0 of retire-minikube: the forgejo-runner becomes a native macOS LaunchAgent on indri (new forgejo_runner ansible role), running jobs against Docker Desktop instead of a DinD sidecar in minikube. New runner identity indri-runner advertising both k8s (compat) and indri labels; zot registry mirror moves into Docker Desktop’s daemon.json; runner logs ship to Loki via alloy.
• retire-minikube phase 1 complete: prowler migrated to ringtail k3s (Nix-built from nixpkgs at 5.12.3 — a step back from 5.23.0 until nixpkgs catches up; same cis_1.11_kubernetes framework). Weekly CIS scans now target ringtail, k3s-appropriate node mounts, trivy + the image/IaC Dockerfile retired.
• Retired the Prowler container-image CVE scan and IaC scan, keeping only the K8s CIS benchmark scan. The two retired scans generated tens of thousands of un-actioned, un-muted findings every week (~20,000 image findings and growing, mostly unpatchable upstream-image CVEs; ~650 systemic Trivy KSV pod-security warnings) — the weekly mise run review-compliance-reports re-surfaced them all as “action needed” though none were ever triaged. The K8s CIS scan is fully mutelisted and runs clean, so it stays. Removed the two CronJobs, the now-unused trivyignore.yaml mutelist, and the grouped-findings rendering in the review tool that existed solely for the high-volume scans.
• Upgraded Prometheus v3.10.0 → v3.12.0. Picks up fixes for stored XSS (CVE-2026-40179, GHSA-fw8g-cg8f-9j28) and remote-write/remote-read snappy decompression DoS (CVE-2026-42154), plus TSDB performance improvements. No breaking changes affect our deployment.
• retire-minikube phase 1 continued: navidrome migrated from minikube to ringtail k3s (Nix-built 0.61.1, exact version lift-and-shift; data PVC copied, music NFS remounted from sifaka).
• retire-minikube phase 4: ArgoCD self-migrated to ringtail k3s — the last workload off minikube. All 32 ringtail app destinations rewritten to in-cluster, the 13 minikube-only Application definitions deleted (their live workloads run unmanaged until phase 5), the argocd metrics job back in-cluster, and the admin password rotated on the fresh install. Minikube’s ArgoCD is scaled to 0 as rollback.
• retire-minikube phase 2 groundwork: ringtail blumeops-pg gains a managed authentik role (password sourced from the same 1Password item the authentik app reads, so the cutover is only a postgresql-host flip), and borgmatic’s authentik entry moves from port 5432 (minikube) to 5434 (ringtail). The authentik database itself is dump/restored at the cutover window per the plan card.
• retire-minikube phase 1 closed out: the seven parked minikube workloads (miniflux, navidrome, torrent, kiwix, unpoller, prowler, the k8s forgejo-runner) are decommissioned — ArgoCD apps cascade-deleted and manifests removed. All seven now serve from ringtail (or, for the runner, as a launchd service on indri). The miniflux database remains on minikube blumeops-pg untouched until the cluster itself retires in phase 2. Also restores the kiwix zim-hash ignoreDifferences that the ringtail app definition dropped in the port (the zim-watcher CronJob patches that annotation at runtime, so without it the app reads permanently OutOfSync).
• retire-minikube phase 5: minikube fully decommissioned. The 12 remaining minikube manifest dirs deleted, ansible minikube/ minikube_metrics roles removed, the Caddy L4 :5432 route and its .pgpass line retired, services-check rewritten for the single-cluster world (ArgoCD app table now reads ringtail), ensure-minikube-indri-kubectl-config deleted, the compliance report tooling’s minikube node-verification removed (k3s equivalent tracked in heph), tailnet tags tag:k8s-api/tag:loki/tag:pg swept from pulumi, and the docs sweep (AGENTS.md rule 2 inverted to k3s-ringtail, restart-indri/architecture/cluster/tailscale-operator/indri/ disaster-recovery cards revised, rebuild-minikube-cluster deleted).
• retire-minikube phase 3 groundwork: Nix default.nix ports for the five LGTM-stack containers (prometheus v3.12.0 with the embedded mantine UI built from source, grafana 12.4.2 from the official release tarball, loki v3.6.7, tempo v2.10.3, grafana-sidecar 2.6.0), all build-verified on ringtail. Their Dockerfile/dagger build paths are retired. The *-ringtail manifests stage the move: prometheus-ringtail gains in-cluster CNPG metrics scrapes for both ringtail pg clusters (previously dark), alloy-k8s repoints to the external LGTM names and absorbs the argocd/kube-state-metrics scrapes, and the PVC data copies
  • cutover follow on this branch.
• Upgraded the nvidia-device-plugin on ringtail from v0.19.0 to v0.19.2 (upstream patch release: CDI/Tegra fixes and dependency bumps, no breaking changes for our manifest-based CDI + RuntimeClass setup).
• Bumped the indri heph hub to v1.2.1, which adds the hub GET /config endpoint and ships the heph-pwa Login with Authentik flow (Authorization Code + PKCE). Pairs with the Authentik heph provider redirect URIs registered earlier.
• Rebuilt all six migrated phase-1 service images from main (-f386e2e-nix tags) and repointed the ringtail deployments at them, closing out the branch-built artifacts.
• Rebuilt the external-secrets images off main and repointed both clusters to the stable main-sha tags (v2.2.0-13895bb arm64 / v2.2.0-13895bb-nix amd64), so the deployed images on indri and ringtail trace to the same main commit rather than earlier feature-branch builds.
• Rebuilt the locally-built external-secrets image from the main branch so the deployed tag (v2.2.0-0e70a1b) traces to a main commit rather than the now-merged feature branch, giving a stable provenance reference.
• Updated ringtail NixOS flake inputs (nixpkgs nixos-25.11, disko) to latest via dagger call flake-update.
• retire-minikube phase 3 close-out: the five LGTM images rebuilt from main (-8b1f89e-nix) and the ringtail apps repointed to them.

Documentation

• Reviewed the five stalest documentation cards (argocd, authentik, grafana, unifi, plan-a-meal): brought ArgoCD’s SSO/dual-cluster/sync-policy story up to date, expanded Authentik’s blueprint and OIDC client inventory to all eight clients, fixed Grafana’s TeslaMate datasource target and dashboard list, and noted UnPoller’s locally-built image.
• Kick off the minikube retirement: umbrella migration plan (retire-minikube) to move all remaining k8s workloads (miniflux, navidrome, torrent, kiwix, unpoller, prowler, the LGTM observability stack, authentik’s database, and ArgoCD itself) from indri/minikube to ringtail/k3s, convert the forgejo-runner to a native launchd service on indri, and decommission minikube.
• Reviewed four never-reviewed reference cards (cluster, ntfy, tempo, alloy) and corrected drift: minikube is now Kubernetes v1.35.0; ntfy, tempo, and alloy-k8s images are now locally-built registry.ops.eblu.me/blumeops/* nix containers (v2.19.2, v2.10.3, v1.16.0) rather than upstream Docker Hub; the Fly.io alloy binary is v1.16.1; and the ringtail workload list reflects the in-progress minikube→k3s migration.
• Corrected the 1Password backup how-to: the desktop app’s export menu item is named after the account (“File > Export > Blume/Davis”), not “All Vaults”. Verified an account export contains all four vaults (Private, blumeops, Payrix, Shared).
• Reviewed the tailscale-operator reference card: documented the dual indri/ringtail deployment, corrected the ArgoCD apps list, pinned the upstream version, and added the ProxyGroup Ingress host: caveat.
• retire-minikube phase 2 cutover complete: authentik now reads its database from ringtail blumeops-pg in-cluster (row-exact restore, SSO verified). Minikube blumeops-pg soaks idle until ~2026-06-18, then retires with cnpg, the Caddy L4 :5432 route, and its .pgpass line.

AI Assistance

• Retired the ai-docs mise task and its mandatory session-start rule: the concatenated docs corpus had grown to ~130K tokens, too large to ingest wholesale. Agents now start tasks by finding and reading the relevant docs (grep + wiki-links); ai-sources remains for opt-in deep codebase context. Also documented the full heph CLI task workflow (read, log, complete, create) in AGENTS.md.

Miscellaneous

• Service review: AutoMounter on indri is current at 1.13.0 (App Store auto-updated from the tracked 1.11.0); all sifaka SMB mounts verified healthy. Fixed the stale tracking-file path shown by mise run service-review.