v1.15

Changelog archive — v1.15

Moved verbatim from CHANGELOG.md; never edited.

[v1.15.7] - 2026-04-18

Bug Fixes

• Fix borgmatic LaunchAgent failing silently due to macOS TCC permission dialogs. LaunchAgents now call borgmatic directly instead of routing through mise x, which triggered “wants to access Documents” dialogs that hung headless sessions. The ansible role now also manages borgmatic installation via mise install.

Infrastructure

• Automate verification of Prowler MANUAL findings (kubelet file perms, kubelet config, etcd CA, RBAC cluster-admin) in review-compliance-reports and mute them with node-config-automated-verification compensating control.
• Migrate transmission and transmission-exporter containers from Dockerfile to native Dagger builds (container.py). Updates base images to Alpine 3.23 and Python 3.14, pins uv to 0.11.6.
• Switched Fly proxy to upstream keepalive pools, reducing forge.eblu.me latency from 35s+ p50 to sub-second. Added mise run fly-reload for DNS re-resolution without redeploy.
• Upgrade Prowler from 5.22.0 to 5.23.0; remove init container workaround for broken --registry flag (upstream fix in PR #10470).
• Added robots.txt to forge.eblu.me blocking crawlers from /mirrors/ to reduce load from Facebook scraping.
• Container builds are now manual-only via mise run container-build-and-release. Removed auto-trigger on push to main — shared Dagger helpers made path-based detection unreliable.
• Migrate devpi container from Dockerfile to native Dagger build; bump devpi-server 6.19.1→6.19.3 and devpi-web 5.0.1→5.0.2.
• Migrated kiwix-serve container from Dockerfile to native Dagger build, bumping Alpine base from 3.22 to 3.23.
• Mitigated Forgejo archive endpoint DoS: redirect public archive requests to tailnet, expanded robots.txt, enabled archive cleanup cron, cached release downloads at proxy.
• Refactored Dagger container pipelines: extended go_build() helper with buildmode and extra_env params, migrated miniflux and forgejo-runner to use it, and standardized all Alpine bases from 3.22 to 3.23.

Miscellaneous

• Review compensating control sso-gated-admin-tools: tightened scope to ArgoCD only, removed Grafana reference.
• container-build-and-release now verifies the commit exists on the remote before dispatching a build.

[v1.15.6] - 2026-04-14

Bug Fixes

• Rotate ArgoCD workflow-bot token and admin password after DR rebuild invalidated signing keys, fixing build-blumeops workflow failures.

[v1.15.5] - 2026-04-14

Features

• Deploy Paperless-ngx document management system at paperless.ops.eblu.me with OCR, Authentik SSO, and NFS storage on sifaka.
• Add ty (Astral) Python typechecker to prek hooks, configured for Dagger SDK and container.py modules. Add type: mise to service-versions.yaml for tracking development tool versions (dagger, ansible-core, prek, pulumi, ty) through the standard service review process.
• Upgrade grafana-sidecar from 1.28.0 to 2.6.0, adding health probes and porting build to native Dagger container.py.
• Upgrade Navidrome to v0.61.1 — major artwork overhaul with per-disc cover art, rebuilt search engine (SQLite FTS5), server-managed transcoding, and WebP performance fix.
• Add mise run review-compliance-reports task for weekly compliance report review with muted/unmuted distinction and week-over-week delta

Bug Fixes

• Add paperless database to borgmatic backup configuration. Previously the only service DB not included in nightly pg_dump backups.
• Fix Fly.io proxy rate limiting to key on real client IP instead of Fly’s internal proxy IP, so crawlers no longer consume the shared rate limit bucket for all clients.
• Fix UnPoller (UniFi) Grafana dashboards failing to load due to UID exceeding Grafana 12’s 40-character limit.
• Fix blumeops-tasks swallowing wiki-link brackets in task descriptions (rich markup escaping)
• Fix dagger flake-update pipeline: replace nonexistent --exclude flag with dynamic input discovery
• Fix services-check to display all firing alerts for a given alert name, not just the first one.
• Pin Fly.io proxy Tailscale to v1.94.1 — the :stable tag pulled v1.96.5 which has a MagicDNS regression (SERVFAIL on tailnet names), breaking all public routing through forge.eblu.me, docs.eblu.me, and cv.eblu.me.
• Rewrite mise run runner-logs CLI: list runs by run number (not task ID), drill into jobs per run, fetch logs via Forgejo web API instead of SSH+filesystem. Fixes broken log retrieval caused by incorrect hex path calculation and stale data directory. Added --repo to query any forge repo (e.g. sporks) and --limit/-n to control listing size (0 for all).
• Route Dagger build telemetry to Tempo, fixing OTEL metrics exporter warnings.
• Switch paperless redis sidecar from amd64-only nix-built authentik-redis image to upstream valkey:8.1-alpine (multi-arch). The nix image was previously running under QEMU emulation on arm64 minikube.

Infrastructure

• Build forgejo-runner container locally via native Dagger pipeline instead of pulling from upstream.
• Build kube-state-metrics container locally (Dockerfile + nix) from forge mirror, replacing upstream registry.k8s.io image on both indri and ringtail.
• Upgrade miniflux from 2.2.17 to 2.2.19 and migrate from Dockerfile to native Dagger container.py build (second container after navidrome). Refactor alpine_runtime() with create_user parameter to support Alpine’s built-in nobody user. Pin all mise.toml tool versions to explicit versions instead of “latest”.
• Migrate Dagger module from .dagger/ to repo root (src/blumeops/) and replace docker_build() with native Dagger pipelines for container builds. Navidrome is the first container migrated, with full build error visibility.
• Migrate teslamate container build from legacy Dockerfile to native Dagger container.py.
• Add seccomp RuntimeDefault profiles to alloy-k8s and immich pods, resolving 4 unmuted Prowler findings
• Full DR recovery from power loss and minikube cluster rebuild. Validated bootstrap procedure, identified circular dependencies (forge.eblu.me, Zot/Authentik OIDC), Tailscale device name collision issues, and documented recovery steps for restart-indri.
• Set Frigate preview quality to CRF 8 (from default 1) to reduce preview file sizes and improve review timeline loading over NFS.
• Track Fly.io proxy component versions (Tailscale, nginx, Alloy) in service-versions.yaml with new fly service type.
• Upgrade ArgoCD from v3.3.2 to v3.3.6 (bug-fix patches), SHA-pin install manifest
• Upgrade authentik 2026.2.0 → 2026.2.2 (bug-fix patch release)
• Upgrade ollama from 0.17.5 to 0.20.4 (adds Gemma 4 support, benchmark tooling, Apple Silicon perf improvements)

Documentation

• Delete outdated install-dagger-on-nix-runner card; add service-versions reference card; clean up zot.md and review-services.md links.
• Enhanced the adding-a-service tutorial with kustomization setup, corrected Tailscale ingress format, updated ArgoCD repoURL, and added a step for creating service reference cards.
• Review gandi.md: add missing forge.eblu.me CNAME, fix program description, stamp review date.

[v1.15.4] - 2026-04-06

Infrastructure

• Migrate 1Password Connect from Helm to kustomize (1.8.1 → 1.8.2), completing the no-helm-policy migration.

Documentation

• Rewrite observability stack tutorial: replace Helm instructions with actual kustomize/ArgoCD patterns, fix typos, document Alloy as core component

[v1.15.3] - 2026-04-05

Infrastructure

• Build Tempo container from source via forge mirror; bump 2.10.1 → 2.10.3
• Pin NixOS service versions (forgejo-runner, snowflake, k3s) via nixpkgs-services overlay in ringtail flake, preventing silent upgrades from nix flake update. Add k3s and minikube to service-versions.yaml tracking. Fix stale nix-container-builder version (was 12.6.4, actually running 12.7.2).
• Migrate Immich from Helm chart to kustomize manifests and upgrade from v2.5.6 to v2.6.3
• Upgrade Grafana from 12.3.3 to 12.4.2 — patches 7 CVEs including an unauthenticated DoS (CVE-2026-27880).

Documentation

• First compensating control review: verified single-user-cluster still in effect. Added aspirational how-to card for PCI DSS evidence collection.
• Prowler --registry fix merged upstream (PR #10470); initContainer workaround documented as pending release.

[v1.15.2] - 2026-03-30

Features

• Build custom Kingfisher container from sporked deploy branch, replacing upstream image with locally-built version including —clone-url-base patch.
• Add Kingfisher secret scanner as a weekly CronJob scanning all Forgejo repos, with HTML and JSON reports written to sifaka NFS.
• Add MongoDB Kingfisher secret scanner as a prek hook alongside TruffleHog for comparative coverage evaluation.
• Add spork strategy: floating-branch soft-fork tooling (mise run spork-create) and documentation for maintaining local patches against upstream projects.

Infrastructure

• Add compensating controls framework: tracking file, review mise task, and how-to doc. Map all Prowler mutelist entries to named controls with CC: prefixes.
• Add Prowler mutelist to suppress expected findings from system components, operator-managed pods, and accepted operational needs. Fix missing seccomp profile on kube-state-metrics.
• Borgmatic photos backup: restrict to library/ and upload/ (skip regenerable dirs), add SSH keepalives and checkpoint interval to prevent broken pipe failures on large initial syncs.
• Upgrade forgejo-runner from 12.7.0 to 12.7.3 (bug fixes, security dep update). Add service reference card.

Documentation

• Add service reference documentation for Kingfisher secret scanner.
• Review and update Ansible reference doc: add missing roles, sibling playbooks, and clarify Ansible’s role in the IaC stack.

[v1.15.1] - 2026-03-28

Features

• Add Tor Snowflake proxy on ringtail as a systemd service to support anti-censorship efforts.
• Add offsite backup for immich photo library to BorgBase, running daily at 4 AM from indri via sifaka SMB mount.
• Add QArt Tuner — a Go tool that generates QR codes whose data modules form a recognizable image, with an interactive web UI for parameter tuning. Based on the QArt technique by Russ Cox. Lives in utils/qart/.

Infrastructure

• Migrate Forgejo from Homebrew to source build with mcquack LaunchAgent, matching the pattern used by zot, caddy, and alloy. Upgrades to v14.0.3 (7 security fixes including PKCE bypass and OAuth scope bypass).
• Add borgmatic pg_dump backups for authentik and immich databases. Authentik uses the existing blumeops-pg cluster on port 5432. Immich requires a new borgmatic role on the immich-pg cluster, a Tailscale service, and Caddy L4 proxy on port 5433.
• Upgrade External Secrets Operator from v1.3.2 to v2.2.0 and migrate from Helm chart to static kustomize manifests.
• Add post-deploy maintenance docs and generation pruning task for ringtail.
• Fix Immich Helm values: resource limits and probe timeouts were silently ignored due to wrong value keys. Resources now actually apply to pods, and liveness/readiness probe timeouts increased from 1s to 5s to prevent kubelet from killing pods during ML inference.
• Reduce PodNotReady alert lookback window from 5m to 60s to clear faster after rollouts.
• Tighten ArgoCDAppOutOfSync alert: reduce pending duration from 30m to 5m and lookback window from 5m to 1m so alerts clear faster after sync.
• Update ringtail flake inputs (nixpkgs, home-manager).
• Upgrade Homepage dashboard from v1.10.1 to v1.11.0
• Upgrade nvidia-device-plugin from v0.18.2 to v0.19.0

Documentation

• Review and fix CV service doc (correct URL, forge domain, container tag link) and add private forge repo review guidance to review-services process.
• Review tailscale-setup tutorial: fix macOS install steps, add --accept-routes tip, correct tag name, add ACL apply instructions, add [[tailscale-operator]] cross-reference.

Miscellaneous

• Add preserve/* branch prefix exclusion to branch-cleanup task; document Pyroscope profiling work and blockers in observability reference.

[v1.15.0] - 2026-03-24

Features

• Deploy Prowler CIS scanner as a weekly CronJob on minikube-indri, with reports written to sifaka NFS share.
• Add Grafana “Alerts” dashboard showing currently firing alerts and recent state changes.
• Add IaC scanning via Prowler IaC provider (Saturday 2am, Dockerfiles and K8s manifests).
• Add container image vulnerability scanning via Prowler image provider (Saturday 3am, all blumeops/* images).

Bug Fixes

• Fix authentik worker OOMKill by setting AUTHENTIK_WORKER_CONCURRENCY=2 (was defaulting to 16 based on CPU count).
• Remove group: "" from tailscale-operator ignoreDifferences — ArgoCD normalizes away the empty string, causing permanent OutOfSync on the apps app.

Infrastructure

• Decommission JobSync service — removed ArgoCD app, k8s manifests, container build, Caddy proxy, Homepage entry, docs, and forge mirror. Replaced by datasette-based job tracking (coming soon).
• Localize authentik-redis container: replace upstream redis:7-alpine with nix-built image from nixpkgs (Redis 8.2.3). Introduces attached service pattern with parent field in service-versions.yaml and version assertion in default.nix to prevent silent version drift.
• Unified Dockerfile and Nix container build workflows into a single workflow that auto-classifies containers by build type and routes to the correct runner (k8s for Dockerfile, nix-container-builder for Nix). Removed nettest container (outgrown). Nix builds now require an explicit version = "..." declaration — no implicit nixpkgs fallback.
• Monthly tooling dependency update: bump prek hooks (trufflehog 3.94.0, ruff 0.15.7, shfmt 3.13.0), Fly.io images (nginx 1.29.6, Alloy 1.14.1), actions/checkout v4.3.1→v6.0.2, tighten mise task Python lower bounds (rich 14, typer 0.24, httpx 0.28.1, pyyaml 6.0.2), and bump ansible-lint/ansible-core floors.
• Upgrade ntfy v2.17.0 → v2.19.2 (adds experimental PostgreSQL support, read replicas, web push fixes)
• Revert Tailscale operator to v1.94.2 (v1.96.3 images not yet published); keep Fly proxy tailscale wait improvement
• Add RuntimeDefault seccomp profiles to all managed deployments, statefulsets, and cronjobs.
• Upgrade Frigate from 0.17.0-rc2 to 0.17.1 (security fixes, bugfixes). Add motion retention tier (365 days), reduce continuous retention from 180 to 30 days.

Documentation

• Review and fix ArgoCD config tutorial: correct sync policy example, fix typo, add missing cross-references and frontmatter.
• Review and update 12 reference docs: fix stale image references to point at kustomization manifests instead of hardcoded tags, correct Prometheus scrape target, expand external-secrets stub, add cross-references between backup/disaster-recovery docs, and remove misleading .ts.net URLs from Quick Reference tables.