BlumeOps Docs

Review Services

8 min read

Review Services

How to periodically review BlumeOps services for version freshness and upgrade opportunities.

Review by Staleness

Show services sorted by when they were last reviewed (most stale first):

mise run service-review

This reads the tracking file at service-versions.yaml (repo root) and sorts by the last-reviewed field. Services without a review date float to the top. The script shows a staleness table and then displays the most stale service with a review checklist.

To show more entries in the table:

mise run service-review --limit 30

To filter by service type:

mise run service-review --type argocd
mise run service-review --type ansible
mise run service-review --type container

See service-versions for the full list of types in use.

Review Process by Service Type

For all service types, start by reading the service’s reference card (docs/reference/services/<service>.md) for architecture, configuration, and endpoint details.

ArgoCD Services (type: argocd)

  1. Check the upstream releases page for new versions
  2. Compare to the image tag in argocd/manifests/<service>/kustomization.yaml (images[].newTag)
  3. Review the upstream changelog for breaking changes
  4. If the service uses a custom-built container, also check the nixpkgs pin / upstream pin in containers/<service>/default.nix and follow build-container-image to rebuild
  5. If upgrading, update the manifest and follow deploy-k8s-service

Ansible Services (type: ansible)

  1. Check the upstream releases page for new versions
  2. Review the role’s vars/defaults for version pins in ansible/roles/<service>/
  3. If upgrading, update the version and dry-run: mise run provision-indri -- --tags <service> --check --diff
  4. Follow add-ansible-role patterns for role changes

NixOS Services (type: nixos)

Versioned NixOS services are pinned via two frozen inputs in nixos/ringtail/flake.nix: forgejo-runner comes from the nixpkgs-runner input; snowflake, k3s, and kubectl come from nixpkgs-services. Both are bare-SHA pins, listed in the ringtail-flake-update SKIP list and rev-checked by flake-lock-check, so nix flake update never moves them — they only change when one of the two inputs is deliberately updated.

  1. Check the upstream project for new releases
  2. Check what version nixpkgs has: nix eval github:NixOS/nixpkgs/<rev>#<pkg>.version with the pin’s rev from flake.nix (from the talos pod you can nix eval the pinned nixpkgs refs directly — it has nix and no ringtail SSH)
  3. To upgrade, update the relevant rev in flake.nix (the nixpkgs-runner rev for forgejo-runner, the nixpkgs-services rev for the other three) to a nixpkgs commit that includes the desired version, then run nix flake lock from nixos/ringtail/ to update the lock entry
  4. Deploy: merge the blumeops PR — ringtail-rebuild auto-applies it to ringtail (break-glass alternative: the local apply in docs/reference/operations/disaster-recovery.md)
  5. Update service-versions.yaml with the new version

Mise Tools (type: mise)

Development tools managed via mise.toml with pinned versions. These are local CLI tools (pulumi, prek, ty, ansible-core) rather than deployed services.

  1. Check the upstream releases page for new versions
  2. Review the changelog for breaking changes
  3. Update the pinned version in mise.toml
  4. Run mise install to verify the new version installs correctly
  5. Update service-versions.yaml with the new version

Container Images (type: container)

The locally built images (tailscale, tailscale-k8s-nameserver) are built from the derivation in containers/<service>/default.nix and pushed to the registry on merge to main. current-version pins the upstream version, and the nix file declares it too (version = "…" plus the fetch rev/hash) — all declaration sites must agree.

  1. Check the version pins in containers/<service>/default.nix and confirm they match service-versions.yaml — mise run container-version-check verifies the consistency
  2. To bump: update the version pins in that file and push — the PR build check resolves the fetch/FOD hashes for you; merge to main triggers the registry push
  3. horkos then opens the kustomization pin-bump PR against argocd/manifests/<service>/kustomization.yaml; merge it to deploy (the app syncs itself)
  4. Confirm the deployed image tag matches what you expected

Fly Proxy Apps (type: fly)

The Fly proxy is one Dockerfile (fly/) with its component versions digest-pinned on the FROM base image and the COPY --from stages (nginx@sha256:…, tailscale/tailscale@sha256:…, grafana/alloy@sha256:…, techarohq/anubis@sha256:…).

  1. The version pins live in fly/Dockerfile — bump the tag and digest there
  2. Deploy via mise run fly-deploy (a [human] task — it needs the fly.io API token); from an agent, file a request: mise run request-run deploy-fly.yaml <sha> --pr N -i revision=<sha>, and the warrant dispatches it after approval
  3. Confirm the deploy landed with the expected image digest

Private Forge Repos (upstream-source under forge.ops.eblu.me/eblume/)

Some services are built from private repos on the forge rather than tracking an external upstream project. When upstream-source points to a forge.ops.eblu.me/eblume/ repo:

  1. Clone the repo to ~/code/personal/ if not already checked out
  2. Review the repo’s dependency pins — uv script metadata, pyproject.toml, package.json, flake.nix inputs, etc.
  3. Update stale dependencies and rebuild locally to verify nothing breaks
  4. If changes were made, commit, push, and trigger a new release from that repo
  5. Back in blumeops, update the container image or release artifact reference as needed

This extends the service review into the source repo’s build-time dependencies, which would otherwise be a blind spot — the blumeops-side review only covers the deployment manifest and container base image.

Self-Released Services (release: self)

First-party services are not reviewed against an upstream — their version is advanced by their own release pipeline (release CI publishes; a pin PR bumps current-version in blumeops). A review of one means:

  1. Confirm the last release actually published (source repo’s release page/tags)
  2. Confirm the pin/deploy it produced landed (pin PR merged, app synced)
  3. Confirm the deployed version matches current-version in service-versions.yaml
  4. Confirm the app is healthy
  5. Refresh build-time dependencies in the source repo, as in the Private Forge Repos steps above

These services stay in the staleness queue — last-reviewed still measures human attention; only the checklist changes.

Attached Services

Some services have auxiliary dependencies that run as separate containers — caches, sidecars, init helpers. These are tracked as attached services with a naming convention and an optional parent field:

- name: authentik-redis
  type: argocd
  parent: authentik
  current-version: "8.2.3"
  upstream-source: https://github.com/redis/redis/releases
  notes: >-
    Attached service: Redis cache/broker for Authentik.

Conventions:

  • Naming: <parent>-<component> (e.g., authentik-redis, grafana-sidecar)
  • parent field: points to the parent service entry. Currently informational — the review task doesn’t use it yet, but it enables future grouping/dependency-aware reviews.
  • notes field: always starts with “Attached service:” to make the relationship clear at a glance.
  • Version tracking: attached services that use nixpkgs packages should include a version assertion in default.nix (assert pkgs.<pkg>.version == version;) so that flake.lock updates that change the package version break the build and force explicit acknowledgment.

Existing attached services: grafana-sidecar, authentik-redis.

Version Tracking Convention

The current-version field in service-versions.yaml tracks the upstream application version, not the container image tag. For services with custom-built containers, the container image tag (e.g., v1.0.0) is decoupled from the contained app version (e.g., v1.10.1). This allows container rebuilds (base image updates, build fixes) without implying an upstream version change.

Scope: One Service Per Review

The recurring review task covers the single most-stale service — the one mise run service-review surfaces in “Up For Review” — and the review includes performing the upgrade, not just flagging it. Stamp last-reviewed only for services actually reviewed-and-bumped; leave the rest in the queue for future runs. If the upgrade is genuinely out of scope for one sitting (e.g. a major train jump with migrations), do the largest safe step now (such as the latest patch on the current train) and file the rest as a follow-up forge issue/PR, noting why in the description.

Remote-Agent Reviews

Remote-agent sessions run the same loop by PR, with CI doing anything that needs nix or a builder:

  1. Work on a recurring-reviews-YYYY-MM-DD branch; the deliverable is a PR containing the version bump.
  2. Check upstream versions without a browser: curl the GitHub API (https://api.github.com/repos/<owner>/<repo>/releases/latest) or PyPI (https://pypi.org/pypi/<pkg>/json).
  3. Upstream-image services: bump the image tag in the manifest directly. Nix-built containers: bump the version/rev pins, set changed fetch/FOD hashes to pkgs.lib.fakeHash and push — the PR build check does the TOFU rounds for you, commenting the real hashes on the PR on failure (no request needed; see build-container-image). After a green check, merge: the push to main builds and pushes the v<version>-<sha7>-nix tag, and horkos opens the kustomization pin PR against argocd/manifests/<service>/kustomization.yaml — merge that to deploy.
  4. Stamp last-reviewed and update current-version. Record what changed and why in the changelog fragment (see AGENTS.md) and the PR description — do not write review findings into notes. notes describes how the service is deployed at all, not a per-review log; only touch it if the deployment relationship itself changed.
  5. Deploys need nothing from you or the human in the common case: auto-sync apps deploy themselves on merge. Only the four manual apps or a revision pin warrant a mise run request-run argocd-deploy.yaml … — file that yourself too. Pre-merge deploy-testing from the PR branch stays human-side.

Marking a Service as Reviewed

After reviewing, edit service-versions.yaml (repo root) and update the service entry:

- name: prometheus
  type: argocd
  last-reviewed: 2026-02-16
  current-version: "v3.9.1"
  upstream-source: https://github.com/prometheus/prometheus/releases

Commit this change alongside any upgrades you make during the review.

Deployment Policy

BlumeOps uses kustomize manifests for all services. Helm charts should not be introduced for new services. See no-helm-policy for rationale and migration history.

Graph View

Backlinks