ArgoCD
GitOps continuous delivery platform for the Kubernetes cluster.
Quick Reference
| Property | Value |
|---|---|
| URL | https://argocd.ops.eblu.me |
| Tailscale URL | https://argocd.tail8d86e.ts.net |
| Namespace | argocd |
| Git Source | ssh://forgejo@forge.ops.eblu.me:2222/eblume/blumeops.git (the tailnet name; the push-webhook story below) |
| Manifests Path | argocd/apps/ (Applications), argocd/manifests/ (workloads) |
Clusters
One ArgoCD instance on ringtail’s k3s, managing that cluster in-place — every Application targets https://kubernetes.default.svc. The indri minikube cluster it used to also manage was decommissioned; see retire-minikube.
Sync Policy
Workload applications sync automatically (automated, selfHeal: false): a merge to main reaches the cluster on its own. The human gate is the PR review and merge — both behind authentik SSO with TOTP, the same factor that gates a privileged dispatch — so the second confirmation a manual sync used to provide was a repeat of a decision already made, not an independent check.
A Forgejo push webhook makes that happen in seconds rather than minutes. eblume/blumeops posts every push to https://argocd.ops.eblu.me/api/webhook; ArgoCD refreshes each Application whose source matches the pushed repo and revision, and an automated app then syncs immediately. Without it the floor is timeout.reconciliation — upstream default 120s plus 60s jitter, so up to ~3 minutes. The webhook is an accelerator, not a dependency: if a delivery is missed the ordinary reconciliation loop still picks the change up.
Deliveries and their responses are visible under Settings -> Webhooks on the repo. A 400 Unknown webhook event means the payload reached ArgoCD but carried no recognised event header; a 200 with nothing syncing almost always means the URL match failed — see below.
Why the Applications say forge.ops.eblu.me
Every Application tracking blumeops uses ssh://forgejo@forge.ops.eblu.me:2222/… — the host Forgejo itself reports for the repo.
ArgoCD decides which apps a push affects by building a regex from the payload’s repository.html_url and matching it against each spec.source.repoURL (util/webhook/webhook.go, GetWebURLRegex / sourceUsesURL). The hostname is regexp.QuoteMeta’d, so it has to be equal — the only host prefix the regex tolerates is (alt)?ssh.. Forgejo builds html_url from its ROOT_URL, so the Applications must name whatever host ROOT_URL names.
The forge’s public/private split (eblume/blumeops#1208) moves ROOT_URL to https://forge.ops.eblu.me/, and every Application already names that same host — matching is direct, no DNS alias involved (the in-cluster CoreDNS rewrite was removed from code 2026-10-02, ahead of the flip, and from the live cluster 2026-10-03). During the transition window between the app re-point and the ROOT_URL flip (now closed), the webhook matched nothing and quietly refreshed zero apps; the timeout.reconciliation loop (up to ~3 minutes) covered delivery in the meantime.
In-cluster HTTPS clients must name forge.ops.eblu.me. It resolves to indri over the tailnet, and indri’s Caddy holds a cert for it. The public name, by contrast, held no cert there — an in-cluster HTTPS request for it died in the TLS handshake with tlsv1 alert internal error, never reaching a 404 or redirect you might notice — and since the CoreDNS alias was retired (eblume/blumeops#1208) it resolves in-cluster to the public static mirror on fly, like anywhere else. The ringtail rebuild of 2026-10-02 did not retire it: the NixOS k3s module left the coredns-custom.yaml symlink in /var/lib/rancher/k3s/server/manifests/, and k3s kept re-applying it until it was deleted by hand on 2026-10-03 — removing a k3s manifest from nix config needs manual cleanup (cluster > k3s Manifests). This bit horkos (silent no-dispatch on approval, 2026-08-26), the homepage Forgejo widget, and the mise-tasks forge clients (request-run and friends — found 2026-08-31 when it blocked the issue #753 warrants); all set the tailnet name for API calls — forge.eblu.me is today the public static mirror (flyio-proxy > Static forge mirror), with no API of its own.
webhook.gogs.secret in argocd-secret is the shared signing secret (argocd-webhook-secret in the blumeops vault, merged in by external-secret-webhook.yaml). Gogs, not GitHub: Forgejo sends X-Gogs-*, X-Gitea-*, X-Forgejo-* and X-Hub-* headers carrying the same digest, and ArgoCD checks the Gogs headers first — upstream’s comment reads “Gogs needs to be checked before GitHub since it carries both Gogs and (incompatible) GitHub headers”. Rotating means changing both the vault item and the hook’s secret in Forgejo. No rollout is needed on the ArgoCD side: the parser is built once at handler construction, but argocd-server watches the setting and restarts itself when it changes ("gogs secret modified. restarting" in the server log).
selfHeal is off everywhere: hand-applied drift is not reverted, and several resources are manual by design (the immich-db Secret).
prune is on for the thirteen generator-backed apps below and off everywhere else. Where it is off, removing a resource from git does not delete it from the cluster, and deletions stay a deliberate argocd app sync --prune — run from a host with the CLI (gilbert or ringtail) or through the prune input on the ArgoCD Deploy workflow.
Never spell the defaults (prune: false, selfHeal: false) in an Application manifest. The application controller round-trips the Application spec through its Go structs — where both fields are omitempty — every time it initiates an automated sync, so explicit-false fields vanish from the live object the first time the app auto-syncs. The apps root then sees a field-level diff it can never reconcile and flaps OutOfSync (the 2026-08-18 ArgoCDAppOutOfSync episode: talos auto-synced during a release blitz and its live CR normalized to automated: {}). Write automated: {} for plain automated sync, and only add prune: true where intended.
Orphan ConfigMaps from generators
Thirteen apps render their config through a kustomize configMapGenerator, which appends a content hash to the ConfigMap name so that editing the content rolls the pods. Under prune: false the superseded ConfigMap was never deleted, ArgoCD counted it as pending-prune, and the app read OutOfSync indefinitely — tripping ArgoCDAppOutOfSync on a merge that deployed exactly as intended, and stranding another ConfigMap on every later content edit.
These thirteen therefore carry prune: true:
alloy-ringtail, alloy-tracing-ringtail, frigate, grafana-ringtail, homepage, kiwix-ringtail, loki-ringtail, ntfy, ollama, prometheus-ringtail, prowler-ringtail, tempo-ringtail, unpoller-ringtail
The cost is the general one: a resource dropped from git under one of these apps is deleted from the cluster on the next sync, rather than lingering. That is the intended trade — the alternative was an alert that flaps on every correct config edit. Nothing else changes, because ArgoCD prunes only what it tracks: resources created by a controller rather than by ArgoCD (an external-secrets Secret, say) carry no tracking label and are not candidates.
Tracking is label-based (app.kubernetes.io/instance; no resourceTrackingMethod is configured, so ArgoCD’s default applies). Five of the thirteen share the monitoring namespace and two share alloy, but each Application has a distinct name and therefore a distinct label, so they do not prune each other’s resources.
The ArgoCD Deploy prune input remains the route for the apps that keep prune: false.
Bootstrap-owned fields survive sync (argocd-cm, argocd-secret)
The argocd app self-manages: it syncs its own manifests — the argocd-cm accounts patch among them — by server-side apply. SSA prunes only the fields its field managers own (per-field managedFields ownership), so a key written by another manager survives, and a field the app’s declared manifests don’t touch is invisible to its diff. The original bootstrap applied argocd-cm with manual kubectl (see argocd/manifests/argocd/README.md), and that field manager still owns whatever it wrote. In argocd-cm, the app’s own argocd-cm-patch.yaml owns the accounts.ci-argocd and accounts.talos-argocd-readonly keys plus the shared config; the bootstrap kubectl manager still owns accounts.workflow-bot, so it survives every sync while the app reads Synced — for every field the app tracks, the live object matches the declared state.
The argocd-secret half works differently, and the app’s manifests declare no argocd-secret data at all (only metadata/type are tracked; the one data key the repo contributes, webhook.gogs.secret, is merged in by the ExternalSecret argocd-webhook-secret). The API-token records accounts.<name>.tokens are written by the argocd-server manager itself (operation Update) when a token is generated — bootstrap owns none of them. Removing an account from argocd-cm never garbage-collects its token records, and since the app never tracked them either, they survive no matter what.
The #1039 principals sweep (eblume/blumeops#1167, verified live 2026-10-16) left exactly this: the stale accounts.workflow-bot key in argocd-cm and the orphaned accounts.workflow-bot.tokens / accounts.agents-readonly.tokens records in argocd-secret. All three are inert (no RBAC grant, no policy.default), removable only by manual kubectl; they were removed by hand on 2026-10-16 (eblume/blumeops#1167). Consequence for any account or credential sweep: Synced does not mean the live object equals the manifest. Compare the live keys against the repo manifests (kubectl -n argocd get cm argocd-cm -o json | jq '.data | keys', kubectl -n argocd get secret argocd-secret -o json | jq '.data | keys') and treat any extra as residue to delete by hand.
Four applications remain manual, each for a stated reason in its manifest:
| Application | Why |
|---|---|
apps | The app-of-apps root. A new Application appearing earns a second look, and automated sync here would revert argocd app set --revision overrides. |
argocd | Self-management: a manifest that breaks argocd-server breaks the reconciler that would repair it. |
cloudnative-pg-ringtail | Tracks a mutable tag on a mirror, not blumeops main — “the revision moved” is not “a human merged”. |
external-secrets-crds-ringtail | Same, plus CRD churn under a live operator. |
The tag-tracking rows deploy by warrant rather than by hand: merge the
version bump in argocd/apps/<app>.yaml, run argocd-sync-apps.yaml so
the root syncs the Application spec to the new tag, then
argocd-deploy.yaml with -i revision=declared to sync the app at its
declared tag without re-pointing it.
horkos was a fifth until 2026-09-04. It is automated now, for the reason the
cloudnative-pg-ringtail row gives in reverse: it tracks blumeops main, so
“the revision moved” does mean “a human merged a reviewed pin PR” — and a
second human act after that guarded nothing. See
warrant-approval-gated-runs > Deploying horkos.
To pick up newly added Application manifests, sync apps explicitly:
argocd app sync appsAgents cannot run that command (the pod’s argocd CLI is read-only). The
agent route is the ArgoCD Sync Apps warrant: it
pins the root to the bound SHA, syncs, waits healthy, and resets the root to
tracking main — a plain SHA sync would leave it pinned, and a pinned root
reads Synced against the pin and silently ignores later argocd/apps/
merges.
Deploying from a branch
argocd app set <app> --revision <ref> still works for pre-merge testing, but on an automated app always pass a full 40-char SHA, never a branch name. A branch revision on an automated app means every subsequent push to that branch deploys itself unreviewed. The ArgoCD Deploy workflow enforces SHA, main, or declared for exactly this reason. The hand-run equivalent is a full SHA, main, or — for declared, which is a workflow-side skip, not a revision value — an argocd app sync with no app set at all. Reset with --revision main when done.
SHA reachability (pre-merge PR heads)
The argocd-sync-apps and argocd-deploy warrants bind a run to an immutable SHA and fetch it from canonical — except argocd-deploy with revision=declared, where the bound SHA is the commit that declares the revision rather than the payload fetched. The open question — recorded here because it was previously documented neither way — is whether a pre-merge PR head is reachable that way.
Verified from the pod (2026-09-17), in a fresh clone of canonical: Forgejo advertises every PR head on the base (canonical) repo as refs/pull/N/head. A git fetch origin <full-40-char-sha> of that head succeeds — git resolves a full SHA by matching it against the tips of advertised refs — as does git fetch origin refs/pull/N/head by name (tested against the current head of open PR eblume/blumeops#1155, 1ddd0ff0, from a fresh clone, with and without --depth 1). So the argocd-sync-apps payload check (which does git fetch origin <bound-sha> from a checkout of canonical) can fetch a pre-merge head when the bound SHA is the PR’s current head (the tip of refs/pull/N/head). A bound SHA that is not the tip of any advertised ref — a stale or mid-PR commit — is not fetchable by SHA and fails the check: the run is refused before the pin, which is the safe direction. The fetchability of a current head is also exactly why the payload check exists: a bound pre-merge head is a real, installable payload, so the root must validate it rather than trust it.
Still unverified: whether the ArgoCD server’s own repo-creds identity and go-git fetch path resolve a raw full SHA the same way (by matching it to refs/pull/N/head). The pod’s argocd CLI is read-only and cannot run a test sync to check. Before binding a warrant to a pre-merge PR head, verify once from gilbert (e.g. argocd app set <app> --revision <pre-merge-sha> && argocd app sync <app> on a scratch app, or confirm ArgoCD’s repo-creds can read refs/pull/*). Post-merge main SHAs and the literal main always resolve — bind to those by default.
Authentication
- SSO via authentik — OIDC with a public PKCE client (
argocd), shared by the web UI and CLI:argocd login argocd.ops.eblu.me --sso. The Authentikadminsgroup maps torole:adminvia the RBAC ConfigMap; the default policy grants no access. - Local admin — break-glass password in 1Password (blumeops vault), for when Authentik is down.
The git deploy key (SSH) is injected via external-secrets.
Related
- argocd-cli - CLI usage and deployment workflows
- Apps - Full application registry
- forgejo - Git source
- authentik - OIDC identity provider for SSO
- federated-login - How authentication works across BlumeOps