Forgejo

Git forge and CI/CD platform. Primary source of truth for blumeops (mirrored to GitHub).

Built from source on indri. The LaunchAgent unit is nix-managed (indri flake, PR 8 of the nix-darwin series); the build, the app.ini and the version stay with the Ansible role. The build pulls from the forge mirror (origin); Codeberg is the upstream remote (codeberg). To upgrade, see upgrade-forgejo.

Quick Reference

Property	Value
URL (primary, ROOT_URL)	https://forge.ops.eblu.me
URL (public static mirror)	https://forge.eblu.me
SSH	ssh://forgejo@forge.ops.eblu.me:2222
Local Ports	3001 (HTTP), 2200 (SSH)
Config	ansible/roles/forgejo/templates/app.ini.j2
Binary	~/code/3rd/forgejo/forgejo (source-built)
Data	~/forgejo
LaunchAgent	mcquack.eblume.forgejo (unit nix-managed — indri §Maintenance Notes)
Source	~/code/3rd/forgejo (origin = forge mirror, codeberg = upstream)

Building from Source

Forgejo is built from source on indri, matching the pattern used by zot, caddy, and alloy.

Remotes: origin → https://forge.ops.eblu.me/mirrors/forgejo.git (the build source), codeberg → https://codeberg.org/forgejo/forgejo.git (upstream). The original clone was from Codeberg to avoid a circular dependency with the forge; the mirror was later promoted to origin.

Version is declared in the Ansible role, not built ad-hoc. forgejo_version (plus forgejo_node_version/forgejo_build_tags) in ansible/roles/forgejo/defaults/main.yml pins the deployed tag. On provision-indri --tags forgejo the role fetches from the mirror, checks out the tag, rebuilds only when the running binary doesn’t match, links ./forgejo, and restarts. Bumping forgejo_version in a PR is therefore the whole upgrade — reproducible and DR-safe. See upgrade-forgejo for the full procedure (DB backup, breaking changes, verification, rollback).

WARNING: Do NOT use make forgejo directly — it rebuilds with empty TAGS, stripping SQLite support. The role passes TAGS explicitly to make build and ln -f gitea forgejo afterwards.

Build tags (forgejo_build_tags): bindata (embed assets), timetzdata (embed timezone data), sqlite sqlite_unlock_notify (SQLite support).

go comes from the indri global mise baseline (declared in the indri flake’s mise config, darwin/indri/configuration.nix) — GOTOOLCHAIN=auto switches per go.mod and that config keeps mise’s GOROOT export off, so a plain make build works at any tag (upgrade-forgejo §Go toolchain). The role removes the checkout’s untracked mise.toml, which used to pin a stale go.

Repositories

The forge has three namespaces:

• eblume/ — first-party repos (blumeops, talos, hephaestus, myeve, gamedev, …). The forge is the primary source of truth for blumeops.
• talos/ — the talos bot’s forks (blumeops, agents, horkos); the push target for repos where the human gate is the point.
• mirrors/ — pull mirrors of external repos (alloy, tesla_auth, …) for supply chain control; see manage-forgejo-mirrors.

CI/CD (Forgejo Actions)

Runners:

Runner	Host	Labels	Purpose
indri-runner	indri (native, host-mode)	indri	Default jobs (host-mode, no container engine)
ringtail-nix-builder-{blumeops,horkos,talos}	ringtail (NixOS)	nix-container-builder	Nix container builds via nix-build + skopeo (one daemon per registered repo)
ringtail-priv-runner	ringtail (NixOS, sandboxed horkos-runner)	priv	Warrant-gated, dispatch-only privileged jobs (warrant-approval-gated-runs)

Workflows in .forgejo/workflows/:

Workflow	Trigger	Runner	Purpose
forge-reconcile	push/PR/schedule/dispatch	indri	Reconcile the talos + horkos-forge bots’ collaborations, webhooks, labels, and branch protections against repos.json + forge/branch-protections.json; the single forge drift check (weekly schedule + same-repo PRs; protections apply only from a human run)
argocd-deploy	dispatch	priv	Warrant-gated ArgoCD deploy of a single app
argocd-sync-apps	dispatch	priv	Warrant-gated sync of the app-of-apps root (apps)
branch-cleanup	cron/dispatch	indri-build	Delete stale branches
release	push (main)	indri-build	Detect a version bump, build the docs tarball, and tag + publish via eblume/release-kit (update-documentation)
build-container	push (main)/PR	indri-build → nix-container-builder	Nix container image builds; classify on the unprivileged runner, build on the nix builder (build-container-image)
deploy-fly	dispatch	priv	Warrant-gated deploy of the Fly.io proxy (flyio-proxy)
docs-checks	PR/push	indri-build	Docs + changelog validation
flake-update	dispatch	nix-container-builder	Ringtail flake input update (native nix on the ringtail nix runner)
lint	PR/push	indri-build	Repo lint (prek hooks)
provision-indri	dispatch	indri-priv	Warrant-gated apply of a bound SHA’s nix-darwin generation; fire-and-forget — green means the switch launched (provision)
run-script	dispatch	priv	Warrant-gated one-off script run

PR jobs additionally end with the shared .forgejo/actions/report-failure composite action: on failure it posts the job’s teed log tail to the PR as forgejo-actions — a triggering COMMENT review on agent-authored PRs (the talos fix loop), a plain comment otherwise — deduped per (workflow, job, matrix leg, head SHA) and capped at three failure reviews per PR (agent-change-process > CI failure notices).

(Until retire-minikube a k8s runner was a minikube DinD pod that also built Dockerfile/Dagger containers; that path is retired.)

Secrets (Forgejo Config)

Server configuration secrets managed via 1Password → Ansible (fetched in the indri playbook pre_tasks):

• lfs-jwt-secret, internal-token, oauth2-jwt-secret — Forgejo server tokens (rendered into app.ini)
• runner_ringtail_priv_token, runner_ringtail_nix_builder_{blumeops,horkos,talos,hephaestus}_token — five repo-scoped registration tokens on the “Forgejo Secrets” item, seated by the human-run seed-secrets ringtail task as /etc/forgejo-runner/{priv,nix-builder-blumeops,nix-builder-horkos,nix-builder-talos,nix-builder-hephaestus}.env (declared in nixos/ringtail/secrets.json; eblume/blumeops#1524). runner_reg (instance-global) is retired — left in 1Password until the old instance-wide registrations are deleted from forge.

Per-runner identity and job-credential secrets live on the runner card (forgejo-runner > Credentials).

Forgejo Actions Secrets

Repository-level Actions secrets are synced from 1Password to Forgejo by the values half of mise run forge-reconcile (human-run from gilbert under biometric op — the values half authenticates with the scoped eblume write:repository PAT and the step never leaves a human). The sync is authoritative: declared secrets are PUT (created/updated), undeclared live ones are DELETEd, and --check reports name-level drift — values are write-only, so value drift is invisible to the sync):

mise run forge-reconcile

Names are declared per repo in forge/actions-secrets.json (see declared-state-files); each name’s 1Password source is the in-code ACTIONS_SECRET_SOURCES allowlist in mise-tasks/forge-reconcile, so a data edit cannot point a repo’s CI secrets at a vault item of its choosing. The CI token (FORGE_REPO_WRITE_TOKEN) is report-only: it never op reads or writes a value, and one dim line plus exit 0 covers it. forge-reconcile --check (weekly schedule and same-repo PR check) compares the declared names against the live forge and fails on a missing or undeclared live name.

Repo	Secrets	Purpose
eblume/blumeops	FORGE_REPO_WRITE_TOKEN, BLUMEOPS_CI_OP_TOKEN, FORGE_ACTIONS_RO, FORGE_PAT_ROTATE_OP_TOKEN	forge-reconcile reconcile + drift check (write:repository,read:user eblume PAT); job-time op read of blumeops-ci items; read-only forge token for the auto-apply preflights; read+write-agents-vault service account for the forge-bot-pat-rotate warrant run
eblume/hephaestus	— (none)	No secrets declared: the MAIN_PUSH_TOKEN changelog main push was retired by the release-PR model (eblume/blumeops#1628) — the live secret was deleted by the 2026-10-15 reconcile apply, and the hephaestus-main-push-token PAT was revoked the same day
eblume/talos, eblume/horkos	ZOT_PUSH_API_KEY	Auto-release CI: per-repo push-only zot identity (ci-zot-talos / ci-zot-horkos), provisioned from the zot master fields by the values half
eblume/cv	— (none)	Release CI is stored-secret-free; the empty declaration makes provisioning authoritative here (first run deletes the stale FORGE_TOKEN)

The per-purpose secrets the retired Ansible role used to sync (argocd token, fly deploy token, zot CI key) are no longer Forgejo secrets: workflows op read the blumeops-ci items at job time with BLUMEOPS_CI_OP_TOKEN — see blumeops-ci-item-migration. The main-push PATs that used to be the exception in both directions — blumeops’s release reading its own blumeops-ci/forge-main-push/token at job time, and hephaestus declaring the item’s hephaestus-scoped hephaestus-token field as MAIN_PUSH_TOKEN — are both retired: the release PR carries changelog and docs_version to main itself (eblume/blumeops#1628). Both sides are fully done: hephaestus’s live secret was deleted and its hephaestus-main-push-token PAT revoked on 2026-10-15, and blumeops’s blumeops-main-push-token PAT was revoked on 2026-10-16 with the item’s token fields removed.

These secrets are injected as ${{ secrets.SECRET_NAME }} in workflow files.

API Tokens

The role authenticates with an eblume PAT scoped to write:repository,read:user (1Password item forge-repo-write-token — the same token CI holds as FORGE_REPO_WRITE_TOKEN). Its repo-admin endpoints (collaborator / Actions-secret ops) work with that scope because eblume owns the repos; the site-admin token is never needed here. Mint it on indri with forgejo admin user generate-access-token … --scopes write:repository,read:user and store the value in 1Password → forge-repo-write-token → token.

The site-admin PAT (api-token in the “Forgejo Secrets” item) is not used by this role anymore. Its remaining consumers:

• mise run runner-logs (reads from 1Password at run time)
• The tea CLI (copies the token into ~/.config/tea/config.yml; re-paste it there after rotation). tea switched to this PAT 2026-07-04 after its OAuth token expired and tea 0.14.2 broke httpsign auth (tea#1046 — fixed in go-sdk but unreleased; httpsign can be re-enabled in tea’s config once 0.14.3 ships).
• the forgejo_metrics role (fetched as forgejo_metrics_api_key in the playbook pre_tasks)

Identity Provider

authentik is the BlumeOps OIDC identity provider and source of truth for user identity. Forgejo authenticates against Authentik as an OIDC client.

Configuration:

• OAuth2 provider and application defined in Authentik blueprints (argocd/manifests/authentik/configmap-blueprint.yaml)
• Auth source created via forgejo admin auth add-oauth with --skip-local-2fa (lives in Forgejo’s SQLite database, not app.ini)
• [oauth2_client] section in app.ini.j2 controls auto-registration and account linking behavior

MFA: SSO logins skip Forgejo’s local 2FA (--skip-local-2fa on the auth source) — Authentik enforces MFA instead. Local password logins still require Forgejo’s own TOTP. Note: the --skip-local-2fa CLI flag has a known bug where it doesn’t persist via update-oauth; it was set directly in the login_source.cfg JSON (SkipLocalTwoFA: true).

Account linking: ACCOUNT_LINKING = login — when an Authentik user’s email matches an existing local account, Forgejo prompts for the local password (and local MFA) to confirm the link. This is a one-time operation that preserves existing accounts, API tokens, SSH keys, and repository ownership.

Group-based admin: The admins group in Authentik maps to Forgejo admin status via --admin-group admins on the auth source. Manage admin access in Authentik, not Forgejo.

Break-glass: Local password login always works (with local MFA). Authentik SSO is additive — if Authentik is down, log in with local credentials.

Access

The instance’s canonical name is https://forge.ops.eblu.me (DOMAIN/ROOT_URL, tailnet-reachable). The public name https://forge.eblu.me serves a read-only static mirror of the exposed repos (stagit HTML + git dumb-HTTP) on the Fly edge via flyio-proxy; every path the mirror does not serve 302s to the private forge (see flyio-proxy > Static forge mirror).

Access Method	URL	Reachable From
HTTPS (primary)	https://forge.ops.eblu.me	Tailnet
HTTPS (public mirror)	https://forge.eblu.me	Public internet (static mirror)
SSH	ssh://forgejo@forge.ops.eblu.me:2222	Tailnet only

The UI shows forge.ops.eblu.me for both HTTPS and SSH clone URLs.

Security Controls

• Registration: Local registration disabled; only authentik SSO login allowed (ALLOW_ONLY_EXTERNAL_REGISTRATION = true)
• Reverse proxy trust: REVERSE_PROXY_LIMIT = 2, REVERSE_PROXY_TRUSTED_PROXIES = * — Forgejo logs the real client IP from X-Real-IP header, not the proxy’s Tailscale IP
• Public surface: the public name serves only the static mirror — no login, no API, no archive generation. The relay-era proxy defenses (login rate limiting, fail2ban, /swagger 403, archive redirect, relay robots.txt, the Authentik dead-end) were retired with it; the mirror vhost carries its own crawler defenses — rate limiting, robots.txt Disallow: /, declared-bot 403, and Anubis proof-of-work (see ai-scraper-mitigation).

Break-glass

mise run fly-shutoff stops all public traffic immediately. forge.ops.eblu.me continues to work from the tailnet. See expose-service-publicly > Break-glass shutoff.

Monitoring

Forgejo exposes a Prometheus /metrics endpoint (enabled via [metrics] in app.ini). Alloy on indri scrapes it at localhost:3001/metrics. Metrics are mostly Go runtime stats and repo counters (no per-request latency histogram).

Request latency is measured at the Fly.io proxy layer via the flyio_nginx_upstream_response_time_seconds histogram, visible on the Forgejo Grafana dashboard under “Mirror: Upstream Response Time”.

CI (Actions) metrics

Forgejo’s /metrics has nothing on Actions, and the API has no queue or timing history. So the indri collector darwin/indri/mcquack-forgejo-metrics.sh reads Forgejo’s sqlite DB read-only every minute and writes forgejo_ci_* series into forgejo.prom:

• Counts: finished job attempts by repo, workflow, runner label and result.
• Histograms: queue wait (ready → picked up, split by gate="none|approval") and job duration, per repo, workflow, job and runner label. Per-combo counts are cumulative, but a combo stops being exported once it has had no task in the last 30 days — so a brand-new combo’s first observation is invisible to increase(), which is fine for attribution.
• Busy time: busy-seconds per runner, against a static forgejo_ci_runner_capacity that mirrors each runner’s configured slots.
• Live state: queued/running jobs, and open PRs blocked on “Approve and run”.

The same query also supplies forgejo_repo_latest_commit_timestamp_seconds (from the branch table) and forgejo_actions_last_success_timestamp_seconds. Workflows with no run in 90 days drop out unless they are scheduled. Only languages, releases and the repo list still come from the API, paged and skipping forks. This brings a cycle down from about 50 s to about 4 s.

The Forgejo Grafana dashboard’s CI rows are built on these metrics. ForgejoRunnerOffline and ForgejoCICollectorDown alert on them (runbook-forgejo-ci).

Archive Cleanup

The [cron.archive_cleanup] section is enabled with OLDER_THAN = 2h and RUN_AT_START = true. This prevents the repo-archive/ directory from growing unboundedly when crawlers or users trigger archive downloads. Without this, the directory grew to 54GB in 2 days during a crawler incident in April 2026.

Database hygiene

The SQLite database (data/forgejo.db) does not shrink on its own: deletes only mark pages reusable. Two crons bound the tables that grow unbounded:

• [cron.delete_old_system_notices] — weekly (@every 168h, RUN_AT_START = true), drops system notices older than forgejo_old_notices_retention (default 720h, 30 days). The notice table had grown to 146k rows of pure history (mirror-sync failures, closed-pipe webhooks) before this.
• [cron.delete_old_actions] — weekly, drops activity-feed (action table) entries older than forgejo_old_actions_retention (default 2160h, 90 days). Caveat: this deletes the whole feed older than the cutoff — dashboard timeline and profile heatmap included, not just the mirror-sync rows — and Forgejo has no per-org switch for mirror-sync feed entries. The mirrors user’s sync events dominated the table (~40M/month of full commit lists). 90 days was chosen because the feed is rarely consulted and git history is the record; raise the retention if a longer feed is wanted.

Neither cron shrinks the file. To reclaim space, stop Forgejo and run sqlite3 ~/forgejo/data/forgejo.db 'VACUUM;' during a quiet window (same window shape as upgrade-forgejo). Upgrade-time forgejo.db.bak-* copies are deleted once the upgrade proves out; borgmatic already holds nightly DB dumps.

[cron.git_gc_repos] is deliberately left disabled (Forgejo default). eblume/blumeops.git had 47 packs / 1,209 loose objects at the time of writing — moderate, and git’s own auto-gc bounds it. If the loose-object count keeps climbing, enable it:

[cron.git_gc_repos]
ENABLED = true
RUN_AT_START = true
SCHEDULE = @every 168h

Mirrors

Forgejo hosts pull mirrors of external repositories (GitHub, etc.) for supply chain control. Mirrors live in the mirrors/ org and sync on a configurable interval. See manage-forgejo-mirrors for operations.

Related

• upgrade-forgejo - Version upgrade procedure (DB backup, breaking changes, rollback)
• forgejo-runner - CI/CD runners (indri + ringtail instances, credentials)
• principals - The principal inventory; the talos bot identity behind the talos/ namespace
• argocd - Uses Forgejo as git source
• authentik - OIDC identity provider
• zot - Container registry for built images