Bootstrap Agent Workspaces

One-time steps to bring up agent-workspaces on ringtail. Ordinary redeploys are just a blumeops merge — ringtail-rebuild auto-applies it — or, when the box can’t reach the merge machinery, the break-glass local apply (docs/reference/operations/disaster-recovery.md); these steps cover the state that lives outside git (the Forgejo bot user, the agent’s OAuth login, and Claude’s first-run consent) — see the status note below — and must be done by a human.

Status (2026-08-19): the containerized stage of the agent workspace — the agent-ws pod and the agent-ws-* / agent-repos-init host services — is retired, superseded by talos. The Forgejo-bot steps (§1–2) and the agent user still apply; the Claude install / OAuth / start-services steps (§4–6) are historical, retained for the pre-talos record.

Do these in order. Steps 1–2 can happen before the config is deployed; steps 4–6 require the agent user to exist (i.e. after the first nixos-rebuild).

1. Create the Forgejo bot user

Status: already done (2026-07-08) — the talos user exists. This section documents how, for reproduction/rotation.

On indri (Forgejo admin CLI), create a non-admin user named talos. Forgejo runs as erichblume with its work path at /Users/erichblume/forgejo:

ssh indri
FJ=/Users/erichblume/code/3rd/forgejo/forgejo
WP=/Users/erichblume/forgejo; CFG=$WP/custom/conf/app.ini
"$FJ" admin user create --username talos --email blume.erich+talos@gmail.com \
    --random-password --must-change-password=false --work-path "$WP" --config "$CFG"

(Email is a +talos gmail alias because eblu.me has no working MX yet.)

Do not grant repo access here. Collaborations come from repos.json (argocd/manifests/talos/), reconciled by the Forge Reconcile workflow, which is authoritative in both directions — a repo absent from that file has its talos collaboration removed, so a grant clicked in the web UI is reverted on the next run. The same file drives the pod’s clone loop, which is the point: a missing grant and a typo look identical from inside the pod (Forgejo 404s rather than 403s on a private repo the bot cannot see). To share a repo, edit repos.json and open a PR. See principals.

agents and blumeops are pinned read-only. The reconciler refuses a write request for either, and repos.json cannot override it — that fence lives in reviewed code. Both are cloned fork: the bot pushes branches to talos/<repo> and opens cross-repo PRs, so a human merge sits in front of the instructions every future session boots with and in front of blumeops CI (and therefore its deploy secrets). Every other pooled repo is write + canonical; write ≠ deploy, since the gates are the blumeops 1Password vault and the root-only kubeconfig, not repo permissions. See agent-workspaces §“blumeops: author-only, not a server”.

main is intentionally not branch-protected against the bot. A username push-whitelist rejects CI’s automatic Forgejo Actions token (Forgejo #11159), which breaks the release workflows. The bot can push to main; the boundary is convention (open PRs) plus the fact that the bot holds no deploy credentials. See agent-workspaces §Isolation.

1b. Create the talos Forgejo PAT (for PR creation)

Status: done (2026-07-08) — item talos-forgejo-token in the agents vault.

Agents open PRs with tea, which needs a token. Mint it as the talos user (not admin): a Forgejo PAT can’t exceed its owner’s permissions, so a talos-owned token is structurally bounded to the repos talos collaborates on — the ownership bound is the guardrail, not the scope list.

ssh indri
FJ=/Users/erichblume/code/3rd/forgejo/forgejo
WP=/Users/erichblume/forgejo; CFG=$WP/custom/conf/app.ini
"$FJ" admin user generate-access-token --username talos \
    --scopes write:repository,write:issue --token-name agent-pr \
    --work-path "$WP" --config "$CFG"

Store it concealed in the agents vault as talos-forgejo-token → api-token (op item create --vault agents --category "API Credential" --title talos-forgejo-token "api-token[password]=<tok>"). The workspace launcher reads it via the op shim and both exports FORGEJO_TOKEN and seeds tea’s config, so PR creation has no blumeops-vault dependency. Scopes are write:repository

• write:issue (tea’s pr create needs the issue scope — PRs are issues in Forgejo) — never write:admin/write:organization/sudo.

2. Confirm vault items exist

The agents vault must contain (created during the prototype, 2026-07-08):

• talos-forgejo-token — Forgejo PAT (api-token field, write:repository).
• The agents-ringtail Service Account item lives in the blumeops vault and holds the agents-ringtail-rw token (read/write on agents only).

3. Deploy the config

The config ships as a blumeops PR: merging to main with a diff touching nixos/ringtail/** makes ringtail-rebuild auto-apply it. A fresh host that cannot reach the merge machinery yet uses the human break-glass local apply instead — see docs/reference/operations/disaster-recovery.md (one-time clone of blumeops into /etc/blumeops, then a direct nixos-rebuild switch on a fresh host — the ringtail-apply@ unit doesn’t exist before a first generation — or the ringtail-apply@<sha> unit on a host that already runs one).

This creates the agent user. On-host secret files are not part of any deploy: /etc/agents/op-token is owned by mise run seed-secrets ringtail. The bot’s SSH key (former /etc/agents/ssh/id_ed25519) was retired in Oct 2026 (eblume/blumeops#1635). (The agent-repos-init + agent-ws-* host services it used to install are retired — see the status note above.)

4. Install Claude Code for the agent user

Remote Control moves fast, so we use the official self-updating installer rather than nixpkgs (see agent-workspaces “Known warts”):

ssh ringtail
sudo -u agent -H bash -lc 'curl -fsSL https://claude.ai/install.sh | bash'
# verify:
sudo -u agent -H bash -lc '~/.local/bin/claude --version'

5. Log in (OAuth) — interactively, then again every 5 days

Remote Control needs a full-scope subscription OAuth login: the credential an interactive claude auth login writes to ~/.claude/.credentials.json on the PVC. It is the only credential Remote Control accepts — a claude setup-token / CLAUDE_CODE_OAUTH_TOKEN token is inference-only and claude refuses to start Remote Control with one (v0.16.0 shipped that and crash-looped; see agent-workspaces > Authentication).

This is not a one-time step: the login’s refresh token expires ~29 days after login. Seed it by performing one rotation, and the recurring 21-day “Rotate Claude OAuth login” heph chore (Blumeops project) kept it alive from then on. (The dedicated rotation runbook retired with the agent-ws pod.)

Two pieces of first-run state live outside the credential and still need seeding on a fresh PVC:

• Remote Control consent is per-config-dir and MUST be seeded, or every service blocks invisibly. On first remote-control launch Claude prompts Enable Remote Control? (y/n). A non-interactive service can’t answer it, so it hangs at the prompt — and because the launcher wraps Claude in script, the hung process still reports active (running) to systemd with 0 restarts. It looks healthy but never connects (this bit us: the only visible environment was a stale ghost from an earlier run). The consent is stored in <config-dir>/.claude.json as "remoteDialogSeen": true. Seed it either by:
  • running ~/.local/bin/claude remote-control once interactively and answering y+Enter (confirm it prints Connected), or
  • pre-seeding non-interactively: set "remoteDialogSeen": true at the top level of .claude.json (same edit pass as the trust flags below), which is the reproducible path. It persists on disk (survives reboots/re-provision), but a fresh agent home needs it re-seeded.
• Trust is per-directory (~/.claude.json → projects["<path>"].hasTrustDialogAccepted). A login trusts only its own cwd — with the single home-base workspace that is the only cwd that needs it (~agent/code/personal/agents). If the workspace cwd ever moves, pre-seed the new path (set hasTrustDialogAccepted=true, then chown agent:agent ~agent/.claude.json && chmod 600 ~agent/.claude.json) or the service crash-loops on the untrusted dir.

6. Start the services

Historical. The agent-ws-* host services are retired (superseded by talos); the commands below are the pre-retirement record.

ssh ringtail 'sudo systemctl start agent-repos-init.service'
ssh ringtail 'sudo systemctl start "agent-ws-*"'
ssh ringtail 'systemctl status "agent-ws-*" --no-pager'

Open the Code tab in the Claude mobile app — ringtail-agent should appear online. Tapping it and starting a session spawns an isolated worktree of the agents home-base repo, with the sibling repos alongside at ~/code/personal/.

7. Seed the heph spoke (one-time)

The agent runs a hephd spoke (see agent-workspaces > The heph spoke (deliberately in-boundary)). The mechanical parts — the nix-store hephd (the hephaestus flake input), the services, owner adoption (--owner-id) — are source-controlled and ride along with the config apply (a merge auto-applies via ringtail-rebuild). These are the irreducible secret/identity steps a human does once. They are fiddlier than they look; read the gotchas.

1. Give talos-heph a password + MFA — logged in AS talos-heph. The blueprint creates the talos-heph user (heph-scoped group, not admins). Set its password (ak shell → u.set_password(...), or the UI) and store it in the blumeops vault (item talos-heph-login) — the human reads blumeops, not the agents vault. Then, in a private/incognito window (see the gotcha), sign in to https://authentik.ops.eblu.me as talos-heph and complete the forced TOTP enrollment (MFA is enforced on the default flow — appropriate, this identity reaches your tasks). Save the TOTP secret into talos-heph-login.

2. Seed the token — approve the device code in that same talos-heph session. The agent’s hephd and the heph CLI are the nix-store package (there is no ~/.cargo build anymore). The spoke unit’s ExecStart is the store wrapper script; its contents carry the heph package bin dir and the heph-token-save shim — resolve both from there and run the login as the agent:

   WRAP=$(ssh ringtail "systemctl show agent-heph-spoke -p ExecStart --value | grep -oE '/nix/store/[^ ;]+' | head -n1")
   BIN=$(ssh ringtail "grep -oE '/nix/store/[a-z0-9]+-heph-[0-9.]+/bin' $WRAP | head -n1")
   SAVE=$(ssh ringtail "grep -oE '/nix/store/[a-z0-9]+-heph-token-save/bin' $WRAP | head -n1")
   ssh ringtail "sudo -u agent -H env HOME=/home/agent $BIN/heph auth login \
     --hub-url http://indri.tail8d86e.ts.net:8787 \
     --issuer https://authentik.ops.eblu.me/application/o/heph/ \
     --client-id heph --no-browser --token-save-cmd $SAVE/heph-token-save"

   Approve the printed URL in the incognito talos-heph session. On success heph-token-save writes the token to the heph-spoke-token item in the agents vault, addressed by the id pinned as hephTokenItemId in nixos/ringtail/agent-heph-spoke.nix. If that id is absent (fresh vault, deleted item), the save creates the item — record the new id in the module afterwards (op item list --vault agents --format json | jq '.[] | select(.title == "heph-spoke-token") | .id'; eblume/blumeops#963).

   If you see duplicate heph-spoke-token items in the agents vault, stop: every title lookup against the item fails while duplicates exist, and the save refuses to write until it is resolved by hand (keep one, delete the rest, record the kept id in the module). eblume/blumeops#963.

   THE trap: the device flow authorizes as whoever the browser is logged into. If you open the URL in your normal browser (logged in as you), it silently binds you as the hub owner and puts your token in the agents vault — not what you want. Always approve from a session logged in as talos-heph. Device codes also expire in a few minutes, so do the TOTP enrollment (step 1) first, then the approval is a single click. If they keep expiring, pull the current code straight from Authentik: DeviceToken.objects.filter(provider__name="Heph").order_by("-pk").first().user_code.

3. Record the authorized sub for the hub. Decode the token’s sub (a hashed_user_id) and store it in the blumeops vault (talos-heph-sub/sub):

   # as the agent; base64URL needs padding — python is more reliable than `base64 -d`
   # <item-id> = hephTokenItemId in nixos/ringtail/agent-heph-spoke.nix
   op read op://agents/<item-id>/token \
     | python3 -c 'import sys,json,base64; t=json.load(sys.stdin)["access_token"].split(".")[1]; t+="="*(-len(t)%4); print(json.loads(base64.urlsafe_b64decode(t))["sub"])'
   op item create --vault blumeops --category "API Credential" \
     --title talos-heph-sub "sub[text]=<the-sub>"

4. Authorize on the hub, then (re)start the spoke on a clean store.

   mise run provision-indri -- --tags heph        # hub picks up --authorized-sub
   # the spoke's fresh store minted its own owner_id before adoption; reset it so
   # it re-pulls under the adopted --owner-id:
   ssh ringtail 'sudo systemctl stop agent-heph-spoke; \
     sudo rm -f /home/agent/.local/share/heph/heph.db*; \
     sudo systemctl start agent-heph-spoke'
   ssh ringtail "sudo -u agent -H env HOME=/home/agent $BIN/heph list --project Blumeops"
   # $BIN = the heph package bin resolved in step 2

   Expect your real Blumeops tasks. heph sync --status should show auth_failure=false, last_error=null.

Revoke by disabling the talos-heph Authentik user, or by setting heph_agents_sub_enabled: false in ansible/inventory/group_vars/all.yml and re-provisioning indri (drops the sub from --authorized-sub) — either cuts the spoke without touching your own logins. Deleting the talos-heph-sub vault item alone is no longer the way: the role’s vault read is fail-closed, so provisioning aborts instead of silently de-authorizing the spoke.

Other gotchas banked while bootstrapping this:

• If git push (or any forge SSH) hangs, it’s an unapproved 1Password SSH-agent biometric prompt, not the network.
• nix build/nixos-rebuild --flake git+https://…?ref=main caches the ref; use --refresh (or ?rev=<full-sha>) to pick up a just-pushed commit.
• Owner model: talos-heph is only the login credential (revocable). The hub still has one owner — you — and the spoke adopts your owner_id (--owner-id) so the agent works your actual nodes. It is not a second owner.

Verifying the secrets path

From a spawned session (or sudo -u agent), plain op should work read/write against the agents vault only:

op vault list                                  # exactly: agents
op item get agent-test-secret --vault agents    # reads

Teardown / rollback

Revert the PR (the revert merge auto-applies via ringtail-rebuild, or apply the revert commit with the local break-glass apply — see docs/reference/operations/disaster-recovery.md); then ssh ringtail 'sudo userdel -r agent' if you want the home gone. Remove the talos Forgejo user and revoke the service account in 1Password if retiring the capability entirely.

Related

• agent-workspaces — design & operations
• principals — the talos bot identity