BlumeOps Docs

Talos

7 min read

Talos

Self-hosted agent workflow service (talos-design): browser-driven pi agent sessions over OpenRouter models, resumable from any tailnet device, behind authentik SSO. The non-Anthropic parallel to Claude remote control on agent-workspaces.

Quick Reference

PropertyValue
URLhttps://talos.ops.eblu.me
Tailscale URLhttps://talos.tail8d86e.ts.net
Namespacetalos
Clusterringtail k3s
Imageregistry.ops.eblu.me/blumeops/talos (first-party Nix, built from default.nix in the talos repo — auto-released on merge to its main)
Sourcehttps://forge.eblu.me/eblume/talos
Manifestsargocd/manifests/talos/
Port3000

Architecture

Bun server embedding the pi runtime (pi-coding-agent SDK): sessions are append-only JSONL trees on the talos-home PVC with per-message token/cost data embedded; the browser UI (JSON + SSE) streams responses, renders tool calls, shows per-session cost, and offers two dictation modes (browser SpeechRecognition, or /api/transcribe through an OpenRouter audio model). OIDC is a confidential client talos, admins only.

Access model = the containerized agent model: userspace Tailscale sidecar (talos-agent, tag:talos) as the only tailnet path, CGNAT egress fence NetworkPolicy, no cluster API, op agents-vault service-account token as the one bootstrap secret, shared hephd spoke socket. Ingress arrives separately via the ProxyGroup (talos MagicDNS name) + Caddy.

Models are env-pinned (TALOS_MODEL, currently qwen/qwen3.8-27b); models newer than pi’s catalog are synthesized from OpenRouter’s live listing with real pricing so cost tracking stays correct. The deployment also carries the instance-config scalars from eblume/talos#314 (TALOS_FORGE_URL, TALOS_FORGE_OWNER, TALOS_FORGE_BOT, TALOS_CREATOR_ALLOWLIST, TALOS_DEFAULT_ASSIGNEE, TALOS_HORKOS_URL, TALOS_ARGOCD_SERVER, TALOS_OIDC_ISSUER, TALOS_OR_REFERER, TALOS_PROMETHEUS_URL, TALOS_GIT_AUTHOR_NAME, TALOS_GIT_AUTHOR_EMAIL); with repos.json/lanes.json (ConfigMap) and these scalars (env), the instance has exactly two config surfaces, deliberately.

At boot, before the server starts, the entrypoint runs a per-clone pool repair (pool-repair.sh: zero-length objects, refs with missing targets, zeroed worktree indexes, commit-graph); a clone still unclean afterwards leaves a marker under $HOME/data/pool-unhealthy/. Pool health is exported as talos_pool_unhealthy{repo,source} (source=“fetch” = runtime corrupt fetch → restart the pod; source=“repair” = boot repair left it unclean → a restart will not fix it) plus the talos_pool_fetch_failures_total{reason} counter (eblume/talos#228).

The image bakes a building nix (following the agent-containerization §“Nix in the pod” precedent): a writable canonical /nix/store in the container’s writable layer (max-jobs = 2), so cache.nixos.org substitutes and the pod can build, not just compute fetchgit hash values for the image’s pinned dependencies (heph, npm deps). Growth is bounded by the watermark gc in the talos hourly sweep (eblume/talos#250: nix store gc below half the limit), with the Deployment’s ephemeral-storage: 40Gi limit as backstop; /tmp is a mount of the talos-home PVC (subPath: .tmp — the same directory the entrypoint’s TMPDIR names, wiped at boot) so throwaway writes never count against the limit. (The image’s own source needs no hash since the auto-release move — the talos repo’s default.nix builds from the checkout itself, and every merge to talos main releases automatically.) Rust builds (hephaestus is the only Rust repo in the pool) use a shared CARGO_TARGET_DIR=/home/talos/.cache/cargo-target on the PVC, set in the deployment env: one incremental tree for every session and warm across pod replacement, instead of a cold rebuild per worktree leaving a multi-GB target/ behind (blumeops#813).

Lane shaping is off (enabled: false) since 2026-10-08: it existed to size a local-inference hardware purchase, and that decision is made (eblume/talos#282). While on, a virtual lane holds every turn it serves to the modeled box’s prefill and decode rates, so it adds real latency. lanes.json sits beside repos.json in the talos kustomization’s talos-repos ConfigMap, mounted at /etc/talos and selected by the TALOS_LANES env, so the virtual local lane and the infinite OpenRouter lane follow the serving params in that file. The file is the source of truth — enabled: false is the off switch — and PUT /api/lanes (admin-gated) hot-swaps the policy at runtime without a restart (eblume/talos#252). The Grafana Talos dashboard’s Lanes (shaping) row charts the dispatch decisions, admission wait, lane load and shadow spend from the talos_lane_* metrics.

Programmatic API

The API (POST /api/run, /api/crons, …) trusts two bearer issuers: the talos OIDC client (browser users, minted server-side) and — since TALOS_OIDC_M2M_ISSUER is set — the fleet’s shared talos-m2m machine identity that agent-health already uses. So a script or service drives talos with the talos-m2m credential, no browser session and no talos-specific secret. TALOS_OIDC_M2M_ISSUER is list-valued (step 4) and carries the fleet’s talos-m2m issuer. A token for any other issuer fails iss/aud (talos src/jwt.ts, verifyBearer).

A third trigger surface is the Forgejo webhook forge loop (talos-design Driver 2): it also starts cycles on CI-failure notices — COMMENT reviews posted by forgejo-actions via the blumeops shared report-failure action — framed as untrusted build output to diagnose, capped at three failure reviews per PR.

No wrapper task by design — it’s just the API. Mint the talos-m2m token the way agent-health does, then call talos. The credential is in the blumeops vault (talos-m2m-app-password); it is also in the agents vault, so this path is reachable from an agent session too. Warrant + human approval remains the gate on every privileged action, so an agent creating a session or cron job never escalates — it only spawns more equally-unprivileged work. One exception: the heph-task watcher kind (below) has a session-facing wrapper, mise run talos-wait-for-task, because a session registers a watcher as its “waiting on a human” primitive and the wrapper keeps that to one command.

TOKEN=$(curl -s https://authentik.ops.eblu.me/application/o/token/ \
  -d grant_type=client_credentials -d client_id=talos-m2m \
  -d username=talos-ringtail \
  --data-urlencode "password=$(op read op://blumeops/oor7os5kapczgpbwv7obkca4y4/talos-m2m-app-password)" \
  -d 'scope=openid profile' | jq -r .access_token)
 
# create the daily doc-review cron (talos#21 — scheduled headless runs)
curl -s -X POST https://talos.ops.eblu.me/api/crons \
  -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
  -d '{"name":"BlumeOps doc review","schedule":{"expr":"0 7 * * *","timezone":"UTC"},"prompt":"…"}'
 
curl -s https://talos.ops.eblu.me/api/crons -H "Authorization: Bearer $TOKEN"   # list

heph-task watcher (one-shot continuation). A job whose schedule.kind is "heph-task" fires a new session when the watched heph task reaches a state in on ("done" and/or "dropped"):

{
  "name": "ringtail rebuild follow-up (blumeops#800)",
  "prompt": "The burn-in task is done: verify the GC config and close out #800.",
  "schedule": { "kind": "heph-task", "nodeId": "01M1J1PRRQGYJNTFFZQSZJARMB", "on": ["done"], "expiresAt": "2026-09-13T08:00:00Z" },
  "origin": { "repo": "eblume/blumeops", "issue": 800 }
}
  • One-shot semantics. The job is consumed before firing — patched enabled: false plus firedReason in one step — so a failed fire never re-fires; the job then stays listed as disabled. Re-arm is an explicit PATCH /api/crons/:id (re-arm clears firedReason); cancel is DELETE /api/crons/:id. firedReason is condition, expired, or disarmed.
  • The talos-watch tag. On arm (create or re-arm) talos adds the talos-watch tag to the watched heph task and exposes tagApplied: true on the job; the tag is removed on consume, delete, or disable. If the tag is gone at poll time, the job is consumed with firedReason: "disarmed" — removing the tag in heph (the PWA’s Unwatch action) is the human-side cancel, and it is where the human sees the watcher (not the talos dashboard).
  • expiresAt (future ISO timestamp): when it passes, the job fires with firedReason: "expired" instead of waiting forever.
  • origin ({repo, issue}) anchors the spawned session to the origin issue; GET /api/crons adds the derived originUrl.

From an agent pod the wrapper mints the token and routes through the sidecar itself:

# register (the prompt comes from a file — the API takes an inline prompt only)
mise run talos-wait-for-task create 01M1J1PRRQGYJNTFFZQSZJARMB \
  --on done --prompt-file ./prompt.md --origin eblume/blumeops#800 \
  --expires 2026-09-13T08:00:00Z
mise run talos-wait-for-task list
mise run talos-wait-for-task show <job-id>
mise run talos-wait-for-task cancel <job-id>

From the tailnet-fenced agent pod, route through the sidecar with ALL_PROXY=socks5://localhost:1055. Creating a job in the UI instead needs an admin browser login.

Cost accounting

First-class (talos-design): per-message Usage in every transcript, per-session totals in the UI. The OpenRouter key (“openrouter (blumeops)”) is a copy of the personal key — replace with a dedicated spend-limited key.

Graph View

Backlinks