pacer cluster-ops docs

← console

The browser UI

Three pages. Start with the estate dashboard — https://ai-ops.testtemplate.org/dashboard — which is the only one that drives everything. The two older consoles, ai-ops at / and cluster-ops at /, each cover part of the estate and remain what they were.

All three are reachable without a token and useless without one.

The estate dashboard — /dashboard on ai-ops

One page, one token, one origin, covering every resource pacer-serverless manages:

PanelWhat it drivesReaches
Clusterthe OKE cluster, all 13 ops, verifycluster-ops
AI resourcesall 6 family resources × 9 verbsai-ops, then its delegates
Stackup, down, drill: preflight, drill: fullai-ops
Starter boxup / downcluster-ops
CI runner boxprovision, start, pause, backup, terminate, per archbuild-ops, then cluster-ops
God boxstart, pause, build, terminate on hermes/godai-ops, then cluster-ops
Runs ×2 + Logboth run shelves, with incremental followai-ops and cluster-ops

How it reaches three Workers from one origin. ai-ops carries named pass-through routes — /cluster/* and /build/*, implemented in src/estate.ts — and holds a separate bearer for each sibling. The browser talks only to ai-ops.

This is not a generic proxy, and the difference is the security argument: every forwarded route is named, there is no path wildcard, and cluster-ops' raw /oci/resources/* surface is deliberately not exposed. A request for a route the allowlist does not carry 404s at ai-ops without ever reaching the sibling.

No CSP client moved. cluster-ops still owns every OCI call and still holds the only OCI signing key; the runner path is ai-ops → build-ops → cluster-ops → OCI, three named hops and one OCI client. What changed is where the browser asks, not who holds the key.

The rejected alternative was CORS — let the page call all three origins with three tokens. It adds no server coupling, which is a real advantage, but it means putting an Access-Control-Allow-Origin on the Worker holding a tenancy-level signing key, and pasting three secrets into one page.

The gates are not reimplemented in the page. spendConfirm, confirm, naming-a-resource-back — each is enforced by the Worker that owns the action. The Confirm panel collects the field so the refusal is not how you discover it exists, and when a refusal does come back the panel stays open and shows the upstream's own message, which names the fix:

teardown requires {"cluster": "pacer-oke-drill"} — this is a DRILL: it
builds a separate cluster on its own tofu state and cannot touch
production DNS

The God box panel

Hermes plus its colocated LiteLLM, on pacer-hermes-god. It went missing from this page for a while: the dashboard covered cluster, gateway, agent and build-ops, and the one box carrying the agent had no panel at all, so the only way to start it was a hand-rolled curl. That is the gap this fills.

Unlike the Starter box, it speaks the generic family routes rather than a bespoke /cluster/* path, because hermes is a family in the registry:

POST /families/hermes/resources/god/start      {"spendConfirm": true}
POST /families/hermes/resources/god/pause
POST /families/hermes/resources/god/terminate  {"resource": "god"}
GET  /families/hermes/resources/god/status

Three things the panel encodes so they are not learned from a refusal:

Starting it does not repoint anything. Hermes reaches LiteLLM at litellm.testtemplate.org by NAME, and the name is what moves — see cluster-ops.md on the gateway's point / unpoint. A god box that comes up finds the name resolving wherever it resolved before.

Read the running pill as tri-state. running / stopped / unknown, and unknown is not stopped — see ai-ops.md § GET /resources. The page says so once, above the cards, and confirm against substrate is the tick that turns unknown into a real answer.

The two older consoles

They still exist and still work; each covers part of the estate.

Neither covers the CI runner box, and neither links to the other. That gap is what the dashboard exists to close (#81).

What the pages are, and what they are not

It is a shell. Static HTML and JavaScript with no secrets and no data in it. Everything it displays comes from the authenticated routes, fetched by your browser with a token you paste in. The page itself ships none.

The token lives in the tab, not on the server. Paste it into the CONTROL_TOKEN box and press Connect; Forget clears it. Nothing is persisted server-side.

The shell has to be reachable unauthenticated because a browser cannot put an Authorization header on a plain navigation — there is no way to open a page that demands one. The better answer is Cloudflare Access in front of the hostname, which is why wrangler.jsonc keeps these on a zone we control. Until that is configured, the shell is public and inert.

Both pages are served with X-Frame-Options: DENY and Cache-Control: no-store — the second so a page does not survive a deploy.

ai-ops — what is on it

cluster-ops — what is on it

Cluster status and history, the ops list, the starter box, and the same runs and log panes. It is the only way to drive a DR rebuild from a browser, which is why a styling regression in it matters more than it looks: nobody notices a broken page until the moment it is needed.

/docs — these documents, in the UI

Both pages link to /docs, which serves this directory, unauthenticated, from the same bytes committed at docs/agents/. shared/docs/ imports the Markdown into the Worker bundle, so there is one copy and it cannot drift.

GET /docs            index
GET /docs/<slug>     one document
GET /docs?format=md  the raw Markdown, for an agent that would rather parse it

<slug> is the filename without its extension — contract, ai-ops, cluster-ops, pacer-llm-control, build-ops, ui. An unknown slug returns 404 with the real list.

An agent should prefer ?format=md: it is the source text, with no HTML to strip.

Changing the UI — the one thing that will bite you

The shared stylesheet lives at shared/ui/ops.css and is composed into each shell at build time by shared/ui/render.ts, which substitutes the __OPS_CSS__ placeholder and then asserts nothing was missed. A __PLACEHOLDER__ that survives into a served page is not cosmetic — it would mean the page renders with no styling at all, which looks like a broken deploy and sends whoever sees it looking in the wrong place.

render.ts uses split/join rather than String.replace on purpose: the CSS contains $ sequences, and $& / $' in a replacement string are substitution patterns.

Before committing a UI change, run:

node scripts/check-ui.mjs
npm run typecheck

check-ui.mjs asserts two things: that the stylesheet composed into the shell is byte-identical to the one that was served inline before the extraction, and that every shell composes with no placeholder left over — so a shell that would throw at runtime, serving a 502 where the UI should be, fails at build time instead.

Two units traps it already knows about, recorded because each cost a round of debugging:

It needs git history to find its baseline, so it cannot run in a --depth 1 clone. That is a clone limitation, not drift: fetch more history, or run it where the history exists.


Rendered from docs/agents/ui.md in pacer-serverless, bundled into this Worker. Raw Markdown: ?format=md