index contract ai-ops cluster-ops pacer-llm-control build-ops ui

`ai-ops` — the front door

Deployed at https://ai-ops.testtemplate.org.

Owns the lease store, run log, watchdog, UI and family registry. Holds no cloud-provider client of its own — every family delegates to the Worker that owns its substrate. If you are starting, stopping or inspecting an AI resource, this is the Worker you want.

Read contract.md first for auth, gates and error shapes.

The model: family / resource / verb

/families/<family>/resources/<resourceId>/<verb>

A family is a named set of resources sharing a substrate and a delegate. It is not a provider interface — each family is implemented in its own module, natively, with no shared supertype. That is a hard rule (#2): proposing a shared driver type is a ddd/modeling escalation, not a refactor.

Family	Resources	Substrate	Delegate
`llm`	`vllm`	Runpod SECURE pod + network volume	`pacer-llm-control`
`media`	`image`, `video`	Runpod SECURE pods, one volume each	`pacer-llm-control`
`agent`	`pacer-ai`	OCI compute + block volume	`cluster-ops`
`gateway`	`litellm`	OCI compute (`A1.Flex`) + block volume	`cluster-ops`
`hermes`	`god`	OCI compute (`E5.Flex`) + block volume	`cluster-ops`

Do not hardcode this table. GET /families returns it live, along with the verb list, and it is the registry the Worker itself dispatches on.

Verbs

start  status  pause  terminate  backup  build  detach  restore  backups

status and backups are GET. Everything else is POST. backups is a GET for a reason that is not tidiness: it is where you get the backupId that restore requires, and a list you can only reach with a POST is a list a browser will not load.

Not every family implements every verb. build, detach and restore exist on hermes and nowhere else. A verb a family does not implement returns 501 with implementedIn and substrate — that is a substrate fact, not a gap to file. See contract.md § Reading the error.

Gates

start and build need {"spendConfirm": true}. terminate, detach and restore need {"resource": "<the id in the path>"}.

The spend-gate refusal quotes a cost profile that matches the family. For Runpod families it names the pinned A100-SXM4-80GB at $1.59/hr in US-KS-2, billed per second, with a cold start measured between 149s and 642s and once 62 minutes. The OCI families answer from their own module instead, because quoting a GPU's price for a ~$0.092/hr CPU box trains people to ignore the gate — and that is the one thing a spend gate cannot survive.

Discovery and state

GET /                        the UI shell (unauthenticated)
GET /docs                    these documents (unauthenticated)
GET /families                the registry above, plus every verb
GET /resources               what exists, and whether it is running
GET /resources?substrate=1   the same, CONFIRMED against each substrate
GET /gpu/availability        GPU capacity before you spend
GET /runs?limit=N            recent runs
GET /runs/:id                one run's status
GET /runs/:id/log?offset=N   that run's log, incrementally

`GET /resources` — read this one carefully

This answers "what is running right now", which is the question asked immediately before a destroy or a spend review. Every row carries:

running — tri-state. true (confirmed, or a live lease claims it), false (confirmed not running by the substrate), null (unknown — nobody asked the substrate).
confidence and why — which of those you have, and how it was reached.
lease, leaseExpiresInSeconds, label, persistence.

false and null are different claims. Conflating them is #59: running was derived from lease state alone, so a box created outside ai-ops read as not running while it was demonstrably running and billing. Answering "nothing is running" about a live instance is wrong in the expensive direction and the dangerous one — trusted, it reports a destroy as a no-op and leaves the box up.

?substrate=1 confirms each row through the family's own status verb. It costs one upstream call per resource, so it is off by default. Turn it on whenever the answer has to be right — before any destroy, and before any claim about spend.

Rows labelled (orphan — no descriptor) are leases with no matching resource in this build. They are surfaced rather than filtered because they are probably still billing.

The stack

A stack is a named, ordered composition of family verbs. There is exactly one, and it names its own members in its own module.

GET  /stacks                 list (there is one)
GET  /stacks/pacer           its current status
POST /stacks/pacer/up        bring it up, in order
POST /stacks/pacer/down      take it down, in order
POST /stacks/pacer/drill     rehearse it

These routes are hand-written for that one stack rather than parameterised over a registry, deliberately. A second stack means a second module and its own routes — that is what keeps this from becoming the data-driven runner ADR 0039 forbids. If you find yourself wanting to generalise them, that is an architecture/adr escalation.

drill takes a preflight mode (check without acting) and a full mode. The UI exposes both as separate buttons, and the full one is styled as destructive because it is.

The watchdog

A cron fires every 10 minutes and reports overdue leases. Read scheduled() in src/index.ts for why it cannot yet reap them — that is a known limit, not an oversight, and the hermes family is deliberately soft-stopped rather than terminated when its lease lapses, because a 90-minute lease sized for a $1.59/hr GPU would otherwise reap an agent mid-task to save about fourteen cents.

Lease budgets are LEASE_MINUTES (90) and HERMES_LEASE_MINUTES (180), both non-secret vars in wrangler.jsonc.

Worked examples

Bring the LLM up, then confirm it against the substrate rather than the lease:

curl -sS -X POST "$AI_OPS/families/llm/resources/vllm/start" \
  -H "Authorization: Bearer $CONTROL_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"spendConfirm": true}'

curl -sS "$AI_OPS/resources?substrate=1" -H "Authorization: Bearer $CONTROL_TOKEN"

Terminate the god box — note the resource named back, and that hermes is the only family with detach, restore, build and backups:

curl -sS -X POST "$AI_OPS/families/hermes/resources/god/terminate" \
  -H "Authorization: Bearer $CONTROL_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"resource": "god"}'

Follow a run to completion:

RUN=$(curl -sS "$AI_OPS/runs?limit=1" -H "Authorization: Bearer $CONTROL_TOKEN" \
  | jq -r '.runs[0].id')
curl -sS "$AI_OPS/runs/$RUN/log?offset=0" -H "Authorization: Bearer $CONTROL_TOKEN"

Adding a family

Write a new module. Name its delegate explicitly and speak that Worker's own routes. Do not implement a shared interface, and do not add a provider discriminant anywhere.

The test ADR 0037 wrote in advance: adding a family should mean writing a new file. It has now run three times — media (second Runpod family) and hermes (third OCI family, reaching the same Worker agent and gateway reach) both passed. hermes is the sharper case: its verb set is different, so a DRIVERS[family][verb] table would have had to carry three verbs for every family or grow a capability flag — the forbidden shape, arrived at from a different direction.

Three things are forbidden here and are cheap to grep for:

a shared ResourceDriver / OpsDriver type two family modules implement
a shared/ function that branches on provider to pick an API
any direct call to a CSP API from inside ai-ops — *.oraclecloud.com, rest.runpod.io, or api.runpod.io (capacity lives in Runpod's GraphQL, not REST, so naming only the REST host would leave the loophole open)

Rendered from docs/agents/ai-ops.md in pacer-serverless, bundled into this Worker. Raw Markdown: ?format=md

ai-ops — the front door