# `ai-ops` — the front door **Deployed** at `https://ai-ops.testtemplate.org`. Owns the **lease store, run log, watchdog, UI and family registry**. Holds **no cloud-provider client of its own** — every family delegates to the Worker that owns its substrate. If you are starting, stopping or inspecting an AI resource, this is the Worker you want. Read [`contract.md`](./contract.md) first for auth, gates and error shapes. ## The model: family / resource / verb /families//resources// A **family** is a named set of resources sharing a substrate and a delegate. It is *not* a provider interface — each family is implemented in its own module, natively, with no shared supertype. That is a hard rule ([#2](https://github.com/ColtsFan07/pacer-serverless/issues/2)): proposing a shared driver type is a `ddd/modeling` escalation, not a refactor. | Family | Resources | Substrate | Delegate | | --- | --- | --- | --- | | `llm` | `vllm` | Runpod SECURE pod + network volume | `pacer-llm-control` | | `media` | `image`, `video` | Runpod SECURE pods, **one volume each** | `pacer-llm-control` | | `agent` | `pacer-ai` | OCI compute + block volume | `cluster-ops` | | `gateway` | `litellm` | OCI compute (`A1.Flex`) + block volume | `cluster-ops` | | `hermes` | `god` | OCI compute (`E5.Flex`) + block volume | `cluster-ops` | Do not hardcode this table. `GET /families` returns it live, along with the verb list, and it is the registry the Worker itself dispatches on. ## Verbs start status pause terminate backup build detach restore backups **`status` and `backups` are `GET`. Everything else is `POST`.** `backups` is a `GET` for a reason that is not tidiness: it is where you get the `backupId` that `restore` requires, and a list you can only reach with a `POST` is a list a browser will not load. **Not every family implements every verb.** `build`, `detach` and `restore` exist on `hermes` and nowhere else. A verb a family does not implement returns `501` with `implementedIn` and `substrate` — that is a substrate fact, not a gap to file. See [`contract.md`](./contract.md) § Reading the error. ### Gates `start` and `build` need `{"spendConfirm": true}`. `terminate`, `detach` and `restore` need `{"resource": ""}`. The spend-gate refusal quotes a **cost profile that matches the family**. For Runpod families it names the pinned A100-SXM4-80GB at **$1.59/hr in US-KS-2**, billed per second, with a cold start measured between **149s and 642s** and once **62 minutes**. The OCI families answer from their own module instead, because quoting a GPU's price for a ~$0.092/hr CPU box trains people to ignore the gate — and that is the one thing a spend gate cannot survive. ## Discovery and state GET / the UI shell (unauthenticated) GET /docs these documents (unauthenticated) GET /families the registry above, plus every verb GET /resources what exists, and whether it is running GET /resources?substrate=1 the same, CONFIRMED against each substrate GET /gpu/availability GPU capacity before you spend GET /runs?limit=N recent runs GET /runs/:id one run's status GET /runs/:id/log?offset=N that run's log, incrementally ### `GET /resources` — read this one carefully This answers "what is running right now", which is the question asked immediately before a destroy or a spend review. Every row carries: - **`running`** — **tri-state.** `true` (confirmed, or a live lease claims it), `false` (confirmed **not** running by the substrate), `null` (**unknown** — nobody asked the substrate). - **`confidence`** and **`why`** — which of those you have, and how it was reached. - `lease`, `leaseExpiresInSeconds`, `label`, `persistence`. `false` and `null` are **different claims**. Conflating them is [#59](https://github.com/ColtsFan07/pacer-serverless/issues/59): `running` was derived from lease state alone, so a box created outside `ai-ops` read as *not running* while it was demonstrably running and billing. Answering "nothing is running" about a live instance is wrong in the expensive direction **and** the dangerous one — trusted, it reports a destroy as a no-op and leaves the box up. `?substrate=1` confirms each row through the family's own `status` verb. It costs one upstream call per resource, so it is off by default. **Turn it on whenever the answer has to be right** — before any destroy, and before any claim about spend. Rows labelled `(orphan — no descriptor)` are leases with no matching resource in this build. They are surfaced rather than filtered **because they are probably still billing**. ## The stack A stack is a **named, ordered composition of family verbs**. There is exactly one, and it names its own members in its own module. GET /stacks list (there is one) GET /stacks/pacer its current status POST /stacks/pacer/up bring it up, in order POST /stacks/pacer/down take it down, in order POST /stacks/pacer/drill rehearse it These routes are hand-written for that one stack rather than parameterised over a registry, deliberately. **A second stack means a second module and its own routes** — that is what keeps this from becoming the data-driven runner ADR 0039 forbids. If you find yourself wanting to generalise them, that is an `architecture/adr` escalation. `drill` takes a `preflight` mode (check without acting) and a full mode. The UI exposes both as separate buttons, and the full one is styled as destructive because it is. ## The watchdog A cron fires every 10 minutes and **reports** overdue leases. Read `scheduled()` in `src/index.ts` for why it cannot yet reap them — that is a known limit, not an oversight, and the `hermes` family is deliberately soft-stopped rather than terminated when its lease lapses, because a 90-minute lease sized for a $1.59/hr GPU would otherwise reap an agent mid-task to save about fourteen cents. Lease budgets are `LEASE_MINUTES` (90) and `HERMES_LEASE_MINUTES` (180), both non-secret vars in `wrangler.jsonc`. ## Worked examples Bring the LLM up, then confirm it against the substrate rather than the lease: ```bash curl -sS -X POST "$AI_OPS/families/llm/resources/vllm/start" \ -H "Authorization: Bearer $CONTROL_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"spendConfirm": true}' curl -sS "$AI_OPS/resources?substrate=1" -H "Authorization: Bearer $CONTROL_TOKEN" ``` Terminate the god box — note the resource named back, and that `hermes` is the only family with `detach`, `restore`, `build` and `backups`: ```bash curl -sS -X POST "$AI_OPS/families/hermes/resources/god/terminate" \ -H "Authorization: Bearer $CONTROL_TOKEN" \ -H 'Content-Type: application/json' \ -d '{"resource": "god"}' ``` Follow a run to completion: ```bash RUN=$(curl -sS "$AI_OPS/runs?limit=1" -H "Authorization: Bearer $CONTROL_TOKEN" \ | jq -r '.runs[0].id') curl -sS "$AI_OPS/runs/$RUN/log?offset=0" -H "Authorization: Bearer $CONTROL_TOKEN" ``` ## Adding a family Write a **new module**. Name its delegate explicitly and speak that Worker's own routes. Do **not** implement a shared interface, and do **not** add a provider discriminant anywhere. The test ADR 0037 wrote in advance: *adding a family should mean writing a new file.* It has now run three times — `media` (second Runpod family) and `hermes` (third OCI family, reaching the same Worker `agent` and `gateway` reach) both passed. `hermes` is the sharper case: its **verb set is different**, so a `DRIVERS[family][verb]` table would have had to carry three verbs for every family or grow a capability flag — the forbidden shape, arrived at from a different direction. Three things are forbidden here and are cheap to grep for: 1. a shared `ResourceDriver` / `OpsDriver` type two family modules implement 2. a `shared/` function that branches on provider to pick an API 3. **any** direct call to a CSP API from inside `ai-ops` — `*.oraclecloud.com`, `rest.runpod.io`, **or `api.runpod.io`** (capacity lives in Runpod's GraphQL, not REST, so naming only the REST host would leave the loophole open)