# The contract every Worker here shares Read this before any of the per-Worker files. Most failed calls against this estate break a rule on this page rather than a rule of the Worker being called. ## Authentication Every route is gated by a bearer token except the ones named below: Authorization: Bearer $CONTROL_TOKEN Each Worker has **its own** `CONTROL_TOKEN`. They are not interchangeable, and `ai-ops` holds separate secrets (`CLUSTER_OPS_TOKEN`, `LLM_CONTROL_TOKEN`) for calling its delegates. Tokens come from the private `envs` checkout. A missing or wrong token is a `401` with no detail — that is deliberate, not a bug to investigate. **The unauthenticated routes, and why.** `ai-ops` and `cluster-ops` serve `GET /` (the UI shell) and `GET /docs*` without a token. A browser cannot put an `Authorization` header on a plain navigation, so a page that demanded one could never be opened. Both carry static text only: no secrets, no data. Everything the page *displays* comes from gated routes the browser calls with a token the operator pastes in. `build-ops` and `pacer-llm-control` gate **everything**, `/docs` included. They serve no page of their own, so an unauthenticated route there would be a gate hole with no user. **If an unauthenticated request to a gated route ever returns `200`, stop.** That means the token gate is not in the request path, which is worse than a failed deploy. `scripts/deploy-worker.sh` asserts this after every deploy. ## The gates on spending and destruction These exist at more than one layer on purpose. `ai-ops` enforces its own gate *and* the delegate enforces its own; that is so `ai-ops` cannot become a way to route around `pacer-llm-control`'s gate. **Spend gate** — `start` and `build` refuse without: ```json { "spendConfirm": true } ``` **Naming back** — `terminate`, `detach` and `restore` refuse unless the request names its own target back: ```json { "resource": "" } ``` This is not theatre. It makes a replayed or copy-pasted request aimed at the wrong resource **fail closed** instead of destroying whichever resource happens to be running. `restore` replaces the disk holding every session, memory and learned skill an agent has; `detach` pulls that disk off a running box. `cluster-ops`' destructive ops take the equivalent, naming the cluster: ```json { "confirm": true, "cluster": "" } ``` **Do not write a wrapper that fills these in automatically.** The gate's whole value is that a human or an agent had to state the target a second time. ## Reading the error, instead of guessing These Workers answer with structured errors that tell you the fix. Read the body before retrying. | Status | Means | Do | | --- | --- | --- | | `400` | a gate refused, or the body was not JSON | read `error` and `hint`; they name the missing field | | `401` | wrong or missing bearer | fix the token; do not retry the same call | | `404` | **the thing you named does not exist** | read `available` / `families` / `resources` — they list the real ones | | `405` | right route, wrong method | reads are `GET`, everything that mutates is `POST` | | `501` | **it exists; this verb is not defined for it** | not a gap to file — see below | | `502` | the upstream (OCI, Runpod, a delegate) failed | the body carries the upstream's own message | **`404` and `501` are different claims and the distinction is load-bearing.** `501` means the family exists and this verb is not defined for it, which for `restore` on a Runpod family is a *substrate fact*, not a missing feature. The `501` body names `implementedIn` and `substrate` so you can see why. Verbs differ by substrate, not by taste. A `404` body is self-describing: it carries the real list. Use it rather than guessing a second spelling. ## Runs and logs Anything long-running returns a `runId` and a `202`, not a result. It is fire-and-forget: the work ships its log to R2 and then terminates itself. GET /runs?limit=N recent runs, newest first GET /runs/:id that run's status GET /runs/:id/log?offset=N the log, incrementally `offset` is how you follow a log without re-reading it: pass back the offset the previous response returned. The UI's "follow" checkbox does exactly this. `ai-ops` owns the `ai-runs/` prefix; `cluster-ops` owns `ops-runs/`. **`build-ops` deliberately reads `cluster-ops`' prefix** rather than owning a second one, because a `provision` run *is* a `cluster-ops` ops box — owning a separate prefix would mean a run that appears in one Worker's list and not the other's. ## Leases, and what they do not prove A lease is a watchdog budget, not a fact about the substrate. It says "something claimed this resource until time T". The cron sweeper reaps on a lapsed lease. **A lease is not evidence that a thing is running, and its absence is not evidence that nothing is.** A box created outside `ai-ops` has no lease and reads as `running: null` — unknown, not stopped. That is the tri-state above, and it is why `?substrate=1` exists. An orphan lease with no matching resource descriptor is surfaced rather than filtered out, labelled `(orphan — no descriptor)`, because it is probably still billing. ## The trap that governs how probes are written **A deployed Worker's `fetch()` silently discards a non-standard port on an HTTPS URL.** No error, no warning — the request goes to `:443`. Any probe or API call against a non-standard port **must** use `connect()` from `cloudflare:sockets`. This matters more than it sounds: the failure mode is a *green* check against the wrong endpoint. Prefer an assertion that names the fix in its failure message over one that returns a bare status code. ## Before you commit a Worker ```bash npm run typecheck npx wrangler deploy --dry-run --outdir=/tmp/build ``` Both are cheap and catch the common failures. **Neither one deploys anything.** "Typechecks and bundles" is a different claim from "deployed and serving", and conflating them has cost this project real debugging time. Say which you did. ## Deploying ```bash scripts/deploy-worker.sh ai-ops # bundle-only, the normal case scripts/deploy-worker.sh cluster-ops # --containers-rollout=none scripts/deploy-worker.sh cluster-ops --containers # needs a Docker daemon scripts/deploy-worker.sh ai-ops --dry-run # typecheck + bundle only ``` **GitHub Actions does not run for this account** — free plan, so hosted runners are never assigned (`runner_id: 0`, no steps, logs 404). This is [#58](https://github.com/ColtsFan07/pacer-serverless/issues/58), closed as won't-fix. There is nothing to debug in the workflow, and "CI is down" does not mean "nothing can ship" — reading it that way left `main` undeployed for eight days. The script runs the workflow's own checks: live credential check, typecheck, dry-run, deploy, then **two verifications** — the account's deployments API must agree with the version wrangler printed, and an unauthenticated request must still be refused. A **container** change still needs a Docker daemon, which a sandbox does not have. That is the self-hosted runner box, and per the standing rule every image build runs there and the box is **torn down afterwards with its volume intact**. A build that leaves the box running is an unfinished build. ## Track what you find Anything discovered — a bug, drift, a follow-up, a missing guard — becomes a GitHub issue in this repo **while the context is fresh**. Work that lives only in a session transcript is lost when the container is reclaimed. Keep issue titles and bodies free of secret values: **name the key, never the value.** A secret that reaches a transcript is published, not merely exposed — say so immediately and treat rotation as required, not optional.