index contract ai-ops cluster-ops pacer-llm-control build-ops ui

`cluster-ops` — the OCI path

Deployed and verified live at https://cluster-ops.testtemplate.org.

Its name says "cluster"; its job is OCI — the OKE cluster and every other OCI compute and volume resource driven on behalf of ai-ops families. There is exactly one OCI client in this estate and it is here. Do not add a second one anywhere.

This Worker holds a tenancy-level signing key. It is never public.

Read contract.md first for auth, gates and error shapes.

Three surfaces, and which one you want

/ops/<op>                    ephemeral OPS BOX — the heavy cluster operations
/starter                     the long-lived dev box
/oci/resources/<f>/<id>/...  one family resource's instance + volume

They are genuinely different mechanisms, not three spellings of one. /ops/* launches a throwaway box that does a job and destroys itself. /oci/resources/* manipulates a named, long-lived instance and its volume directly.

`POST /ops/<op>` — the ops box

Launches an ephemeral OCI instance in the pacer-ops compartment that runs one operation, ships its log to R2, and terminates itself. Fire-and-forget: you get 202 with a runId, a deadlineIso, and the R2 paths for its log and status. You do not get a result.

Operations and their watchdog deadlines (minutes):

Op	Deadline	Op	Deadline
`status`	15	`longhorn`	45
`diagnose`	15	`backup`	45
`repair-webhooks`	15	`e2e`	45
`node-journal`	15	`teardown`	60
`openhands`	—	`rebuild`	120
`runner`	—	`restore`	120
`stack-drill`	—

The deadlines are generous on purpose. The watchdog guards against a wedged box, not a slow one: killing a live tofu apply leaves a lock in R2 and a half-built cluster, which is worse than paying for twenty idle minutes.

teardown, restore and rebuild need:

{ "confirm": true, "cluster": "<cluster name>" }

GET / on a 404 returns the live op list. Use it rather than this table if the two disagree — and then fix this file.

The `runner` op — building images

This is the path that has actually provisioned runner boxes. build-ops — the Worker ADR 0038 gives this box — is deployed and serving (correcting a claim that stood in three documents until 2026-09-17, #50/#79), but its provision and start are unexercised, so this op remains the proven route.

RUNNER_MODE=status|create|destroy|destroy-all    (default: status)
RUNNER_ARCH=x86|arm                              (default: x86)
CONFIRM=yes                                      required for anything but status

RUNNER_ARCH is load-bearing on destroy, not cosmetic. The box name encodes the architecture — pacer-runner-oci vs pacer-runner-oci-arm — so a destroy with the wrong arch destroys the other box, or reports nothing and looks like it already ran.

Architecture matters for a second reason: x86 (VM.Standard.E5.Flex) costs money and is the default because the OKE nodes are amd64 — an arm64 image built for them fails at deploy time with an exec-format error a long way from its cause. arm (VM.Standard.A1.Flex, Ampere) is Always Free and is viable for images that are not compiled artefacts. Prefer arm where the image genuinely permits it; the saving is the whole cost of the box.

Every first-party image goes to ghcr.io/coltsfan07/. Pushing one anywhere else is a decision to raise, not a default to fall into.

Tearing the box down is part of the job. Provision → build → push → destroy, volume intact. A build that leaves the box running is an unfinished build — a billable instance nobody is watching, which is exactly the orphan-that-still-bills shape ADR 0036 names. The volume is never deleted: it carries the TLS certificate and the runner registration, which are what make the next provision cheap.

The box lives outside the cluster's VCN deliberately (pacer-runner-vcn). A runner inside the pacer-oke VCN is destroyed by teardown.sh's tofu destroy, and because queued jobs never fail, its disappearance is invisible. Do not move it inside.

Cluster reads

GET /cluster/status      the active cluster
GET /cluster/history     what has been done to it
GET /verify              a verification pass
GET /version             what this Worker is

`/starter` — the dev box

GET    /starter          is it up
POST   /starter          bring it up
DELETE /starter          take it down
GET|POST /starter/<sub>  proxied sub-routes

The starter box carries no deadline in its name, so the cron sweeper never touches it. It is reaped on a lapsed lease instead — which is what keeps a long operation from being killed mid-flight, because a busy box keeps renewing.

`/oci/resources/<family>/<resourceId>` — long-lived resources

GET  /oci/resources/:f/:id                 instance + volume, present or not
POST /oci/resources/:f/:id/build           create from scratch
POST /oci/resources/:f/:id/start           make it be running
POST /oci/resources/:f/:id/stop
POST /oci/resources/:f/:id/detach
POST /oci/resources/:f/:id/terminate
POST /oci/resources/:f/:id/attach-clone
POST /oci/resources/:f/:id/detach-clone
GET  /oci/resources/:f/:id/backups
POST /oci/resources/:f/:id/backup
POST /oci/resources/:f/:id/restore

family and resourceId are validated before any OCI call — lowercase alphanumeric plus . _ -, 1–64 characters. An id like ../starter would otherwise reach a display-name lookup, and the derived names are the only link between a resource and its OCI objects.

`build` refuses where `start` adopts

This asymmetry is deliberate and the two failures are opposite:

start is idempotent by adoption. It takes whatever instance carries the derived name, in whatever state. That is right for a verb whose job is *make it be running*.
build refuses if the instance already exists. That is right for a verb whose job is make it exist. Adopting on a build would hide that the box was never rebuilt; creating on a start would launch a second billable instance that can never attach the first one's volume.

build is defined only where a cloud-init defines the box. There is one today (hermes/god). A second means writing its own profile module and adding a branch — deliberately not a registry keyed on a provider field, which is the forbidden table.

The compartment trap

These routes read OCI_COMPARTMENT_OCID, not OPS_COMPARTMENT_OCID (#40). The estate uses two compartments: OPS_COMPARTMENT_OCID is pacer-ops, where ephemeral ops boxes launch; long-lived family resources live in OCI_COMPARTMENT_OCID.

The consequence of the old value was not a 404. ensureVolumeNamed would have found nothing and created a second, empty volume in pacer-ops while the real data sat elsewhere — the orphan-that-still-bills failure, again.

The sweeper

A cron every 10 minutes terminates overdue ops boxes (whose deadline is encoded in the box name) and reaps the starter box on an expired lease. It never throws: a failed sweep must not stop the next one being scheduled.

Cron Triggers work on the Cloudflare free plan, which is the whole reason this architecture is affordable.

Errors from OCI

A 502 body carries OCI's own message, truncated to 500 characters. That is deliberate: a 401 from OCI does not say which signed header was wrong, so the raw text is worth more than a tidy wrapper.

Deploying this one

cluster-ops has a container, so read contract.md § Deploying before shipping it:

scripts/deploy-worker.sh cluster-ops              # --containers-rollout=none
scripts/deploy-worker.sh cluster-ops --containers # needs a Docker daemon

A bundle-only change ships from a session. The cloud-init files are a text import into the bundle, so a box-definition change is a bundle change — it does not need a daemon. A change to the container image does, and that is the self-hosted runner box.

Rendered from docs/agents/cluster-ops.md in pacer-serverless, bundled into this Worker. Raw Markdown: ?format=md

cluster-ops — the OCI path