# `cluster-ops` — the OCI path **Deployed and verified live** at `https://cluster-ops.testtemplate.org`. Its name says "cluster"; **its job is OCI** — the OKE cluster *and* every other OCI compute and volume resource driven on behalf of `ai-ops` families. There is exactly one OCI client in this estate and it is here. Do not add a second one anywhere. This Worker **holds a tenancy-level signing key**. It is never public. Read [`contract.md`](./contract.md) first for auth, gates and error shapes. ## Three surfaces, and which one you want /ops/ ephemeral OPS BOX — the heavy cluster operations /starter the long-lived dev box /oci/resources///... one family resource's instance + volume They are genuinely different mechanisms, not three spellings of one. `/ops/*` launches a throwaway box that does a job and destroys itself. `/oci/resources/*` manipulates a **named, long-lived** instance and its volume directly. ## `POST /ops/` — the ops box Launches an ephemeral OCI instance in the `pacer-ops` compartment that runs one operation, ships its log to R2, and **terminates itself**. Fire-and-forget: you get `202` with a `runId`, a `deadlineIso`, and the R2 paths for its log and status. You do not get a result. Operations and their watchdog deadlines (minutes): | Op | Deadline | Op | Deadline | | --- | --- | --- | --- | | `status` | 15 | `longhorn` | 45 | | `diagnose` | 15 | `backup` | 45 | | `repair-webhooks` | 15 | `e2e` | 45 | | `node-journal` | 15 | `teardown` | 60 | | `openhands` | — | `rebuild` | 120 | | `runner` | — | `restore` | 120 | | `stack-drill` | — | | | The deadlines are **generous on purpose**. The watchdog guards against a *wedged* box, not a slow one: killing a live `tofu apply` leaves a lock in R2 and a half-built cluster, which is worse than paying for twenty idle minutes. `teardown`, `restore` and `rebuild` need: ```json { "confirm": true, "cluster": "" } ``` `GET /` on a `404` returns the live op list. Use it rather than this table if the two disagree — and then fix this file. ### The `runner` op — building images This is the path that has actually provisioned runner boxes. `build-ops` — the Worker ADR 0038 gives this box — **is deployed and serving** (correcting a claim that stood in three documents until 2026-09-17, #50/#79), but its `provision` and `start` are unexercised, so this op remains the proven route. RUNNER_MODE=status|create|destroy|destroy-all (default: status) RUNNER_ARCH=x86|arm (default: x86) CONFIRM=yes required for anything but status **`RUNNER_ARCH` is load-bearing on destroy, not cosmetic.** The box name encodes the architecture — `pacer-runner-oci` vs `pacer-runner-oci-arm` — so a destroy with the wrong arch **destroys the other box**, or reports nothing and looks like it already ran. Architecture matters for a second reason: **x86** (`VM.Standard.E5.Flex`) costs money and is the default because the OKE nodes are amd64 — an arm64 image built for them fails at deploy time with an exec-format error a long way from its cause. **arm** (`VM.Standard.A1.Flex`, Ampere) is Always Free and is viable for images that are not compiled artefacts. Prefer arm where the image genuinely permits it; the saving is the whole cost of the box. Every first-party image goes to **`ghcr.io/coltsfan07/`**. Pushing one anywhere else is a decision to raise, not a default to fall into. **Tearing the box down is part of the job.** Provision → build → push → **destroy, volume intact**. A build that leaves the box running is an unfinished build — a billable instance nobody is watching, which is exactly the orphan-that-still-bills shape ADR 0036 names. The volume is never deleted: it carries the TLS certificate and the runner registration, which are what make the next provision cheap. The box lives **outside the cluster's VCN** deliberately (`pacer-runner-vcn`). A runner inside the pacer-oke VCN is destroyed by `teardown.sh`'s `tofu destroy`, and because queued jobs never fail, **its disappearance is invisible**. Do not move it inside. ## Cluster reads GET /cluster/status the active cluster GET /cluster/history what has been done to it GET /verify a verification pass GET /version what this Worker is ## `/starter` — the dev box GET /starter is it up POST /starter bring it up DELETE /starter take it down GET|POST /starter/ proxied sub-routes The starter box **carries no deadline in its name**, so the cron sweeper never touches it. It is reaped on a **lapsed lease** instead — which is what keeps a long operation from being killed mid-flight, because a busy box keeps renewing. ## `/oci/resources//` — long-lived resources GET /oci/resources/:f/:id instance + volume, present or not POST /oci/resources/:f/:id/build create from scratch POST /oci/resources/:f/:id/start make it be running POST /oci/resources/:f/:id/stop POST /oci/resources/:f/:id/detach POST /oci/resources/:f/:id/terminate POST /oci/resources/:f/:id/attach-clone POST /oci/resources/:f/:id/detach-clone GET /oci/resources/:f/:id/backups POST /oci/resources/:f/:id/backup POST /oci/resources/:f/:id/restore `family` and `resourceId` are validated **before any OCI call** — lowercase alphanumeric plus `. _ -`, 1–64 characters. An id like `../starter` would otherwise reach a display-name lookup, and the derived names are the only link between a resource and its OCI objects. ### `build` refuses where `start` adopts This asymmetry is deliberate and the two failures are opposite: - **`start` is idempotent by adoption.** It takes whatever instance carries the derived name, in whatever state. That is right for a verb whose job is *make it be running*. - **`build` refuses if the instance already exists.** That is right for a verb whose job is *make it exist*. Adopting on a build would hide that the box was never rebuilt; creating on a start would launch a second billable instance that **can never attach the first one's volume**. `build` is defined only where a cloud-init defines the box. There is one today (`hermes/god`). A second means writing its own profile module and adding a branch — deliberately **not** a registry keyed on a provider field, which is the forbidden table. ### The compartment trap These routes read `OCI_COMPARTMENT_OCID`, **not** `OPS_COMPARTMENT_OCID` ([#40](https://github.com/ColtsFan07/pacer-serverless/issues/40)). The estate uses two compartments: `OPS_COMPARTMENT_OCID` is `pacer-ops`, where ephemeral ops boxes launch; long-lived family resources live in `OCI_COMPARTMENT_OCID`. The consequence of the old value was **not a 404**. `ensureVolumeNamed` would have found nothing and **created a second, empty volume** in `pacer-ops` while the real data sat elsewhere — the orphan-that-still-bills failure, again. ## The sweeper A cron every 10 minutes terminates overdue ops boxes (whose deadline is encoded in the box name) and reaps the starter box on an expired lease. It **never throws**: a failed sweep must not stop the next one being scheduled. Cron Triggers work on the Cloudflare free plan, which is the whole reason this architecture is affordable. ## Errors from OCI A `502` body carries **OCI's own message**, truncated to 500 characters. That is deliberate: a `401` from OCI does not say which signed header was wrong, so the raw text is worth more than a tidy wrapper. ## Deploying this one `cluster-ops` has a **container**, so read [`contract.md`](./contract.md) § Deploying before shipping it: ```bash scripts/deploy-worker.sh cluster-ops # --containers-rollout=none scripts/deploy-worker.sh cluster-ops --containers # needs a Docker daemon ``` A **bundle-only** change ships from a session. The cloud-init files are a text import into the bundle, so **a box-definition change is a bundle change** — it does not need a daemon. A change to the container image does, and that is the self-hosted runner box.