# `build-ops` — the CI runner box

**DEPLOYED and serving** at `https://build-ops.testtemplate.org`.

Be careful with older notes about this Worker: `README.md`, `CLAUDE.md` and
`pacer-agent-system-doctrine.md` § 7 have all said "not deployed; has never
served a request", and **this file said it too until 2026-09-17**. It is wrong.
The Worker answers on its custom domain and enforces its own token gate —
recorded as [#50](https://github.com/ColtsFan07/pacer-serverless/issues/50) on
2026-09-10 and confirmed again on 2026-09-17. When it first went live, and by
what route, is still not established.

**What is verified, and what is not.** `status`, `backups` and `terminate` were
exercised live on 2026-09-17, after
[#80](https://github.com/ColtsFan07/pacer-serverless/issues/80) corrected how
this Worker addresses the box. `provision` and `start` have **not** been run —
they spend money, and nothing has needed a runner box since. So this Worker
reports correctly; whether it can successfully *build* a box end to end is
untested.

`cluster-ops`' `runner` op remains the path ADR 0038's own notes point at, and
is the one that has actually provisioned boxes — see
[`cluster-ops.md`](./cluster-ops.md) § The `runner` op.

ADR 0038 gives this Worker the lease, run log, watchdog and UI for the
ephemeral OCI instance that runs Docker builds outside the cluster and hosts the
`self-hosted` runners. It holds **no cloud-provider client**: it delegates its
OCI calls to `cluster-ops`, exactly as `ai-ops` does.

**Everything is gated**, `/docs` included — it serves no page of its own yet, so
an unauthenticated route added "for the UI" before then would be a gate hole with
no user.

## Routes

    GET  /version
    GET  /status?arch=x86|arm
    GET  /backups?arch=x86|arm
    POST /provision    {"arch": "...", "mode": "...", "confirm": true}
    POST /start        {"spendConfirm": true, "arch": "..."}
    POST /pause        {"arch": "..."}
    POST /terminate    {"resource": "runner", "arch": "..."}
    POST /backup       {"arch": "..."}
    POST /restore      {"resource": "runner", "backupId": "...", "arch": "..."}
    GET  /runs?limit=N
    GET  /runs/:id
    GET  /runs/:id/log?offset=N

**`arch` is on every lifecycle route, and it is not optional detail.** The two
boxes have separate instances, separate volumes and separate backup lists, so a
call that omits it acts on **x86** and never touches the ARM box. `x86` or
`arm`; anything else is a `400` naming the valid set.

This Worker **serves no page**. `GET /` is a `404` — the UI ADR 0038 gives it
was never built, and the estate dashboard at
`https://ai-ops.testtemplate.org/dashboard` is this box's screen.

## Two names for one box, and why both are right

This Worker calls the box **`build/runner`** — its own namespace, where its
lease and run log live. `cluster-ops` knows the same box as **`runner/oci`** (or
`runner/oci-arm`), because that is the key its name overrides carry:

    runner/oci      ->  instance pacer-runner-oci      volume pacer-runner-oci-data
    runner/oci-arm  ->  instance pacer-runner-oci-arm  volume pacer-runner-oci-arm-data

`/status` reports the translation as `ociResource`, so a future mismatch is
visible in the response rather than only in a volume name nobody recognises.

**This was broken until 2026-09-17**
([#80](https://github.com/ColtsFan07/pacer-serverless/issues/80)). Every volume
call went to `/oci/resources/build/runner`, which derives `build-runner-data` —
a name nothing has ever created — so `/status?arch=arm` reported the volume
**absent** while `pacer-runner-oci-arm-data` existed with 50 GB of real state.
Had `provision` run, it would have created a second empty volume while the TLS
certificate and runner registration sat on the first.

**Two name-backs, and they are different strings.** You name the box back as
`runner`; this Worker translates that to `oci` / `oci-arm` for cluster-ops,
whose gate compares the name-back against its own path segment. Forwarding your
word verbatim would fail that gate and read as your typo.

## It reads `cluster-ops`' run prefix, deliberately

`/runs*` here reads the **`ops-runs/`** prefix that `cluster-ops` owns, not a
prefix of its own. That is correct rather than sloppy: a `provision` run *is*
literally a `cluster-ops` ops box, launched by `cluster-ops` and reporting where
every other ops box reports. Owning a second prefix would produce a run that
appears in one Worker's list and not the other's.

## The standing rule this Worker exists to serve

Anything that **builds or pushes a Docker image** runs on the **self-hosted OCI
runner box**, and the box is **torn down afterwards with its volume intact**.

    build / push an image   ->  the self-hosted runner box      DEFAULT
    GitHub-hosted runners   ->  an option, chosen deliberately   NOT the default

This is a default, not a prohibition. `ubuntu-latest` remains available and is
sometimes right — but choosing it is **a decision to state** in the PR or the
log, not something to fall into because a workflow already said
`runs-on: ubuntu-latest`. A `runs-on:` line already sitting in a workflow is not
a decision anyone made recently.

Provision → build → push → **destroy, volume intact**. A build that leaves the
box running is an unfinished build. The volume is never deleted: it carries the
TLS certificate and the runner registration, which make the next provision cheap.

## What is still unsettled

Stated here so a session does not discover them mid-build and assume it broke
something:

- **Nobody knows when this Worker first went live.** It was believed undeployed
  for a week while answering on its custom domain and running a `*/10 * * * *`
  watchdog. Three documents still say it is not deployed
  ([#50](https://github.com/ColtsFan07/pacer-serverless/issues/50),
  [#79](https://github.com/ColtsFan07/pacer-serverless/issues/79)).
- **`provision` and `start` are unexercised.** They spend money and nothing has
  needed a box since the addressing was fixed.
- **`cluster-ops` deploys were themselves broken** —
  [#58](https://github.com/ColtsFan07/pacer-serverless/issues/58). GitHub Actions
  does not run for this account, so a change to the `runner` op could not reach
  production through CI. `scripts/deploy-worker.sh` is the answer; see
  [`contract.md`](./contract.md) § Deploying.

Two related corrections worth carrying, because both were wrong in the obvious
direction before being measured:

- **The persistent volume is implemented; the attach raced.**
  `runner-box-provision.sh` has always had `DATA_VOL_NAME`, `DATA_MOUNT`, and a
  destroy that detaches and never deletes. What was wrong is subtler: it
  attached immediately after launch, and **OCI refuses an attach while the
  instance is still `PROVISIONING`**, so the single attempt lost the race and the
  box came up with no persistent store — behind a warning whose text was garbled
  because the error had been sent to `/dev/null`. It now waits for `RUNNING` and
  retries.
- **Two names for one disk.** The script names the volume `${BOX_NAME}-data`
  (`pacer-runner-oci-data`); `cluster-ops`' deriver produced `runner-oci-data`.
  `start` would have built an empty duplicate, and `terminate` would have
  terminated the box with the real disk still attached and unmounted. Both
  arches now carry `VOLUME_NAME_OVERRIDES` entries.

`pacer-runner-oci` was terminated 2026-09-11 at the owner's instruction to stop
billing. **The next build provisions a fresh one — that is the intended
lifecycle, not an error state.**