# `pacer-llm-control` — the Runpod path

**Deployed** at `https://pacer-llm-control.testtemplate.org`.

Its name says "llm"; **its job is Runpod** — the LLM pod *and* `pacer-media`'s
image and video pods (ADR 0041). There is exactly one Runpod client in this
estate and it is here.

**Everything is gated**, `/docs` included. This Worker serves no page of its own.

Read [`contract.md`](./contract.md) first for auth, gates and error shapes.

## Routes

    GET  /ready                          can a pod be started at all
    GET  /resources                      the three specs, substrate-confirmed
    GET  /pods                           live pods
    GET  /gpu/availability?resource=<id> capacity, before you spend
    POST /resources/:id/up               {"spendConfirm": true}
    GET  /resources/:id/status
    POST /resources/:id/down
    POST /up                             alias for resource `vllm`
    GET  /status?id=<workflow-id>        alias for resource `vllm`
    POST /down                           alias for resource `vllm`

The bare `/up`, `/status` and `/down` are **kept as aliases for `vllm`** because
`ai-ops`' `llm.ts` and `stack.ts` call them, and breaking the running stack to
add media is not an improvement. Prefer the `/resources/:id/*` forms in new code.

## The spec registry

`src/specs.ts` holds three pod specs:

| `id` | GPU | Serves |
| --- | --- | --- |
| `vllm` | NVIDIA A100-SXM4-80GB | GLM-4.5-Air |
| `media-image` | NVIDIA L40S | vLLM-Omni, text-to-image |
| `media-video` | NVIDIA A100-SXM4-80GB | vLLM-Omni, LTX-2.3 text/image-to-video |

**Adding a resource is a registry entry, not a second Runpod client.**

Two hard rules about this file, both from ADR 0041:

1. **No provider discriminant.** The moment an entry carries
   `provider: "oci" | "runpod"`, the registry has become the forbidden dispatch
   table. It is data for *one* provider's client.
2. **It is never read from outside this Worker.** `ai-ops` does not import it.

Proposing either is a `ddd/modeling` escalation, not a refactor.

The module **asserts at load time that no pod name is a prefix of another's**.
That is not fussiness: `/down` deletes by pod name, and a prefix collision would
make one resource's teardown delete another's pod.

## `/down` is per resource

`ai-ops`' `llm.ts` documented its own `/down` as coarse and correct "only while
the family has one member". A second Runpod resource exists now, so the per-pod
delete it named as the fix is what landed.

`/down` returns `deleted`, `failed`, and `volumeKept: true`. It deletes pods
**one at a time and reports which ones failed** — a batch that half-succeeds
silently leaves you paying for pods you believe are gone. A non-empty `failed`
comes back as `502`.

**The volume is always kept.** It carries the model weights and the TLS
certificate.

## Volumes, and why each resource has its own

A Runpod network volume is **data-centre-pinned** and mounts read-write to
**exactly one pod**. Two models that must run concurrently therefore cannot
share one. So each media resource has its own volume, hostname and certificate —
that is a substrate constraint, not a design preference.

The volume attaches **only in the data centre it was created in**. The LLM
volume `pacer-llm-weights` (`tggdlsm22u`, 70 GB) is in **`US-KS-2`**, and
`wrangler.jsonc` carries the real `NETWORK_VOLUME_ID` / `DATA_CENTER_ID`.

Weights live at `/workspace/model`, the TLS cert at `/workspace/certs`. Both
outlive the pod, so a cycle reuses them — which matters because it avoids
spending one of Let's Encrypt's **5 duplicate certificates per host per week**.

## Certificates and addresses — be exact

The loose version of this claim is false, and it has been written down wrongly
more than once:

> A **certificate** makes the **origin verifiable**.
> **LiteLLM** makes the **address steady**.

Runpod reassigns a pod's external TCP port on **every** pod, and a DNS record
cannot carry a port. So **nothing about a certificate stabilises an address.**
What a cert buys is a direct route under our own name with verification on.
What makes an address steady is LiteLLM — `point-litellm.sh` /
`register-litellm.sh` rewrite one field.

**TLS-from-volume is ON for every Runpod resource, `llm` included.** It is one
pattern, not a media exception:

- `pacer-llm`'s `make runpod-live` sets `TLS_FROM_VOLUME=1 ATTACH_DOMAIN=1`, and
  its `measurements.md` records the full chain green — cert (volume reuse no-op,
  third consecutive run) → serve with TLS from the volume → domain attach →
  remote-prober verification at `https://llm.testtemplate.org:14806/v1` with
  `HTTP:200` and `CERT_VERIFY:0`.
- A ~6-minute cold wake already put that workload out of reach of the Runpod
  proxy's **100-second** Cloudflare cap.
- `serve.sh`'s `:-0` is a **bare-script default every real bring-up overrides**.
- `point-litellm.sh`'s `UPSTREAM_MODE=proxy` governs how LiteLLM *addresses* a
  pod, **not** how the pod *serves*.

Media only makes the cap bite on every request rather than on a cold one.

## Spend, measured

The pinned profile is an **A100-SXM4-80GB at $1.59/hr in US-KS-2**, billed per
second. Cold start has measured **149s–642s**, and once **62 minutes**.

Ignore the **$4.41 H100** figure quoted throughout the older docs: the volume is
in US-KS-2 and **that data centre does not carry an H100 80GB HBM3 at all**.

Check capacity before you spend — `GET /gpu/availability?resource=<id>`. Note
that Runpod's capacity data lives in its **GraphQL** API (`api.runpod.io`), not
the REST one (`rest.runpod.io`). Both hosts are forbidden from `ai-ops`, which
is why the forbidden-call rule names them both.

## Known limits

- **`GET /ready` returns `canStart: true`**, and
  [#24](https://github.com/ColtsFan07/pacer-serverless/issues/24) is closed —
  the Runpod credential exists and the volume is created.
- **That is not the same as serving, and the gap is the point.** A live GPU run
  on 2026-09-08 never served: the bring-up workflow's retry budget **exceeds
  Cloudflare Free's 50-subrequest cap**
  ([#29](https://github.com/ColtsFan07/pacer-serverless/issues/29)). That issue
  stands.
- [#30](https://github.com/ColtsFan07/pacer-serverless/issues/30) **has been
  overtaken by measurement and its stated fix is now wrong.** It concluded the
  certificate "was likely never issued" and that the Worker should default to
  plain HTTP "so it matches the proven path". The measurements above record the
  opposite. Hardcoding TLS-from-volume *matches* the proven path; defaulting it
  off would be the regression. What survives of #30 is narrower: `/ready` should
  reflect whether the cert is actually present.