docs/specs/2026-09-05-runner-isolation-design.md

  1# Runner isolation with rootless podman
  2
  3Ref #144. Milestone v1.14.0. Splits from #115, whose concurrency half (`-jobs N`)
  4is done.
  5
  6## Problem
  7
  8A build runs whatever a repository's `.gitbay/ci.yml` says, through `sh -c`, as
  9the runner process's own user (`cmd/gitbay-runner/main.go:309`).
 10
 11`deploy/gitbay-runner.override.conf` constrains the *service* —
 12`NoNewPrivileges`, `ProtectSystem=full`, `RestrictSUIDSGID`, CPU and IO weight —
 13and the runner's key carries scope `runner`, so a build cannot administer the
 14instance. Three things it can still do:
 15
 16- **Read the runner's SSH private key.** It is on disk, owned by `ci-runner`,
 17  and a step runs as `ci-runner`. Scope `runner` is not nothing: it claims
 18  builds, posts logs, and marks builds done, for every repository the runner
 19  serves.
 20- **Read the runner's environment.** `cmd.Env = append(os.Environ(), …)`
 21  (`main.go:311`) hands each step the runner's entire environment, not a
 22  constructed one.
 23- **See other builds in flight.** `-jobs N` puts N workspaces under one
 24  `-workdir`, all readable by the same user.
 25
 26Acceptable while every repository on the instance is the operator's. It stops
 27being acceptable the moment a fork's CI runs, which `registration = "open"`
 28makes reachable.
 29
 30## Decision
 31
 32Rootless podman, chosen over bubblewrap and a hand-rolled `unshare` sandbox.
 33
 34The deciding factor was not isolation strength — bubblewrap would have been
 35enough for that, at one small package with no daemon. It was `image:` per job.
 36An OCI runtime is the only option that gets there, and a CI system where every
 37build runs against whatever the host happens to have installed is a CI system
 38people work around rather than with.
 39
 40The cost is real and should be stated rather than discovered: podman is a
 41substantially larger dependency on bay1 than anything the runner has needed so
 42far, it needs subuid/subgid delegation for `ci-runner`, and it introduces image
 43storage that grows without pruning. The runner stops being a single Go binary
 44plus `git`.
 45
 46## Design
 47
 48### What runs where
 49
 50The split matters more than the flags:
 51
 52- **The runner clones**, outside any container, using its own key. The container
 53  never sees `GIT_SSH_COMMAND`, the key, or the runner's environment.
 54- **The container runs the steps**, with the already-cloned workspace bind
 55  mounted read-write at a fixed path.
 56
 57That alone closes the key-theft path, independently of how good the sandbox is.
 58
 59### Per-step or per-job
 60
 61Per job, one container for all of a job's steps. Steps in a job share state
 62today — a build step writes what a test step reads — and per-step containers
 63would break that or force a layer-caching scheme nobody asked for. `sh -c` per
 64step stays, inside the one container.
 65
 66### The environment
 67
 68Stop inheriting. Build the step environment explicitly: `PATH`, `HOME`, `CI`,
 69the build's own variables, and secrets when `b.Trusted`. `os.Environ()` must not
 70appear in the container's environment.
 71
 72Secrets stay env vars inside the container. They are visible to `podman inspect`
 73on the host, which is the runner's own user — the same user that already holds
 74them in memory, so this is not a new exposure. It is worth a comment saying so,
 75because it looks like one.
 76
 77### `image:` in ci.yml
 78
 79`Job` gains `Image string`. Empty means the instance default, which is
 80configuration on the runner (`-image`), not a value baked into the binary.
 81
 82Validation belongs in `ci.Parse` beside the existing job checks: a reference,
 83not a command line. Reject anything with a shell metacharacter or whitespace —
 84this string reaches `podman run`, and the whole point is that a repository's
 85config file cannot become an argument injection.
 86
 87### Network
 88
 89Default on. A build that cannot fetch dependencies is useless to most projects,
 90and this design's threat model is about what a build can *reach on the host*,
 91not about exfiltration. `network: none` per job is a plausible later addition
 92and explicitly out of scope here.
 93
 94### Failure modes
 95
 96**Podman missing or broken.** The runner must refuse to run builds rather than
 97falling back to running them unsandboxed. A fallback that silently drops
 98isolation is worse than a stopped runner, because nothing surfaces it. Check at
 99startup, fail loudly, and say what is wrong.
100
101**Image pull failure.** Fail the build with the pull error in the log. Do not
102retry indefinitely; do not fall back to another image.
103
104**Storage growth.** Images accumulate. Prune on a schedule, and document it —
105an unbounded cache on a 40GB host is a slow outage.
106
107### Host changes on bay1
108
109- `apt install podman` (Debian 13 has it; kernel 6.12, cgroup2, user namespaces
110  enabled with `max_user_namespaces` = 31556 — all already verified).
111- `/etc/subuid` and `/etc/subgid` entries for `ci-runner`.
112- The systemd override needs `Delegate=yes` for rootless cgroup management, and
113  `ProtectSystem=full` must be checked against podman's storage under
114  `~/.local/share/containers` — it may need an explicit `ReadWritePaths`.
115
116These are deployment changes, not code, and belong in `deploy/` with the rest.
117
118## Sequencing
119
120Three merge requests, because the middle one is the risky one and should be
121reviewable on its own:
122
1231. **Environment hygiene.** Stop inheriting `os.Environ()`; construct the step
124   environment explicitly. Independently valuable, no new dependency, and
125   testable today.
1262. **Podman execution.** `Image` in `ci.yml` with validation, the clone/run
127   split, container invocation, startup check, failure modes.
1283. **Deployment.** `deploy/` changes, subuid/subgid, the systemd override,
129   pruning, and the operational notes.
130
131## Tests
132
133- A step cannot read the runner's SSH key.
134- A step's environment contains what was constructed and nothing from the
135  runner's own environment.
136- Two concurrent builds cannot see each other's workspaces.
137- An `image:` containing a shell metacharacter is refused by `ci.Parse`.
138- A missing podman fails the runner at startup rather than running a build
139  unsandboxed — the fallback that must not exist.
140- A pull failure fails the build with the error in the log.
141
142The isolation tests need podman on the machine running them. Gate them so the
143suite still passes without it, and make the skip visible: a silently skipped
144isolation test is how this regresses.
145
146## What this does not do
147
148No `network: none`, no per-step images, no image caching strategy beyond
149pruning, no resource limits per build beyond what the service already applies.
150Each is a separate decision.