runner: run build steps in a rootless podman container !300

merged merged by cmc on 2026-09-06 22:25 UTC · krz/gitbay:runner-podman-exec-144 into main

Discussion

cmc

The execution half of #144, following docs/specs/2026-09-05-runner-isolation-design.md. bay1 is prepared, so the podman-gated tests actually run in CI there rather than skipping.

The split that does the work. The runner clones outside any container, with its own key, and bind mounts the finished workspace at /workspace. The container never sees GIT_SSH_COMMAND, the key, or the runner's environment. That closes the key-theft path independently of how good the sandbox is.

One container per job, podman exec per step: steps share state — a build step writes what a test step reads — and per-step containers would break that.

image: per job, defaulting to -image and then to docker.io/library/debian:stable-slim. Validated in ci.Parse as a reference, not a command line: the string becomes a podman run argument, so anything with whitespace or a shell metacharacter is refused. TestParseImageValidation covers alpine; rm -rf /, --privileged, -v /:/host and friends. It rides the build row (migration 0044) so it is the image the config named at that commit.

Secrets do not go in argv. --env-file with a 0600 file, written outside the bind-mounted workspace — outside because a file of secrets sitting in the checkout is one cat from the build's own log. /proc is world-readable and this codebase keeps secrets off argv everywhere else. A value containing a newline is refused rather than truncated, since the format cannot carry one.

No fallback. -isolation podman (the default) checks podman at start-up and log.Fatals if it is missing or broken, naming runner-podman-setup.sh. -isolation none is an explicit operator choice for an instance where every repository is trusted, and logs a warning at start-up. An unknown mode is refused. A pull failure fails the build with podman's own message; no retry, no second image.

Tests. TestRunnerRefusesToStartWithoutPodman is the one that matters — the fallback that must not exist — and it needs no podman, so it runs everywhere. TestRunnerRefusesUnknownIsolation likewise. The two that need podman (TestPodmanStepCannotReachTheRunnersKey, which has a step try to cat the runner's key, and TestPodmanPullFailureFailsTheBuild) skip with a loud message saying the container path was not covered — a silently skipped isolation test is how this regresses. The existing CI e2e tests pass -isolation none explicitly, since they exercise claiming, logs and cancellation rather than the sandbox.

Deploying this needs the host prepared first, which bay1 now is (podman 5.4.2, Linger=yes, cgroup manager systemd, rootless verified). A runner binary with this change deployed to an unprepared host stops every build on the instance, by design.

Not doing: network: none, per-step images, resource limits per build. Each is its own decision, as the design says.

Stacked on !298.

Closes #144

retargeted from runner-buildhome to main: !298 merged

2026-09-06 22:25 UTC