CICD eval #184

A write-up, not a feature. State what CI is good for, what it is not, and which gaps are worth closing versus documenting. Facts to start from:

• Every runner change so far cost an outage. The drain in #179 shipped broken (KillMode) and was caught by reading a journal by hand. No test exercises a real systemd stop.
• Dedupe (#177), path filters (#169, #171, #172) and the reaper are three mechanisms with their own edge cases; #176 was a case where they combined into a green build that never ran.
• Nothing measures it: no queue latency, no time-to-claim, no count of builds ended by the reaper rather than by report.
• One runner, one host, shared with the forge.

End with a decision: CICD is a feature of the forge. That sets the bar for everything else on the list.

Risks of widening the runner's scope, as of 2026-09-07

1. The build home is one directory shared by every build on the runner, mounted read-write into every container as HOME. A stranger's build can poison the module cache or plant a .gitconfig that the forge's own build honours. #144 mounted the same shared home. Fixed under A below.
2. No network flag: a build gets podman's default network, so bay1 is free compute with a clean IP, and 127.0.0.1:22 is reachable from inside.
3. No memory cap, on the forge's own host. Fixed under A below.
4. Queue starvation: one runner, oldest-first, 45 minutes per build, no per-account cap on builds queued, schedules down to a minute.
5. Rootless podman needed NoNewPrivileges, ProtectKernelTunables and RestrictSUIDSGID relaxed; an escape is on the forge's host.

Handled already: the runner's key and other workspaces are unreachable (the nightly canary proves it), fork heads build without secrets, images are provisioned never pulled, logs are capped at 2 MiB.

A, done regardless: CI stays scoped, and two bugs go

• A build home per repository, not one for the runner.
• A memory cap that leaves the forge alive.
• The Admin page and FAQ say that CI on gitbay.org builds only the repositories the operator names; self-hosters get the runner for their own.

B, if CI is a feature: a second machine and a tier per trust level

The runner already polls over SSH from anywhere, admin runners lists several, and -repos scopes each, which is most of a tiered design:

• The runner on bay1 keeps building krz/gitbay and the canary.
• A second small VPS runs a runner with no scope, --network none or an egress allowlist, a memory cap, a lower CPU share, and nothing else on it. Disposable: a compromise costs a reinstall, not the forge.
• Per-account limits server-side: builds queued at once, build minutes per day, a floor on schedule intervals. Same shape and place as admin user limits.
• Claim order stops being oldest-first across the instance: each runner takes the oldest build in its scope, and an unscoped runner rotates across accounts so one cannot hold the queue.

Under any answer

• One table on the wiki that says, for every push shape, which builds dedupe, path filters and the reaper produce and what status the commit gets; a property test over push shapes asserting it. #176 was found by accident.
• A real stop test: the runner under a throwaway systemd unit, claim, SIGTERM, assert the build reports. The drain shipped broken because only the process was tested, never the unit.
• Three numbers on admin runners or the dashboard: queue-to-claim time, builds ended by the reaper rather than by report, queue depth.

A landed in 0caaaed and runs on bay1 as of 19:32 UTC:

• build home per repository, asserted nightly by the canary
• MemoryMax=6G, CPUQuota=300%, OOMPolicy=continue on the unit; the e2e suite (build 1207) passed under the cap
• Admin page and FAQ state the scope

The caps are on the unit rather than the container because podman's -memory and -cpus have never applied here (#188). B stays open above.

"Three numbers on admin runners", from the list under any answer: !350 and !351, deployed.

admin runners now heads its output with the queue: builds pending now, and over the last 24 hours the builds claimed, the wait from creation to claim (average and worst), and the builds the reaper ended instead of a runner reporting them. Migration 0049 adds builds.reaped_at, set by ReapStaleBuilds, so the last one is countable. The JSON is {queue, runners} where it was a bare list.

First production read: the claim wait averages about 15 minutes with a worst case around 30, on a day of MR stacks. That is one runner working oldest-first through pairs of build and test jobs, so a test job waits for the build ahead of it plus whatever the previous branch queued. Not a fault, but it is the number a second runner or -jobs 2 would move.

Still open from the same list: the push-shape table with its property test, and the real systemd stop test.

Correction to the numbers above, read after the fix deployed: over the last 24 hours, 103 claimed, wait average 882 s, worst 4924 s (about 82 minutes, not 30), 0 reaped.

"A real stop test", from the list under any answer: !352, merged.

Three tests in e2e/runnerstop_test.go. TestRunnerDrainsOnSIGTERM signals a runner mid-step and asserts exit 0, the build a success, and the step run to its end. TestRunnerDropInLetsTheDrainHappen reads deploy/gitbay-runner.override.conf and fails unless KillMode=mixed and TimeoutStopSec outlasts ExecStart's -timeout plus the report retries. TestRunnerDrainUnderSystemd runs the runner as a transient user unit through systemd-run --user, stops it with systemctl --user stop, and asserts the build is a success under the drop-in's mode and not under control-group, which is the case that shipped broken.

The systemd test skips where there is no user manager, which includes the container CI runs in, so it has passed only its skip so far. It needs one run on a Linux host with a user manager, from a checkout:

go test ./e2e -run TestRunnerDrainUnderSystemd -count=1 -v

Left on the list: the push-shape table with its property test.

"One table on the wiki that says, for every push shape, which builds dedupe, path filters and the reaper produce": !353, merged. That closes the list under any answer.

The CI page carries fifteen push shapes against five jobs that each isolate one rule, with what the push produces per job and the commit status that follows. TestPushShapes in internal/hookd runs every row through the real entry points: branch push, merge request head, tag push, the scheduler's tick, the reaper. TestPushShapesTableOnWiki checks the page carries each row as the code has it, so a row cannot change without the test naming it.

Everything under "under any answer" is now in: the queue numbers on admin runners (!350, !351), the stop tests (!352, with the systemd variant still to be run once on a host with a user manager), and this table. What remains on this issue is B, the second machine and a tier per trust level, which is a decision.

B, decided and shipped as attachment rather than a second machine: v1.17.0, !354, deployed to bay1 2026-09-09.

• A runner key claims builds only for the repositories it is attached to. repo runner add <owner/name> < key.pub (a new key lands on the caller's account with scope runner), repo runner list, repo runner remove <fingerprint>, and a Runners section on the settings page. A runner-scoped key with nothing attached claims nothing, whoever owns it; a full-scope admin key still claims any repository. runner log and runner done refuse a build outside the key's attachments. Migration 0050 (runner_repos; runner_seen rekeyed by key, so two runners on one account are two rows and admin runners shows fingerprints).
• Closed on the way: any account could keys add --scope runner and runner next handed it the oldest pending build on the instance, secrets included. The wiki's "a runner account is admin by necessity" was never enforced.
• Fork merge request heads are claimed only with runner next --untrusted, which the runner sends when started with -untrusted. The bay1 unit sets it because it isolates in podman; a runner on someone's laptop builds their own commits by default.
• gitbay-runner init writes a key and config.toml under ~/.config/gitbay-runner/ and prints the attach command; the daemon reads that file, flags override it, -identity pins its own key. brew install krz/tap/gitbay-runner and brew services start from krz/homebrew-tap; releases now ship gitbay-runner binaries.
• bay1's runner (ci, non-admin) is attached to krz/gitbay and cmc/ci-smoke; the first canary build after the attach was claimed in seconds. The six builds pending from other repositories stay pending until their owners attach a runner, which is the point.

Spec: docs/specs/2026-09-08-user-runners-design.md. Plan: docs/plans/2026-09-08-user-runners.md.

Left on this issue from B: per-account limits (builds queued at once, minutes per day, a floor on schedule intervals) and claim order. Both matter less now that compute is the owner's; each is a decision on its own.

The two items left from B are their own issues now: #206 (per-account limits on queued builds and schedule intervals) and #207 (claim order when one runner serves many repositories). Nothing else remains here.

TestRunnerDrainUnderSystemd has now run for real, on bay1 as root under its user manager, from a checkout at 8e2dbc6 with a temporary Go 1.27.1: mixed passes (14.5 s, the build reports a success), control-group fails the build as the test expects (2.3 s). The toolchain and checkout were removed afterwards. Nothing on the "under any answer" list is untested now.