Commit `0caaaedf8f`

0caaaedf8fa3d930b6b4fb83e249b1e0e3265c0a

parent: 606c4ec959

Verified · cmc ci/build: success ci/test: success

cmc <hello@cleberg.net> · 2026-09-07 18:47 UTC

runner, deploy, wiki: a build home per repository, and caps on the runner unit

One build home was shared by every repository the runner built and
mounted read-write into each container, so a step could poison the
module cache or plant a .gitconfig that another repository's build
would honour. The home is now <workdir>/home/<owner>/<name>.

bay1's unit gains MemoryMax=6G, CPUQuota=300% and OOMPolicy=continue.
The host has 7.7GB and no swap, and an uncapped build that allocates
without bound takes the forge with it; the cap sits above the e2e
suite's 5GB peak. The caps are on the unit because podman's --memory
and --cpus never applied here: under rootless cgroupfs the container
runs inside the service's own cgroup (#188). -cpus 3 leaves ExecStart.

The Admin page and FAQ say that gitbay.org's runner builds only the
forge's own repositories and the canary, and why.

Ref #184, #188

.gitbay/wiki/Admin.org +19 −2

		@@ -391,6 +391,11 @@ gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
391	391	-workdir /var/lib/gitbay-runner/work
392	392	#+end_src
393	393
	394	gitbay.org's runner is scoped: it builds the forge's own repositories
	395	and the isolation canary, nothing else, because it shares the host with
	396	the forge. A =.gitbay/ci.yml= in another repository there queues builds
	397	no runner claims. Whether that changes is krz/gitbay#184.
	398
394	399	Naming no repositories is the old behaviour and stays the right choice for
395	400	the runner on the server itself. The scoping is what the runner asks for,
396	401	not an ACL the server holds over it: a runner account is admin by
		@@ -444,8 +449,20 @@ overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
444	449	so a config file cannot turn it into podman arguments.
445	450
446	451	=-cpus= and =-memory= cap one build's container (podman's own units,
447		e.g. =-cpus 2 -memory 4g=); unset means uncapped. bay1 runs =-cpus 3=
448		and no memory cap, since the e2e suite needs most of the host's 7GB.
	452	e.g. =-cpus 2 -memory 4g=); unset means uncapped. They only take effect
	453	where podman can create a cgroup for the container. Under rootless
	454	podman with the cgroupfs manager, which is what a system service gets,
	455	it cannot: the container runs inside the service's own cgroup and both
	456	flags are accepted and ignored (krz/gitbay#188). Cap the unit instead.
	457	bay1's drop-in sets =MemoryMax=6G= and =CPUQuota=300%= on a 7.7GB
	458	four-core host with no swap: the memory cap is what keeps the forge
	459	alive when a build allocates without bound, and it sits above the e2e
	460	suite's 5GB peak rather than at a fair share. =OOMPolicy=continue= keeps
	461	systemd from stopping the runner when a build is OOM-killed.
	462
	463	Each repository gets its own build home under the runner's workdir,
	464	mounted into its containers as =HOME=. Caches persist between builds of
	465	one repository and are never read by another's.
449	466
450	467	Images are provisioned, never pulled by a build. The runner passes
451	468	=--pull=never=. Two reasons, and the second is the better one: the

.gitbay/wiki/FAQ.org +6

		@@ -17,3 +17,9 @@
17	17	email patch flow (revisit only if sourcehut-style demand appears),
18	18	federation and Postgres (no need at this scale). Recorded so the
19	19	absence reads as a decision, not an oversight.
	20	- Does CI run for my repository on gitbay.org? :: Not yet. The runner
	21	there is scoped to the forge's own repositories and its isolation
	22	canary, since it shares the host with the forge. A =.gitbay/ci.yml=
	23	in your repository queues builds nothing claims. krz/gitbay#184 is
	24	where that gets decided. A self-hosted instance runs the same runner
	25	for whichever repositories its operator names.

.gitbay/wiki/Threat-Model.org +7 −5

		@@ -166,11 +166,13 @@ runner, polling over SSH, clones the commit and runs its steps.
166	166
167	167	Under =-isolation none=, anything a step can do as the runner's user a
168	168	pushed =ci.yml= can do. Under podman a step is confined to its
169		container and the bind-mounted workspace, but the build home's caches
170		are shared between builds, so one build can still leave something a
171		later build reads. Treat the runner host as executing untrusted code:
172		keep it off the daemon's host where the database lives, or scope it to
173		repositories whose writers you trust.
	169	container, the bind-mounted workspace and the repository's own build
	170	home, so what a build leaves in a cache is read only by later builds of
	171	the same repository. Treat the runner host as executing untrusted code
	172	all the same: keep it off the daemon's host where the database lives,
	173	or scope it to repositories whose writers you trust. gitbay.org does
	174	the latter — its runner builds only the repositories the operator
	175	names.
174	176
175	177	* What has not been audited
176	178

cmd/gitbay-runner/home_test.go added +53

		@@ -0,0 +1,53 @@
	1	package main
	2
	3	import (
	4	"os"
	5	"path/filepath"
	6	"testing"
	7	)
	8
	9	// The build home is per repository: one shared home let a step poison
	10	// the module cache or plant a .gitconfig that another repository's build
	11	// would honour (#184).
	12	func TestBuildHomeIsPerRepository(t *testing.T) {
	13	work := t.TempDir()
	14	a, err := buildHomeFor(work, "alice/app")
	15	if err != nil {
	16	t.Fatal(err)
	17	}
	18	b, err := buildHomeFor(work, "bob/app")
	19	if err != nil {
	20	t.Fatal(err)
	21	}
	22	if a == b {
	23	t.Fatalf("two repositories share a build home: %s", a)
	24	}
	25	for _, dir := range []string{a, b} {
	26	rel, err := filepath.Rel(filepath.Join(work, "home"), dir)
	27	if err != nil \|\| rel == "." \|\| filepath.IsAbs(rel) \|\| rel[0] == '.' {
	28	t.Fatalf("build home %s is not under %s/home", dir, work)
	29	}
	30	st, err := os.Stat(dir)
	31	if err != nil {
	32	t.Fatalf("build home not created: %v", err)
	33	}
	34	if st.Mode().Perm() != 0o700 {
	35	t.Fatalf("build home mode %o, want 0700", st.Mode().Perm())
	36	}
	37	}
	38	// The same repository gets the same home back: that is what makes it
	39	// a cache.
	40	again, _ := buildHomeFor(work, "alice/app")
	41	if again != a {
	42	t.Fatalf("build home moved between builds: %s then %s", a, again)
	43	}
	44	}
	45
	46	// A repository path is server-validated, but the home must still never
	47	// resolve outside the runner's home root.
	48	func TestBuildHomeRefusesTraversal(t *testing.T) {
	49	work := t.TempDir()
	50	if _, err := buildHomeFor(work, "../../etc"); err == nil {
	51	t.Fatal("a traversing repository path produced a build home")
	52	}
	53	}

cmd/gitbay-runner/main.go +26 −12

		@@ -267,13 +267,12 @@ func (r *runner) run(j job) bool {
267	267	// credential dotfiles are. A directory beside the workspaces is
268	268	// neither.
269	269	//
270		// It is shared by every build on this runner, so a step can poison a
271		// cache another repository's build will read. That is already true of
272		// anything a step can reach as this user — see the wiki's
273		// Threat-Model on the runner — and is what container isolation (#144)
274		// is for; -repos is the control until then.
275		buildHome := filepath.Join(r.workdir, "home")
276		if err := os.MkdirAll(buildHome, 0o700); err != nil {
	270	// One per repository: shared across repositories, a step could poison
	271	// the module cache or plant a .gitconfig that another repository's
	272	// build would honour, and the container mounts the home read-write
	273	// (#184).
	274	buildHome, err := buildHomeFor(r.workdir, j.Repo)
	275	if err != nil {
277	276	log.Printf("build %d: build home: %v", j.ID, err)
278	277	return false
279	278	}
		@@ -388,14 +387,29 @@ func (r *runner) run(j job) bool {
388	387	// the runner's entire environment, including anything an operator set on
389	388	// the service (#144).
390	389	//
391		// HOME is a build home shared by this runner's builds, not the runner's
392		// own: tools read credentials out of dotfiles — .netrc, .npmrc,
393		// .gitconfig — and a build has no business finding the runner's. It is
394		// not the workspace either, because the workspace is deleted after every
395		// build and every tool cache lives under HOME.
	390	// HOME is the repository's build home, not the runner's own: tools read
	391	// credentials out of dotfiles — .netrc, .npmrc, .gitconfig — and a build
	392	// has no business finding the runner's. It is not the workspace either,
	393	// because the workspace is deleted after every build and every tool
	394	// cache lives under HOME.
396	395	//
397	396	// PATH is the one thing carried over: without it a step cannot find the
398	397	// tools the host was provisioned with.
	398	// buildHomeFor is the build home for one repository: <workdir>/home/<owner>/<name>,
	399	// created on first use. The repository path comes from the server, but a
	400	// home must still never resolve outside the home root.
	401	func buildHomeFor(workdir, repo string) (string, error) {
	402	root := filepath.Join(workdir, "home")
	403	dir := filepath.Join(root, filepath.FromSlash(repo))
	404	if rel, err := filepath.Rel(root, dir); err != nil \|\| rel == "." \|\| strings.HasPrefix(rel, "..") {
	405	return "", fmt.Errorf("repository path %q escapes the build home root", repo)
	406	}
	407	if err := os.MkdirAll(dir, 0o700); err != nil {
	408	return "", err
	409	}
	410	return dir, nil
	411	}
	412
399	413	func stepEnv(j job, home string) []string {
400	414	path := os.Getenv("PATH")
401	415	if path == "" {

deploy/gitbay-runner.override.conf +17 −4

		@@ -27,9 +27,22 @@
27	27	# repositories never claims a build it is not scoped to, so the canary
28	28	# must be listed or its scheduled build waits forever.
29	29	#
30		# -cpus 3 of the host's 4 leaves a core for gitbayd and sshd while a build
31		# runs. No -memory: the e2e suite peaks past 5GB of the 7GB, and a cap
32		# that kills it is an outage rather than a limit (#144).
	30	# Resource caps are on the unit, not on the container. Under rootless
	31	# podman with the cgroupfs manager the container runs inside this
	32	# service's own cgroup: no child cgroup is created, so podman's --cpus
	33	# and --memory are accepted and never applied (#188). MemoryMax and
	34	# CPUQuota below bound the runner and every build together, which is
	35	# what keeps the forge alive when a build allocates without bound.
	36	#
	37	# 6G of the host's 7.7GB, no swap: the e2e suite peaks past 5GB, so the
	38	# cap sits above that rather than at a fair share. CPUQuota=300% is
	39	# three of the four cores, leaving one for gitbayd and sshd (#144, #184).
	40	# OOMPolicy=continue: systemd's default stops the whole service when any
	41	# process in it is OOM-killed, which would end the runner mid-build; the
	42	# build fails and the runner carries on.
	43	MemoryMax=6G
	44	CPUQuota=300%
	45	OOMPolicy=continue
33	46	#
34	47	# ExecStart is overridden here rather than left in the unit so the flags
35	48	# and the sandboxing that has to match them live in one file: -isolation
		@@ -57,7 +70,7 @@ TimeoutStopSec=50min
57	70	# when it exits is killed.
58	71	KillMode=mixed
59	72	ExecStart=
60		ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -repos krz/gitbay,cmc/ci-smoke -isolation podman -image localhost/gitbay-ci:1 -cpus 3
	73	ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -repos krz/gitbay,cmc/ci-smoke -isolation podman -image localhost/gitbay-ci:1
61	74	Nice=10
62	75	CPUWeight=30
63	76	IOWeight=30

Commit 0caaaedf8f

Commit `0caaaedf8f`