Commit `748122ef10`

748122ef1028a1ae48e703996414cd25ae938508

parent: 7acfcebdf5

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-29 03:46 UTC

runner host: flood test asserts the table's effect; exact rollback of the builds table

Ref #260

Layout: unified · split

.gitbay/wiki/Admin.org +30 −8

		@@ -976,7 +976,12 @@ private range, allow it in the file first or builds resolve nothing.
976	976	A restart of =nftables.service= flushes both tables;
977	977	=systemctl reload gitbay-runner-egress= restores both.
978	978
979		Checking the tables on a running host, during a build:
	979	The egress unit's stop and the rollback below use =nft destroy=,
	980	which needs nftables 1.0.8 or later; check =nft --version= on a new
	981	host.
	982
	983	Checking the tables on a running host, during a build (the flood test
	984	below waits a minute before its first probe, for this):
980	985
981	986	#+begin_src sh
982	987	nft list table inet gitbay_builds # counters on the reject rules
		@@ -986,15 +991,32 @@ for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
986	991
987	992	A pasta process anywhere else — the runner's own =runner= cgroup, a
988	993	user slice — means the builds table does not see that build's traffic
989		and only the first table applies. Then run
	994	and only the first table applies. Run
990	995	=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
991		once as a push and once with =--untrusted=; it prints what the build
992		reached and fails if the runner was locked out.
	996	once as a push and once with =--untrusted=. It fails if the runner was
	997	locked out or if the build log lacks the lines only a working table
	998	produces. The authoritative proof that pasta's sockets are in the
	999	build's cgroup is the untrusted run: =169.254.1.2:22= and
	1000	=github.com:22= refused, all twelve logins refused, and the counters on
	1001	the =untrusted= chain's rejects rising. The first table lets
	1002	=ci-runner= reach both of those addresses.
	1003
	1004	To take the builds table out, on the host:
	1005
	1006	#+begin_src sh
	1007	sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
	1008	rm /etc/gitbay-runner/builds.nft
	1009	systemctl daemon-reload
	1010	nft destroy table inet gitbay_builds
	1011	#+end_src
993	1012
994		To take the builds table out: delete the three =ExecStartPre= lines
995		that name =builds= from the drop-in, =systemctl daemon-reload=, and
996		=nft destroy table inet gitbay_builds=. The runner needs no restart,
997		and builds keep the first table.
	1013	The runner needs no restart. It still places builds under
	1014	=builds/trusted= and =builds/untrusted=, but with the file gone neither
	1015	a runner start nor =systemctl reload gitbay-runner-egress= loads the
	1016	table again, and builds keep the first table and =--no-map-gw=. A
	1017	runner that takes =-untrusted= or polls over loopback refuses to start
	1018	without build cgroups at all, since the table would match nothing. The
	1019	next =make deploy-runner= installs the file and the lines again.
998	1020
999	1021	The drop-in sets =NoNewPrivileges=no=, without which rootless podman
1000	1022	cannot call =newuidmap= and the runner refuses to start. That is a

deploy/gitbay-runner-egress.service +5 −4

		@@ -15,9 +15,10 @@
15	15	# flush ruleset removes it); delete would fail and leave the unit failed.
16	16	#
17	17	# The builds table (gitbay-runner-builds.nft) is loaded by the runner's
18		# own start, against its cgroups. Reload loads it again when those
19		# cgroups exist, so after a restart of nftables.service one reload puts
20		# both tables back.
	18	# own start, against its cgroups. Reload loads it again when the file is
	19	# installed and those cgroups exist, so after a restart of
	20	# nftables.service one reload puts both tables back; without the file
	21	# (taken out per the Admin page) the reload skips it and succeeds.
21	22	[Unit]
22	23	Description=Host egress rule for CI builds
23	24	After=nftables.service ufw.service
		@@ -28,7 +29,7 @@ Type=oneshot
28	29	RemainAfterExit=yes
29	30	ExecStart=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
30	31	ExecReload=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
31		ExecReload=/bin/sh -c 'if [ -d /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/untrusted ]; then exec /usr/sbin/nft -f /etc/gitbay-runner/builds.nft; fi'
	32	ExecReload=/bin/sh -c 'if [ -f /etc/gitbay-runner/builds.nft ] && [ -d /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/untrusted ]; then exec /usr/sbin/nft -f /etc/gitbay-runner/builds.nft; fi'
32	33	ExecStop=/usr/sbin/nft destroy table inet gitbay_runner
33	34	ExecStop=/usr/sbin/nft destroy table inet gitbay_builds
34	35

deploy/runner-auth-flood-test.sh +33 −9

		@@ -10,14 +10,15 @@
10	10	# deploy/runner-auth-flood-test.sh cmc/runner-scratch # trusted: a push to main
11	11	# deploy/runner-auth-flood-test.sh cmc/runner-scratch --untrusted # a merge request from a fork
12	12	#
13		# The build's logins use a key registered with --ttl 1s and expired by
14		# the time the build runs. With registration open an unknown key is
15		# admitted to run register and never counts against the SSH auth
16		# limiter; an expired key counts (internal/sshd/sshd.go, authenticate).
	13	# The build's logins use a git-scoped key registered with --ttl 1s and
	14	# expired by the time the build runs. With registration open an
	15	# unknown key is admitted to run register and never counts against the
	16	# SSH auth limiter; an expired key counts (internal/sshd/sshd.go, authenticate).
17	17	# The key is removed from the account when the script exits.
18	18	#
19		# The step probes what the build reaches, then makes 12 logins without
20		# pause, so the limiter (ssh_auth_rate, 10 a minute per address) locks
	19	# The step waits a minute, so the operator can find pasta's cgroup
	20	# (Admin page), probes what the build reaches, then makes 12 logins
	21	# without pause, so the limiter (ssh_auth_rate, 10 a minute per address) locks
21	22	# the address those logins come from for most of the next minute. The
22	23	# runner reports the result right after the step; its report retries
23	24	# for half a minute.
		@@ -34,6 +35,14 @@
34	35	# names that address, the build succeeds and the runner keeps polling.
35	36	# Untrusted: every login is refused by the builds table before it
36	37	# reaches sshd, and no auth.* entry comes from the build at all.
	38	#
	39	# The script also requires lines in the build log, so a missing ssh or
	40	# bash in the image, or a table that matches nothing, fails rather than
	41	# passes. Trusted: "logins 12 denied" (every login reached sshd) and
	42	# 10.0.0.1:80 refused, not timed out. Untrusted: 169.254.1.2:22 and
	43	# github.com:22 refused and "12 refused". The untrusted lines are the
	44	# proof that pasta's sockets are in the build's cgroup: the uid table
	45	# lets ci-runner reach both.
37	46	set -eu
38	47
39	48	repo=${1:-}
		@@ -48,13 +57,15 @@ tmp=$(mktemp -d)
48	57	fp=
49	58	cleanup() {
50	59	if [ -n "$fp" ]; then gitbay keys remove "$fp" >/dev/null \|\| echo "remove key $fp by hand" >&2; fi
	60	fp=
51	61	rm -rf "$tmp"
52	62	}
53	63	trap cleanup EXIT
	64	trap 'cleanup; exit 130' INT TERM
54	65
55	66	echo "==> an expired key"
56	67	ssh-keygen -q -t ed25519 -N '' -C auth-flood-260 -f "$tmp/key"
57		gitbay keys add --label auth-flood-260 --ttl 1s <"$tmp/key.pub" >/dev/null
	68	gitbay keys add --scope git --label auth-flood-260 --ttl 1s <"$tmp/key.pub" >/dev/null
58	69	fp=$(ssh-keygen -lf "$tmp/key.pub" \| awk '{print $2}')
59	70	sleep 2
60	71
		@@ -87,6 +98,7 @@ cat >.gitbay/flood.sh <<'EOF'
87	98	#!/bin/sh
88	99	# Written by deploy/runner-auth-flood-test.sh (#260).
89	100	set -u
	101	sleep 60
90	102	key=/tmp/flood.key
91	103	cp .gitbay/flood.key "$key"
92	104	chmod 600 "$key"
		@@ -104,7 +116,7 @@ probe() {
104	116	esac
105	117	}
106	118	getent hosts proxy.golang.org >/dev/null && echo "dns ok" \|\| echo "dns failed"
107		for t in 127.0.0.1:22 127.0.0.1:2222 "$host:22" "$host:80" "$host:443" "$host:2222" \
	119	for t in "$host:22" "$host:80" "$host:443" "$host:2222" \
108	120	10.0.0.1:80 192.168.0.1:80 proxy.golang.org:443 github.com:22; do
109	121	probe "${t%:}" "${t##:}"
110	122	done
		@@ -159,7 +171,8 @@ second=$(seen)
159	171	echo " $account last seen $first, then $second"
160	172
161	173	echo "==> build log"
162		gitbay build log "$repo" "$n" \| sed -n '/^dns /,$p'
	174	log=$(gitbay build log "$repo" "$n")
	175	printf '%s\n' "$log" \| sed -n '/dns /,$p'
163	176
164	177	echo "==> auth audit, last 15 minutes"
165	178	gitbay audit --action auth. --since 15m --json \|
		@@ -173,5 +186,16 @@ if gitbay audit --action auth.throttled --since 15m --json \| jq -e '.data[] \| se
173	186	echo "FAIL: 127.0.0.1, the runner's address, was throttled"
174	187	fail=1
175	188	fi
	189	need() {
	190	printf '%s\n' "$log" \| grep -Eq "$1" \|\| { echo "FAIL: the build log lacks \"$2\""; fail=1; }
	191	}
	192	if [ "$mode" = trusted ]; then
	193	need 'logins +12 denied' "logins 12 denied"
	194	need 'refused +10\.0\.0\.1:80( \|$)' "refused 10.0.0.1:80"
	195	else
	196	need 'refused +169\.254\.1\.2:22( \|$)' "refused 169.254.1.2:22"
	197	need 'refused +github\.com:22( \|$)' "refused github.com:22"
	198	need '12 refused' "12 refused"
	199	fi
176	200	[ $fail = 0 ] && echo "PASS ($mode): the build's failed logins did not lock the runner out"
177	201	exit $fail

Commit 748122ef10

Commit `748122ef10`