!519 · krz/gitbay

.gitbay/wiki/Admin.org +61 −2

		@@ -956,8 +956,67 @@ unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
956	956	restarting the runner: =127.0.0.1:22= and the public 22 must answer,
957	957	2222 must not. The table limits the runner's user to =127.0.0.1:22=,
958	958	DNS on loopback, and 22, 80 and 443 on the host's public address; the
959		Threat-Model page says why. A restart of =nftables.service= flushes it;
960		=systemctl reload gitbay-runner-egress= restores it.
	959	Threat-Model page says why.
	960
	961	That table cannot tell a build from the runner, since both run as
	962	=ci-runner=. =deploy/gitbay-runner-builds.nft=, shipped as
	963	=/etc/gitbay-runner/builds.nft=, matches by cgroup instead: the runner
	964	starts each build under =builds/trusted= or =builds/untrusted= in its
	965	service cgroup, and the table closes the host's loopback to every
	966	build, limits an untrusted build to TCP 80 and 443 and DNS, and closes
	967	private ranges to both (the CI page has the table). nftables resolves a
	968	cgroup path to its id when the table loads, and the service cgroup is
	969	new on every start, so the drop-in's =ExecStartPre= creates the two
	970	cgroups, hands them to =ci-runner= and loads the table before the
	971	runner starts; a table that does not load stops the start. =make
	972	deploy-runner= checks the file's syntax before the restart and that the
	973	table is loaded after it. If =/etc/resolv.conf= names a nameserver in a
	974	private range, allow it in the file first or builds resolve nothing.
	975
	976	A restart of =nftables.service= flushes both tables;
	977	=systemctl reload gitbay-runner-egress= restores both.
	978
	979	The egress unit's stop and the rollback below use =nft destroy=,
	980	which needs nftables 1.0.8 or later; check =nft --version= on a new
	981	host.
	982
	983	Checking the tables on a running host, during a build (the flood test
	984	below waits a minute before its first probe, for this):
	985
	986	#+begin_src sh
	987	nft list table inet gitbay_builds # counters on the reject rules
	988	for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
	989	# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>
	990	#+end_src
	991
	992	A pasta process anywhere else — the runner's own =runner= cgroup, a
	993	user slice — means the builds table does not see that build's traffic
	994	and only the first table applies. Run
	995	=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
	996	once as a push and once with =--untrusted=. It fails if the runner was
	997	locked out or if the build log lacks the lines only a working table
	998	produces. The authoritative proof that pasta's sockets are in the
	999	build's cgroup is the untrusted run: =169.254.1.2:22= and
	1000	=github.com:22= refused, all twelve logins refused, and the counters on
	1001	the =untrusted= chain's rejects rising. The first table lets
	1002	=ci-runner= reach both of those addresses.
	1003
	1004	To take the builds table out, on the host:
	1005
	1006	#+begin_src sh
	1007	sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
	1008	rm /etc/gitbay-runner/builds.nft
	1009	systemctl daemon-reload
	1010	nft destroy table inet gitbay_builds
	1011	#+end_src
	1012
	1013	The runner needs no restart. It still places builds under
	1014	=builds/trusted= and =builds/untrusted=, but with the file gone neither
	1015	a runner start nor =systemctl reload gitbay-runner-egress= loads the
	1016	table again, and builds keep the first table and =--no-map-gw=. A
	1017	runner that takes =-untrusted= or polls over loopback refuses to start
	1018	without build cgroups at all, since the table would match nothing. The
	1019	next =make deploy-runner= installs the file and the lines again.
961	1020
962	1021	The drop-in sets =NoNewPrivileges=no=, without which rootless podman
963	1022	cannot call =newuidmap= and the runner refuses to start. That is a

.gitbay/wiki/Architecture/03-Deployment.org +4 −2

		@@ -67,8 +67,10 @@ a database check.
67	67	* Host firewall
68	68
69	69	=deploy/cloud-init.yaml= opens 22, 80, 443 and 2222 inbound with ufw.
70		Outbound traffic is not restricted, including from CI containers. See
71		[[file:10-Known-Gaps.org][10. Known gaps]].
	70	Outbound traffic from the host is not restricted. CI builds are, by
	71	two nftables tables on the runner's host: trusted builds keep the
	72	internet, untrusted ones get TCP 80 and 443 and DNS, and neither
	73	reaches private ranges or the host's loopback beyond DNS (the CI wiki page, #260).
72	74
73	75	* Change path
74	76

.gitbay/wiki/Architecture/04-Trust-Boundaries.org +1 −1

		@@ -24,7 +24,7 @@
24	24	\| TB4 \| Z1 → Z3 git \| argv, repository path, stdin packs \| argv built by code, never a shell; repository path from the database, not the request (=internal/gitutil=) \|
25	25	\| TB5 \| Z3 → Z1 hook socket \| ref updates, repository id, user id, key scope, push token, commit objects \| the socket is mode 0600 and, on Linux, refuses a peer whose uid is not the daemon's; a request must carry the token sshd minted for its receive-pack (stored hashed in =push_tokens=) and name the same repository, account and scope. The daemon then decides with =policy.CheckPush= and =sig.VerifyCommit= (=internal/hookd/hookd.go=) \|
26	26	\| TB6 \| Z4 ↔ Z1 runner channel \| build claims (with secrets for trusted builds), logs, results \| runner-scoped SSH key; claims limited to attached repositories; secrets only when the build is trusted (=internal/control/build.go=) \|
27		\| TB7 \| Z5 → Z4 container \| build steps, workspace, build home \| rootless podman, operator-provisioned image, cgroup limits; a trusted build's home is its repository's, an untrusted build's is discarded with it; outbound is open; on the host only the forge's public ports (#260) \|
	27	\| TB7 \| Z5 → Z4 container \| build steps, workspace, build home \| rootless podman, operator-provisioned image, cgroup limits; a trusted build's home is its repository's, an untrusted build's is discarded with it; private ranges and the host's loopback (but DNS) closed; trusted: internet open, host public 22/80/443; untrusted: internet TCP 80/443 and DNS, no host (#260) \|
28	28	\| TB8 \| Z1 → Z0 outbound \| webhooks, mirrors, mail, push \| address checks on user-supplied URLs; HMAC on webhooks; no redirects ([[file:03-Deployment.org][3]]) \|
29	29	\| TB9 \| user content → browser \| Markdown and Org bodies, READMEs, filenames \| HTML sanitised (=ugcHTML=, =internal/httpd/web.go=, bluemonday); CSP =script-src 'none'= \|
30	30	\| TB10\| Z6 → everything \| host shell \| operator SSH on 2222, keys only, fail2ban; append-only offsite backup credentials \|

.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org +1 −1

		@@ -69,7 +69,7 @@ Who may do what:
69	69	\| Build home \| trusted: =<workdir>/trusted-home/<owner>/<name>=, one per repository, persistent; untrusted: =<workdir>/build-<id>-home=, removed with the build (=main.go=) \|
70	70	\| Secrets \| env file 0600 outside the workspace, or =--env NAME= for multi-line values \|
71	71	\| Resources \| per-build cgroup with =memory.max= and =cpu.max= written by the runner; unit-level =MemoryMax=6G=, =CPUQuota=300%= \|
72		\| Network \| pasta; outbound open; a loopback runner's builds run with =--no-map-gw= (=main.go=); on the host only public 22/80/443 (=gitbay-runner-egress.nft=, #260) \|
	72	\| Network \| pasta; a loopback runner's builds run with =--no-map-gw= (=main.go=); host limited by user (=gitbay-runner-egress.nft=) and by build cgroup, trusted or untrusted (=gitbay-runner-builds.nft=, #260) \|
73	73	\| Shutdown \| SIGTERM stops claiming and drains in-flight builds; the unit uses =KillMode=mixed= \|
74	74
75	75	* Integrations

.gitbay/wiki/Architecture/09-Controls.org +1 −1

		@@ -88,7 +88,7 @@ chapter names of OWASP ASVS 4.0 where one fits.
88	88	\| No secrets for untrusted builds \| in place \| =internal/control/build.go= \|
89	89	\| Runner limited to attached repositories \| in place \| =runnerMayBuild= (=build.go=) \|
90	90	\| Build images fixed by the operator \| in place \| =--pull=never= \|
91		\| Build network egress restricted \| partial \| host: loopback closed, public 22/80/443 only (=gitbay-runner-egress.nft=); internet outbound open by decision (#260) \|
	91	\| Build network egress restricted \| partial \| by build cgroup (=gitbay-runner-builds.nft=): host loopback (but DNS) and private ranges closed; trusted: host public 22/80/443, internet open by decision; untrusted: internet TCP 80/443 and DNS only. Not yet measured from a build (#260) \|
92	92	\| Build results reused only across equal trust \| in place \| =SuccessBuildForTree=, =SuccessBuildFor= (=internal/store/builds.go=) \|
93	93
94	94	** Availability and operations

.gitbay/wiki/Architecture/10-Known-Gaps.org +2 −2

		@@ -11,7 +11,7 @@ what the 2026-09-27 review found; remove a row when its issue closes.
11	11	\| Issue \| Area \| Gap \| Severity \|
12	12	\|-------+------------------+-----------------------------------------------------------------------+----------\|
13	13	\| #259 \| Recovery \| No restore has been exercised; the procedure and tooling (=admin restore-drill=, =backup --verify=) are in place, the clean-host drill is pending \| high \|
14		\| #260 \| CI network \| Builds share the runner's source address; no egress policy \| medium \|
	14	\| #260 \| CI network \| Separation and egress tables written; not yet measured from a build on bay1 (=deploy/runner-auth-flood-test.sh=) \| medium \|
15	15	\| #261 \| Various \| Migration foreign-key check after commit; three web writes bypass dispatch; documentation drift \| medium \|
16	16
17	17	* Not filed
		@@ -30,5 +30,5 @@ what the 2026-09-27 review found; remove a row when its issue closes.
30	30	\|-----------------------------------------------------------+------------------------------------------\|
31	31	\| What is the measured recovery time? \| unmeasured (#259) \|
32	32	\| How many concurrent clones does the host sustain? \| unmeasured (#262) \|
33		\| What can a build reach on the host's network? \| configuration inspected, reachability untested (#260) \|
	33	\| What can a build reach on the host's network? \| policy on the CI page; reachability untested until the flood test runs (#260) \|
34	34	\| Have the collaboration features been used by independent users? \| no; one human user, tests only \|

.gitbay/wiki/CI.org +23 −7

		@@ -77,13 +77,29 @@ failed one.
77	77
78	78	* What a build can reach
79	79
80		Builds have outbound internet access, trusted and untrusted alike. On
81		the runner's host an nftables table limits them to the forge's public
82		ports 22, 80 and 443, which closes the operator's sshd. The forge is
83		reached at the address in =GITBAY_SSH=: on a runner that polls the
84		daemon over loopback, =169.254.1.2=, which pasta translates to the
85		host's public address. See the Threat-Model page, "What a build can
86		reach", for how and why (krz/gitbay#260).
	80	\| Build \| Internet \| Runner's host \| Private ranges \|
	81	\|-----------+------------------+---------------------------------------------+----------------\|
	82	\| trusted \| open \| forge's public 22, 80, 443; DNS on loopback \| closed \|
	83	\| untrusted \| TCP 80, 443; DNS \| DNS on loopback \| closed \|
	84
	85	An untrusted build is a merge request head from a fork. It can fetch
	86	modules and packages over HTTPS but cannot reach the forge, send mail,
	87	or open ssh elsewhere. The forge is reached at the address in
	88	=GITBAY_SSH=: on a runner that polls the daemon over loopback,
	89	=169.254.1.2=, which pasta translates to the host's public address, so
	90	a build's logins never arrive from =127.0.0.1=, where the runner polls.
	91	Two nftables tables on the runner's host enforce this, one by the
	92	runner's user and one by the cgroup each build runs in; the Threat-Model
	93	page, "What a build can reach", says how and why, and the Admin page
	94	how to install them (krz/gitbay#260). A runner off the daemon's host
	95	gets none of this unless its operator adds it.
	96
	97	To check a runner, run =deploy/runner-auth-flood-test.sh= against a
	98	scratch repository, once as a push and once with =--untrusted=: the
	99	build probes what it reaches, then fails SSH logins with an expired key
	100	until the limiter locks its address, and the script checks that the
	101	runner still reported the build and kept polling, and that the build's
	102	log shows what the table allowed and refused.
87	103
88	104	* The table
89	105

.gitbay/wiki/Threat-Model.org +33 −21

		@@ -203,31 +203,43 @@ runner, polling over SSH, clones the commit and runs its steps.
203	203	already has rather than naming anything on the internet. On an
204	204	instance with open registration that is the difference between a
205	205	curated set and arbitrary code from a registry nobody vetted.
206		- What a build can reach. Outbound internet, trusted or not: a fork's
207		merge request to a Go repository has to fetch its modules. On the
208		runner's host, the rules below limit it to the forge's public ports
209		22, 80 and 443. Under pasta a build's container holds the host's own
210		public address, and pasta translates =169.254.1.2= (its
211		=--map-guest-addr=) to that address. A runner that polls the daemon
212		over loopback starts its containers with =--network
213		pasta:--no-map-gw=, which is podman's default stated explicitly, and
214		gives them =GITBAY_SSH= at =169.254.1.2=, with the forge's port when
215		it is not 22. The runner's source address is =127.0.0.1=. An nftables
216		table (=deploy/gitbay-runner-egress.nft=) rejects every connection
217		the runner's user makes to the host's own addresses except
	206	- What a build can reach. A trusted build has the internet; an
	207	untrusted one — a fork's merge request head — has TCP 80 and 443 and
	208	DNS, enough to fetch modules and packages. Neither reaches private
	209	address ranges (RFC 1918, CGNAT, link-local, ULA). On the runner's
	210	host a trusted build reaches the forge's public ports 22, 80 and 443
	211	and the resolver on loopback; an untrusted build reaches only the
	212	resolver. Under pasta a build's container holds the host's own public
	213	address, and pasta translates =169.254.1.2= (its =--map-guest-addr=)
	214	to that address. A runner that polls the daemon over loopback starts
	215	its containers with =--network pasta:--no-map-gw=, which is podman's
	216	default stated explicitly, and gives them =GITBAY_SSH= at
	217	=169.254.1.2=, with the forge's port when it is not 22. The runner's
	218	source address is =127.0.0.1=; a trusted build's is the host's public
	219	address. Two nftables tables enforce this. The first
	220	(=deploy/gitbay-runner-egress.nft=) matches the runner's user and
	221	rejects every connection it makes to the host's own addresses except
218	222	=127.0.0.1:22=, DNS on loopback, and 22, 80 and 443 on the public
219	223	address: the operator's sshd on 2222 and anything bound to loopback
220		are closed to it. Under rootless podman a build's connections are
221		made by pasta as the runner's user, so the table cannot tell a build
222		from its runner and leaves =127.0.0.1:22= open; that no build reaches
223		the host's loopback is measured from inside a build by runbook R3.
224		The runner does not start without the table. The SSH auth limiter
	224	are closed. Under rootless podman a build's connections are made by
	225	pasta as that same user, so this table cannot tell a build from its
	226	runner. The second (=deploy/gitbay-runner-builds.nft=) can: the runner
	227	starts every podman process for a build, pasta included, inside
	228	=builds/trusted/build-<id>= or =builds/untrusted/build-<id>= under its
	229	service cgroup, and the table matches sockets by those cgroups
	230	(=socket cgroupv2=). It closes the host's loopback, =127.0.0.1:22=
	231	included, to every build except for DNS, and applies the per-trust
	232	rules above. The runner does not start without either table. The SSH auth limiter
225	233	counts failures per source address and, once an address is over the
226	234	limit, refuses every key from it until the window passes, the
227		runner's included; with registration open or by invite an unknown
228		key never counts (krz/gitbay#260). Under =-isolation none= a build
229		runs on the host and shares its loopback; the table still applies,
230		since it runs as the same user.
	235	runner's included; an unknown key counts only with registration
	236	closed, an expired key always (krz/gitbay#260). No build shares
	237	=127.0.0.1= with the runner, and an untrusted build cannot reach sshd
	238	at all. Trusted builds share the public address with one another, so
	239	one that fails logins can throttle another's push for a minute. Under
	240	=-isolation none= a build runs on the host in the runner's cgroup and
	241	shares its loopback; only the first table applies, and such a runner
	242	must not take =-untrusted=.
231	243
232	244	Under =-isolation none=, anything a step can do as the runner's user a
233	245	pushed =ci.yml= can do. Under podman a step is confined to its

.gitbay/wiki/Users.org +3 −1

		@@ -577,7 +577,9 @@ the destination that runner polls (its =-remote=, such as
577	577	=ssh ssh://$GITBAY_SSH …=, which work in either form. A job that
578	578	talks back to the instance — a release asset, a comment, a push to a
579	579	pages branch — uses =$GITBAY_SSH= with a key it holds as a secret;
580		the build's container has no key of its own. Two things about that
	580	the build's container has no key of its own. On the forge's own runner a
	581	build from a fork's merge request cannot reach the instance at all, and
	582	reaches the internet only over HTTPS, HTTP and DNS (CI page). Two things about that
581	583	container: a secret with newlines (a private key) arrives intact, and
582	584	=ssh= expands =~= from the passwd entry, =/root=, not from =$HOME=,
583	585	which is the build home — so keep an ssh config in the workspace and

CHANGELOG.org +14

		@@ -15,6 +15,20 @@ anything beyond "replace the binary and restart" is needed.
15	15	recovered and the elapsed time. The Admin wiki's Restore drill
16	16	section lists what to restore, what to check and where to record it
17	17	(#259).
	18	- The runner starts each podman build under =builds/trusted= or
	19	=builds/untrusted= in its service cgroup, and a second nftables
	20	table (=deploy/gitbay-runner-builds.nft=) matches build traffic by
	21	that cgroup: no build reaches the host's loopback beyond DNS, or a
	22	private range; a trusted build keeps the internet and the forge's public 22,
	23	80 and 443; an untrusted build gets TCP 80 and 443 and DNS, and not
	24	the forge. The runner's drop-in creates the cgroups and loads the
	25	table on every start, and the start fails without it. A runner
	26	with =-untrusted= or a loopback =-remote= refuses to start without
	27	build cgroups, which the table needs to match anything. =make
	28	deploy-runner= installs it. =deploy/runner-auth-flood-test.sh= runs
	29	the scratch-repository test: a build failing SSH logins must not
	30	lock the runner out. Run it before pointing the runner back at real
	31	repositories; the Admin page has the steps and the rollback. (#260)
18	32
19	33	* v1.37.0 — 2026-09-29
20	34

Makefile +5

		@@ -77,9 +77,13 @@ deploy-runner: preflight
77	77	./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-prune.timer /etc/systemd/system/gitbay-runner-prune.timer
78	78	./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-egress.nft /etc/gitbay-runner/egress.nft
79	79	./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-egress.service /etc/systemd/system/gitbay-runner-egress.service
	80	./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-builds.nft /etc/gitbay-runner/builds.nft
80	81	@echo "==> loading the egress rule"
	82	@# builds.nft names the runner's class cgroups, which may not exist yet;
	83	@# its syntax is checked against system.slice, which always does.
81	84	ssh -p $(PORT) root@$(HOST) 'set -eu; \
82	85	nft -c -f /etc/gitbay-runner/egress.nft; \
	86	sed "s\|level 4 \"system.slice/gitbay-runner.service/builds/[a-z]*\"\|level 1 \"system.slice\"\|" /etc/gitbay-runner/builds.nft \| nft -c -f /dev/stdin; \
83	87	systemctl daemon-reload; \
84	88	systemctl enable gitbay-runner-egress.service; \
85	89	systemctl reload-or-restart gitbay-runner-egress.service'
		@@ -89,5 +93,6 @@ deploy-runner: preflight
89	93	mv /usr/local/bin/gitbay-runner.new /usr/local/bin/gitbay-runner; \
90	94	systemctl enable --now gitbay-runner-prune.timer; \
91	95	systemctl restart gitbay-runner; \
	96	nft list table inet gitbay_builds >/dev/null; \
92	97	systemctl --no-pager --lines=3 status gitbay-runner; \
93	98	systemctl --no-pager list-timers gitbay-runner-prune.timer'

cmd/gitbay-runner/cgroup.go +33

		@@ -17,6 +17,39 @@ import (
17	17	// process for that build from inside it with podman's own cgroup handling
18	18	// off. What follows is the portable half: parsing and the file writes.
19	19
	20	// buildClasses are the cgroups a build is placed under, by the claim's
	21	// trust flag. deploy/gitbay-runner-builds.nft matches a build's sockets,
	22	// pasta's included, by these two cgroups (#260), so the names are fixed.
	23	var buildClasses = []string{"trusted", "untrusted"}
	24
	25	// buildCgroupDir is build id's cgroup under the runner's builds cgroup.
	26	func buildCgroupDir(builds string, id int64, trusted bool) string {
	27	class := buildClasses[1]
	28	if trusted {
	29	class = buildClasses[0]
	30	}
	31	return filepath.Join(builds, class, fmt.Sprintf("build-%d", id))
	32	}
	33
	34	// buildCgroupsRequired names what makes build cgroups mandatory, or ""
	35	// when a podman runner may run its builds in its own service cgroup.
	36	// Limits that cannot be applied are refused, not dropped: a runner that
	37	// accepted -memory and ran uncapped is what #188 was. A runner taking
	38	// untrusted builds, or polling over loopback on the daemon's host, is
	39	// the shape the builds nftables table guards, and that table matches
	40	// builds by these cgroups; without them it matches nothing (#260).
	41	func buildCgroupsRequired(memory, cpus string, untrusted, loopback bool) string {
	42	switch {
	43	case memory != "" \|\| cpus != "":
	44	return "-memory/-cpus"
	45	case untrusted:
	46	return "-untrusted"
	47	case loopback:
	48	return "a loopback -remote"
	49	}
	50	return ""
	51	}
	52
20	53	// memoryBytes parses podman's memory units — a whole number with an
21	54	// optional b, k, m or g suffix — into bytes.
22	55	func memoryBytes(s string) (int64, error) {

cmd/gitbay-runner/cgroup_linux.go +21 −6

		@@ -17,7 +17,8 @@ import (
17	17	// Delegate=yes hands the runner its service cgroup; the runner parks
18	18	// itself in a leaf so the service cgroup can enable controllers for
19	19	// children (a cgroup may hold processes or controller-enabled children,
20		// not both), and creates one child per build under builds/.
	20	// not both), and creates one child per build under builds/trusted or
	21	// builds/untrusted.
21	22	type buildCgroups struct {
22	23	builds string // <service cgroup>/builds
23	24	}
		@@ -25,7 +26,11 @@ type buildCgroups struct {
25	26	const cgroupControllers = "+cpu +memory +pids"
26	27
27	28	// prepareBuildCgroups moves the runner into <own>/runner, enables the
28		// controllers on its original cgroup, and creates builds/. It fails
	29	// controllers on its original cgroup, and creates builds/ with a child
	30	// per trust class. The unit's drop-in may have created those before the
	31	// runner started, to load the builds nftables table against them; they
	32	// are used as found, never recreated, since the table holds their ids.
	33	// It fails
29	34	// where the cgroup is not writable, which is a unit without
30	35	// Delegate=yes; the caller decides whether that is fatal.
31	36	func prepareBuildCgroups() (*buildCgroups, error) {
		@@ -55,13 +60,23 @@ func prepareBuildCgroups() (*buildCgroups, error) {
55	60	if err := os.WriteFile(filepath.Join(builds, "cgroup.subtree_control"), []byte(cgroupControllers), 0o644); err != nil {
56	61	return nil, fmt.Errorf("enabling controllers on %s: %w", builds, err)
57	62	}
	63	for _, class := range buildClasses {
	64	dir := filepath.Join(builds, class)
	65	if err := os.MkdirAll(dir, 0o755); err != nil {
	66	return nil, err
	67	}
	68	if err := os.WriteFile(filepath.Join(dir, "cgroup.subtree_control"), []byte(cgroupControllers), 0o644); err != nil {
	69	return nil, fmt.Errorf("enabling controllers on %s: %w", dir, err)
	70	}
	71	}
58	72	return &buildCgroups{builds: builds}, nil
59	73	}
60	74
61		// create makes the cgroup for one build with its limits written, and
62		// returns its path and an open directory fd for placing processes.
63		func (c buildCgroups) create(id int64, memory, cpus string) (string, os.File, error) {
64		dir := filepath.Join(c.builds, fmt.Sprintf("build-%d", id))
	75	// create makes the cgroup for one build under its trust class with its
	76	// limits written, and returns its path and an open directory fd for
	77	// placing processes.
	78	func (c buildCgroups) create(id int64, trusted bool, memory, cpus string) (string, os.File, error) {
	79	dir := buildCgroupDir(c.builds, id, trusted)
65	80	if err := os.Mkdir(dir, 0o755); err != nil {
66	81	return "", nil, err
67	82	}

cmd/gitbay-runner/cgroup_other.go +1 −1

		@@ -14,7 +14,7 @@ func prepareBuildCgroups() (*buildCgroups, error) {
14	14	return nil, errors.New("build cgroups need Linux")
15	15	}
16	16
17		func (c buildCgroups) create(id int64, memory, cpus string) (string, os.File, error) {
	17	func (c buildCgroups) create(id int64, trusted bool, memory, cpus string) (string, os.File, error) {
18	18	return "", nil, errors.New("build cgroups need Linux")
19	19	}
20	20

cmd/gitbay-runner/cgroup_test.go +67

		@@ -1,8 +1,10 @@
1	1	package main
2	2
3	3	import (
	4	"fmt"
4	5	"os"
5	6	"path/filepath"
	7	"strings"
6	8	"testing"
7	9	)
8	10
		@@ -90,3 +92,68 @@ func TestWriteLimits(t *testing.T) {
90	92	t.Error("a bad memory limit was accepted")
91	93	}
92	94	}
	95
	96	// A build's cgroup sits under its trust class, which is what the builds
	97	// nftables table matches on (#260).
	98	func TestBuildCgroupDir(t *testing.T) {
	99	builds := "/sys/fs/cgroup/system.slice/gitbay-runner.service/builds"
	100	if got := buildCgroupDir(builds, 7, true); got != builds+"/trusted/build-7" {
	101	t.Errorf("trusted: %s", got)
	102	}
	103	if got := buildCgroupDir(builds, 8, false); got != builds+"/untrusted/build-8" {
	104	t.Errorf("untrusted: %s", got)
	105	}
	106	}
	107
	108	// The builds table and the drop-in that creates its cgroups and loads it
	109	// name the same class cgroups the runner places builds in. A rename on
	110	// one side alone would leave builds unmatched, with only the uid table
	111	// between them and the host.
	112	func TestBuildsTableNamesTheClassCgroups(t *testing.T) {
	113	read := func(name string) string {
	114	t.Helper()
	115	b, err := os.ReadFile(filepath.Join("..", "..", "deploy", name))
	116	if err != nil {
	117	t.Fatal(err)
	118	}
	119	return string(b)
	120	}
	121	const unit = "system.slice/gitbay-runner.service"
	122	table, dropin := read("gitbay-runner-builds.nft"), read("gitbay-runner.override.conf")
	123	for _, class := range buildClasses {
	124	match := fmt.Sprintf(`socket cgroupv2 level 4 "%s/builds/%s" jump %s`, unit, class, class)
	125	if !strings.Contains(table, match) {
	126	t.Errorf("gitbay-runner-builds.nft lacks %q", match)
	127	}
	128	dir := "/sys/fs/cgroup/" + unit + "/builds/" + class
	129	if !strings.Contains(dropin, dir) {
	130	t.Errorf("the drop-in does not create %s", dir)
	131	}
	132	}
	133	if !strings.Contains(dropin, "ExecStartPre=+/usr/sbin/nft -f /etc/gitbay-runner/builds.nft") {
	134	t.Error("the drop-in does not load the builds table")
	135	}
	136	}
	137
	138	// Without build cgroups a podman runner may carry on only where nothing
	139	// depends on them: no limits, no untrusted builds, and not on the
	140	// daemon's host, where the builds table matches by cgroup (#260).
	141	func TestBuildCgroupsRequired(t *testing.T) {
	142	cases := []struct {
	143	memory, cpus string
	144	untrusted, loopback bool
	145	want string
	146	}{
	147	{"", "", false, false, ""},
	148	{"6g", "", false, false, "-memory/-cpus"},
	149	{"", "3", false, false, "-memory/-cpus"},
	150	{"", "", true, false, "-untrusted"},
	151	{"", "", false, true, "a loopback -remote"},
	152	{"", "", true, true, "-untrusted"},
	153	}
	154	for _, c := range cases {
	155	if got := buildCgroupsRequired(c.memory, c.cpus, c.untrusted, c.loopback); got != c.want {
	156	t.Errorf("buildCgroupsRequired(%q, %q, %v, %v) = %q, want %q", c.memory, c.cpus, c.untrusted, c.loopback, got, c.want)
	157	}
	158	}
	159	}

cmd/gitbay-runner/isolate.go +1 −1

		@@ -134,7 +134,7 @@ func (r *runner) runStepsPodman(j job, dir string, env []string, sink io.Writer,
134	134	// from the runner's cgroup would run the step outside the limit.
135	135	var cgroupFD *os.File
136	136	if r.cgroups != nil {
137		dir, f, err := r.cgroups.create(j.ID, r.memory, r.cpus)
	137	dir, f, err := r.cgroups.create(j.ID, j.Trusted, r.memory, r.cpus)
138	138	if err != nil {
139	139	fmt.Fprintf(sink, "preparing the build cgroup: %v\n", err)
140	140	return &failure{Reason: "preparing the build cgroup failed"}

cmd/gitbay-runner/main.go +13 −14

		@@ -127,22 +127,12 @@ func main() {
127	127	memory: *memory,
128	128	cpus: *cpus,
129	129	}
	130	var cgErr error
130	131	if r.isolation == isolationPodman {
131	132	// Before podman runs anything: its pause process lands in the
132	133	// cgroup of the first invocation, and that must be the runner's
133	134	// leaf, not a build's.
134		cg, err := prepareBuildCgroups()
135		switch {
136		case err == nil:
137		r.cgroups = cg
138		case r.memory != "" \|\| r.cpus != "":
139		// Limits that cannot be applied are refused, not dropped:
140		// a runner that accepted -memory and ran uncapped is what
141		// #188 was.
142		log.Fatalf("-memory/-cpus: build cgroups unavailable: %v", err)
143		default:
144		log.Printf("build cgroups unavailable (%v); builds run unconfined in the service cgroup", err)
145		}
	135	r.cgroups, cgErr = prepareBuildCgroups()
146	136	}
147	137	if err := r.checkIsolation(); err != nil {
148	138	// Refusing to start is the point. A runner that quietly fell back
		@@ -152,6 +142,14 @@ func main() {
152	142	// one of them is honest about why (#144).
153	143	log.Fatalf("isolation: %v", err)
154	144	}
	145	if cgErr != nil {
	146	// After checkIsolation, so a missing image or podman is reported
	147	// as that rather than as the cgroups it would also lack.
	148	if why := buildCgroupsRequired(r.memory, r.cpus, *untrusted, r.loopbackRemote()); why != "" {
	149	log.Fatalf("%s: build cgroups unavailable: %v", why, cgErr)
	150	}
	151	log.Printf("build cgroups unavailable (%v); builds run unconfined in the service cgroup", cgErr)
	152	}
155	153	if *sshOpts != "" {
156	154	r.sshOpts = strings.Fields(*sshOpts)
157	155	}
		@@ -557,8 +555,9 @@ func (r *runner) buildSSH(public string) string {
557	555	// polls from; the SSH auth limiter counts failures per source address
558	556	// (#260). podman passes --no-map-gw to pasta by default; it is stated
559	557	// here so the build's view of the host does not depend on that default.
560		// The host's nftables table (deploy/gitbay-runner-egress.nft) limits
561		// what a build reaches on the host to 22, 80 and 443.
	558	// The host's nftables tables (deploy/gitbay-runner-egress.nft and
	559	// gitbay-runner-builds.nft) limit what a build reaches on the host to
	560	// 22, 80 and 443, and an untrusted build to nothing but DNS.
562	561	func (r *runner) buildNetwork() []string {
563	562	if !r.loopbackRemote() {
564	563	return nil

deploy/gitbay-runner-builds.nft added +94

		@@ -0,0 +1,94 @@
	1	#!/usr/sbin/nft -f
	2	# Egress for CI builds by trust (#260). Installed as
	3	# /etc/gitbay-runner/builds.nft by `make deploy-runner` and loaded by the
	4	# runner unit's ExecStartPre (gitbay-runner.override.conf), after the
	5	# cgroups it names exist.
	6	#
	7	# gitbay-runner-egress.nft cannot tell a build from its runner: both run
	8	# as ci-runner. This table can. The runner starts every podman process
	9	# for a build inside builds/trusted/build-<id> or
	10	# builds/untrusted/build-<id> under its service cgroup, and pasta,
	11	# which podman starts, inherits that cgroup; so every socket pasta opens
	12	# for a build carries it. The runner itself, its clone and its log
	13	# stream run in <service>/runner and never match here.
	14	#
	15	# nftables resolves a cgroup path to the cgroup's id when the table
	16	# loads. The runner's service cgroup is new on every start, so the drop-in
	17	# creates builds/trusted and builds/untrusted and loads this file on
	18	# every start, and the runner never recreates them. A table loaded
	19	# against an earlier start's cgroups matches nothing, and builds then
	20	# get only the uid table.
	21	#
	22	# The uid table still applies to builds; a packet must pass both. This
	23	# table only takes away. Both hook output with policy accept, so their
	24	# relative order does not matter.
	25	#
	26	# Trusted builds (a branch of the repository, with its secrets):
	27	# internet open, any port.
	28	# host, public 22, 80 and 443: the forge at GITBAY_SSH
	29	# (169.254.1.2, which pasta translates to the public
	30	# address). hutch and orgo publish over 22.
	31	# host, loopback 53 only: the host's resolver, where pasta forwards
	32	# a build's DNS when the host's nameserver is a
	33	# loopback address.
	34	# private ranges closed (RFC 1918, CGNAT, link-local, ULA).
	35	# Untrusted builds (a fork's merge request head, no secrets):
	36	# internet 80 and 443 over TCP, and 53: enough to fetch
	37	# modules and packages, not to send mail or reach
	38	# ssh elsewhere.
	39	# host loopback 53 only. No forge: an untrusted build has
	40	# no key to use there, and a failing login from it
	41	# would count against the host's public address.
	42	# private ranges closed.
	43	# -isolation none builds run in the runner's own cgroup and get only the
	44	# uid table; such a runner must not take -untrusted.
	45	#
	46	# A host whose /etc/resolv.conf names a nameserver in a private range
	47	# needs that address let through here, or builds resolve nothing.
	48	#
	49	# The first line creates the table if it is missing, so the delete never
	50	# fails; the file then replaces it in one transaction.
	51
	52	table inet gitbay_builds
	53	delete table inet gitbay_builds
	54
	55	table inet gitbay_builds {
	56	set private4 {
	57	type ipv4_addr
	58	flags interval
	59	elements = { 0.0.0.0/8, 10.0.0.0/8, 100.64.0.0/10, 169.254.0.0/16, 172.16.0.0/12, 192.168.0.0/16 }
	60	}
	61
	62	set private6 {
	63	type ipv6_addr
	64	flags interval
	65	elements = { fc00::/7, fe80::/10 }
	66	}
	67
	68	chain output {
	69	type filter hook output priority filter; policy accept;
	70	socket cgroupv2 level 4 "system.slice/gitbay-runner.service/builds/trusted" jump trusted
	71	socket cgroupv2 level 4 "system.slice/gitbay-runner.service/builds/untrusted" jump untrusted
	72	}
	73
	74	chain trusted {
	75	oifname "lo" ip daddr 127.0.0.0/8 meta l4proto { tcp, udp } th dport 53 return
	76	oifname "lo" ip6 daddr ::1 meta l4proto { tcp, udp } th dport 53 return
	77	oifname "lo" ip daddr != 127.0.0.0/8 tcp dport { 22, 80, 443 } return
	78	oifname "lo" ip6 daddr != ::1 tcp dport { 22, 80, 443 } return
	79	oifname "lo" counter reject
	80	ip daddr @private4 counter reject
	81	ip6 daddr @private6 counter reject
	82	}
	83
	84	chain untrusted {
	85	oifname "lo" ip daddr 127.0.0.0/8 meta l4proto { tcp, udp } th dport 53 return
	86	oifname "lo" ip6 daddr ::1 meta l4proto { tcp, udp } th dport 53 return
	87	oifname "lo" counter reject
	88	ip daddr @private4 counter reject
	89	ip6 daddr @private6 counter reject
	90	meta l4proto { tcp, udp } th dport 53 return
	91	tcp dport { 80, 443 } return
	92	counter reject
	93	}
	94	}

deploy/gitbay-runner-egress.nft +2 −1

		@@ -9,7 +9,8 @@
9	9	# nftables sees them exactly as it sees the runner's own ssh, so this
10	10	# table cannot tell a build from its runner. It limits what that user
11	11	# reaches on this host, and the runner needs little: 127.0.0.1:22, to
12		# poll, clone and stream logs.
	12	# poll, clone and stream logs. gitbay-runner-builds.nft tells them apart
	13	# by cgroup and narrows this per build, trusted or not.
13	14	#
14	15	# Every packet to one of the host's own addresses, loopback or public,
15	16	# leaves through lo, so the output hook sees host-bound traffic as

deploy/gitbay-runner-egress.service +8

		@@ -13,6 +13,12 @@
13	13	#
14	14	# Stop uses destroy, which succeeds when the table is already gone (a
15	15	# flush ruleset removes it); delete would fail and leave the unit failed.
	16	#
	17	# The builds table (gitbay-runner-builds.nft) is loaded by the runner's
	18	# own start, against its cgroups. Reload loads it again when the file is
	19	# installed and those cgroups exist, so after a restart of
	20	# nftables.service one reload puts both tables back; without the file
	21	# (taken out per the Admin page) the reload skips it and succeeds.
16	22	[Unit]
17	23	Description=Host egress rule for CI builds
18	24	After=nftables.service ufw.service
		@@ -23,7 +29,9 @@ Type=oneshot
23	29	RemainAfterExit=yes
24	30	ExecStart=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
25	31	ExecReload=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
	32	ExecReload=/bin/sh -c 'if [ -f /etc/gitbay-runner/builds.nft ] && [ -d /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/untrusted ]; then exec /usr/sbin/nft -f /etc/gitbay-runner/builds.nft; fi'
26	33	ExecStop=/usr/sbin/nft destroy table inet gitbay_runner
	34	ExecStop=/usr/sbin/nft destroy table inet gitbay_builds
27	35
28	36	[Install]
29	37	WantedBy=multi-user.target

deploy/gitbay-runner.override.conf +13 −1

		@@ -25,7 +25,8 @@
25	25	[Unit]
26	26	# The host egress rule (#260, gitbay-runner-egress.nft) limits what this
27	27	# unit's user reaches on the host: 127.0.0.1:22 for the runner, the
28		# forge's public 22, 80 and 443 for builds, nothing else. Required, so
	28	# forge's public 22, 80 and 443 for builds, nothing else. The builds
	29	# table (ExecStartPre below) narrows that per build. Required, so
29	30	# the runner does not start without it: a table that failed to load must
30	31	# not mean builds reach the admin sshd.
31	32	Requires=gitbay-runner-egress.service
		@@ -71,6 +72,17 @@ OOMPolicy=continue
71	72	# start joins its namespaces — including a /tmp that no longer exists.
72	73	# End it with the service.
73	74	ExecStopPost=-/usr/bin/pkill -u ci-runner -x catatonit
	75	# The builds table (#260, gitbay-runner-builds.nft) matches a build's
	76	# traffic by the cgroup the runner starts it in, and nftables turns a
	77	# cgroup path into the cgroup's id when the table loads. This unit's
	78	# cgroup is new on every start, so the two class cgroups are created
	79	# here, handed to the runner's user, and the table loaded against them,
	80	# before the runner starts. As root (+), so the mkdir does not depend on
	81	# when systemd hands the delegated cgroup to the unit's user. A table
	82	# that fails to load stops the start, as the uid table's Requires= does.
	83	ExecStartPre=+/usr/bin/mkdir -p /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/trusted /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/untrusted
	84	ExecStartPre=+/usr/bin/chown -R ci-runner:ci-runner /sys/fs/cgroup/system.slice/gitbay-runner.service/builds
	85	ExecStartPre=+/usr/sbin/nft -f /etc/gitbay-runner/builds.nft
74	86	# On stop the runner drains: it claims nothing more and finishes the
75	87	# build in flight, then exits. Give it long enough — the per-build limit
76	88	# is 45m plus half a minute of report retries — before systemd kills it.

deploy/runner-auth-flood-test.sh added +201

		@@ -0,0 +1,201 @@
	1	#!/bin/sh
	2	# Scratch-repository test for #260: a build fails SSH logins while the
	3	# runner works, and the runner must not be locked out with it.
	4	#
	5	# Run from a machine with an admin gitbay identity, after the Admin
	6	# page's scratch procedure: the runner's key attached to the scratch
	7	# repository and the runner scoped to it with -repos. The script
	8	# replaces the repository's .gitbay/ci.yml.
	9	#
	10	# deploy/runner-auth-flood-test.sh cmc/runner-scratch # trusted: a push to main
	11	# deploy/runner-auth-flood-test.sh cmc/runner-scratch --untrusted # a merge request from a fork
	12	#
	13	# The build's logins use a git-scoped key registered with --ttl 1s and
	14	# expired by the time the build runs. With registration open an
	15	# unknown key is admitted to run register and never counts against the
	16	# SSH auth limiter; an expired key counts (internal/sshd/sshd.go, authenticate).
	17	# The key is removed from the account when the script exits.
	18	#
	19	# The step waits a minute, so the operator can find pasta's cgroup
	20	# (Admin page), probes what the build reaches, then makes 12 logins
	21	# without pause, so the limiter (ssh_auth_rate, 10 a minute per address) locks
	22	# the address those logins come from for most of the next minute. The
	23	# runner reports the result right after the step; its report retries
	24	# for half a minute.
	25	#
	26	# Before #260 a build reached the host's loopback through pasta, and its
	27	# logins arrived from 127.0.0.1, where the runner polls: the report is
	28	# refused ("too many authentication attempts" in the runner's journal),
	29	# the build is failed by the server two minutes after its log stream
	30	# ended, the runner's last-seen stops advancing for up to a minute, and
	31	# the audit log has auth.throttled for 127.0.0.1.
	32	#
	33	# With the fix, trusted: GITBAY_SSH is git@169.254.1.2, the logins are
	34	# denied and arrive from the host's public address, auth.throttled
	35	# names that address, the build succeeds and the runner keeps polling.
	36	# Untrusted: every login is refused by the builds table before it
	37	# reaches sshd, and no auth.* entry comes from the build at all.
	38	#
	39	# The script also requires lines in the build log, so a missing ssh or
	40	# bash in the image, or a table that matches nothing, fails rather than
	41	# passes. Trusted: "logins 12 denied" (every login reached sshd) and
	42	# 10.0.0.1:80 refused, not timed out. Untrusted: 169.254.1.2:22 and
	43	# github.com:22 refused and "12 refused". The untrusted lines are the
	44	# proof that pasta's sockets are in the build's cgroup: the uid table
	45	# lets ci-runner reach both.
	46	set -eu
	47
	48	repo=${1:-}
	49	[ -n "$repo" ] \|\| { echo "usage: $0 <owner/name> [--untrusted]" >&2; exit 2; }
	50	mode=trusted
	51	[ "${2:-}" = --untrusted ] && mode=untrusted
	52	host=${GITBAY_HOST:-gitbay.org}
	53	account=${RUNNER_ACCOUNT:-ci}
	54	job=flood260
	55
	56	tmp=$(mktemp -d)
	57	fp=
	58	cleanup() {
	59	if [ -n "$fp" ]; then gitbay keys remove "$fp" >/dev/null \|\| echo "remove key $fp by hand" >&2; fi
	60	fp=
	61	rm -rf "$tmp"
	62	}
	63	trap cleanup EXIT
	64	trap 'cleanup; exit 130' INT TERM
	65
	66	echo "==> an expired key"
	67	ssh-keygen -q -t ed25519 -N '' -C auth-flood-260 -f "$tmp/key"
	68	gitbay keys add --scope git --label auth-flood-260 --ttl 1s <"$tmp/key.pub" >/dev/null
	69	fp=$(ssh-keygen -lf "$tmp/key.pub" \| awk '{print $2}')
	70	sleep 2
	71
	72	if [ "$mode" = untrusted ]; then
	73	src="${repo%/}/${repo#/}-fork260"
	74	if ! gitbay repo show "$src" --json >/dev/null 2>&1; then
	75	echo "==> forking $repo to $src"
	76	gitbay repo fork "$repo" --name "${repo#*/}-fork260" >/dev/null
	77	fi
	78	branch="flood-260-$(date +%s)"
	79	else
	80	src=$repo
	81	branch=main
	82	fi
	83
	84	echo "==> committing the $job job to $src ($branch)"
	85	git clone -q "ssh://git@$host/$src.git" "$tmp/repo"
	86	cd "$tmp/repo"
	87	[ "$branch" = main ] \|\| git checkout -q -b "$branch"
	88	mkdir -p .gitbay
	89	cp "$tmp/key" .gitbay/flood.key
	90	cat >.gitbay/ci.yml <<EOF
	91	jobs:
	92	$job:
	93	steps:
	94	- echo "GITBAY_SSH=\$GITBAY_SSH"
	95	- sh .gitbay/flood.sh
	96	EOF
	97	cat >.gitbay/flood.sh <<'EOF'
	98	#!/bin/sh
	99	# Written by deploy/runner-auth-flood-test.sh (#260).
	100	set -u
	101	sleep 60
	102	key=/tmp/flood.key
	103	cp .gitbay/flood.key "$key"
	104	chmod 600 "$key"
	105	dest=${GITBAY_SSH#*@}
	106	host=${dest%:*}
	107	port=${dest##*:}
	108	[ "$host" = "$dest" ] && port=22
	109
	110	probe() {
	111	timeout 5 bash -c "exec 3<>/dev/tcp/$1/$2" 2>/dev/null
	112	case $? in
	113	0) echo "open $1:$2" ;;
	114	124) echo "timeout $1:$2" ;;
	115	*) echo "refused $1:$2" ;;
	116	esac
	117	}
	118	getent hosts proxy.golang.org >/dev/null && echo "dns ok" \|\| echo "dns failed"
	119	for t in "$host:22" "$host:80" "$host:443" "$host:2222" \
	120	10.0.0.1:80 192.168.0.1:80 proxy.golang.org:443 github.com:22; do
	121	probe "${t%:}" "${t##:}"
	122	done
	123
	124	denied=0 refused=0 other=0 i=0
	125	while [ $i -lt 12 ]; do
	126	i=$((i + 1))
	127	out=$(ssh -F /dev/null -i "$key" -o IdentitiesOnly=yes -o BatchMode=yes \
	128	-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5 \
	129	-p "$port" "git@$host" whoami 2>&1)
	130	case $out in
	131	"Permission denied") denied=$((denied + 1)) ;;
	132	"Connection refused") refused=$((refused + 1)) ;;
	133	*) other=$((other + 1)); echo "login $i: $out" ;;
	134	esac
	135	done
	136	echo "logins $denied denied, $refused refused, $other other"
	137	EOF
	138	git add .gitbay
	139	git commit -q -m "ci: auth flood test (#260)"
	140	git push -q origin "$branch"
	141	cd - >/dev/null
	142
	143	if [ "$mode" = untrusted ]; then
	144	gitbay mr create "$repo" --source "$src:$branch" --target main --title "#260 auth flood, untrusted" >/dev/null
	145	fi
	146
	147	# One gitbay call per tick: the CLI shares one connection, and a burst of
	148	# logins is what the limiter is for.
	149	echo "==> waiting for the build"
	150	n=
	151	for _ in $(seq 1 12); do
	152	sleep 5
	153	n=$(gitbay build list "$repo" --job "$job" --limit 1 --json \| jq -r '.data \| (.items // .) \| .[0].number // empty')
	154	[ -n "$n" ] && break
	155	done
	156	[ -n "$n" ] \|\| { echo "no $job build queued on $repo" >&2; exit 1; }
	157	echo " build $n"
	158	status=
	159	for _ in $(seq 1 60); do
	160	sleep 10
	161	status=$(gitbay build show "$repo" "$n" --json \| jq -r .data.status)
	162	case $status in success \| failure) break ;; esac
	163	done
	164	echo " $status"
	165
	166	echo "==> the runner after the build"
	167	seen() { gitbay admin runners --json \| jq -r --arg a "$account" '[.data.runners[] \| select(.username == $a) \| .last_seen] \| max // empty'; }
	168	first=$(seen)
	169	sleep 15
	170	second=$(seen)
	171	echo " $account last seen $first, then $second"
	172
	173	echo "==> build log"
	174	log=$(gitbay build log "$repo" "$n")
	175	printf '%s\n' "$log" \| sed -n '/dns /,$p'
	176
	177	echo "==> auth audit, last 15 minutes"
	178	gitbay audit --action auth. --since 15m --json \|
	179	jq -r '.data[] \| "\(.action) \(.data \| fromjson \| .ip // "-")"' \| sort \| uniq -c
	180
	181	echo
	182	fail=0
	183	[ "$status" = success ] \|\| { echo "FAIL: build $n is $status; the runner could not report it"; fail=1; }
	184	[ -n "$second" ] && [ "$second" != "$first" ] \|\| { echo "FAIL: the runner did not poll in 15 seconds after the build"; fail=1; }
	185	if gitbay audit --action auth.throttled --since 15m --json \| jq -e '.data[] \| select((.data \| fromjson \| .ip) == "127.0.0.1")' >/dev/null; then
	186	echo "FAIL: 127.0.0.1, the runner's address, was throttled"
	187	fail=1
	188	fi
	189	need() {
	190	printf '%s\n' "$log" \| grep -Eq "$1" \|\| { echo "FAIL: the build log lacks \"$2\""; fail=1; }
	191	}
	192	if [ "$mode" = trusted ]; then
	193	need 'logins +12 denied' "logins 12 denied"
	194	need 'refused +10\.0\.0\.1:80( \|$)' "refused 10.0.0.1:80"
	195	else
	196	need 'refused +169\.254\.1\.2:22( \|$)' "refused 169.254.1.2:22"
	197	need 'refused +github\.com:22( \|$)' "refused github.com:22"
	198	need '12 refused' "12 refused"
	199	fi
	200	[ $fail = 0 ] && echo "PASS ($mode): the build's failed logins did not lock the runner out"
	201	exit $fail

runner: build egress by trust class; auth-flood scratch test !519