Commit `748122ef10`

748122ef1028a1ae48e703996414cd25ae938508

parent: 7acfcebdf5

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-29 03:46 UTC

runner host: flood test asserts the table's effect; exact rollback of the builds table

Ref #260

Layout: unified · split

.gitbay/wiki/Admin.org +30 −8

 A restart of =nftables.service= flushes both tables;
 =systemctl reload gitbay-runner-egress= restores both.
-Checking the tables on a running host, during a build:
+The egress unit's stop and the rollback below use =nft destroy=,
+which needs nftables 1.0.8 or later; check =nft --version= on a new
+host.
+Checking the tables on a running host, during a build (the flood test
+below waits a minute before its first probe, for this):
 #+begin_src sh
 nft list table inet gitbay_builds          # counters on the reject rules
 A pasta process anywhere else — the runner's own =runner= cgroup, a
 user slice — means the builds table does not see that build's traffic
-and only the first table applies. Then run
+and only the first table applies. Run
 =deploy/runner-auth-flood-test.sh= on the scratch repository (below),
-once as a push and once with =--untrusted=; it prints what the build
+once as a push and once with =--untrusted=. It fails if the runner was
-reached and fails if the runner was locked out.
+locked out or if the build log lacks the lines only a working table
+produces. The authoritative proof that pasta's sockets are in the
+build's cgroup is the untrusted run: =169.254.1.2:22= and
+=github.com:22= refused, all twelve logins refused, and the counters on
+the =untrusted= chain's rejects rising. The first table lets
+=ci-runner= reach both of those addresses.
+To take the builds table out, on the host:
+#+begin_src sh
+sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
+rm /etc/gitbay-runner/builds.nft
+systemctl daemon-reload
+nft destroy table inet gitbay_builds
+#+end_src
-To take the builds table out: delete the three =ExecStartPre= lines
+The runner needs no restart. It still places builds under
-that name =builds= from the drop-in, =systemctl daemon-reload=, and
+=builds/trusted= and =builds/untrusted=, but with the file gone neither
-=nft destroy table inet gitbay_builds=. The runner needs no restart,
+a runner start nor =systemctl reload gitbay-runner-egress= loads the
-and builds keep the first table.
+table again, and builds keep the first table and =--no-map-gw=. A
+runner that takes =-untrusted= or polls over loopback refuses to start
+without build cgroups at all, since the table would match nothing. The
+next =make deploy-runner= installs the file and the lines again.
 The drop-in sets =NoNewPrivileges=no=, without which rootless podman
 cannot call =newuidmap= and the runner refuses to start. That is a

deploy/gitbay-runner-egress.service +5 −4

 # flush ruleset removes it); delete would fail and leave the unit failed.
 #
 # The builds table (gitbay-runner-builds.nft) is loaded by the runner's
-# own start, against its cgroups. Reload loads it again when those
+# own start, against its cgroups. Reload loads it again when the file is
-# cgroups exist, so after a restart of nftables.service one reload puts
+# installed and those cgroups exist, so after a restart of
-# both tables back.
+# nftables.service one reload puts both tables back; without the file
+# (taken out per the Admin page) the reload skips it and succeeds.
 [Unit]
 Description=Host egress rule for CI builds
 After=nftables.service ufw.service
 RemainAfterExit=yes
 ExecStart=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
 ExecReload=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
-ExecReload=/bin/sh -c 'if [ -d /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/untrusted ]; then exec /usr/sbin/nft -f /etc/gitbay-runner/builds.nft; fi'
+ExecReload=/bin/sh -c 'if [ -f /etc/gitbay-runner/builds.nft ] && [ -d /sys/fs/cgroup/system.slice/gitbay-runner.service/builds/untrusted ]; then exec /usr/sbin/nft -f /etc/gitbay-runner/builds.nft; fi'
 ExecStop=/usr/sbin/nft destroy table inet gitbay_runner
 ExecStop=/usr/sbin/nft destroy table inet gitbay_builds

deploy/runner-auth-flood-test.sh +33 −9

 #   deploy/runner-auth-flood-test.sh cmc/runner-scratch              # trusted: a push to main
 #   deploy/runner-auth-flood-test.sh cmc/runner-scratch --untrusted  # a merge request from a fork
 #
-# The build's logins use a key registered with --ttl 1s and expired by
+# The build's logins use a git-scoped key registered with --ttl 1s and
-# the time the build runs. With registration open an unknown key is
+# expired by the time the build runs. With registration open an
-# admitted to run register and never counts against the SSH auth
+# unknown key is admitted to run register and never counts against the
-# limiter; an expired key counts (internal/sshd/sshd.go, authenticate).
+# SSH auth limiter; an expired key counts (internal/sshd/sshd.go, authenticate).
 # The key is removed from the account when the script exits.
 #
-# The step probes what the build reaches, then makes 12 logins without
+# The step waits a minute, so the operator can find pasta's cgroup
-# pause, so the limiter (ssh_auth_rate, 10 a minute per address) locks
+# (Admin page), probes what the build reaches, then makes 12 logins
+# without pause, so the limiter (ssh_auth_rate, 10 a minute per address) locks
 # the address those logins come from for most of the next minute. The
 # runner reports the result right after the step; its report retries
 # for half a minute.
 # names that address, the build succeeds and the runner keeps polling.
 # Untrusted: every login is refused by the builds table before it
 # reaches sshd, and no auth.* entry comes from the build at all.
+#
+# The script also requires lines in the build log, so a missing ssh or
+# bash in the image, or a table that matches nothing, fails rather than
+# passes. Trusted: "logins 12 denied" (every login reached sshd) and
+# 10.0.0.1:80 refused, not timed out. Untrusted: 169.254.1.2:22 and
+# github.com:22 refused and "12 refused". The untrusted lines are the
+# proof that pasta's sockets are in the build's cgroup: the uid table
+# lets ci-runner reach both.
 set -eu
 repo=${1:-}
 fp=
 cleanup() {
     if [ -n "$fp" ]; then gitbay keys remove "$fp" >/dev/null || echo "remove key $fp by hand" >&2; fi
+    fp=
     rm -rf "$tmp"
 }
 trap cleanup EXIT
+trap 'cleanup; exit 130' INT TERM
 echo "==> an expired key"
 ssh-keygen -q -t ed25519 -N '' -C auth-flood-260 -f "$tmp/key"
-gitbay keys add --label auth-flood-260 --ttl 1s <"$tmp/key.pub" >/dev/null
+gitbay keys add --scope git --label auth-flood-260 --ttl 1s <"$tmp/key.pub" >/dev/null
 fp=$(ssh-keygen -lf "$tmp/key.pub" | awk '{print $2}')
 sleep 2
 #!/bin/sh
 # Written by deploy/runner-auth-flood-test.sh (#260).
 set -u
+sleep 60
 key=/tmp/flood.key
 cp .gitbay/flood.key "$key"
 chmod 600 "$key"
     esac
 }
 getent hosts proxy.golang.org >/dev/null && echo "dns      ok" || echo "dns      failed"
-for t in 127.0.0.1:22 127.0.0.1:2222 "$host:22" "$host:80" "$host:443" "$host:2222" \
+for t in "$host:22" "$host:80" "$host:443" "$host:2222" \
     10.0.0.1:80 192.168.0.1:80 proxy.golang.org:443 github.com:22; do
     probe "${t%:*}" "${t##*:}"
 done
 echo "   $account last seen $first, then $second"
 echo "==> build log"
-gitbay build log "$repo" "$n" | sed -n '/^dns /,$p'
+log=$(gitbay build log "$repo" "$n")
+printf '%s\n' "$log" | sed -n '/dns /,$p'
 echo "==> auth audit, last 15 minutes"
 gitbay audit --action auth. --since 15m --json |
     echo "FAIL: 127.0.0.1, the runner's address, was throttled"
     fail=1
 fi
+need() {
+    printf '%s\n' "$log" | grep -Eq "$1" || { echo "FAIL: the build log lacks \"$2\""; fail=1; }
+}
+if [ "$mode" = trusted ]; then
+    need 'logins +12 denied' "logins 12 denied"
+    need 'refused +10\.0\.0\.1:80( |$)' "refused 10.0.0.1:80"
+else
+    need 'refused +169\.254\.1\.2:22( |$)' "refused 169.254.1.2:22"
+    need 'refused +github\.com:22( |$)' "refused github.com:22"
+    need '12 refused' "12 refused"
+fi
 [ $fail = 0 ] && echo "PASS ($mode): the build's failed logins did not lock the runner out"
 exit $fail

Commit 748122ef10

Commit `748122ef10`