Commit `94bf85e690`

94bf85e69090d8a4163e3941523d86effca6b687

parent: 5476d28c71

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-07 01:23 UTC

deploy: pin podman's run root under the runner's home; end the pause process with the service

Left to podman the run root is /tmp/storage-run-<uid>, which under the
unit's PrivateTmp is a per-instance tmpfs. The cgroupfs manager places
podman's pause process outside the service cgroup, so it outlives a
restart holding a dead /tmp, and every later podman command in any
context fails with 'mkdir ...: no such file or directory'. storage.conf
pins graphroot and runroot under the home directory, and ExecStopPost
ends the pause process with the service.

The setup script's reset no longer swallows its own failure, and the
final verification fails instead of passing with || true.

Ref #144

Layout: unified · split

deploy/gitbay-runner.override.conf +9 −5

		@@ -27,11 +27,15 @@
27	27	# and the sandboxing that has to match them live in one file: -isolation
28	28	# podman needs NoNewPrivileges=no below, and -image needs an image the
29	29	# host has been given (deploy/runner-podman-setup.sh, Containerfile.ci).
30		# Deliberately no XDG_RUNTIME_DIR. podman records its run root in its
31		# database at first use, so setting one later fails with "database
32		# configuration mismatch"; the runner's storage was initialised without
33		# it and works. Change it only together with `podman system reset` and a
34		# rebuild of the images (#144).
	30	# podman's run root is pinned under the runner's home by storage.conf
	31	# (runner-podman-setup.sh), not taken from XDG_RUNTIME_DIR or /tmp: this
	32	# unit has PrivateTmp, so a /tmp run root is a per-instance tmpfs.
	33	#
	34	# The cgroupfs manager puts podman's pause process under the user slice,
	35	# outside this unit's cgroup, so a stop does not end it and the next
	36	# start joins its namespaces — including a /tmp that no longer exists.
	37	# End it with the service.
	38	ExecStopPost=-/usr/bin/pkill -u ci-runner -x catatonit
35	39	ExecStart=
36	40	ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -repos krz/gitbay -isolation podman -image localhost/gitbay-ci:1
37	41	Nice=10

deploy/runner-podman-setup.sh +27 −4

		@@ -52,13 +52,36 @@ if [ "$max_ns" -lt 1 ]; then
52	52	fi
53	53	echo " max_user_namespaces=$max_ns"
54	54
55		# Lingering keeps the user's systemd session — and so podman's storage
56		# and any running container — alive when nobody is logged in.
	55	# podman's storage paths are pinned in storage.conf, both graphroot and
	56	# runroot, under the runner's home. Left to podman, the run root is
	57	# $XDG_RUNTIME_DIR or /tmp/storage-run-<uid>; the service runs with
	58	# PrivateTmp, so that is a per-instance tmpfs, and podman's pause process
	59	# (which the cgroupfs manager places outside the service cgroup) can
	60	# outlive a restart holding a dead /tmp — after which every podman
	61	# command, in any context, fails with "mkdir ...: no such file or
	62	# directory". A run root under the home directory is valid in every
	63	# namespace and needs neither lingering nor /tmp.
	64	#
	65	# podman records the run root at first use. Changing it later needs
	66	# `podman system reset --force` as the runner user and a rebuild of the
	67	# images; this script does not do that for you.
	68	home=$(getent passwd "$RUNNER_USER" \| cut -d: -f6)
	69	conf="$home/.config/containers/storage.conf"
	70	echo "==> storage config in $conf"
	71	install -d -o "$RUNNER_USER" -g "$RUNNER_USER" -m 700 "$home/.config/containers"
	72	printf '[storage]\ndriver = "overlay"\ngraphroot = "%s/.local/share/containers/storage"\nrunroot = "%s/.local/share/containers/run"\n' "$home" "$home" >"$conf"
	73	chown "$RUNNER_USER:$RUNNER_USER" "$conf"
	74	echo " written"
	75
	76	# Lingering keeps the user's systemd session alive when nobody is logged
	77	# in, which podman's pause process relies on.
57	78	echo "==> lingering for $RUNNER_USER"
58	79	loginctl enable-linger "$RUNNER_USER"
59	80
60	81	echo "==> verifying rootless podman as $RUNNER_USER"
61		su - "$RUNNER_USER" -s /bin/sh -c 'podman info --format "{{.Host.Security.Rootless}}"'
	82	# The verification fails rather than passing with \|\| true: a host that
	83	# reports ready and is not is the outage this script exists to prevent.
	84	su - "$RUNNER_USER" -s /bin/sh -c "podman info --format 'rootless={{.Host.Security.Rootless}} runroot={{.Store.RunRoot}}'"
62	85
63	86	echo
64		echo "host is ready; now: make deploy-runner"
	87	echo "host is ready. Build the CI image (deploy/Containerfile.ci), then: make deploy-runner"

Commit 94bf85e690

Commit `94bf85e690`