Commit `94bf85e690`

94bf85e69090d8a4163e3941523d86effca6b687

parent: 5476d28c71

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-07 01:23 UTC

deploy: pin podman's run root under the runner's home; end the pause process with the service

Left to podman the run root is /tmp/storage-run-<uid>, which under the
unit's PrivateTmp is a per-instance tmpfs. The cgroupfs manager places
podman's pause process outside the service cgroup, so it outlives a
restart holding a dead /tmp, and every later podman command in any
context fails with 'mkdir ...: no such file or directory'. storage.conf
pins graphroot and runroot under the home directory, and ExecStopPost
ends the pause process with the service.

The setup script's reset no longer swallows its own failure, and the
final verification fails instead of passing with || true.

Ref #144

Layout: unified · split

deploy/gitbay-runner.override.conf +9 −5

 # and the sandboxing that has to match them live in one file: -isolation
 # podman needs NoNewPrivileges=no below, and -image needs an image the
 # host has been given (deploy/runner-podman-setup.sh, Containerfile.ci).
-# Deliberately no XDG_RUNTIME_DIR. podman records its run root in its
+# podman's run root is pinned under the runner's home by storage.conf
-# database at first use, so setting one later fails with "database
+# (runner-podman-setup.sh), not taken from XDG_RUNTIME_DIR or /tmp: this
-# configuration mismatch"; the runner's storage was initialised without
+# unit has PrivateTmp, so a /tmp run root is a per-instance tmpfs.
-# it and works. Change it only together with `podman system reset` and a
+#
-# rebuild of the images (#144).
+# The cgroupfs manager puts podman's pause process under the user slice,
+# outside this unit's cgroup, so a stop does not end it and the next
+# start joins its namespaces — including a /tmp that no longer exists.
+# End it with the service.
+ExecStopPost=-/usr/bin/pkill -u ci-runner -x catatonit
 ExecStart=
 ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -repos krz/gitbay -isolation podman -image localhost/gitbay-ci:1
 Nice=10

deploy/runner-podman-setup.sh +27 −4

	@@ -52,13 +52,36 @@ if [ "$max_ns" -lt 1 ]; then
52	fi	52	fi
53	echo " max_user_namespaces=$max_ns"	53	echo " max_user_namespaces=$max_ns"
54		54
55	# Lingering keeps the user's systemd session — and so podman's storage	55	# podman's storage paths are pinned in storage.conf, both graphroot and
56	# and any running container — alive when nobody is logged in.	56	# runroot, under the runner's home. Left to podman, the run root is
		57	# $XDG_RUNTIME_DIR or /tmp/storage-run-<uid>; the service runs with
		58	# PrivateTmp, so that is a per-instance tmpfs, and podman's pause process
		59	# (which the cgroupfs manager places outside the service cgroup) can
		60	# outlive a restart holding a dead /tmp — after which every podman
		61	# command, in any context, fails with "mkdir ...: no such file or
		62	# directory". A run root under the home directory is valid in every
		63	# namespace and needs neither lingering nor /tmp.
		64	#
		65	# podman records the run root at first use. Changing it later needs
		66	# `podman system reset --force` as the runner user and a rebuild of the
		67	# images; this script does not do that for you.
		68	home=$(getent passwd "$RUNNER_USER" \| cut -d: -f6)
		69	conf="$home/.config/containers/storage.conf"
		70	echo "==> storage config in $conf"
		71	install -d -o "$RUNNER_USER" -g "$RUNNER_USER" -m 700 "$home/.config/containers"
		72	printf '[storage]\ndriver = "overlay"\ngraphroot = "%s/.local/share/containers/storage"\nrunroot = "%s/.local/share/containers/run"\n' "$home" "$home" >"$conf"
		73	chown "$RUNNER_USER:$RUNNER_USER" "$conf"
		74	echo " written"
		75
		76	# Lingering keeps the user's systemd session alive when nobody is logged
		77	# in, which podman's pause process relies on.
57	echo "==> lingering for $RUNNER_USER"	78	echo "==> lingering for $RUNNER_USER"
58	loginctl enable-linger "$RUNNER_USER"	79	loginctl enable-linger "$RUNNER_USER"
59		80
60	echo "==> verifying rootless podman as $RUNNER_USER"	81	echo "==> verifying rootless podman as $RUNNER_USER"
61	su - "$RUNNER_USER" -s /bin/sh -c 'podman info --format "{{.Host.Security.Rootless}}"'	82	# The verification fails rather than passing with \|\| true: a host that
		83	# reports ready and is not is the outage this script exists to prevent.
		84	su - "$RUNNER_USER" -s /bin/sh -c "podman info --format 'rootless={{.Host.Security.Rootless}} runroot={{.Store.RunRoot}}'"
62		85
63	echo	86	echo
64	echo "host is ready; now: make deploy-runner"	87	echo "host is ready. Build the CI image (deploy/Containerfile.ci), then: make deploy-runner"

Commit 94bf85e690

Commit `94bf85e690`