Why podman on bay1 worked, then stopped, then refused every invocation.
gitbay-runner.service has PrivateTmp=yes. Left to itself, rootless podman
puts its run root at /tmp/storage-run-<uid>, so under the service that is a
per-instance tmpfs. podman also keeps a pause process for its user namespace, and
with the cgroupfs manager that process sits under the user slice, outside the
service cgroup — a service stop does not end it. After a restart the pause
process still holds the old mount namespace, whose /tmp backing directory
systemd has removed. Every later podman command joins that namespace via
/run/user/<uid>/libpod/tmp/pause.pid and fails with mkdir /tmp/storage-run-999: no such file or directory, whatever context it runs from.
That is the failure the setup script's own verification then hit.
Two changes:
storage.confpinsgraphrootandrunrootunder the runner's home. A run root there is valid in every namespace and depends on neither/tmpnor lingering. Rootless podman honours both keys; the earlier appearance that it ignoredrunrootwas the stale namespace.ExecStopPost=-pkill -u ci-runner -x catatonit, so a pause process ends with the service and the next start creates a fresh one in its own namespace.
The setup script no longer swallows a failed podman system reset behind
|| true, and its verification fails instead of passing. Applied on bay1 with a
real reset and image rebuild; podman info reports the pinned run root and
rootless=true from both a login shell and a systemd-run context with only
HOME set.
Stacked on !307.
Ref #144
retargeted from runner-provisioned-images to main: !307 merged
2026-09-07 01:23 UTC