deploy: pin podman's run root; end the pause process with the service !310

merged merged by cmc on 2026-09-07 01:52 UTC · krz/gitbay:runner-podman-storage into main

Discussion

cmc

Why podman on bay1 worked, then stopped, then refused every invocation.

gitbay-runner.service has PrivateTmp=yes. Left to itself, rootless podman puts its run root at /tmp/storage-run-<uid>, so under the service that is a per-instance tmpfs. podman also keeps a pause process for its user namespace, and with the cgroupfs manager that process sits under the user slice, outside the service cgroup — a service stop does not end it. After a restart the pause process still holds the old mount namespace, whose /tmp backing directory systemd has removed. Every later podman command joins that namespace via /run/user/<uid>/libpod/tmp/pause.pid and fails with mkdir /tmp/storage-run-999: no such file or directory, whatever context it runs from. That is the failure the setup script's own verification then hit.

Two changes:

  • storage.conf pins graphroot and runroot under the runner's home. A run root there is valid in every namespace and depends on neither /tmp nor lingering. Rootless podman honours both keys; the earlier appearance that it ignored runroot was the stale namespace.
  • ExecStopPost=-pkill -u ci-runner -x catatonit, so a pause process ends with the service and the next start creates a fresh one in its own namespace.

The setup script no longer swallows a failed podman system reset behind || true, and its verification fails instead of passing. Applied on bay1 with a real reset and image rebuild; podman info reports the pinned run root and rootless=true from both a login shell and a systemd-run context with only HOME set.

Stacked on !307.

Ref #144

retargeted from runner-provisioned-images to main: !307 merged

2026-09-07 01:23 UTC
cmc 2026-09-07 01:32 UTC

Three commits added after the description, each from a smoke build on cmc/ci-smoke with the runner scoped to it by -repos, so none of them touched krz/gitbay CI:

  • containers.conf with default_sysctls = []: podman's default net.ipv4.ping_group_range cannot be written under ProtectKernelTunables (/proc/sys read-only) and the container failed to start. A build has no use for ping; the default goes, not the hardening.
  • ProtectKernelTunables=no: it overmounts /proc in the unit's namespace and the kernel then refuses a proc mount in any child user namespace — crun's "mount proc to proc: Operation not permitted". No podman setting avoids it; the container masks the same paths for the build. Documented in the drop-in and the Threat-Model, same treatment as NoNewPrivileges.
  • The Admin page gains the scratch-repository validation procedure.

With these applied on bay1 the smoke job ran inside the container: Go from the image, /workspace and the build home the only host mounts, the runner's key unreachable, sibling workspaces hidden. Two green runs, then the runner was pointed back at krz/gitbay in podman mode and is draining its queue.