CI: daily-pull fails at clone and cannot run on the podman runner #1

closed cmc opened this on 2026-09-07 19:04 UTC

Discussion

cmc 2026-09-07 19:04 UTC

Every daily-pull build since number 15 on 2026-08-30 fails at clone:

fatal: raw_data/Incidents_2015.csv: smudge filter lfs failed
warning: Clone succeeded, but checkout failed.

That is before any step runs and independent of the runner change below. Builds 23 to 30 were cancelled on 2026-09-07 after waiting since 2026-09-03.

Since 2026-09-06 the gitbay.org runner runs builds in localhost/gitbay-ci:1, which has no python3 or sqlite3, and is scoped to named repositories. This one is not listed until the clone works and an image: with the job's tools exists on bay1. The job also uploads release assets and force-pushes pages over ssh; inside the container there is no key, and the runner's key is read-only, so it needs a write-scoped deploy key for the push and an account key or API token for release asset add, delivered as build secrets.

cmc 2026-09-10 07:12 UTC

Fixed as of build 42 (2026-09-10), which ran to completion on the laptop runner.

  • LFS pointers whose objects existed nowhere were dropped from main, so the clone checks out.
  • The job runs on a gitbay-runner on the laptop attached to this repository alone, since Sarpy County geo-blocks bay1. Its image carries python3-venv and sqlite3 (gitbay-ci:2).
  • Publishing happens as the blotter-ci account with the BOT_SSH_KEY secret, passed through podman's environment and used via ssh -F with a workspace config.
  • The long step moved to ci/pull-publish.sh (steps cap at 4096 bytes).

Shipped in gitbay v1.18.0.