a build stays running forever when the runner stops working on it #53

closed cmc opened this on 2026-08-30 17:48 UTC

Discussion

cmc 2026-08-30 17:48 UTC

A build the runner stops working on stays running forever. Two on krz/orgo:

17	test	running	5448d39270	runner-ssh-host
18	pages	running	5448d39270	main

Build 17's log contains every step including the last, so it finished its work and never reported. Build 18's log stops mid-compile; the runner moved on to builds 19 and 20 without it. Neither has a finished timestamp:

$ gitbay build show krz/orgo 18
build 18	pages	running
5448d39270 on main
queued 2026-08-30T17:10:15Z

Both coincide with systemctl restart gitbay-runner. The runner reports terminal state with runner done, so a process that dies between claiming a build and that call leaves the row claimed and never resolves it.

The queue keeps moving, because runner next claims by pending rather than by an absence of running builds, so the cost is not a stall — it is that build list cannot be trusted to say what is happening, and the ci/<job> commit status stays pending against the commit forever.

Worth considering: a claim deadline the server enforces (the runner already takes a -timeout, currently 45m, that only it knows about), or having the runner resolve builds it finds claimed by itself at startup.

closed by commit dfc4fecaa3 by cmc: Resolve builds a dead runner abandoned, and forward bare-redirect stdin

2026-08-30 18:11 UTC

referenced in commit 81bb0d1408 by cmc: ci: reap stale builds on the scheduler tick

2026-09-02 03:29 UTC