Force-pushing leaves queued builds that fail on a missing object #152

Found while releasing v1.13.1.

Amending a commit and force-pushing the branch left the builds already queued against the abandoned commit in the queue. They then ran and failed:

$ git clone ssh://git@127.0.0.1/krz/gitbay.git (d69a1511bf)
fatal: unable to read tree (d69a1511bf753ba1605f024c4e8c2c2958a7438a)
git -C: step failed: exit status 128

Five builds across two force-pushes did this in one afternoon (655–658 and 662). They show as failure in build list alongside real results, and the message names a git internal error rather than the actual cause, which is that nothing points at that commit any more.

Two costs. A red build that means nothing trains a reader to ignore red, which is the state a gating CI exists to prevent. And it actively misleads: I read 662 failure vuln and spent time believing a new advisory had landed against unchanged code, because vuln failing is exactly what that looks like.

QueueBranchBuilds records sha at push time, and nothing revisits that queue when the ref moves. The runner claims the build later and clones a commit no ref reaches.

Remedy: when a ref update leaves a queued build's sha unreachable from any ref, cancel it as obsolete rather than letting it run. build cancel already exists and cancelled is already a status the UI renders, so this is a matter of noticing at ref-update time — post-receive already has the old and new values.

Worth deciding alongside: whether a running build for a superseded commit should also be cancelled. A force-push while the suite is running is common, and finishing a build of code nobody will merge costs a runner slot. Cancelling it is defensible, but so is letting it finish, since its log may be the reason someone force-pushed. Cancelling only queued builds is the smaller, safer change.

A live instance, build 771 on login-followups:

$ git clone ssh://git@127.0.0.1/krz/gitbay.git (debbd7c440)
fatal: unable to read tree (debbd7c4400ce7a42467e687f6f31efeba4128fd)
git -C: step failed: exit status 128

debbd7c was the branch head before a rebase; the force-push orphaned it while four builds for that sha were still queued. Each then failed on a sha the repository no longer reaches, and the failures sit in the build list looking like real ones.

Rebasing a branch onto a moved main is routine here — signed commits mean only fast-forward merges, so any branch whose target advances gets rebased and force-pushed. Every one of those produces a batch of doomed builds.

Worth noting for whoever picks this up: the failure is indistinguishable from a genuine one in build list, which is the part that costs time. Cancelling queued builds for an orphaned sha on force-push would fix both the noise and the wasted runner slots.