Second of three for #179.
ReapStaleBuilds keyed only on started_at past a fixed deadline, which is why
build 1037 sat running for over half an hour after its runner was restarted
out from under it. The server already has a better signal: the runner streams
the log over one ssh session and reports the outcome right after closing it. A
stream that ends with no outcome following means the runner is gone.
runRunnerLog now records log_closed_at when the stream ends (migration 0046,
running builds only). The reaper fails a running build whose stream closed more
than StaleLogGrace (two minutes) ago — long enough to outlast !320's thirty
seconds of report retries — and keeps the deadline rule for a build whose stream
was never seen.
TestReapStaleBuildsAfterLogClosed covers inside and past the grace period, a
build with an open stream staying untouched, and marking being a no-op once the
build is no longer running. TestRunnerLogMarksStreamClosed drives
runRunnerLog with a stream that ends and checks the mark lands while the build
stays running for the report.
Stacked on !320.
Ref #179
retargeted from runner-report-retry-179 to main
2026-09-07 05:07 UTC