Supervise the collectors: restart, contain panics, bound git #10

closed cmc opened this on 2026-09-04 15:14 UTC

cmc 2026-09-04 15:14 UTC

The daemon degrades silently and never recovers in three ways:

  • The hardware collector is spawned once and never respawned (lib.rs:409). If the child dies, hardware signals stop permanently; KeepAlive does not help because signald is still alive.
  • The producer thread is unsupervised (main.rs:94). A panic in git::collect or terminal.collect kills it while the socket keeps serving a frozen last-value cache.
  • run_git uses .output() with no timeout, so a stuck lock or a network mount hangs the producer thread forever while health would read "up".

Done when:

  • signald/src/supervisor.rs owns collector liveness and is the only publisher of CollectorUp.
  • Git and terminal collector calls are wrapped in catch_unwind; a panic publishes that source down, the tick continues, the next tick retries, and the terminal Collector is replaced with a fresh one.
  • The hardware child is respawned with backoff from 1s doubling to a 60s cap, reset after a run surviving 30s, with no retry limit. A collector that was never configured publishes down once and starts no retry loop.
  • Health is published on transitions plus once at startup, not every tick.
  • run_git kills the subprocess on timeout and treats it as a failed call.

closed by commit d1942a01f0 by cmc: Supervise the collectors and bound the ways they hang

2026-09-04 15:56 UTC