Shutdown waits on idle SSH connections, and a restart locks the CLI out via the auth rate limit #141

closed cmc opened this on 2026-09-03 23:39 UTC · cli ops security · milestone v1.10.0

Discussion

cmc 2026-09-03 23:39 UTC

Two things seen deploying v1.9.0, both consequences of the drain and the CLI's shared connection together.

Idle connections hold the drain. sshd.Shutdown waits for every accepted connection, including a CLI control master with no session open. The deploy restart waited the full 30 s for one idle multiplexed connection (cmd/gitbayd/main.go, internal/sshd/sshd.go). A connection with no active session should be closed immediately; only sessions with a command running need the drain.

A restart re-authenticates everything at once. When the daemon restarts, every client's control master dies and each following command opens a fresh connection. A script issuing a few commands a minute (a CI poll, a status check) then crosses limits.ssh_auth_rate (10 per minute per IP) and gets "Permission denied (publickey)" for a minute, with nothing telling it why. Options: exempt or raise the limit for a key that already authenticated successfully in the window, have the CLI back off and retry on exit 5 once, and say "rate limited" on stderr rather than a bare denial. The server-side message is the important part.

Ref #94, #105

closed by commit d299dba2d3 by cmc: sshd: close idle connections on shutdown, and do not count a store failure as a bad key

2026-09-04 00:10 UTC