security hardening: remaining items #28

Umbrella for security work beyond #14. The original scope is done except for the three items below; everything else was verified closed out in the comment thread and re-verified 2026-08-31.

1. Host monitoring reports nowhere

gitbay-monitor.timer fires hourly on bay1 and has done so since !28, but /etc/gitbay/monitor.url has never existed, so gitbay-monitor.sh exits 0 at its second line every run. Disk, service-failure, and cert-expiry results are computed and discarded. An unset webhook and a healthy host are indistinguishable from the outside, and systemctl --failed stays clean either way.

The fix is not just to set a URL: the script should log its status to journald unconditionally and fail the unit on an alert condition, so the checks are visible on the host itself and the webhook is an optional second channel. A swallowed curl ... || true should also stop being silent.

2. govulncheck is not in the test loop

The original item asked for govulncheck "wired into the test loop". It lives only in deploy/audit.sh, which is run by hand. .gitbay/ci.yml has build and test jobs and neither invokes it. Dependency freshness is automated now (the deps checker, which filed #60), but nothing scans for vulnerabilities on a schedule or per-change.

3. Database durability and file mode

• /var/lib/gitbay/gitbay.db is mode 0644. The parent directory is drwxr-x--- gitbay:gitbay, so traversal blocks other local users and this is not a live exposure — but the file mode is wrong as defence in depth and store.Open should set it.
• Continuous replication was raised as "consider" and never decided either way. gitbay-backup.timer and gitbay-offsite.timer are both daily, so the current RPO is roughly 24 hours. Either adopt a litestream-style continuous copy or record the decision not to.

Concrete items shipped in !28 and applied to bay1. Software: fuzz targets for every attacker-facing parser (the SSHSIG armor fuzzer found a real decodeArmor negative-slice crash — fixed, crasher kept as a regression seed); CSP + X-Frame-Options/nosniff/no-referrer/HSTS on every response (live, verified on gitbay.org); govulncheck wired into deploy/audit.sh, which flagged CIRCL GO-2026-4550 reachable via PGP key parsing (bumped 1.6.2->1.6.3, now clean); token-comparison audit (every secret checked by SHA-256 hash DB lookup, never a Go-level compare — no timing oracle); docs/threat-model.org; optional minisign over release manifests. Host: systemd sandbox (SystemCallFilter=@system-service, PrivateDevices, LockPersonality, MemoryDenyWriteExecute, RestrictAddressFamilies) — verified clone, blob render, archive, and a signed push all still work under it; unattended-upgrades; fail2ban + MaxStartups/MaxAuthTries on the admin sshd (already banning 2 real attackers); hourly disk/service/cert monitoring (dormant until /etc/gitbay/monitor.url is set). Leaving this umbrella open for ongoing security work; the gating items for open registration are done.

All three remaining items are done, merged and live on bay1.

1. Host monitoring reports nowhere — !145, !146. gitbay-monitor.sh writes disk=… service=… cert_expires=… cert_days=… to journald every run, reports a failed webhook post instead of swallowing it, and exits non-zero on an alert so the unit surfaces in systemctl --failed. The webhook is now a second channel rather than the only one, so an unset monitor.url no longer makes a sick host look quiet.

Making it speak immediately exposed a second bug: the cert check read /var/lib/gitbay/autocert, but the ACME cache is /var/lib/gitbay/acme, so it had reported n/a on every run since it was written. It also took whichever filename sorted first, which says nothing about the certificate actually about to lapse. It now reports the soonest expiry across the cache and alerts under 21 days, inside Let's Encrypt's 30-day renewal window. Live on bay1: cert_expires=Nov 22 00:22:23 2026 GMT cert_days=81, correctly picking gitbay.org out of 8 certificates.

2. govulncheck is not in the test loop — !145. A vuln job in .gitbay/ci.yml, separate from test so an advisory published against unchanged code cannot be mistaken for a test failure. Confirmed genuinely scanning rather than passing vacuously. Clean today; the one module-graph finding, GO-2026-5932 on golang.org/x/crypto/openpgp, is unreachable since internal/sig uses ProtonMail/go-crypto.

3. Database durability and file mode — !145, !147.

store.Open chmods the database to 0640. SQLite gives -wal and -shm the main file's mode, so all three follow; a test asserts it and fails at 0644. The existing files on bay1 were chmodded by hand, since SQLite only sets the mode on files it creates.

admin backup --db-only writes the SQLite snapshot without the repositories, and gitbay-db-backup.timer runs it hourly keeping 48. The nightly full archive and its 7-day retention are untouched. Verified on bay1: 1.9MB per snapshot, pragma integrity_check ok, all 71 repositories present.

Plain hourly full backups were the obvious reading and do not work here. Each is ~550MB, and the retention is "keep the last 7", so hourly would have silently cut the history window from 7 days to 7 hours; holding 7 days at that rate needs about 92GB. The split costs ~91MB and puts the tighter recovery point on the data that has no second copy — repositories are git and are mirrored, issues and merge requests and comments are not.

Continuous replication was considered and rejected: it would take the database to seconds but leave the repositories on the nightly archive, so a restore could produce a database referencing commits the repository backup does not have. Consistency between the two halves is worth more than latency on one. Recorded on the wiki's Admin page. Revisit if repository replication ever becomes continuous.

Both backup scripts also wrote their archives 0644, including the 550MB nightly ones carrying the whole database. umask 027 in both, and db/ is created 0750.

Closing. Two things found alongside this work and deliberately not folded in:

• The CI runner loses test output on failure — 6 lines captured against 29 on a successful run, with no FAIL line — so an intermittent e2e failure cannot be diagnosed. Predates this work (builds 127 and 131 failed the same way). Worth its own issue.
• gitbay-offsite.timer and its restic script exist on bay1 but are not in deploy/cloud-init.yaml, so an instance rebuilt from cloud-init comes up with no offsite backup.