backup: restore drill on a clean host; verify git connectivity #259

closed cmc opened this on 2026-09-28 02:36 UTC · admin ops

Discussion

cmc 2026-09-28 02:36 UTC

Restore has never been exercised.

Backups run hourly (db-only) and nightly (full), with a nightly restic copy offsite. No restore onto a clean host is recorded, and recovery time is unmeasured. --verify (cmd/gitbayd/backup.go:227) checks SQLite integrity and that each repository's HEAD is present, not that its objects are.

The backup snapshots SQLite, then walks the live filesystem with no coordination against repository deletion (internal/control/repo.go:611). A repository deleted between the two is named in the database and absent from the archive.

Restore drill on a clean host: database integrity, git fsck --connectivity-only per repository, LFS objects, release assets, config, host keys, secrets. Record time to service and the recovered timestamps.
--verify runs a connectivity check on each repository.
Hold repository deletion and rename off while a backup runs, or snapshot the filesystem.
Record the drill and its numbers on the Admin wiki page; repeat on a schedule.

referenced in commit 7a6343d02d by cmc: wiki: architecture and security pages

2026-09-28 04:30 UTC

referenced in commit 4e0958a163 by cmc: plans: open issues from the architecture and UX reviews

2026-09-28 05:43 UTC

referenced in commit b711b70da5 by cmc: plans: apply decisions on the open questions

2026-09-28 06:16 UTC

referenced in commit 8994b7770e by cmc: backup: verify skips commondir; refs/ before packed-refs; tighter stale match

2026-09-28 23:09 UTC

referenced in commit 32fb00e679 by cmc: changelog: upgrade notes first, duplicates dropped, backup fixes; system-mode comment names the gitbay user

2026-09-28 23:09 UTC

referenced in commit 72d22fe4f7 by cmc: backup: verify runs fsck on the git directory alone and ignores alternates

2026-09-28 23:09 UTC

referenced in commit b383802109 by cmc: backup: remove a killed run's leftovers; refuse an archive inside server.root

2026-09-28 23:09 UTC

referenced in commit 884a5c558b by cmc: backup: open the database without the key file

2026-09-28 23:09 UTC

referenced in commit 9e4827a050 by cmc: docs: restore puts the secret key file back before starting gitbayd

2026-09-28 23:09 UTC

referenced in commit 5f1152bca4 by cmc: backup: gc and mr prune refuse during a full backup; skip objects removed mid-walk

2026-09-28 23:09 UTC

referenced in commit d201d6acb1 by cmc: backup: archive each repository's refs before its objects

2026-09-28 23:09 UTC

referenced in commit 55cbdaf81e by cmc: wiki: backup verify, backup lock, restore drill procedure

2026-09-28 23:09 UTC

referenced in commit 4906a493f2 by cmc: backup: hold repository moves off during a full backup; verify git connectivity

2026-09-28 23:09 UTC

referenced in commit 6576de5182 by cmc: control: repository delete, rename, transfer and org rename wait out a full backup

2026-09-28 23:09 UTC

referenced in commit a166de255d by cmc: backuplock: flock between a full backup and repository moves

2026-09-28 23:09 UTC

referenced in commit 690a0542e1 by cmc: gitutil: FsckConnectivity

2026-09-28 23:09 UTC

referenced in commit 18da8c33b3 by cmc: backup: verify skips LFS upload staging files; asset label names a missing repository by id

2026-09-29 03:37 UTC

referenced in commit 504b4d488c by cmc: wiki, changelog: restore-drill, what a drill restores and checks

2026-09-29 03:37 UTC

referenced in commit a27671383a by cmc: gitbayd: admin restore-drill restores an archive, verifies it, reports elapsed time and the newest activity

2026-09-29 03:37 UTC

referenced in commit c09a124633 by cmc: backup: verify checks release assets and LFS object digests

2026-09-29 03:37 UTC

closed by cmc in commit 415c793921: wiki: first restore drill recorded

2026-09-29 05:57 UTC

referenced in commit c0f2228f98 by cmc: wiki: backup --verify needs TMPDIR on disk where /tmp is a small tmpfs

2026-09-29 16:19 UTC