Commit `55cbdaf81e`

55cbdaf81ef6d24d44beb5c8ba220618c6769407

parent: 4906a493f2

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-28 22:15 UTC

wiki: backup verify, backup lock, restore drill procedure

Ref #259

Layout: unified · split

.gitbay/wiki/Admin.org +49 −3

	@@ -447,9 +447,31 @@ hook socket, regenerated hook scripts, WAL files. Safe to run against a
447	live daemon.	447	live daemon.
448		448
449	=--verify= reads an archive back: the snapshot must pass SQLite's	449	=--verify= reads an archive back: the snapshot must pass SQLite's
450	integrity check, and every repository the snapshot names must be in the	450	integrity check, every repository the snapshot names must be in the
451	archive. A database-only archive is checked for integrity and says so.	451	archive, and each must pass =git fsck --connectivity-only=. It
452	Exit is non-zero on damage or a missing repository.	452	extracts the repositories to a temporary directory for that, so it
		453	needs free space the size of the repositories. A database-only archive
		454	is checked for integrity and says so. Exit is non-zero on damage, a
		455	missing repository or a missing object.
		456
		457	A full backup holds =<root>/backup.lock= from its database snapshot to
		458	its last repository. While it runs, =repo delete=, =repo rename=,
		459	=repo transfer=, =admin repo delete= and =org rename= refuse with "a
		460	backup is running"; retry when it finishes. Database-only backups take
		461	no lock.
		462
		463	Verifying an archive of unknown origin is safe as an unprivileged
		464	user: the connectivity check runs git against the archived
		465	repositories' own extracted copies, nothing under =server.root=.
		466
		467	Archives carry a directory entry for every directory, including an
		468	empty one, so a bare repository whose refs are all packed restores as
		469	a repository. Archives written before this release do not: extracting
		470	one can leave a repository's =refs/= directory missing, which stops
		471	git from recognizing it as a repository at all. =gitbayd admin backup
		472	--verify <archive>= names the repositories this affects; the fix is
		473	=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
		474	which it opens normally.
453		475
454	With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=	476	With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
455	and =--verify= needs the private key:	477	and =--verify= needs the private key:
	@@ -566,6 +588,30 @@ gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
566	- Push devices are looked up by the SHA-256 of their token	588	- Push devices are looked up by the SHA-256 of their token
567	(=push_devices.token_hash=), since two seals of one token differ.	589	(=push_devices.token_hash=), since two seals of one token differ.
568		590
		591	** Restore drill
		592
		593	A restore onto a clean host, run quarterly and after any change to the
		594	backup code (=cmd/gitbayd/backup.go=, the offsite job), and recorded
		595	below. The disaster it rehearses is losing bay1, so the local archives
		596	are gone with it and the sources are the main offsite restic
		597	repository (repositories, LFS, the staged database, =config.toml=),
		598	the keys repository (=secret.key=, =apns.p8=), and the operator's
		599	password manager (=offsite.env=, the keys repository's password and
		600	token, =backup-identity.txt=). The steps are in the data-at-rest
		601	plan's operator runbook
		602	(=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
		603
		604	Time to service runs from the clean host's first root login to the
		605	first successful =git clone= over SSH from it. The recovery point is
		606	the time of the newest restic snapshot restored.
		607
		608	No drill has been run yet; the procedure above is written but
		609	unexercised, and #259 stays open until the first row below is
		610	recorded.
		611
		612	\| Date \| Host \| Snapshot restored (UTC) \| Time to service \| DB integrity \| Connectivity \| LFS \| Release assets \| Host key \| Secrets \| Notes \|
		613	\|------+------+-------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------\|
		614
569	* Upgrades	615	* Upgrades
570		616
571	Replace the binary, restart the unit. Migrations apply automatically and	617	Replace the binary, restart the unit. Migrations apply automatically and

.gitbay/wiki/Architecture/08-Operations.org +6 −5

   (=cmd/gitbayd/backup.go=).
 - Excluded: WAL files, the hook socket, askpass scripts, generated
   hooks.
-- =gitbayd admin backup --verify= checks SQLite integrity and that every
+- =gitbayd admin backup --verify= checks SQLite integrity, that every
-  repository the database names is present (=backup.go=). It does
+  repository the database names is present, and =git fsck
-  not check git object connectivity.
+  --connectivity-only= on each (=backup.go=).
+- Repository deletes, renames and transfers refuse while a full backup
+  runs (=internal/backuplock=), so the snapshot and the walk agree.
 - The host's restic credentials are append-only; the key that can
   delete or prune snapshots is held off the host, so a compromised host
   cannot destroy its own history (documented: Admin wiki).
 - Recovery point: about one hour for database-only data (issues, merge
   requests, reviews), one day for repositories.
-- Recovery time: not measured. No restore onto a clean host has been
+- Recovery time: see the Admin wiki's Restore drill table.
-  recorded (#259).
 Restore procedure: extract the archive into an empty directory, point
 =server.root= at it, start =gitbayd=; hooks regenerate and the host key

.gitbay/wiki/Architecture/10-Known-Gaps.org +1 −1

	@@ -10,7 +10,7 @@ what the 2026-09-27 review found; remove a row when its issue closes.
10		10
11	\| Issue \| Area \| Gap \| Severity \|	11	\| Issue \| Area \| Gap \| Severity \|
12	\|-------+------------------+-----------------------------------------------------------------------+----------\|	12	\|-------+------------------+-----------------------------------------------------------------------+----------\|
13	\| #259 \| Recovery \| No restore has been exercised; verification does not check git connectivity \| high \|	13	\| #259 \| Recovery \| No restore has been exercised; the drill is written (Admin wiki) and not yet run \| high \|
14	\| #260 \| CI network \| Builds share the runner's source address; no egress policy \| medium \|	14	\| #260 \| CI network \| Builds share the runner's source address; no egress policy \| medium \|
15	\| #261 \| Various \| Migration foreign-key check after commit; three web writes bypass dispatch; documentation drift \| medium \|	15	\| #261 \| Various \| Migration foreign-key check after commit; three web writes bypass dispatch; documentation drift \| medium \|
16	\| #262 \| Availability \| No limit on concurrent git pack generation \| high \|	16	\| #262 \| Availability \| No limit on concurrent git pack generation \| high \|

.gitbay/wiki/Threat-Model.org +3 −3

	@@ -262,9 +262,9 @@ assume has been checked.
262	pixel against a viewer. A profile is the wider surface of the two: it	262	pixel against a viewer. A profile is the wider surface of the two: it
263	is linked from every commit and issue its owner touches. Documented;	263	is linked from every commit and issue its owner touches. Documented;
264	proxying is future work.	264	proxying is future work.
265	- Backups are consistent per the DB-snapshot-first ordering but are not a	265	- Backups are consistent per the DB-snapshot-first ordering, and
266	single atomic snapshot; a few orphaned git objects are possible and	266	repository deletes and moves wait out a full backup; a push during
267	harmless (see [[Admin]]).	267	one leaves only unreferenced objects (see [[Admin]]).
268	- The audit log lives in the database the daemon writes, so anyone with	268	- The audit log lives in the database the daemon writes, so anyone with
269	the daemon user's access can change it. The hash chain makes an edited	269	the daemon user's access can change it. The hash chain makes an edited
270	or removed row show as a break under =gitbayd admin audit verify=,	270	or removed row show as a break under =gitbayd admin audit verify=,

CHANGELOG.org +12

	@@ -180,6 +180,18 @@ secret.
180	- =gitbayd admin backup= encrypts archives to =[backup] age_recipients=	180	- =gitbayd admin backup= encrypts archives to =[backup] age_recipients=
181	when set (#274); =--verify= takes =--identity <file>=. Archive names	181	when set (#274); =--verify= takes =--identity <file>=. Archive names
182	gain =.age=; the shipped backup scripts and monitor match both.	182	gain =.age=; the shipped backup scripts and monitor match both.
		183	- A full backup holds =<root>/backup.lock= from its database snapshot
		184	to its last repository; =repo delete=, =repo rename=, =repo
		185	transfer=, =admin repo delete= and =org rename= refuse with "a
		186	backup is running" while it runs. =--verify= now also runs =git
		187	fsck --connectivity-only= on each archived repository and names any
		188	that fail. (#259)
		189	Upgrade note. Archives now carry a directory entry for every
		190	directory, so a bare repository whose refs are all packed restores as
		191	a repository. An archive written before this release lacks those
		192	entries; if extracting one leaves a repository's =refs/= directory
		193	missing, =gitbayd admin backup --verify <archive>= names it, and
		194	=mkdir -p <root>/repos/<owner>/<name>.git/refs= fixes it.
183		195
184	* v1.36.0 — 2026-09-23	196	* v1.36.0 — 2026-09-23
185		197

Commit 55cbdaf81e

Commit `55cbdaf81e`