Commit `6f5a52ae63`

6f5a52ae63074a9989129b53f65b6b7cb3dd9cbc

parent: a7c9352033

Verified · cmc ci/build: success ci/test: skipped

cmc <hello@cleberg.net> · 2026-09-18 03:54 UTC

wiki: offsite restic copies and the snapshot rewrite procedure

Admin page: where the offsite snapshots go, which key can remove them
and why it is not on bay1, and how to take one repository out of every
snapshot with restic rewrite --forget followed by prune, including the
renamed-repository case.

Ref #227

.gitbay/wiki/Admin.org +46 −1

		@@ -295,7 +295,8 @@ MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
295	295	reviews and head sha; =mr diff= and the MR page say the head is gone.
296	296	Run it when nothing is pushing to that repository: without the grace, a
297	297	push caught between leaving quarantine and writing its ref loses its
298		objects. Objects also survive in offsite backups until those are pruned.
	298	objects. Objects also survive in offsite backups until those are
	299	rewritten; see "Removing a repository's history from every snapshot".
299	300
300	301	=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
301	302	weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
		@@ -347,6 +348,50 @@ have. Consistency between the two halves is worth more here than latency
347	348	on one of them. Revisit if repository replication becomes continuous
348	349	too.
349	350
	351	** Offsite copies
	352
	353	bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
	354	=/var/lib/gitbay-stage= (the staged database copy) to an S3 bucket at
	355	Scaleway, with a key that can only add snapshots. The key that can
	356	remove them lives on the operator's machine, in
	357	=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
	358	cannot destroy its own history. Forgetting, pruning and rewriting all
	359	run from there.
	360
	361	*** Removing a repository's history from every snapshot
	362
	363	A history rewrite plus =admin mr prune= takes commits off the server,
	364	but every snapshot taken before it still holds them, and the retention
	365	window is the only thing that ages them out. To remove them now,
	366	rewrite the snapshots without that repository rather than forgetting
	367	the snapshots: everything else in them stays restorable. The next
	368	nightly run adds the repository back in its current state.
	369
	370	Repositories are stored under the name they had on disk when each
	371	snapshot was taken, so a renamed repository needs every name it has
	372	carried. Check what an older snapshot holds before choosing the paths:
	373
	374	#+begin_src sh
	375	set -a; . ~/.config/gitbay/offsite.env; set +a
	376	restic $RESTIC_OPTS snapshots
	377	restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
	378	#+end_src
	379
	380	Then dry-run, apply, prune, and confirm nothing matches:
	381
	382	#+begin_src sh
	383	EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
	384	restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
	385	restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
	386	restic $RESTIC_OPTS prune # drops the data nothing references
	387	restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
	388	#+end_src
	389
	390	=--forget= is what makes the originals go; without it the rewritten
	391	snapshots sit beside them and the data stays referenced. Snapshot IDs
	392	change; their times do not. Done for krz/keycask (formerly rust-pass)
	393	on 2026-09-18, across 22 snapshots.
	394
350	395	* Upgrades
351	396
352	397	Replace the binary, restart the unit. Migrations apply automatically and

Commit 6f5a52ae63

Commit `6f5a52ae63`