Commit `504b4d488c`

504b4d488c600b733a3dc0cea6a8f485f83ff9a6

parent: a27671383a

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-29 03:30 UTC

wiki, changelog: restore-drill, what a drill restores and checks

Ref #259

Layout: unified · split

.gitbay/wiki/Admin.org +66 −14

	@@ -660,28 +660,80 @@ gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
660		660
661	** Restore drill	661	** Restore drill
662		662
663	A restore onto a clean host, run quarterly and after any change to the	663	A restore onto a clean host, run quarterly (January, April, July,
664	backup code (=cmd/gitbayd/backup.go=, the offsite job), and recorded	664	October) and after any change to the backup code
665	below. The disaster it rehearses is losing bay1, so the local archives	665	(=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite
666	are gone with it and the sources are the main offsite restic	666	job), and recorded below. The disaster it rehearses is losing bay1, so
667	repository (repositories, LFS, the staged database, =config.toml=),	667	the local archives are gone with it and the sources are the main
668	the off-host copy of =secret.key= and =apns.p8= (a keys repository once	668	offsite restic repository (repositories, LFS, the staged database,
669	runbook D creates it; until then the operator's hand-made copy), and	669	=config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys
670	the operator's password manager (=offsite.env=, the keys repository's	670	repository once runbook D creates it; until then the operator's
671	password and token once it exists, =backup-identity.txt=). The steps are in the data-at-rest	671	hand-made copy), and the operator's password manager (=offsite.env=,
672	plan's operator runbook	672	the keys repository's password and token once it exists,
673	(=docs/plans/2026-09-27-data-at-rest-and-backup.md=).	673	=backup-identity.txt=). The full steps are runbook C of the
		674	data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
		675
		676	=gitbayd admin restore-drill= does the archive half. It extracts a
		677	full archive into an empty or absent directory, runs every =--verify=
		678	check on the extracted copy, and prints what was restored, the newest
		679	issue, issue comment, merge request comment and push in the restored
		680	database, and the elapsed time. Exit is non-zero if any check fails.
		681
		682	#+begin_src sh
		683	gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
		684	gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill
		685	#+end_src
		686
		687	What to restore, and from where:
		688
		689	\| Item \| Source \| Path on the drill host \|
		690	\|-------------------------------+-------------------------------------------------------------+------------------------------------------------\|
		691	\| Database \| staged copy in =/var/lib/gitbay-stage= (restic), or an archive \| =<root>/gitbay.db= \|
		692	\| Repositories \| =/var/lib/gitbay/repos= (restic), or an archive \| =<root>/repos= \|
		693	\| LFS objects \| =/var/lib/gitbay/lfs= (restic), or an archive \| =<root>/lfs= (or =[lfs] root=) \|
		694	\| Release assets \| inside each repository (=gitbay-releases/=) \| with the repositories \|
		695	\| Host keys \| =/var/lib/gitbay/ssh= (restic), or an archive \| =<root>/ssh= (or =[ssh] host_keys=) \|
		696	\| =config.toml= \| =/var/lib/gitbay-stage/config.toml= (restic) \| =/etc/gitbay/config.toml= \|
		697	\| =secret.key=, =apns.p8= \| the off-host copy; no archive or main snapshot carries them \| =/etc/gitbay/=, mode 0600, owned by =gitbay= \|
		698
		699	From the offsite path (restic is not run by any gitbay command):
		700
		701	#+begin_src sh
		702	restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
		703	cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
		704	gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
		705	gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root
		706	#+end_src
		707
		708	The second archive is how the restic tree gets the same checks and
		709	timestamps; =/tmp/drill-root= is discarded afterwards.
		710
		711	What to check, each a column below:
		712
		713	- DB integrity, connectivity, release assets, LFS: =restore-drill=
		714	prints =integrity ok=, =connectivity ok on N repositories=, =release
		715	assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin
		716	stats --json= on the source at the snapshot time.
		717	- Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets
		718	check= opens every value.
		719	- Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's
		720	fingerprint.
		721	- Config: =gitbayd --config /etc/gitbay/config.toml check-config=.
		722	- Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over
		723	SSH succeed.
674		724
675	Time to service runs from the clean host's first root login to the	725	Time to service runs from the clean host's first root login to the
676	first successful =git clone= over SSH from it. The recovery point is	726	first successful =git clone= over SSH from it. The recovery point is
677	the time of the newest restic snapshot restored.	727	the time of the newest restic snapshot restored; record beside it the
		728	newest issue, comment and push =restore-drill= printed, which show how
		729	much activity the restore carries.
678		730
679	No drill has been run yet; the procedure above is written but	731	No drill has been run yet; the procedure above is written but
680	unexercised, and #259 stays open until the first row below is	732	unexercised, and #259 stays open until the first row below is
681	recorded.	733	recorded.
682		734
683	\| Date \| Host \| Snapshot restored (UTC) \| Time to service \| DB integrity \| Connectivity \| LFS \| Release assets \| Host key \| Secrets \| Notes \|	735	\| Date \| Host \| Snapshot restored (UTC) \| Newest issue / comment / push \| Time to service \| DB integrity \| Connectivity \| LFS \| Release assets \| Host key \| Secrets \| Notes \|
684	\|------+------+-------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------\|	736	\|------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------\|
685		737
686	* Upgrades	738	* Upgrades
687		739

.gitbay/wiki/Architecture/08-Operations.org +6 −2

 - Excluded: WAL files, the hook socket, askpass scripts, generated
   hooks.
 - =gitbayd admin backup --verify= checks SQLite integrity, that every
-  repository the database names is present, and =git fsck
+  repository the database names is present, =git fsck
-  --connectivity-only= on each (=backup.go=).
+  --connectivity-only= on each, release assets against their recorded
+  sha256, and LFS objects against their names (=backup.go=).
+  =gitbayd admin restore-drill= runs the same checks on a full
+  extraction and reports elapsed time and the newest recovered
+  activity (=restoredrill.go=).
 - Repository deletes, renames and transfers refuse while a full backup
   runs (=internal/backuplock=), so the snapshot and the walk agree.
 - The host's restic credentials are append-only; the key that can

.gitbay/wiki/Architecture/09-Controls.org +1 −1

	@@ -99,6 +99,6 @@ chapter names of OWASP ASVS 4.0 where one fits.
99	\| Concurrency limit on git pack generation \| in place \| global, per-principal, bounded queue across SSH, HTTP and git:// (=internal/packlimit=); not in system SSH mode \|	99	\| Concurrency limit on git pack generation \| in place \| global, per-principal, bounded queue across SSH, HTTP and git:// (=internal/packlimit=); not in system SSH mode \|
100	\| Service hardening \| in place \| systemd sandboxing ([[file:03-Deployment.org][3]]) \|	100	\| Service hardening \| in place \| systemd sandboxing ([[file:03-Deployment.org][3]]) \|
101	\| Backups offsite and append-only \| in place \| restic with append-only credentials (documented) \|	101	\| Backups offsite and append-only \| in place \| restic with append-only credentials (documented) \|
102	\| Restore tested \| gap \| #259 \|	102	\| Restore tested \| gap \| tooling in place (=admin restore-drill=, Admin wiki "Restore drill"); clean-host drill pending (#259) \|
103	\| Migrations validated before commit \| gap \| foreign-key check runs after commit (#261) \|	103	\| Migrations validated before commit \| gap \| foreign-key check runs after commit (#261) \|
104	\| Signed, reviewed changes to production \| in place \| signed commits, =require-mr=, ff-only merges, clean-tree deploys \|	104	\| Signed, reviewed changes to production \| in place \| signed commits, =require-mr=, ff-only merges, clean-tree deploys \|

.gitbay/wiki/Architecture/10-Known-Gaps.org +1 −1

	@@ -10,7 +10,7 @@ what the 2026-09-27 review found; remove a row when its issue closes.
10		10
11	\| Issue \| Area \| Gap \| Severity \|	11	\| Issue \| Area \| Gap \| Severity \|
12	\|-------+------------------+-----------------------------------------------------------------------+----------\|	12	\|-------+------------------+-----------------------------------------------------------------------+----------\|
13	\| #259 \| Recovery \| No restore has been exercised; the drill is written (Admin wiki) and not yet run \| high \|	13	\| #259 \| Recovery \| No restore has been exercised; the procedure and tooling (=admin restore-drill=, =backup --verify=) are in place, the clean-host drill is pending \| high \|
14	\| #260 \| CI network \| Builds share the runner's source address; no egress policy \| medium \|	14	\| #260 \| CI network \| Builds share the runner's source address; no egress policy \| medium \|
15	\| #261 \| Various \| Migration foreign-key check after commit; three web writes bypass dispatch; documentation drift \| medium \|	15	\| #261 \| Various \| Migration foreign-key check after commit; three web writes bypass dispatch; documentation drift \| medium \|
16		16

CHANGELOG.org +12

	@@ -4,6 +4,18 @@ Versioning follows semver from v0.1.0. Database migrations run
4	automatically on daemon start; upgrade notes appear per release when	4	automatically on daemon start; upgrade notes appear per release when
5	anything beyond "replace the binary and restart" is needed.	5	anything beyond "replace the binary and restart" is needed.
6		6
		7	* Unreleased
		8
		9	- =gitbayd admin backup --verify= also checks every release asset the
		10	database names against its recorded size and sha256, and every
		11	archived LFS object against its name (#259).
		12	- =gitbayd admin restore-drill <archive> --into <dir>= extracts a full
		13	archive into an empty directory, runs the =--verify= checks on the
		14	extracted copy, and prints the newest issue, comment and push it
		15	recovered and the elapsed time. The Admin wiki's Restore drill
		16	section lists what to restore, what to check and where to record it
		17	(#259).
		18
7	* v1.37.0 — 2026-09-29	19	* v1.37.0 — 2026-09-29
8		20
9	Findings from the 2026-09-27 architecture review.	21	Findings from the 2026-09-27 architecture review.

Commit 504b4d488c

Commit `504b4d488c`