Commit 55cbdaf81e
Verified · cmc
Layout: unified · split
.gitbay/wiki/Admin.org +49 −3
| @@ -447,9 +447,31 @@ hook socket, regenerated hook scripts, WAL files. Safe to run against a | ||
| 447 | 447 | live daemon. |
| 448 | 448 | |
| 449 | 449 | =--verify= reads an archive back: the snapshot must pass SQLite's |
| 450 | integrity check, and every repository the snapshot names must be in the | |
| 451 | archive. A database-only archive is checked for integrity and says so. | |
| 452 | Exit is non-zero on damage or a missing repository. | |
| 450 | integrity check, every repository the snapshot names must be in the | |
| 451 | archive, and each must pass =git fsck --connectivity-only=. It | |
| 452 | extracts the repositories to a temporary directory for that, so it | |
| 453 | needs free space the size of the repositories. A database-only archive | |
| 454 | is checked for integrity and says so. Exit is non-zero on damage, a | |
| 455 | missing repository or a missing object. | |
| 456 | ||
| 457 | A full backup holds =<root>/backup.lock= from its database snapshot to | |
| 458 | its last repository. While it runs, =repo delete=, =repo rename=, | |
| 459 | =repo transfer=, =admin repo delete= and =org rename= refuse with "a | |
| 460 | backup is running"; retry when it finishes. Database-only backups take | |
| 461 | no lock. | |
| 462 | ||
| 463 | Verifying an archive of unknown origin is safe as an unprivileged | |
| 464 | user: the connectivity check runs git against the archived | |
| 465 | repositories' own extracted copies, nothing under =server.root=. | |
| 466 | ||
| 467 | Archives carry a directory entry for every directory, including an | |
| 468 | empty one, so a bare repository whose refs are all packed restores as | |
| 469 | a repository. Archives written before this release do not: extracting | |
| 470 | one can leave a repository's =refs/= directory missing, which stops | |
| 471 | git from recognizing it as a repository at all. =gitbayd admin backup | |
| 472 | --verify <archive>= names the repositories this affects; the fix is | |
| 473 | =mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after | |
| 474 | which it opens normally. | |
| 453 | 475 | |
| 454 | 476 | With =[backup] age_recipients= set the archive is =<name>.tar.gz.age= |
| 455 | 477 | and =--verify= needs the private key: |
| @@ -566,6 +588,30 @@ gitbayd admin secrets rotate # new key, reseal, retire the old one (as root) | ||
| 566 | 588 | - Push devices are looked up by the SHA-256 of their token |
| 567 | 589 | (=push_devices.token_hash=), since two seals of one token differ. |
| 568 | 590 | |
| 591 | ** Restore drill | |
| 592 | ||
| 593 | A restore onto a clean host, run quarterly and after any change to the | |
| 594 | backup code (=cmd/gitbayd/backup.go=, the offsite job), and recorded | |
| 595 | below. The disaster it rehearses is losing bay1, so the local archives | |
| 596 | are gone with it and the sources are the main offsite restic | |
| 597 | repository (repositories, LFS, the staged database, =config.toml=), | |
| 598 | the keys repository (=secret.key=, =apns.p8=), and the operator's | |
| 599 | password manager (=offsite.env=, the keys repository's password and | |
| 600 | token, =backup-identity.txt=). The steps are in the data-at-rest | |
| 601 | plan's operator runbook | |
| 602 | (=docs/plans/2026-09-27-data-at-rest-and-backup.md=). | |
| 603 | ||
| 604 | Time to service runs from the clean host's first root login to the | |
| 605 | first successful =git clone= over SSH from it. The recovery point is | |
| 606 | the time of the newest restic snapshot restored. | |
| 607 | ||
| 608 | No drill has been run yet; the procedure above is written but | |
| 609 | unexercised, and #259 stays open until the first row below is | |
| 610 | recorded. | |
| 611 | ||
| 612 | | Date | Host | Snapshot restored (UTC) | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes | | |
| 613 | |------+------+-------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------| | |
| 614 | ||
| 569 | 615 | * Upgrades |
| 570 | 616 | |
| 571 | 617 | Replace the binary, restart the unit. Migrations apply automatically and |
.gitbay/wiki/Architecture/08-Operations.org +6 −5
| @@ -57,16 +57,17 @@ the product activity feed, not an audit trail. | ||
| 57 | 57 | (=cmd/gitbayd/backup.go=). |
| 58 | 58 | - Excluded: WAL files, the hook socket, askpass scripts, generated |
| 59 | 59 | hooks. |
| 60 | - =gitbayd admin backup --verify= checks SQLite integrity and that every | |
| 61 | repository the database names is present (=backup.go=). It does | |
| 62 | not check git object connectivity. | |
| 60 | - =gitbayd admin backup --verify= checks SQLite integrity, that every | |
| 61 | repository the database names is present, and =git fsck | |
| 62 | --connectivity-only= on each (=backup.go=). | |
| 63 | - Repository deletes, renames and transfers refuse while a full backup | |
| 64 | runs (=internal/backuplock=), so the snapshot and the walk agree. | |
| 63 | 65 | - The host's restic credentials are append-only; the key that can |
| 64 | 66 | delete or prune snapshots is held off the host, so a compromised host |
| 65 | 67 | cannot destroy its own history (documented: Admin wiki). |
| 66 | 68 | - Recovery point: about one hour for database-only data (issues, merge |
| 67 | 69 | requests, reviews), one day for repositories. |
| 68 | - Recovery time: not measured. No restore onto a clean host has been | |
| 69 | recorded (#259). | |
| 70 | - Recovery time: see the Admin wiki's Restore drill table. | |
| 70 | 71 | |
| 71 | 72 | Restore procedure: extract the archive into an empty directory, point |
| 72 | 73 | =server.root= at it, start =gitbayd=; hooks regenerate and the host key |
.gitbay/wiki/Architecture/10-Known-Gaps.org +1 −1
| @@ -10,7 +10,7 @@ what the 2026-09-27 review found; remove a row when its issue closes. | ||
| 10 | 10 | |
| 11 | 11 | | Issue | Area | Gap | Severity | |
| 12 | 12 | |-------+------------------+-----------------------------------------------------------------------+----------| |
| 13 | | #259 | Recovery | No restore has been exercised; verification does not check git connectivity | high | | |
| 13 | | #259 | Recovery | No restore has been exercised; the drill is written (Admin wiki) and not yet run | high | | |
| 14 | 14 | | #260 | CI network | Builds share the runner's source address; no egress policy | medium | |
| 15 | 15 | | #261 | Various | Migration foreign-key check after commit; three web writes bypass dispatch; documentation drift | medium | |
| 16 | 16 | | #262 | Availability | No limit on concurrent git pack generation | high | |
.gitbay/wiki/Threat-Model.org +3 −3
| @@ -262,9 +262,9 @@ assume has been checked. | ||
| 262 | 262 | pixel against a viewer. A profile is the wider surface of the two: it |
| 263 | 263 | is linked from every commit and issue its owner touches. Documented; |
| 264 | 264 | proxying is future work. |
| 265 | - Backups are consistent per the DB-snapshot-first ordering but are not a | |
| 266 | single atomic snapshot; a few orphaned git objects are possible and | |
| 267 | harmless (see [[Admin]]). | |
| 265 | - Backups are consistent per the DB-snapshot-first ordering, and | |
| 266 | repository deletes and moves wait out a full backup; a push during | |
| 267 | one leaves only unreferenced objects (see [[Admin]]). | |
| 268 | 268 | - The audit log lives in the database the daemon writes, so anyone with |
| 269 | 269 | the daemon user's access can change it. The hash chain makes an edited |
| 270 | 270 | or removed row show as a break under =gitbayd admin audit verify=, |
CHANGELOG.org +12
| @@ -180,6 +180,18 @@ secret. | ||
| 180 | 180 | - =gitbayd admin backup= encrypts archives to =[backup] age_recipients= |
| 181 | 181 | when set (#274); =--verify= takes =--identity <file>=. Archive names |
| 182 | 182 | gain =.age=; the shipped backup scripts and monitor match both. |
| 183 | - A full backup holds =<root>/backup.lock= from its database snapshot | |
| 184 | to its last repository; =repo delete=, =repo rename=, =repo | |
| 185 | transfer=, =admin repo delete= and =org rename= refuse with "a | |
| 186 | backup is running" while it runs. =--verify= now also runs =git | |
| 187 | fsck --connectivity-only= on each archived repository and names any | |
| 188 | that fail. (#259) | |
| 189 | *Upgrade note.* Archives now carry a directory entry for every | |
| 190 | directory, so a bare repository whose refs are all packed restores as | |
| 191 | a repository. An archive written before this release lacks those | |
| 192 | entries; if extracting one leaves a repository's =refs/= directory | |
| 193 | missing, =gitbayd admin backup --verify <archive>= names it, and | |
| 194 | =mkdir -p <root>/repos/<owner>/<name>.git/refs= fixes it. | |
| 183 | 195 | |
| 184 | 196 | * v1.36.0 — 2026-09-23 |
| 185 | 197 | |