| @@ -660,28 +660,80 @@ gitbayd admin secrets rotate # new key, reseal, retire the old one (as root) |
| 660 | 660 | |
| 661 | 661 | ** Restore drill |
| 662 | 662 | |
| 663 | | A restore onto a clean host, run quarterly and after any change to the |
| 664 | | backup code (=cmd/gitbayd/backup.go=, the offsite job), and recorded |
| 665 | | below. The disaster it rehearses is losing bay1, so the local archives |
| 666 | | are gone with it and the sources are the main offsite restic |
| 667 | | repository (repositories, LFS, the staged database, =config.toml=), |
| 668 | | the off-host copy of =secret.key= and =apns.p8= (a keys repository once |
| 669 | | runbook D creates it; until then the operator's hand-made copy), and |
| 670 | | the operator's password manager (=offsite.env=, the keys repository's |
| 671 | | password and token once it exists, =backup-identity.txt=). The steps are in the data-at-rest |
| 672 | | plan's operator runbook |
| 673 | | (=docs/plans/2026-09-27-data-at-rest-and-backup.md=). |
| 663 | A restore onto a clean host, run quarterly (January, April, July, |
| 664 | October) and after any change to the backup code |
| 665 | (=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite |
| 666 | job), and recorded below. The disaster it rehearses is losing bay1, so |
| 667 | the local archives are gone with it and the sources are the main |
| 668 | offsite restic repository (repositories, LFS, the staged database, |
| 669 | =config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys |
| 670 | repository once runbook D creates it; until then the operator's |
| 671 | hand-made copy), and the operator's password manager (=offsite.env=, |
| 672 | the keys repository's password and token once it exists, |
| 673 | =backup-identity.txt=). The full steps are runbook C of the |
| 674 | data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=). |
| 675 | |
| 676 | =gitbayd admin restore-drill= does the archive half. It extracts a |
| 677 | full archive into an empty or absent directory, runs every =--verify= |
| 678 | check on the extracted copy, and prints what was restored, the newest |
| 679 | issue, issue comment, merge request comment and push in the restored |
| 680 | database, and the elapsed time. Exit is non-zero if any check fails. |
| 681 | |
| 682 | #+begin_src sh |
| 683 | gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill |
| 684 | gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill |
| 685 | #+end_src |
| 686 | |
| 687 | What to restore, and from where: |
| 688 | |
| 689 | | Item | Source | Path on the drill host | |
| 690 | |-------------------------------+-------------------------------------------------------------+------------------------------------------------| |
| 691 | | Database | staged copy in =/var/lib/gitbay-stage= (restic), or an archive | =<root>/gitbay.db= | |
| 692 | | Repositories | =/var/lib/gitbay/repos= (restic), or an archive | =<root>/repos= | |
| 693 | | LFS objects | =/var/lib/gitbay/lfs= (restic), or an archive | =<root>/lfs= (or =[lfs] root=) | |
| 694 | | Release assets | inside each repository (=gitbay-releases/=) | with the repositories | |
| 695 | | Host keys | =/var/lib/gitbay/ssh= (restic), or an archive | =<root>/ssh= (or =[ssh] host_keys=) | |
| 696 | | =config.toml= | =/var/lib/gitbay-stage/config.toml= (restic) | =/etc/gitbay/config.toml= | |
| 697 | | =secret.key=, =apns.p8= | the off-host copy; no archive or main snapshot carries them | =/etc/gitbay/=, mode 0600, owned by =gitbay= | |
| 698 | |
| 699 | From the offsite path (restic is not run by any gitbay command): |
| 700 | |
| 701 | #+begin_src sh |
| 702 | restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage |
| 703 | cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db |
| 704 | gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz |
| 705 | gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root |
| 706 | #+end_src |
| 707 | |
| 708 | The second archive is how the restic tree gets the same checks and |
| 709 | timestamps; =/tmp/drill-root= is discarded afterwards. |
| 710 | |
| 711 | What to check, each a column below: |
| 712 | |
| 713 | - DB integrity, connectivity, release assets, LFS: =restore-drill= |
| 714 | prints =integrity ok=, =connectivity ok on N repositories=, =release |
| 715 | assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin |
| 716 | stats --json= on the source at the snapshot time. |
| 717 | - Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets |
| 718 | check= opens every value. |
| 719 | - Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's |
| 720 | fingerprint. |
| 721 | - Config: =gitbayd --config /etc/gitbay/config.toml check-config=. |
| 722 | - Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over |
| 723 | SSH succeed. |
| 674 | 724 | |
| 675 | 725 | Time to service runs from the clean host's first root login to the |
| 676 | 726 | first successful =git clone= over SSH from it. The recovery point is |
| 677 | | the time of the newest restic snapshot restored. |
| 727 | the time of the newest restic snapshot restored; record beside it the |
| 728 | newest issue, comment and push =restore-drill= printed, which show how |
| 729 | much activity the restore carries. |
| 678 | 730 | |
| 679 | 731 | No drill has been run yet; the procedure above is written but |
| 680 | 732 | unexercised, and #259 stays open until the first row below is |
| 681 | 733 | recorded. |
| 682 | 734 | |
| 683 | | | Date | Host | Snapshot restored (UTC) | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes | |
| 684 | | |------+------+-------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------| |
| 735 | | Date | Host | Snapshot restored (UTC) | Newest issue / comment / push | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes | |
| 736 | |------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------| |
| 685 | 737 | |
| 686 | 738 | * Upgrades |
| 687 | 739 | |