docs/plans/2026-09-27-data-at-rest-and-backup.md
3655 lines · 124875 bytes
1# Data at rest and backup implementation plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Seal the four secret columns under a key file outside the
6database (#273), encrypt backup archives to an age recipient (#274),
7and make `--verify` check git connectivity while repository moves and
8deletions wait for a running backup (#259). The operator runbook moves
9the offsite copy from Scaleway to Cloudflare R2 under bucket locks,
10adds a separate keys repository for `apns.p8` and the secret key file,
11and holds the restore drill, which is deferred.
12
13**Architecture:** A new `internal/seal` package holds AES-256-GCM keys
14read from `server.secret_key_file` (default `/etc/gitbay/secret.key`,
15mode 0600, outside `server.root`). The store seals on write and opens
16on read, so no caller above `internal/store` changes. Every value
17carries `gbs1:<key id>:`; `serve` seals leftover clear values and
18values under retired keys at startup, and `gitbayd admin secrets
19rotate` adds a key, reseals, and retires the old one. Backups wrap the
20tar.gz in `filippo.io/age` when `[backup] age_recipients` is set.
21`--verify` extracts repositories and runs `git fsck
22--connectivity-only` on each. A `flock(2)` on `<root>/backup.lock`
23keeps deletes, renames and transfers (daemon process) out of a full
24backup (separate `gitbayd admin backup` process).
25
26**Tech Stack:** Go 1.27, `crypto/aes` + `crypto/cipher` (GCM),
27`filippo.io/age` (new dependency), `syscall.Flock`, SQLite via
28`modernc.org/sqlite`, cobra.
29
30**Spec:** the issue texts of #273, #274 and #259 on krz/gitbay, and the
31decisions recorded in the brief: AES-GCM, key file under `/etc/gitbay`
32mode 0600 excluded from backups, key id prefix on each value, rotation
33command, re-encryption of existing rows; age recipients, `--verify`
34takes an identity file; the clean-host drill is an operator runbook
35recorded on the Admin wiki page, deferred by the operator (#259 stays
36open), repeated quarterly and after any backup code change; the
37offsite restic repository moves to Cloudflare R2, and `apns.p8` and
38`secret.key` go to a separate small restic repository on R2 with its
39own bucket and password.
40
41## Global Constraints
42
43- Each MR on its own branch off `main`. Commits are signed (the repo
44 refuses unsigned), messages reference issues (`Ref #N`, and
45 `Closes #N` on the commit that finishes one). No attribution to any
46 assistant, model or AI anywhere: commits, MR bodies, comments.
47- MR: `gitbay mr create --source <branch> --target main --title "..."`;
48 merge with `gitbay mr merge <n> --strategy ff` once CI is green, then
49 delete the branch locally and on the remote. Behind main → rebase,
50 force-push, merge again.
51- Locally: `go build ./...`, `go vet ./...`, unit tests of touched
52 packages, and at most the one e2e test being written
53 (`go test ./e2e -run TestName -count=1`). CI on bay1 runs the full suite.
54- Registries that fail CI when a new thing lacks its row: top-level route
55 word in `internal/policy/names.go`; new page template in the width map
56 of `TestMainWidthClass` (`internal/web/web_test.go`); new `ReadOnly`
57 command in `readArgs` in `e2e/readonly_test.go`; new control command
58 needs a `pass()` entry in `cmd/gitbay/main.go` (coverage test);
59 a command reading stdin needs `ReadsStdin: true`. This plan adds no
60 control command, route or template: `gitbayd admin secrets` is a
61 host-local cobra command like `admin backup` and `admin gc`, so none
62 of the registries gains a row.
63- Migrations: the highest today is 0059. This plan owns 0072–0074 and
64 uses one, `0072_push_token_hash`. Plans 1–3 own 0060–0071; whoever
65 lands second renumbers to the next free number at execution time
66 (`loadMigrations` in `internal/store/store.go` refuses a gap, so 0072
67 cannot land before 0060–0071 exist: rename it to the next free
68 number). Migrations come in `.up.sql`/`.down.sql` pairs. Hand-written
69 SQL, no ORM.
70- Secrets travel on stdin, never argv; never logged or echoed. The key
71 file's contents are never printed; commands print key ids only.
72- Wiki pages live in `.gitbay/wiki/`. Update the page in the same MR
73 that changes the behaviour it describes, and close the matching
74 Known-Gaps row (`.gitbay/wiki/Architecture/10-Known-Gaps.org`) and
75 controls-matrix row (`Architecture/09-Controls.org`).
76- Writing style: plain, direct, no hype; code comments match the
77 surrounding density. Comments and docs state facts, never
78 before/after narration.
79- Plan-specific:
80 - Sealed value format: `gbs1:<8 lowercase hex key id>:<base64 raw
81 std (nonce ‖ ciphertext ‖ tag)>`. Additional data is
82 `<table>.<column>`. An empty value is never sealed (empty means
83 "none" for `webhooks.secret` and `mirrors.token`).
84 - Key file format: one `<id> <base64 32 bytes>` per line, `#`
85 comments; the last key seals, all keys open. Mode must be 0600 or
86 stricter; anything group- or world-readable is refused.
87 - `server.secret_key_file` must not be inside `server.root`
88 (validation error), so neither the archive nor the main restic
89 snapshot of `/var/lib/gitbay` can carry it. Its offsite copy is the
90 separate keys repository (runbook D), never the main one.
91 - A missing key file is fatal for every process that opens the
92 database through `openStore` (serve, shell, authorized-keys, host
93 admin commands), with a message naming the path and
94 `gitbayd admin secrets init`. `gitbayd migrate` does not need it.
95 - Encrypted archives end in `.age`; `--out` without the suffix gets
96 it appended when `[backup] age_recipients` is set.
97
98## Order and dependencies
99
100| MR | Branch | Issue | Contents |
101|----|--------|-------|----------|
102| 1 | `secrets-at-rest` | Closes #273 | `internal/seal`, `server.secret_key_file`, store sealing, migration 0072, reseal at startup, `admin secrets init/rotate/check`, install.sh, e2e harness key, wiki |
103| 2 | `backup-age` | Closes #274 | `[backup] age_recipients`, age-wrapped archives, `--verify --identity`, backup script globs, wiki |
104| 3 | `backup-verify-lock` | Ref #259 | `gitutil.FsckConnectivity`, `--verify` extracts and checks each repository, `internal/backuplock`, delete/rename/transfer/org rename refused during a full backup, drill procedure and record table on the Admin page |
105| — | `offsite-r2-wiki` (operator) | Ref the R2 move issue (runbook D.1) | Admin, Threat-Model and Architecture pages describe R2 with bucket locks and the keys repository, after Scaleway is retired |
106| — | `restore-drill-record` (operator, deferred) | Closes #259 | the first drill's numbers in the Admin page, Known-Gaps and Controls rows closed |
107
108#259 stays open after MR 3: its commits say `Ref #259`, and only the
109drill record closes it. The operator has deferred the drill.
110
111The offsite job is not in the repository. `/usr/local/bin/gitbay-offsite`
112and its `gitbay-offsite.service`/`.timer` exist only on bay1;
113`deploy/cloud-init.yaml` does not template them (its
114`gitbay-backup.sh` carries only the comment "To ship offsite, add an
115rclone/s3 upload of $out here."). The move to R2 is therefore a
116runbook section (D) with the script edits written out, not a code MR.
117
118MR 2 and MR 3 both edit `cmd/gitbayd/backup.go`; land them in order.
119MR 2's `testConfig` helper comes from MR 1.
120
121Other plans: no hard dependency. Soft overlaps, resolved by rebase:
122plan 3 (#279) changes `internal/mirror/mirror.go`, which reads
123`store.Mirror.Token` — the field stays a plain string after this plan,
124so its code is unaffected. Plan 5 (#261) touches the migration runner
125in `internal/store/store.go`; migration 0072 does not use the
126`-- foreign_keys: off` directive.
127
128## File map
129
130| File | MR | Responsibility |
131|---|---|---|
132| `internal/seal/seal.go` (create) | 1 | key file read/write, `Keyring`, `Seal`/`Open`, `KeyID` |
133| `internal/seal/seal_test.go` (create) | 1 | round trip, AAD binding, reload on change, mode refusal |
134| `internal/config/config.go` | 1, 2 | `Server.SecretKeyFile`, `Backup.AgeRecipients`, validation |
135| `internal/store/secrets.go` (create) | 1 | `SetKeyring`, `sealValue`/`openValue`, `ResealSecrets`, `SecretKeyUse`, `tokenHash` |
136| `internal/store/store.go` | 1 | `secrets` field on `Store` |
137| `internal/store/cisecrets.go`, `webhooks.go`, `mirrors.go`, `push.go` | 1 | seal in the write transaction, open on read, token hash lookups |
138| `internal/store/migrations/0072_push_token_hash.{up,down}.sql` (create) | 1 | `push_devices.token_hash` + unique index |
139| `cmd/gitbayd/main.go` | 1 | `openStore` loads the keyring; `serve` reseals; `admin secrets` wired |
140| `cmd/gitbayd/secrets.go` (create) | 1 | `admin secrets init|rotate|check` |
141| `cmd/gitbayd/testconfig_test.go` (create) | 1 | `testConfig` helper |
142| `cmd/gitbayd/backup.go` | 2, 3 | age wrap, `--identity`, lock, connectivity |
143| `internal/gitutil/merge.go` | 3 | `FsckConnectivity` |
144| `internal/backuplock/backuplock.go` (create) | 3 | `Hold`, `TryShared`, `ErrBusy` |
145| `internal/control/repo.go`, `org.go` | 3 | `holdOffBackup` in delete/rename/transfer/org rename |
146| `deploy/install.sh` | 1 | create the key file once |
147| `deploy/cloud-init.yaml` | 2 | backup script and monitor globs for `.age` |
148| `e2e/ssh_test.go`, `acme_test.go`, `system_test.go`, `backup_test.go` | 1, 3 | key file in every config; sealed-at-rest and connectivity checks |
149| `.gitbay/wiki/Admin.org`, `Threat-Model.org`, `Architecture/03,06,08,09,10` | 1–3 | docs |
150| `CHANGELOG.org` | 1, 2 | upgrade notes |
151
152---
153
154# MR 1: secrets at rest (branch `secrets-at-rest`, closes #273)
155
156### Task 1.1: `internal/seal`
157
158**Files:**
159- Create: `internal/seal/seal.go`
160- Test: `internal/seal/seal_test.go`
161
162**Interfaces:**
163- Produces:
164 - `const Prefix = "gbs1:"`
165 - `type Key struct { ID string; Secret []byte }`
166 - `func NewKey() (Key, error)`
167 - `func ReadKeys(path string) ([]Key, error)` — refuses a mode with any group/other bit
168 - `func WriteKeys(path string, keys []Key) error` — atomic, 0600, keeps an existing file's owner
169 - `type Keyring`; `func Load(path string) (*Keyring, error)`
170 - `func (k *Keyring) Seal(aad, plain string) (string, error)`
171 - `func (k *Keyring) Open(aad, sealed string) (string, error)`
172 - `func (k *Keyring) CurrentID() (string, error)`
173 - `func IsSealed(v string) bool`, `func KeyID(v string) (string, bool)`
174
175- [ ] **Step 1: Write the failing tests**
176
177```go
178package seal
179
180import (
181 "os"
182 "path/filepath"
183 "strings"
184 "testing"
185)
186
187func keyFile(t *testing.T, keys ...Key) string {
188 t.Helper()
189 path := filepath.Join(t.TempDir(), "secret.key")
190 if err := WriteKeys(path, keys); err != nil {
191 t.Fatal(err)
192 }
193 return path
194}
195
196func newKey(t *testing.T) Key {
197 t.Helper()
198 k, err := NewKey()
199 if err != nil {
200 t.Fatal(err)
201 }
202 return k
203}
204
205func TestSealOpenRoundTrip(t *testing.T) {
206 k := newKey(t)
207 ring, err := Load(keyFile(t, k))
208 if err != nil {
209 t.Fatal(err)
210 }
211 v, err := ring.Seal("build_secrets.value", "hunter2")
212 if err != nil {
213 t.Fatal(err)
214 }
215 if !strings.HasPrefix(v, Prefix+k.ID+":") || strings.Contains(v, "hunter2") {
216 t.Fatalf("sealed value %q", v)
217 }
218 if id, ok := KeyID(v); !ok || id != k.ID {
219 t.Fatalf("KeyID = %q, %v", id, ok)
220 }
221 got, err := ring.Open("build_secrets.value", v)
222 if err != nil || got != "hunter2" {
223 t.Fatalf("Open = %q, %v", got, err)
224 }
225 // Two seals of one value differ: the nonce is random.
226 if w, _ := ring.Seal("build_secrets.value", "hunter2"); w == v {
227 t.Fatal("two seals produced the same value")
228 }
229}
230
231// A value moved to another column does not open there.
232func TestOpenChecksAdditionalData(t *testing.T) {
233 ring, err := Load(keyFile(t, newKey(t)))
234 if err != nil {
235 t.Fatal(err)
236 }
237 v, _ := ring.Seal("mirrors.token", "tok")
238 if _, err := ring.Open("webhooks.secret", v); err == nil {
239 t.Fatal("opened under the wrong column")
240 }
241}
242
243// A running daemon sees a rotation without a restart: the ring re-reads
244// the file when it changes.
245func TestKeyringFollowsTheFile(t *testing.T) {
246 old, next := newKey(t), newKey(t)
247 path := keyFile(t, old)
248 ring, err := Load(path)
249 if err != nil {
250 t.Fatal(err)
251 }
252 before, _ := ring.Seal("webhooks.secret", "s")
253 if err := WriteKeys(path, []Key{old, next}); err != nil {
254 t.Fatal(err)
255 }
256 after, err := ring.Seal("webhooks.secret", "s")
257 if err != nil {
258 t.Fatal(err)
259 }
260 if id, _ := KeyID(after); id != next.ID {
261 t.Fatalf("sealed under %s after rotation, want %s", id, next.ID)
262 }
263 if got, err := ring.Open("webhooks.secret", before); err != nil || got != "s" {
264 t.Fatalf("old value after rotation: %q, %v", got, err)
265 }
266 if err := WriteKeys(path, []Key{next}); err != nil {
267 t.Fatal(err)
268 }
269 if _, err := ring.Open("webhooks.secret", before); err == nil || !strings.Contains(err.Error(), old.ID) {
270 t.Fatalf("a retired key's value opened, or the error does not name the key: %v", err)
271 }
272}
273
274func TestReadKeysRefusesAReadableFile(t *testing.T) {
275 path := keyFile(t, newKey(t))
276 if err := os.Chmod(path, 0o640); err != nil {
277 t.Fatal(err)
278 }
279 if _, err := ReadKeys(path); err == nil || !strings.Contains(err.Error(), "0600") {
280 t.Fatalf("group-readable key file: %v", err)
281 }
282}
283
284func TestWriteKeysMode(t *testing.T) {
285 path := keyFile(t, newKey(t))
286 fi, err := os.Stat(path)
287 if err != nil {
288 t.Fatal(err)
289 }
290 if fi.Mode().Perm() != 0o600 {
291 t.Fatalf("mode %04o", fi.Mode().Perm())
292 }
293}
294
295func TestReadKeysRejectsMalformedLines(t *testing.T) {
296 for _, body := range []string{
297 "",
298 "# only a comment\n",
299 "XYZ12345 AAAA\n",
300 "0123abcd bm90IDMyIGJ5dGVz\n",
301 } {
302 path := filepath.Join(t.TempDir(), "k")
303 if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
304 t.Fatal(err)
305 }
306 if _, err := ReadKeys(path); err == nil {
307 t.Errorf("accepted %q", body)
308 }
309 }
310}
311```
312
313- [ ] **Step 2: Run the tests to verify they fail**
314
315Run: `go test ./internal/seal/ -count=1`
316Expected: FAIL, package does not compile (`undefined: WriteKeys`).
317
318- [ ] **Step 3: Write the implementation**
319
320```go
321// Package seal encrypts the secret columns of the database with
322// AES-256-GCM under keys held in a file outside the database and outside
323// server.root, so neither a copy of the database nor a backup opens them
324// (#273).
325package seal
326
327import (
328 "bufio"
329 "bytes"
330 "crypto/aes"
331 "crypto/cipher"
332 "crypto/rand"
333 "encoding/base64"
334 "encoding/hex"
335 "errors"
336 "fmt"
337 "os"
338 "path/filepath"
339 "strings"
340 "sync"
341 "syscall"
342)
343
344// Prefix marks a sealed value: "gbs1:<key id>:<base64 nonce||ciphertext>".
345const Prefix = "gbs1:"
346
347// Key is one line of the key file.
348type Key struct {
349 ID string // 8 lowercase hex characters
350 Secret []byte // 32 bytes
351}
352
353// NewKey returns a key with a random id and secret.
354func NewKey() (Key, error) {
355 id := make([]byte, 4)
356 secret := make([]byte, 32)
357 if _, err := rand.Read(id); err != nil {
358 return Key{}, err
359 }
360 if _, err := rand.Read(secret); err != nil {
361 return Key{}, err
362 }
363 return Key{ID: hex.EncodeToString(id), Secret: secret}, nil
364}
365
366// ReadKeys reads the key file. The last key seals; every key opens.
367func ReadKeys(path string) ([]Key, error) {
368 fi, err := os.Stat(path)
369 if err != nil {
370 return nil, err
371 }
372 if perm := fi.Mode().Perm(); perm&0o077 != 0 {
373 return nil, fmt.Errorf("%s is mode %04o; it must be readable by its owner alone (0600)", path, perm)
374 }
375 data, err := os.ReadFile(path)
376 if err != nil {
377 return nil, err
378 }
379 var keys []Key
380 seen := map[string]bool{}
381 sc := bufio.NewScanner(bytes.NewReader(data))
382 for n := 1; sc.Scan(); n++ {
383 line := strings.TrimSpace(sc.Text())
384 if line == "" || strings.HasPrefix(line, "#") {
385 continue
386 }
387 f := strings.Fields(line)
388 if len(f) != 2 || !validID(f[0]) {
389 return nil, fmt.Errorf("%s:%d: want \"<8 hex id> <base64 32-byte key>\"", path, n)
390 }
391 secret, err := base64.StdEncoding.DecodeString(f[1])
392 if err != nil || len(secret) != 32 {
393 return nil, fmt.Errorf("%s:%d: key is not 32 bytes of base64", path, n)
394 }
395 if seen[f[0]] {
396 return nil, fmt.Errorf("%s:%d: key id %s appears twice", path, n, f[0])
397 }
398 seen[f[0]] = true
399 keys = append(keys, Key{ID: f[0], Secret: secret})
400 }
401 if err := sc.Err(); err != nil {
402 return nil, err
403 }
404 if len(keys) == 0 {
405 return nil, fmt.Errorf("%s holds no keys", path)
406 }
407 return keys, nil
408}
409
410// WriteKeys replaces the key file: a temporary file in the same
411// directory, mode 0600, given the existing file's owner when there is
412// one (rotation runs as root; the daemon reads the file as its own
413// user), then renamed over it.
414func WriteKeys(path string, keys []Key) error {
415 var b strings.Builder
416 b.WriteString("# gitbay secret keys, \"<id> <base64 key>\" per line. The last line seals\n")
417 b.WriteString("# new values; the others open values sealed before a rotation.\n")
418 b.WriteString("# Keep a copy off this host: backups do not carry this file.\n")
419 for _, k := range keys {
420 fmt.Fprintf(&b, "%s %s\n", k.ID, base64.StdEncoding.EncodeToString(k.Secret))
421 }
422 tmp, err := os.CreateTemp(filepath.Dir(path), ".secret-key-*")
423 if err != nil {
424 return err
425 }
426 defer os.Remove(tmp.Name())
427 fail := func(err error) error {
428 tmp.Close()
429 return err
430 }
431 if err := tmp.Chmod(0o600); err != nil {
432 return fail(err)
433 }
434 if fi, err := os.Stat(path); err == nil {
435 if st, ok := fi.Sys().(*syscall.Stat_t); ok {
436 if err := tmp.Chown(int(st.Uid), int(st.Gid)); err != nil {
437 return fail(err)
438 }
439 }
440 }
441 if _, err := tmp.WriteString(b.String()); err != nil {
442 return fail(err)
443 }
444 if err := tmp.Sync(); err != nil {
445 return fail(err)
446 }
447 if err := tmp.Close(); err != nil {
448 return err
449 }
450 return os.Rename(tmp.Name(), path)
451}
452
453// Keyring is the loaded key file. It re-reads the file whenever the file
454// changes, so a running daemon follows a rotation without a restart.
455type Keyring struct {
456 path string
457
458 mu sync.Mutex
459 fi os.FileInfo
460 cur string
461 aead map[string]cipher.AEAD
462}
463
464func Load(path string) (*Keyring, error) {
465 k := &Keyring{path: path}
466 if err := k.refresh(); err != nil {
467 return nil, err
468 }
469 return k, nil
470}
471
472// refresh reloads the file unless it is the one last read. Callers hold k.mu.
473func (k *Keyring) refresh() error {
474 fi, err := os.Stat(k.path)
475 if err != nil {
476 return err
477 }
478 if k.fi != nil && os.SameFile(k.fi, fi) && fi.ModTime().Equal(k.fi.ModTime()) && fi.Size() == k.fi.Size() {
479 return nil
480 }
481 keys, err := ReadKeys(k.path)
482 if err != nil {
483 return err
484 }
485 aead := make(map[string]cipher.AEAD, len(keys))
486 for _, key := range keys {
487 block, err := aes.NewCipher(key.Secret)
488 if err != nil {
489 return err
490 }
491 g, err := cipher.NewGCM(block)
492 if err != nil {
493 return err
494 }
495 aead[key.ID] = g
496 }
497 k.fi, k.cur, k.aead = fi, keys[len(keys)-1].ID, aead
498 return nil
499}
500
501// CurrentID is the id of the key that seals new values.
502func (k *Keyring) CurrentID() (string, error) {
503 k.mu.Lock()
504 defer k.mu.Unlock()
505 if err := k.refresh(); err != nil {
506 return "", err
507 }
508 return k.cur, nil
509}
510
511// Seal encrypts plain under the current key. aad names the column, so a
512// value copied into another column does not open there.
513func (k *Keyring) Seal(aad, plain string) (string, error) {
514 k.mu.Lock()
515 defer k.mu.Unlock()
516 if err := k.refresh(); err != nil {
517 return "", err
518 }
519 g := k.aead[k.cur]
520 nonce := make([]byte, g.NonceSize())
521 if _, err := rand.Read(nonce); err != nil {
522 return "", err
523 }
524 ct := g.Seal(nonce, nonce, []byte(plain), []byte(aad))
525 return Prefix + k.cur + ":" + base64.RawStdEncoding.EncodeToString(ct), nil
526}
527
528// Open decrypts a value Seal produced under any key the file holds.
529func (k *Keyring) Open(aad, sealed string) (string, error) {
530 id, body, ok := split(sealed)
531 if !ok {
532 return "", errors.New("not a sealed value")
533 }
534 k.mu.Lock()
535 defer k.mu.Unlock()
536 if err := k.refresh(); err != nil {
537 return "", err
538 }
539 g, ok := k.aead[id]
540 if !ok {
541 return "", fmt.Errorf("sealed with key %s, which %s does not hold", id, k.path)
542 }
543 ct, err := base64.RawStdEncoding.DecodeString(body)
544 if err != nil || len(ct) < g.NonceSize() {
545 return "", fmt.Errorf("value sealed with key %s is malformed", id)
546 }
547 plain, err := g.Open(nil, ct[:g.NonceSize()], ct[g.NonceSize():], []byte(aad))
548 if err != nil {
549 return "", fmt.Errorf("value sealed with key %s does not open: wrong key or altered value", id)
550 }
551 return string(plain), nil
552}
553
554// IsSealed reports whether v carries the sealed prefix.
555func IsSealed(v string) bool { return strings.HasPrefix(v, Prefix) }
556
557// KeyID is the id of the key that sealed v.
558func KeyID(v string) (string, bool) {
559 id, _, ok := split(v)
560 return id, ok
561}
562
563func split(v string) (id, body string, ok bool) {
564 rest, ok := strings.CutPrefix(v, Prefix)
565 if !ok {
566 return "", "", false
567 }
568 id, body, ok = strings.Cut(rest, ":")
569 return id, body, ok && validID(id)
570}
571
572func validID(s string) bool {
573 if len(s) != 8 || strings.ToLower(s) != s {
574 return false
575 }
576 _, err := hex.DecodeString(s)
577 return err == nil
578}
579```
580
581- [ ] **Step 4: Run the tests to verify they pass**
582
583Run: `go test ./internal/seal/ -count=1 && go vet ./internal/seal/`
584Expected: PASS.
585
586- [ ] **Step 5: Commit**
587
588```bash
589git add internal/seal
590git commit -S -m "seal: AES-256-GCM keyring for secret columns
591
592Ref #273"
593```
594
595### Task 1.2: `server.secret_key_file`
596
597**Files:**
598- Modify: `internal/config/config.go:4-16` (imports), `:48-57` (`Server`), `:271-273` (`Default`), `:319-335` (`Validate`)
599- Test: `internal/config/config_test.go`
600
601**Interfaces:**
602- Produces: `Config.Server.SecretKeyFile string` (`toml:"secret_key_file"`), default `/etc/gitbay/secret.key`.
603
604- [ ] **Step 1: Write the failing test** (append to `config_test.go`)
605
606```go
607func TestSecretKeyFile(t *testing.T) {
608 cfg, err := Load(writeConfig(t, minimal))
609 if err != nil {
610 t.Fatal(err)
611 }
612 if cfg.Server.SecretKeyFile != "/etc/gitbay/secret.key" {
613 t.Errorf("default secret_key_file = %q", cfg.Server.SecretKeyFile)
614 }
615 for body, want := range map[string]string{
616 minimal + "secret_key_file = \"/var/lib/gitbay/secret.key\"\n": "inside server.root",
617 minimal + "secret_key_file = \"/var/lib/gitbay\"\n": "inside server.root",
618 minimal + "secret_key_file = \"\"\n": "server.secret_key_file is required",
619 } {
620 if _, err := Load(writeConfig(t, body)); err == nil || !strings.Contains(err.Error(), want) {
621 t.Errorf("%q: got %v, want an error containing %q", body, err, want)
622 }
623 }
624 if _, err := Load(writeConfig(t, minimal+"secret_key_file = \"/var/lib/gitbay-keys/secret.key\"\n")); err != nil {
625 t.Errorf("a sibling directory of the root is outside it: %v", err)
626 }
627}
628```
629
630- [ ] **Step 2: Run it to verify it fails**
631
632Run: `go test ./internal/config/ -run TestSecretKeyFile -count=1`
633Expected: FAIL, `unknown config key "server.secret_key_file"`.
634
635- [ ] **Step 3: Implement**
636
637Add `"path/filepath"` to the imports. In `Server`, after `SiteURL`:
638
639```go
640 // SecretKeyFile holds the keys that seal the secret columns of the
641 // database (internal/seal). It lives outside Root, so neither a
642 // backup archive nor a snapshot of Root carries it.
643 SecretKeyFile string `toml:"secret_key_file"`
644```
645
646In `Default()`:
647
648```go
649 Server: Server{Root: "/var/lib/gitbay", SecretKeyFile: "/etc/gitbay/secret.key"},
650```
651
652In `Validate()`, after the `server.site_url is required` check:
653
654```go
655 switch {
656 case c.Server.SecretKeyFile == "":
657 errs = append(errs, errors.New("server.secret_key_file is required"))
658 case within(c.Server.Root, c.Server.SecretKeyFile):
659 errs = append(errs, fmt.Errorf("server.secret_key_file %q is inside server.root: backups of the root would carry the key beside the values it seals", c.Server.SecretKeyFile))
660 }
661```
662
663And below `oneOf`:
664
665```go
666// within reports whether path is dir or below it.
667func within(dir, path string) bool {
668 rel, err := filepath.Rel(filepath.Clean(dir), filepath.Clean(path))
669 return err == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator))
670}
671```
672
673- [ ] **Step 4: Run the package tests**
674
675Run: `go test ./internal/config/ -count=1`
676Expected: PASS (existing tests use `minimal`, which gets the default).
677
678- [ ] **Step 5: Commit**
679
680```bash
681git add internal/config
682git commit -S -m "config: server.secret_key_file, outside server.root
683
684Ref #273"
685```
686
687### Task 1.3: the store seals and opens
688
689**Files:**
690- Create: `internal/store/secrets.go`, `internal/store/migrations/0072_push_token_hash.up.sql`, `internal/store/migrations/0072_push_token_hash.down.sql`
691- Modify: `internal/store/store.go:23-30` (`Store`), `internal/store/cisecrets.go:3-58`, `internal/store/webhooks.go:41-113`, `internal/store/mirrors.go:23-66`, `internal/store/push.go:32-78,153-202`
692- Test: `internal/store/secrets_test.go` (create)
693
694**Interfaces:**
695- Consumes: `seal.Keyring`, `seal.IsSealed`, `seal.KeyID`, `seal.NewKey`, `seal.WriteKeys`, `seal.Load` (Task 1.1).
696- Produces:
697 - `func (s *Store) SetKeyring(k *seal.Keyring)`
698 - `func (s *Store) ResealSecrets() (int, error)` — seals clear values, reseals values not under the current key, fills missing `push_devices.token_hash`; one transaction; returns values rewritten.
699 - `func (s *Store) SecretKeyUse() (map[string]int, error)` — count per key id (`""` = clear), opening each value.
700 - Unchanged signatures for every existing store function; `Mirror.Token`, `Webhook.Secret`, `Delivery.Secret`, `PushDevice.Token`, `QueuedPush.Token` hold plaintext as before.
701
702- [ ] **Step 1: Write the failing tests**
703
704```go
705package store
706
707import (
708 "path/filepath"
709 "strings"
710 "testing"
711
712 "gitbay.org/gitbay/internal/seal"
713)
714
715// keyedStore is a migrated store with a key file of one key.
716func keyedStore(t *testing.T) (*Store, string, int64, int64) {
717 t.Helper()
718 s := open(t)
719 if err := s.MigrateUp(); err != nil {
720 t.Fatal(err)
721 }
722 path := filepath.Join(t.TempDir(), "secret.key")
723 k, err := seal.NewKey()
724 if err != nil {
725 t.Fatal(err)
726 }
727 if err := seal.WriteKeys(path, []seal.Key{k}); err != nil {
728 t.Fatal(err)
729 }
730 ring, err := seal.Load(path)
731 if err != nil {
732 t.Fatal(err)
733 }
734 s.SetKeyring(ring)
735 uid, err := s.CreateUser("alice", false)
736 if err != nil {
737 t.Fatal(err)
738 }
739 repoID, err := s.CreateRepo("user", uid, "app", "public")
740 if err != nil {
741 t.Fatal(err)
742 }
743 return s, path, uid, repoID
744}
745
746// raw reads every stored value of the secret columns.
747func raw(t *testing.T, s *Store) []string {
748 t.Helper()
749 var out []string
750 for _, sc := range secretColumns {
751 rows, err := s.DB.Query("SELECT " + sc.column + " FROM " + sc.table + " WHERE " + sc.column + " != ''")
752 if err != nil {
753 t.Fatal(err)
754 }
755 for rows.Next() {
756 var v string
757 if err := rows.Scan(&v); err != nil {
758 t.Fatal(err)
759 }
760 out = append(out, v)
761 }
762 rows.Close()
763 }
764 return out
765}
766
767func TestSecretColumnsAreSealed(t *testing.T) {
768 s, _, uid, repoID := keyedStore(t)
769 if err := s.SetBuildSecret(repoID, "DEPLOY", "ci-secret"); err != nil {
770 t.Fatal(err)
771 }
772 if _, err := s.AddWebhook(repoID, "https://hook.example/x", "hook-secret", "*"); err != nil {
773 t.Fatal(err)
774 }
775 if _, err := s.AddMirror(repoID, "push", "https://mirror.example/r.git", "u", "mirror-token"); err != nil {
776 t.Fatal(err)
777 }
778 if _, err := s.AddPushDevice(uid, "apns-token", "phone"); err != nil {
779 t.Fatal(err)
780 }
781
782 vals := raw(t, s)
783 if len(vals) != 4 {
784 t.Fatalf("stored %d values, want 4: %v", len(vals), vals)
785 }
786 for _, v := range vals {
787 if !seal.IsSealed(v) {
788 t.Errorf("stored in clear: %q", v)
789 }
790 for _, plain := range []string{"ci-secret", "hook-secret", "mirror-token", "apns-token"} {
791 if strings.Contains(v, plain) {
792 t.Errorf("%q carries %q", v, plain)
793 }
794 }
795 }
796
797 secrets, err := s.BuildSecrets(repoID)
798 if err != nil || secrets["DEPLOY"] != "ci-secret" {
799 t.Fatalf("BuildSecrets = %v, %v", secrets, err)
800 }
801 hooks, err := s.ListWebhooks(repoID)
802 if err != nil || len(hooks) != 1 || hooks[0].Secret != "hook-secret" {
803 t.Fatalf("ListWebhooks = %+v, %v", hooks, err)
804 }
805 ms, err := s.ListMirrors(repoID)
806 if err != nil || len(ms) != 1 || ms[0].Token != "mirror-token" {
807 t.Fatalf("ListMirrors = %+v, %v", ms, err)
808 }
809 ds, err := s.PushDevices(uid)
810 if err != nil || len(ds) != 1 || ds[0].Token != "apns-token" {
811 t.Fatalf("PushDevices = %+v, %v", ds, err)
812 }
813
814 // An empty webhook secret or mirror token stays empty: it means none.
815 if _, err := s.AddWebhook(repoID, "https://hook.example/y", "", "*"); err != nil {
816 t.Fatal(err)
817 }
818 var empty int
819 s.DB.QueryRow("SELECT COUNT(*) FROM webhooks WHERE secret = ''").Scan(&empty)
820 if empty != 1 {
821 t.Errorf("empty secret stored as %d rows of ''", empty)
822 }
823}
824
825// A token re-registered under another account changes hands by its
826// hash, since two seals of one token differ.
827func TestPushDeviceUpsertBySealedToken(t *testing.T) {
828 s, _, uid, _ := keyedStore(t)
829 bob, err := s.CreateUser("bob", false)
830 if err != nil {
831 t.Fatal(err)
832 }
833 first, err := s.AddPushDevice(uid, "tok", "phone")
834 if err != nil {
835 t.Fatal(err)
836 }
837 second, err := s.AddPushDevice(bob, "tok", "ipad")
838 if err != nil {
839 t.Fatal(err)
840 }
841 if first != second {
842 t.Fatalf("re-registration made row %d beside %d", second, first)
843 }
844 if err := s.DeletePushDeviceByToken("tok"); err != nil {
845 t.Fatal(err)
846 }
847 if d, _ := s.PushDevices(bob); len(d) != 0 {
848 t.Fatalf("device left after delete by token: %+v", d)
849 }
850}
851
852// Rows written before sealing existed, and rows under a retired key,
853// end up under the current key.
854func TestResealSecrets(t *testing.T) {
855 s, path, uid, repoID := keyedStore(t)
856 if _, err := s.DB.Exec("INSERT INTO build_secrets (repo_id, name, value) VALUES (?, 'OLD', 'clear-value')", repoID); err != nil {
857 t.Fatal(err)
858 }
859 if _, err := s.DB.Exec("INSERT INTO push_devices (user_id, token, label) VALUES (?, 'clear-token', '')", uid); err != nil {
860 t.Fatal(err)
861 }
862 n, err := s.ResealSecrets()
863 if err != nil || n != 2 {
864 t.Fatalf("ResealSecrets = %d, %v; want 2", n, err)
865 }
866 for _, v := range raw(t, s) {
867 if !seal.IsSealed(v) {
868 t.Errorf("still clear: %q", v)
869 }
870 }
871 var hash string
872 s.DB.QueryRow("SELECT COALESCE(token_hash, '') FROM push_devices").Scan(&hash)
873 if hash != tokenHash("clear-token") {
874 t.Errorf("token_hash = %q", hash)
875 }
876 if n, _ := s.ResealSecrets(); n != 0 {
877 t.Errorf("second reseal rewrote %d values", n)
878 }
879
880 // Rotation: add a key, reseal, drop the old key; the value still opens.
881 old, err := seal.ReadKeys(path)
882 if err != nil {
883 t.Fatal(err)
884 }
885 next, _ := seal.NewKey()
886 if err := seal.WriteKeys(path, append(old, next)); err != nil {
887 t.Fatal(err)
888 }
889 if n, err := s.ResealSecrets(); err != nil || n != 2 {
890 t.Fatalf("reseal after rotation = %d, %v; want 2", n, err)
891 }
892 if err := seal.WriteKeys(path, []seal.Key{next}); err != nil {
893 t.Fatal(err)
894 }
895 use, err := s.SecretKeyUse()
896 if err != nil || use[next.ID] != 2 || len(use) != 1 {
897 t.Fatalf("SecretKeyUse = %v, %v", use, err)
898 }
899 if got, _ := s.BuildSecrets(repoID); got["OLD"] != "clear-value" {
900 t.Fatalf("value after rotation: %v", got)
901 }
902}
903
904func TestSealedValueWithoutKeyFails(t *testing.T) {
905 s, _, _, repoID := keyedStore(t)
906 if err := s.SetBuildSecret(repoID, "X", "v"); err != nil {
907 t.Fatal(err)
908 }
909 s.SetKeyring(nil)
910 if _, err := s.BuildSecrets(repoID); err == nil {
911 t.Fatal("opened a sealed value with no key loaded")
912 }
913}
914```
915
916- [ ] **Step 2: Run them to verify they fail**
917
918Run: `go test ./internal/store/ -run 'TestSecretColumnsAreSealed|TestPushDeviceUpsertBySealedToken|TestResealSecrets|TestSealedValueWithoutKeyFails' -count=1`
919Expected: FAIL to compile, `s.SetKeyring undefined`.
920
921- [ ] **Step 3: Migration 0072**
922
923`internal/store/migrations/0072_push_token_hash.up.sql`:
924
925```sql
926-- APNs tokens are sealed with a random nonce (internal/seal), so two
927-- stores of one token differ; lookups and the re-registration upsert go
928-- by this SHA-256 of the token instead. Rows from before it are filled
929-- by Store.ResealSecrets when the daemon starts.
930ALTER TABLE push_devices ADD COLUMN token_hash TEXT;
931CREATE UNIQUE INDEX push_devices_token_hash ON push_devices(token_hash);
932```
933
934`internal/store/migrations/0072_push_token_hash.down.sql`:
935
936```sql
937DROP INDEX push_devices_token_hash;
938ALTER TABLE push_devices DROP COLUMN token_hash;
939```
940
941- [ ] **Step 4: `Store.secrets` and `internal/store/secrets.go`**
942
943In `store.go`, add the import `"gitbay.org/gitbay/internal/seal"` and a field on `Store` after `logWait`:
944
945```go
946 // secrets seals and opens the secret columns (secrets.go). Nil
947 // stores values as given; only tests leave it nil, since openStore
948 // in cmd/gitbayd refuses to run without a key file.
949 secrets *seal.Keyring
950```
951
952`internal/store/secrets.go`:
953
954```go
955package store
956
957import (
958 "crypto/sha256"
959 "database/sql"
960 "encoding/hex"
961 "errors"
962 "fmt"
963
964 "gitbay.org/gitbay/internal/seal"
965)
966
967// The additional data of each sealed value is its "<table>.<column>".
968const (
969 aadBuildSecret = "build_secrets.value"
970 aadWebhook = "webhooks.secret"
971 aadMirror = "mirrors.token"
972 aadPushToken = "push_devices.token"
973)
974
975type secretColumn struct{ table, column string }
976
977func (c secretColumn) aad() string { return c.table + "." + c.column }
978
979// secretColumns are the columns sealed under the key file (#273).
980var secretColumns = []secretColumn{
981 {"build_secrets", "value"},
982 {"webhooks", "secret"},
983 {"mirrors", "token"},
984 {"push_devices", "token"},
985}
986
987func (s *Store) SetKeyring(k *seal.Keyring) { s.secrets = k }
988
989// sealValue seals v for storage. An empty value stays empty: for
990// webhooks and mirrors it means there is no secret.
991func (s *Store) sealValue(aad, v string) (string, error) {
992 if s.secrets == nil || v == "" {
993 return v, nil
994 }
995 return s.secrets.Seal(aad, v)
996}
997
998// openValue returns a stored value in clear. A value not yet sealed is
999// returned as stored: rows from before sealing existed stay readable
1000// until ResealSecrets reaches them.
1001func (s *Store) openValue(aad, v string) (string, error) {
1002 if !seal.IsSealed(v) {
1003 return v, nil
1004 }
1005 if s.secrets == nil {
1006 return "", errors.New("value is sealed and no secret key is loaded")
1007 }
1008 return s.secrets.Open(aad, v)
1009}
1010
1011// tokenHash is the lookup key for a push device token.
1012func tokenHash(token string) string {
1013 sum := sha256.Sum256([]byte(token))
1014 return hex.EncodeToString(sum[:])
1015}
1016
1017type secretRow struct {
1018 rowid int64
1019 value string
1020}
1021
1022type queryer interface {
1023 Query(query string, args ...any) (*sql.Rows, error)
1024}
1025
1026func secretRows(q queryer, c secretColumn) ([]secretRow, error) {
1027 rows, err := q.Query(fmt.Sprintf("SELECT rowid, %s FROM %s WHERE %s != ''", c.column, c.table, c.column))
1028 if err != nil {
1029 return nil, err
1030 }
1031 defer rows.Close()
1032 var out []secretRow
1033 for rows.Next() {
1034 var r secretRow
1035 if err := rows.Scan(&r.rowid, &r.value); err != nil {
1036 return nil, err
1037 }
1038 out = append(out, r)
1039 }
1040 return out, rows.Err()
1041}
1042
1043// ResealSecrets seals every clear value in the secret columns and
1044// reseals every value not under the key file's current key, then fills
1045// push_devices.token_hash where it is missing. It runs in one write
1046// transaction: every store write of a secret seals inside its own
1047// transaction, so a write either lands before this one and is resealed,
1048// or after it and is sealed under the key this one saw. It returns how
1049// many values it rewrote.
1050func (s *Store) ResealSecrets() (int, error) {
1051 if s.secrets == nil {
1052 return 0, errors.New("no secret key loaded")
1053 }
1054 tx, err := s.DB.Begin()
1055 if err != nil {
1056 return 0, err
1057 }
1058 defer tx.Rollback()
1059 cur, err := s.secrets.CurrentID()
1060 if err != nil {
1061 return 0, err
1062 }
1063 n := 0
1064 for _, c := range secretColumns {
1065 rows, err := secretRows(tx, c)
1066 if err != nil {
1067 return 0, err
1068 }
1069 for _, r := range rows {
1070 if id, ok := seal.KeyID(r.value); ok && id == cur {
1071 continue
1072 }
1073 plain, err := s.openValue(c.aad(), r.value)
1074 if err != nil {
1075 return 0, fmt.Errorf("%s row %d: %w", c.aad(), r.rowid, err)
1076 }
1077 sealed, err := s.secrets.Seal(c.aad(), plain)
1078 if err != nil {
1079 return 0, err
1080 }
1081 if _, err := tx.Exec(fmt.Sprintf("UPDATE %s SET %s = ? WHERE rowid = ?", c.table, c.column), sealed, r.rowid); err != nil {
1082 return 0, err
1083 }
1084 n++
1085 }
1086 }
1087 rows, err := tx.Query("SELECT id, token FROM push_devices WHERE token_hash IS NULL")
1088 if err != nil {
1089 return 0, err
1090 }
1091 var missing []secretRow
1092 for rows.Next() {
1093 var r secretRow
1094 if err := rows.Scan(&r.rowid, &r.value); err != nil {
1095 rows.Close()
1096 return 0, err
1097 }
1098 missing = append(missing, r)
1099 }
1100 rows.Close()
1101 if err := rows.Err(); err != nil {
1102 return 0, err
1103 }
1104 for _, r := range missing {
1105 plain, err := s.openValue(aadPushToken, r.value)
1106 if err != nil {
1107 return 0, fmt.Errorf("push_devices row %d: %w", r.rowid, err)
1108 }
1109 if _, err := tx.Exec("UPDATE push_devices SET token_hash = ? WHERE id = ?", tokenHash(plain), r.rowid); err != nil {
1110 return 0, err
1111 }
1112 }
1113 return n, tx.Commit()
1114}
1115
1116// SecretKeyUse counts the values in the secret columns by the id of the
1117// key that sealed them ("" for a value still in clear), opening each
1118// one, so a wrong or incomplete key file is an error naming the row.
1119func (s *Store) SecretKeyUse() (map[string]int, error) {
1120 use := map[string]int{}
1121 for _, c := range secretColumns {
1122 rows, err := secretRows(s.DB, c)
1123 if err != nil {
1124 return nil, err
1125 }
1126 for _, r := range rows {
1127 if _, err := s.openValue(c.aad(), r.value); err != nil {
1128 return nil, fmt.Errorf("%s row %d: %w", c.aad(), r.rowid, err)
1129 }
1130 id, _ := seal.KeyID(r.value)
1131 use[id]++
1132 }
1133 }
1134 return use, nil
1135}
1136```
1137
1138- [ ] **Step 5: Seal in the writers, open in the readers**
1139
1140`cisecrets.go` — replace `SetBuildSecret` and `BuildSecrets`:
1141
1142```go
1143// SetBuildSecret stores or replaces one secret. The value never leaves the
1144// server except inside a claimed build's environment. It is sealed inside
1145// the write transaction; see ResealSecrets.
1146func (s *Store) SetBuildSecret(repoID int64, name, value string) error {
1147 tx, err := s.DB.Begin()
1148 if err != nil {
1149 return err
1150 }
1151 defer tx.Rollback()
1152 sealed, err := s.sealValue(aadBuildSecret, value)
1153 if err != nil {
1154 return err
1155 }
1156 if _, err := tx.Exec(`
1157 INSERT INTO build_secrets (repo_id, name, value) VALUES (?, ?, ?)
1158 ON CONFLICT (repo_id, name) DO UPDATE SET value = excluded.value`,
1159 repoID, name, sealed); err != nil {
1160 return err
1161 }
1162 return tx.Commit()
1163}
1164```
1165
1166```go
1167// BuildSecrets returns the values, for injection into a claimed build.
1168func (s *Store) BuildSecrets(repoID int64) (map[string]string, error) {
1169 rows, err := s.DB.Query("SELECT name, value FROM build_secrets WHERE repo_id = ?", repoID)
1170 if err != nil {
1171 return nil, err
1172 }
1173 defer rows.Close()
1174 out := map[string]string{}
1175 for rows.Next() {
1176 var n, v string
1177 if err := rows.Scan(&n, &v); err != nil {
1178 return nil, err
1179 }
1180 if out[n], err = s.openValue(aadBuildSecret, v); err != nil {
1181 return nil, fmt.Errorf("build secret %s: %w", n, err)
1182 }
1183 }
1184 return out, rows.Err()
1185}
1186```
1187
1188(add `import "fmt"` to `cisecrets.go`).
1189
1190`webhooks.go` — `AddWebhook`:
1191
1192```go
1193func (s *Store) AddWebhook(repoID int64, url, secret, events string) (int64, error) {
1194 tx, err := s.DB.Begin()
1195 if err != nil {
1196 return 0, err
1197 }
1198 defer tx.Rollback()
1199 sealed, err := s.sealValue(aadWebhook, secret)
1200 if err != nil {
1201 return 0, err
1202 }
1203 res, err := tx.Exec(
1204 "INSERT INTO webhooks (repo_id, url, secret, events) VALUES (?, ?, ?, ?)",
1205 repoID, url, sealed, events)
1206 if err != nil {
1207 return 0, err
1208 }
1209 id, err := res.LastInsertId()
1210 if err != nil {
1211 return 0, err
1212 }
1213 return id, tx.Commit()
1214}
1215```
1216
1217In `ListWebhooks`, after the `rows.Scan(...)` of `&w.Secret`:
1218
1219```go
1220 if w.Secret, err = s.openValue(aadWebhook, w.Secret); err != nil {
1221 return nil, fmt.Errorf("webhook %d: %w", w.ID, err)
1222 }
1223```
1224
1225In `DueDeliveries`, after its `rows.Scan(...)`:
1226
1227```go
1228 if d.Secret, err = s.openValue(aadWebhook, d.Secret); err != nil {
1229 return nil, fmt.Errorf("webhook %d: %w", d.WebhookID, err)
1230 }
1231```
1232
1233(`webhooks.go` imports `"fmt"` and `"time"`.)
1234
1235`mirrors.go` — `AddMirror`:
1236
1237```go
1238func (s *Store) AddMirror(repoID int64, direction, url, username, token string) (int64, error) {
1239 tx, err := s.DB.Begin()
1240 if err != nil {
1241 return 0, err
1242 }
1243 defer tx.Rollback()
1244 sealed, err := s.sealValue(aadMirror, token)
1245 if err != nil {
1246 return 0, err
1247 }
1248 res, err := tx.Exec(
1249 "INSERT INTO mirrors (repo_id, direction, url, username, token) VALUES (?, ?, ?, ?, ?)",
1250 repoID, direction, url, username, sealed)
1251 if err != nil {
1252 if isUniqueErr(err) {
1253 return 0, ErrExists
1254 }
1255 return 0, err
1256 }
1257 id, err := res.LastInsertId()
1258 if err != nil {
1259 return 0, err
1260 }
1261 return id, tx.Commit()
1262}
1263```
1264
1265`scanMirror` becomes a method that opens the token, and `mirrorQuery` calls `s.scanMirror(rows)`:
1266
1267```go
1268func (s *Store) scanMirror(row interface{ Scan(...any) error }) (Mirror, error) {
1269 var m Mirror
1270 if err := row.Scan(&m.ID, &m.RepoID, &m.Direction, &m.URL, &m.Username, &m.Token,
1271 &m.Dirty, &m.LastSync, &m.LastError); err != nil {
1272 return m, err
1273 }
1274 var err error
1275 if m.Token, err = s.openValue(aadMirror, m.Token); err != nil {
1276 return m, fmt.Errorf("mirror %d: %w", m.ID, err)
1277 }
1278 return m, nil
1279}
1280```
1281
1282(`mirrors.go` imports `"errors"` and `"fmt"`.)
1283
1284`push.go` — `AddPushDevice` body between `defer tx.Rollback()` and `if prev != 0 ...`:
1285
1286```go
1287 h := tokenHash(token)
1288 var prev int64
1289 if err := tx.QueryRow("SELECT user_id FROM push_devices WHERE token_hash = ?", h).Scan(&prev); err != nil && !errors.Is(err, sql.ErrNoRows) {
1290 return 0, err
1291 }
1292 sealed, err := s.sealValue(aadPushToken, token)
1293 if err != nil {
1294 return 0, err
1295 }
1296 if _, err := tx.Exec(`
1297 INSERT INTO push_devices (user_id, token, token_hash, label) VALUES (?, ?, ?, ?)
1298 ON CONFLICT(token_hash) DO UPDATE SET user_id = excluded.user_id, label = excluded.label`,
1299 userID, sealed, h, label); err != nil {
1300 return 0, err
1301 }
1302 var id int64
1303 if err := tx.QueryRow("SELECT id FROM push_devices WHERE token_hash = ?", h).Scan(&id); err != nil {
1304 return 0, err
1305 }
1306```
1307
1308Extend the doc comment's second sentence: "The token is sealed (secrets.go), so the lookup and the upsert go by its hash."
1309
1310`PushDevices`, after `rows.Scan(...)`:
1311
1312```go
1313 if d.Token, err = s.openValue(aadPushToken, d.Token); err != nil {
1314 return nil, fmt.Errorf("push device %d: %w", d.ID, err)
1315 }
1316```
1317
1318`DuePush`, after `rows.Scan(...)`:
1319
1320```go
1321 if p.Token, err = s.openValue(aadPushToken, p.Token); err != nil {
1322 return nil, fmt.Errorf("push device %d: %w", p.DeviceID, err)
1323 }
1324```
1325
1326`DeletePushDeviceByToken`:
1327
1328```go
1329 _, err := s.DB.Exec("DELETE FROM push_devices WHERE token_hash = ?", tokenHash(token))
1330```
1331
1332(`push.go` adds `"fmt"` to its imports.)
1333
1334- [ ] **Step 6: Run the store tests**
1335
1336Run: `go test ./internal/store/ -count=1`
1337Expected: PASS, including `TestMigrateUpDown` (0072 down drops the index before the column) and the existing `TestPushDevices` (nil keyring, hash lookups).
1338
1339- [ ] **Step 7: Build and vet everything above the store**
1340
1341Run: `go build ./... && go vet ./...`
1342Expected: no output. No caller changed signature.
1343
1344- [ ] **Step 8: Commit**
1345
1346```bash
1347git add internal/store
1348git commit -S -m "store: seal CI secrets, webhook secrets, mirror tokens and device tokens
1349
1350Values are AES-256-GCM under the key file; push devices are looked up
1351by token hash (migration 0072).
1352
1353Ref #273"
1354```
1355
1356### Task 1.4: `openStore` loads the key; `serve` reseals; `admin secrets`
1357
1358**Files:**
1359- Modify: `cmd/gitbayd/main.go:40-66` (`openStore`), `:138-142` (`serve`), `:408-422` (`adminCmd`)
1360- Create: `cmd/gitbayd/secrets.go`, `cmd/gitbayd/testconfig_test.go`, `cmd/gitbayd/secrets_test.go`
1361- Modify tests: `cmd/gitbayd/main_test.go:17`, `cmd/gitbayd/backup_test.go:46-47`
1362
1363**Interfaces:**
1364- Consumes: `seal.Load`, `seal.ReadKeys`, `seal.WriteKeys`, `seal.NewKey`, `Store.SetKeyring`, `Store.ResealSecrets`, `Store.SecretKeyUse`.
1365- Produces: `func testConfig(t *testing.T) config.Config` (test helper, cmd/gitbayd), `func rotateSecrets(cfg config.Config) error`, `func secretsCmd() *cobra.Command`.
1366
1367- [ ] **Step 1: The test helper and failing tests**
1368
1369`cmd/gitbayd/testconfig_test.go`:
1370
1371```go
1372package main
1373
1374import (
1375 "path/filepath"
1376 "testing"
1377
1378 "gitbay.org/gitbay/internal/config"
1379 "gitbay.org/gitbay/internal/seal"
1380)
1381
1382// testConfig is a config with a fresh root and a key file outside it,
1383// the minimum openStore accepts.
1384func testConfig(t *testing.T) config.Config {
1385 t.Helper()
1386 key := filepath.Join(t.TempDir(), "secret.key")
1387 k, err := seal.NewKey()
1388 if err != nil {
1389 t.Fatal(err)
1390 }
1391 if err := seal.WriteKeys(key, []seal.Key{k}); err != nil {
1392 t.Fatal(err)
1393 }
1394 return config.Config{Server: config.Server{Root: t.TempDir(), SecretKeyFile: key}}
1395}
1396```
1397
1398In `main_test.go:17` replace the config line with `cfg := testConfig(t)`.
1399In `backup_test.go:46-47` replace the two lines with:
1400
1401```go
1402 cfg := testConfig(t)
1403 root := cfg.Server.Root
1404```
1405
1406Both files then no longer use `internal/config`; drop that import from
1407each (MR 2 adds it back to `backup_test.go` for `TestArchivePath`).
1408
1409`cmd/gitbayd/secrets_test.go`:
1410
1411```go
1412package main
1413
1414import (
1415 "path/filepath"
1416 "strings"
1417 "testing"
1418
1419 "gitbay.org/gitbay/internal/seal"
1420)
1421
1422func TestOpenStoreRefusesWithoutKeyFile(t *testing.T) {
1423 cfg := testConfig(t)
1424 cfg.Server.SecretKeyFile = filepath.Join(t.TempDir(), "absent.key")
1425 _, err := openStore(cfg)
1426 if err == nil || !strings.Contains(err.Error(), "gitbayd admin secrets init") {
1427 t.Fatalf("openStore without a key file: %v", err)
1428 }
1429}
1430
1431func TestRotateSecrets(t *testing.T) {
1432 cfg := testConfig(t)
1433 before, err := seal.ReadKeys(cfg.Server.SecretKeyFile)
1434 if err != nil {
1435 t.Fatal(err)
1436 }
1437 st, err := openStore(cfg)
1438 if err != nil {
1439 t.Fatal(err)
1440 }
1441 uid, err := st.CreateUser("alice", false)
1442 if err != nil {
1443 t.Fatal(err)
1444 }
1445 repoID, err := st.CreateRepo("user", uid, "app", "public")
1446 if err != nil {
1447 t.Fatal(err)
1448 }
1449 if err := st.SetBuildSecret(repoID, "TOKEN", "v1"); err != nil {
1450 t.Fatal(err)
1451 }
1452 st.Close()
1453
1454 if err := rotateSecrets(cfg); err != nil {
1455 t.Fatalf("rotate: %v", err)
1456 }
1457 after, err := seal.ReadKeys(cfg.Server.SecretKeyFile)
1458 if err != nil {
1459 t.Fatal(err)
1460 }
1461 if len(after) != 1 || after[0].ID == before[0].ID {
1462 t.Fatalf("key file after rotation holds %v, before %v", after, before)
1463 }
1464 st, err = openStore(cfg)
1465 if err != nil {
1466 t.Fatal(err)
1467 }
1468 defer st.Close()
1469 use, err := st.SecretKeyUse()
1470 if err != nil || use[after[0].ID] != 1 || len(use) != 1 {
1471 t.Fatalf("SecretKeyUse after rotation = %v, %v", use, err)
1472 }
1473 if got, _ := st.BuildSecrets(repoID); got["TOKEN"] != "v1" {
1474 t.Fatalf("value after rotation: %v", got)
1475 }
1476}
1477```
1478
1479- [ ] **Step 2: Run them to verify they fail**
1480
1481Run: `go test ./cmd/gitbayd/ -run 'TestOpenStore|TestRotateSecrets|TestBackupDBOnly' -count=1`
1482Expected: FAIL to compile, `undefined: rotateSecrets`.
1483
1484- [ ] **Step 3: `openStore` and `serve`**
1485
1486`openStore` begins with the key file (add imports `"errors"`, `"io/fs"` and `"gitbay.org/gitbay/internal/seal"`):
1487
1488```go
1489func openStore(cfg config.Config) (*store.Store, error) {
1490 // The key file seals the secret columns (#273). Without it the
1491 // database's secrets cannot be read or written, so nothing that
1492 // opens the database runs.
1493 keys, err := seal.Load(cfg.Server.SecretKeyFile)
1494 if errors.Is(err, fs.ErrNotExist) {
1495 return nil, fmt.Errorf("secret key file %s does not exist: create it with gitbayd admin secrets init, or restore it from its off-host copy (backups do not carry it)", cfg.Server.SecretKeyFile)
1496 }
1497 if err != nil {
1498 return nil, fmt.Errorf("secret key file: %w", err)
1499 }
1500 s, err := store.Open(filepath.Join(cfg.Server.Root, "gitbay.db"))
1501 if err != nil {
1502 return nil, err
1503 }
1504 s.SetKeyring(keys)
1505```
1506
1507(the rest of the function is unchanged).
1508
1509In `serve`, after `defer st.Close()` (line 142):
1510
1511```go
1512 // Values stored before sealing existed, or under a key a
1513 // rotation retired, are sealed under the current key before
1514 // anything reads them. A value the key file cannot open
1515 // stops the start here rather than failing each delivery.
1516 if n, err := st.ResealSecrets(); err != nil {
1517 return fmt.Errorf("sealing secrets: %w", err)
1518 } else if n > 0 {
1519 slog.Info("sealed secret values", "count", n)
1520 }
1521```
1522
1523- [ ] **Step 4: `cmd/gitbayd/secrets.go`**
1524
1525```go
1526package main
1527
1528import (
1529 "fmt"
1530 "os"
1531 "sort"
1532 "strings"
1533
1534 "github.com/spf13/cobra"
1535
1536 "gitbay.org/gitbay/internal/config"
1537 "gitbay.org/gitbay/internal/seal"
1538)
1539
1540// secretsCmd manages the key file that seals CI secrets, webhook
1541// secrets, mirror tokens and push device tokens in the database.
1542func secretsCmd() *cobra.Command {
1543 cmd := &cobra.Command{
1544 Use: "secrets",
1545 Short: "the key file that seals secrets stored in the database",
1546 }
1547 cmd.AddCommand(
1548 &cobra.Command{
1549 Use: "init",
1550 Short: "create the key file (server.secret_key_file) with one new key",
1551 RunE: func(cmd *cobra.Command, args []string) error {
1552 cfg, err := config.Load(configPath)
1553 if err != nil {
1554 return err
1555 }
1556 path := cfg.Server.SecretKeyFile
1557 if _, err := os.Stat(path); err == nil {
1558 return fmt.Errorf("%s already exists; gitbayd admin secrets rotate replaces its key", path)
1559 }
1560 k, err := seal.NewKey()
1561 if err != nil {
1562 return err
1563 }
1564 if err := seal.WriteKeys(path, []seal.Key{k}); err != nil {
1565 return err
1566 }
1567 fmt.Printf("wrote %s (key %s). Copy it off this host: backups do not carry it, and a restored database's secrets do not open without it.\n", path, k.ID)
1568 return nil
1569 },
1570 },
1571 &cobra.Command{
1572 Use: "rotate",
1573 Short: "seal every secret under a new key and retire the old ones",
1574 Long: `Adds a new key to the key file, reseals every value under it in one
1575transaction, then removes the old keys from the file. A running daemon
1576re-reads the file when it changes, so no restart is needed. Run as the
1577user that can replace the key file (root, for /etc/gitbay); the file
1578keeps its owner. Copy the new file off the host afterwards.`,
1579 RunE: func(cmd *cobra.Command, args []string) error {
1580 cfg, err := config.Load(configPath)
1581 if err != nil {
1582 return err
1583 }
1584 return rotateSecrets(cfg)
1585 },
1586 },
1587 &cobra.Command{
1588 Use: "check",
1589 Short: "open every sealed value and count them by key",
1590 RunE: func(cmd *cobra.Command, args []string) error {
1591 cfg, err := config.Load(configPath)
1592 if err != nil {
1593 return err
1594 }
1595 st, err := openStore(cfg)
1596 if err != nil {
1597 return err
1598 }
1599 defer st.Close()
1600 use, err := st.SecretKeyUse()
1601 if err != nil {
1602 return err
1603 }
1604 ids := make([]string, 0, len(use))
1605 for id := range use {
1606 ids = append(ids, id)
1607 }
1608 sort.Strings(ids)
1609 for _, id := range ids {
1610 if id == "" {
1611 fmt.Printf("clear: %d values (sealed when the daemon next starts)\n", use[id])
1612 } else {
1613 fmt.Printf("key %s: %d sealed\n", id, use[id])
1614 }
1615 }
1616 if len(ids) == 0 {
1617 fmt.Println("no secrets stored")
1618 }
1619 return nil
1620 },
1621 },
1622 )
1623 return cmd
1624}
1625
1626// rotateSecrets adds a key, reseals under it, then drops the old keys.
1627// Each step leaves a file that opens every stored value: after the first
1628// write the file holds old and new keys; the reseal is one transaction;
1629// the last write happens only after the reseal committed. Interrupted
1630// anywhere, running it again finishes the job.
1631func rotateSecrets(cfg config.Config) error {
1632 path := cfg.Server.SecretKeyFile
1633 old, err := seal.ReadKeys(path)
1634 if err != nil {
1635 return err
1636 }
1637 next, err := seal.NewKey()
1638 if err != nil {
1639 return err
1640 }
1641 if err := seal.WriteKeys(path, append(old, next)); err != nil {
1642 return err
1643 }
1644 st, err := openStore(cfg)
1645 if err != nil {
1646 return err
1647 }
1648 defer st.Close()
1649 n, err := st.ResealSecrets()
1650 if err != nil {
1651 return fmt.Errorf("resealing: %w (the key file holds the old keys and %s; run rotate again)", err, next.ID)
1652 }
1653 if err := seal.WriteKeys(path, []seal.Key{next}); err != nil {
1654 return err
1655 }
1656 retired := make([]string, len(old))
1657 for i, k := range old {
1658 retired[i] = k.ID
1659 }
1660 fmt.Printf("key %s: resealed %d values; retired %s. Copy %s off this host.\n", next.ID, n, strings.Join(retired, ", "), path)
1661 return nil
1662}
1663```
1664
1665In `adminCmd` add `secretsCmd(),` after `backupCmd(),` (line 417).
1666
1667- [ ] **Step 5: Run the package tests**
1668
1669Run: `go test ./cmd/gitbayd/ -count=1 && go vet ./cmd/gitbayd/`
1670Expected: PASS.
1671
1672- [ ] **Step 6: Commit**
1673
1674```bash
1675git add cmd/gitbayd
1676git commit -S -m "gitbayd: load the secret key file; admin secrets init, rotate, check
1677
1678serve seals values still in clear, or under a retired key, before it
1679starts listening.
1680
1681Ref #273"
1682```
1683
1684### Task 1.5: e2e harness key and a sealed-at-rest check through a backup
1685
1686**Files:**
1687- Modify: `e2e/ssh_test.go:17-27` (`instance`), `:81-117` (`startInstanceWith`)
1688- Modify: `e2e/acme_test.go:28-39`, `e2e/system_test.go:32-41`, `e2e/backup_test.go:14-157`
1689
1690**Interfaces:**
1691- Produces: `instance.keyFile string`.
1692
1693- [ ] **Step 1: Harness**
1694
1695Add to `instance`:
1696
1697```go
1698 keyFile string // server.secret_key_file, outside root
1699```
1700
1701In `startInstanceWith`, before building `cfg`:
1702
1703```go
1704 inst.keyFile = filepath.Join(t.TempDir(), "secret.key")
1705```
1706
1707and change the config's `[server]` table and its `Sprintf` arguments:
1708
1709```go
1710[server]
1711root = %q
1712site_url = "https://gitbay.test"
1713secret_key_file = %q
1714```
1715
1716```go
1717`, inst.root, inst.keyFile, inst.port, inst.httpPort, inst.gitPort)
1718```
1719
1720After writing the config file and before `inst.proc = exec.Command(...)`:
1721
1722```go
1723 inst.admin(t, "admin", "secrets", "init")
1724```
1725
1726In `acme_test.go` and `system_test.go`, add `secret_key_file = %q` under `site_url` and `inst.keyFile` as the second `Sprintf` argument. In `backup_test.go`'s restore config (line 76-85) do the same.
1727
1728- [ ] **Step 2: Extend `TestAdminBackup`**
1729
1730Add `"bytes"` to its imports. After the issue create (line 37):
1731
1732```go
1733 // A build secret, to show the archive carries it sealed and the key
1734 // file not at all.
1735 if _, errOut, code := inst.ssh(t, aliceKey, "hunter2-at-rest", "repo", "secret", "set", "alice/keep", "DEPLOY_TOKEN"); code != 0 {
1736 t.Fatalf("secret set: %s", errOut)
1737 }
1738```
1739
1740After the transient-state loop (line 66):
1741
1742```go
1743 if strings.Contains(names, "secret.key") {
1744 t.Fatalf("archive carries the key file:\n%s", names)
1745 }
1746 db, err := exec.Command("tar", "-xzOf", archive, "gitbay.db").Output()
1747 if err != nil {
1748 t.Fatal(err)
1749 }
1750 if bytes.Contains(db, []byte("hunter2-at-rest")) {
1751 t.Fatal("the archived database carries the build secret in clear")
1752 }
1753```
1754
1755After the restored instance answers `whoami` (line 151):
1756
1757```go
1758 // With the original key the restored secrets open; with another key
1759 // they do not.
1760 if out, err := exec.Command(inst.gitbayd, "--config", config2, "admin", "secrets", "check").CombinedOutput(); err != nil || !strings.Contains(string(out), ": 1 sealed") {
1761 t.Fatalf("secrets check on the restored instance: %v\n%s", err, out)
1762 }
1763 config3 := filepath.Join(root2, "config-wrong-key.toml")
1764 wrong := strings.Replace(cfg, fmt.Sprintf("secret_key_file = %q", inst.keyFile),
1765 fmt.Sprintf("secret_key_file = %q", filepath.Join(t.TempDir(), "other.key")), 1)
1766 if err := os.WriteFile(config3, []byte(wrong), 0o600); err != nil {
1767 t.Fatal(err)
1768 }
1769 if out, err := exec.Command(inst.gitbayd, "--config", config3, "admin", "secrets", "init").CombinedOutput(); err != nil {
1770 t.Fatalf("init the wrong key: %v\n%s", err, out)
1771 }
1772 if out, err := exec.Command(inst.gitbayd, "--config", config3, "admin", "secrets", "check").CombinedOutput(); err == nil || !strings.Contains(string(out), "does not hold") {
1773 t.Fatalf("secrets check with the wrong key: %v\n%s", err, out)
1774 }
1775```
1776
1777- [ ] **Step 3: Run the one e2e test**
1778
1779Run: `go test ./e2e -run TestAdminBackup -count=1`
1780Expected: PASS. (The other e2e tests pick up the harness change in CI.)
1781
1782- [ ] **Step 4: Commit**
1783
1784```bash
1785git add e2e
1786git commit -S -m "e2e: a key file per instance; the archive carries secrets sealed and no key
1787
1788Ref #273"
1789```
1790
1791### Task 1.6: install.sh, wiki, changelog
1792
1793**Files:**
1794- Modify: `deploy/install.sh:12-20`
1795- Modify: `.gitbay/wiki/Admin.org` (Install block lines 17-22, `** [server]` 55-64, a new `** Secret key` under `* Backup and restore`)
1796- Modify: `.gitbay/wiki/Architecture/06-Data-and-Cryptography.org` (inventory rows 17-19, "Outside the database" table, "At rest" table and the paragraph after it, migration count line 5)
1797- Modify: `.gitbay/wiki/Architecture/03-Deployment.org` (file table, after the `config.toml` row)
1798- Modify: `.gitbay/wiki/Architecture/09-Controls.org:59`, `.gitbay/wiki/Architecture/10-Known-Gaps.org` (#273 row)
1799- Modify: `CHANGELOG.org`
1800
1801- [ ] **Step 1: install.sh creates the key once**
1802
1803Replace the remote script with:
1804
1805```sh
1806ssh -p "$port" "root@$host" '
1807 set -eu
1808 chmod 755 /usr/local/bin/gitbayd.new
1809 mv /usr/local/bin/gitbayd.new /usr/local/bin/gitbayd
1810 /usr/local/bin/gitbayd --config /etc/gitbay/config.toml check-config --no-host-checks
1811 # The key that seals secrets in the database. Created on the first
1812 # install, never replaced here; gitbayd refuses to start without it.
1813 if [ ! -e /etc/gitbay/secret.key ]; then
1814 /usr/local/bin/gitbayd --config /etc/gitbay/config.toml admin secrets init
1815 chown gitbay:gitbay /etc/gitbay/secret.key
1816 fi
1817 systemctl restart gitbayd
1818 sleep 1
1819 systemctl --no-pager --lines=5 status gitbayd
1820'
1821```
1822
1823- [ ] **Step 2: Admin.org**
1824
1825Install block becomes:
1826
1827```sh
1828install -m 755 gitbayd /usr/local/bin/
1829adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
1830install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
1831gitbayd --config /etc/gitbay/config.toml check-config
1832gitbayd --config /etc/gitbay/config.toml admin secrets init
1833chown gitbay:gitbay /etc/gitbay/secret.key
1834```
1835
1836Under `** [server]`, after `site_url`:
1837
1838```org
1839- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
1840 seal CI secrets, webhook secrets, mirror tokens and push device
1841 tokens in the database. Must be outside =root=, mode 0600, readable
1842 by the daemon's user. See "Secret key" below.
1843```
1844
1845New subsection at the end of `* Backup and restore` (before `* Upgrades`):
1846
1847```org
1848** Secret key
1849
1850CI secrets, webhook secrets, mirror tokens and APNs device tokens are
1851stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
1852each value prefixed with the id of the key that sealed it
1853(=gbs1:<id>:=). The key file is not in the database, not under
1854=server.root=, and therefore in neither the local archives nor the
1855main restic repository. Its offsite copy is a separate restic
1856repository that holds only keys (see "Offsite copies"), with its own
1857bucket and password, so a leak of the main backup does not expose the
1858key that opens its secrets. Without the key a restored database's
1859secrets cannot be opened, and gitbayd refuses to start against them.
1860
1861#+begin_src sh
1862gitbayd admin secrets init # once; deploy/install.sh does it on first install
1863gitbayd admin secrets check # open every value, count by key
1864gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
1865#+end_src
1866
1867- Missing file: every gitbayd process that opens the database refuses
1868 to run and names the path, including =serve= and, in system mode,
1869 =authorized-keys=. =migrate= does not need it.
1870- Wrong key: =serve= stops at startup naming the first row that does
1871 not open; =secrets check= does the same without starting anything.
1872- Upgrade: the first start after the upgrade seals every value still
1873 in clear and logs =sealed secret values=.
1874- Rotation: =rotate= adds a key, reseals every value under it in one
1875 transaction, then removes the old keys. The daemon re-reads the file
1876 when it changes, so it needs no restart. Back the new file up to the
1877 keys repository afterwards.
1878- Push devices are looked up by the SHA-256 of their token
1879 (=push_devices.token_hash=), since two seals of one token differ.
1880```
1881
1882If runbook D (the keys repository) has not run when this MR lands,
1883leave out the sentences naming the keys repository here and in
1884`06-Data-and-Cryptography.org` below; the `offsite-r2-wiki` MR adds
1885them.
1886
1887- [ ] **Step 3: Architecture pages**
1888
1889`06-Data-and-Cryptography.org`: in the inventory, the CI secrets note becomes `sealed (AES-256-GCM)`, Integrations `webhook secret and mirror token sealed`, Notifications `device tokens sealed; looked up by SHA-256`. Add a row to "Outside the database":
1890
1891```org
1892| Secret key file | =server.secret_key_file= (=/etc/gitbay/secret.key=) | C |
1893```
1894
1895In "At rest", replace the secrets row with:
1896
1897```org
1898| CI secrets, webhook secrets, mirror tokens, APNs device tokens | AES-256-GCM under a key file outside the database and outside =server.root=; additional data is the column name; key id on each value (=internal/seal=, =internal/store/secrets.go=) |
1899```
1900
1901and replace the paragraph "The code base contains no symmetric encryption. ..." with:
1902
1903```org
1904The database file or a backup read by anyone other than the =gitbay=
1905user discloses no CI secret, webhook secret, mirror token or device
1906token without the key file, which neither carries; the key file's
1907only offsite copy is a separate keys repository. Rotation:
1908=gitbayd admin secrets rotate= (Admin wiki).
1909```
1910
1911Update line 5's migration count to the number of files in
1912`internal/store/migrations/` divided by two after this MR.
1913
1914`03-Deployment.org`, file table, after the `config.toml` row:
1915
1916```org
1917| =/etc/gitbay/secret.key= | keys sealing secret columns | 0600, owner =gitbay= (=deploy/install.sh=) |
1918```
1919
1920`09-Controls.org:59`:
1921
1922```org
1923| Secrets encrypted at rest | in place | AES-256-GCM, key file outside the database and the main backups (=internal/seal=) |
1924```
1925
1926`10-Known-Gaps.org`: delete the `#273` row.
1927
1928- [ ] **Step 4: CHANGELOG.org**
1929
1930If the file has no `* Unreleased` heading above the latest version, add one under the header paragraph; under it:
1931
1932```org
1933*Upgrade note.* gitbayd needs =server.secret_key_file= (default
1934=/etc/gitbay/secret.key=) and refuses to start without it. Before
1935replacing the binary, run =gitbayd admin secrets init= as root and
1936=chown gitbay:gitbay /etc/gitbay/secret.key= (=deploy/install.sh= does
1937both when the file is missing). The first start seals the stored
1938secrets. Back the key file up separately: =admin backup= archives do
1939not carry it (see the Admin wiki, "Secret key").
1940
1941- CI secrets, webhook secrets, mirror tokens and push device tokens are
1942 stored sealed with AES-256-GCM (#273). =gitbayd admin secrets
1943 init|rotate|check=.
1944```
1945
1946- [ ] **Step 5: Verify and commit**
1947
1948Run: `go build ./... && go vet ./... && go test ./internal/seal/ ./internal/config/ ./internal/store/ ./cmd/gitbayd/ -count=1`
1949Expected: PASS.
1950
1951```bash
1952git add deploy/install.sh .gitbay/wiki CHANGELOG.org
1953git commit -S -m "deploy, wiki: provision and document the secret key file
1954
1955Closes #273"
1956```
1957
1958- [ ] **Step 6: MR**
1959
1960```bash
1961git push -u origin secrets-at-rest
1962gitbay mr create --source secrets-at-rest --target main --title "Seal secret columns under a key file outside the database"
1963```
1964
1965After CI is green: `gitbay mr merge <n> --strategy ff`, delete the
1966branch locally and remotely. The operator steps for bay1 are in
1967"Operator runbook", part A.
1968
1969---
1970
1971# MR 2: encrypted backup archives (branch `backup-age`, closes #274)
1972
1973### Task 2.1: `[backup] age_recipients`
1974
1975**Files:**
1976- Modify: `go.mod`, `go.sum` (`filippo.io/age`)
1977- Modify: `internal/config/config.go` (`Config`, new `Backup` type, `Validate`)
1978- Test: `internal/config/config_test.go`
1979
1980**Interfaces:**
1981- Produces: `Config.Backup Backup` (`toml:"backup"`), `type Backup struct { AgeRecipients []string }`, `func (b Backup) Recipients() ([]age.Recipient, error)`.
1982
1983- [ ] **Step 1: Add the dependency**
1984
1985Run: `go get filippo.io/age@latest && go mod tidy`
1986Expected: `filippo.io/age` in `go.mod`'s require block. Record the version in the commit message.
1987
1988- [ ] **Step 2: Write the failing test**
1989
1990```go
1991func TestBackupRecipients(t *testing.T) {
1992 id, err := age.GenerateX25519Identity()
1993 if err != nil {
1994 t.Fatal(err)
1995 }
1996 cfg, err := Load(writeConfig(t, minimal+"[backup]\nage_recipients = [\""+id.Recipient().String()+"\"]\n"))
1997 if err != nil {
1998 t.Fatal(err)
1999 }
2000 rs, err := cfg.Backup.Recipients()
2001 if err != nil || len(rs) != 1 {
2002 t.Fatalf("Recipients = %v, %v", rs, err)
2003 }
2004 if _, err := Load(writeConfig(t, minimal+"[backup]\nage_recipients = [\"age1notakey\"]\n")); err == nil || !strings.Contains(err.Error(), "backup.age_recipients") {
2005 t.Fatalf("a malformed recipient: %v", err)
2006 }
2007 if cfg, err := Load(writeConfig(t, minimal)); err != nil || len(cfg.Backup.AgeRecipients) != 0 {
2008 t.Fatalf("default: %v, %v", cfg.Backup, err)
2009 }
2010}
2011```
2012
2013(add `"filippo.io/age"` to the test imports).
2014
2015- [ ] **Step 3: Run it to verify it fails**
2016
2017Run: `go test ./internal/config/ -run TestBackupRecipients -count=1`
2018Expected: FAIL, `cfg.Backup undefined`.
2019
2020- [ ] **Step 4: Implement**
2021
2022In `Config`, after `Push`:
2023
2024```go
2025 Backup Backup `toml:"backup"`
2026```
2027
2028After the `Push` type's methods:
2029
2030```go
2031// Backup configures gitbayd admin backup.
2032type Backup struct {
2033 // AgeRecipients, when set, encrypts every archive to these age
2034 // public keys (age1...). The matching identities stay off the host,
2035 // so the host writes archives it cannot read.
2036 AgeRecipients []string `toml:"age_recipients"`
2037}
2038
2039// Recipients parses AgeRecipients.
2040func (b Backup) Recipients() ([]age.Recipient, error) {
2041 var rs []age.Recipient
2042 for _, s := range b.AgeRecipients {
2043 r, err := age.ParseX25519Recipient(s)
2044 if err != nil {
2045 return nil, fmt.Errorf("backup.age_recipients: %q: %w", s, err)
2046 }
2047 rs = append(rs, r)
2048 }
2049 return rs, nil
2050}
2051```
2052
2053In `Validate`, before `// Contradictions.`:
2054
2055```go
2056 if _, err := c.Backup.Recipients(); err != nil {
2057 errs = append(errs, err)
2058 }
2059```
2060
2061Import `"filippo.io/age"`.
2062
2063- [ ] **Step 5: Run and commit**
2064
2065Run: `go test ./internal/config/ -count=1`
2066Expected: PASS.
2067
2068```bash
2069git add go.mod go.sum internal/config
2070v=$(go list -m -f '{{.Version}}' filippo.io/age)
2071git commit -S -m "config: [backup] age_recipients (filippo.io/age $v)
2072
2073Ref #274"
2074```
2075
2076### Task 2.2: age-wrapped archives and `--verify --identity`
2077
2078**Files:**
2079- Modify: `cmd/gitbayd/backup.go:28-64` (`backupCmd`), `:66-151` (`runBackup`), `:187-197` (`verifyBackup` opening)
2080- Test: `cmd/gitbayd/backup_test.go`
2081
2082**Interfaces:**
2083- Consumes: `cfg.Backup.Recipients()` (Task 2.1), `testConfig` (Task 1.4).
2084- Produces: `func archivePath(out string, cfg config.Config, now time.Time) string`; `func verifyBackup(path, identity string) error` (was `verifyBackup(path string)`); `func archiveReader(f io.Reader, path, identity string) (io.Reader, error)`.
2085
2086- [ ] **Step 1: Write the failing tests**
2087
2088```go
2089func TestBackupEncryptedToAgeRecipient(t *testing.T) {
2090 cfg := testConfig(t)
2091 id, err := age.GenerateX25519Identity()
2092 if err != nil {
2093 t.Fatal(err)
2094 }
2095 cfg.Backup.AgeRecipients = []string{id.Recipient().String()}
2096 s, err := openStore(cfg)
2097 if err != nil {
2098 t.Fatal(err)
2099 }
2100 s.Close()
2101
2102 out := filepath.Join(t.TempDir(), "b.tar.gz.age")
2103 if err := runBackup(cfg, out, true); err != nil {
2104 t.Fatal(err)
2105 }
2106 head := make([]byte, 22)
2107 f, err := os.Open(out)
2108 if err != nil {
2109 t.Fatal(err)
2110 }
2111 io.ReadFull(f, head)
2112 f.Close()
2113 if string(head) != "age-encryption.org/v1\n" {
2114 t.Fatalf("archive is not age-encrypted: %q", head)
2115 }
2116
2117 if err := verifyBackup(out, ""); err == nil || !strings.Contains(err.Error(), "--identity") {
2118 t.Fatalf("verify without an identity: %v", err)
2119 }
2120 idFile := filepath.Join(t.TempDir(), "backup-identity.txt")
2121 if err := os.WriteFile(idFile, []byte(id.String()+"\n"), 0o600); err != nil {
2122 t.Fatal(err)
2123 }
2124 if err := verifyBackup(out, idFile); err != nil {
2125 t.Fatalf("verify with the identity: %v", err)
2126 }
2127 other, _ := age.GenerateX25519Identity()
2128 otherFile := filepath.Join(t.TempDir(), "other.txt")
2129 os.WriteFile(otherFile, []byte(other.String()+"\n"), 0o600)
2130 if err := verifyBackup(out, otherFile); err == nil {
2131 t.Fatal("verify with another identity succeeded")
2132 }
2133}
2134
2135func TestArchivePath(t *testing.T) {
2136 now := time.Date(2026, 9, 27, 9, 0, 0, 0, time.UTC)
2137 plain := testConfig(t)
2138 enc := plain
2139 enc.Backup.AgeRecipients = []string{"age1x"}
2140 for _, c := range []struct {
2141 out string
2142 cfg config.Config
2143 want string
2144 }{
2145 {"", plain, "gitbay-backup-20260927-090000.tar.gz"},
2146 {"", enc, "gitbay-backup-20260927-090000.tar.gz.age"},
2147 {"/b/x.tar.gz", enc, "/b/x.tar.gz.age"},
2148 {"/b/x.tar.gz.age", enc, "/b/x.tar.gz.age"},
2149 {"/b/x.tar.gz", plain, "/b/x.tar.gz"},
2150 } {
2151 if got := archivePath(c.out, c.cfg, now); got != c.want {
2152 t.Errorf("archivePath(%q) = %q, want %q", c.out, got, c.want)
2153 }
2154 }
2155}
2156```
2157
2158(add `"filippo.io/age"`, `"strings"`, `"time"` and
2159`"gitbay.org/gitbay/internal/config"` to the test imports).
2160
2161- [ ] **Step 2: Run them to verify they fail**
2162
2163Run: `go test ./cmd/gitbayd/ -run 'TestBackupEncrypted|TestArchivePath' -count=1`
2164Expected: FAIL to compile (`undefined: archivePath`; `verifyBackup` takes one argument).
2165
2166- [ ] **Step 3: Implement**
2167
2168`backupCmd`: add `identity` to the `var` line, replace the `RunE` body and add a flag:
2169
2170```go
2171 RunE: func(cmd *cobra.Command, args []string) error {
2172 if verify != "" {
2173 return verifyBackup(verify, identity)
2174 }
2175 cfg, err := config.Load(configPath)
2176 if err != nil {
2177 return err
2178 }
2179 return runBackup(cfg, archivePath(out, cfg, time.Now()), dbOnly)
2180 },
2181```
2182
2183```go
2184 cmd.Flags().StringVar(&identity, "identity", "", "with --verify: an age identity file that opens an encrypted archive")
2185```
2186
2187Append to the `Long` text:
2188
2189```
2190With [backup] age_recipients set, the archive is encrypted to those age
2191public keys and its name ends in .age. --verify then needs --identity
2192<file> holding a matching private key, which is kept off the host.
2193```
2194
2195New function:
2196
2197```go
2198// archivePath is where the archive goes: out, or a timestamped name,
2199// ending in .age when the archive is encrypted.
2200func archivePath(out string, cfg config.Config, now time.Time) string {
2201 if out == "" {
2202 out = fmt.Sprintf("gitbay-backup-%s.tar.gz", now.UTC().Format("20060102-150405"))
2203 }
2204 if len(cfg.Backup.AgeRecipients) > 0 && !strings.HasSuffix(out, ".age") {
2205 out += ".age"
2206 }
2207 return out
2208}
2209```
2210
2211In `runBackup`, replace lines 86-87 (`gz := gzip.NewWriter(f)` and `tw := tar.NewWriter(gz)`) with:
2212
2213```go
2214 var sink io.Writer = f
2215 var enc io.WriteCloser
2216 if len(cfg.Backup.AgeRecipients) > 0 {
2217 rs, err := cfg.Backup.Recipients()
2218 if err != nil {
2219 return err
2220 }
2221 if enc, err = age.Encrypt(f, rs...); err != nil {
2222 return err
2223 }
2224 sink = enc
2225 }
2226 gz := gzip.NewWriter(sink)
2227 tw := tar.NewWriter(gz)
2228```
2229
2230and after `gz.Close()` (line 137-139):
2231
2232```go
2233 if enc != nil {
2234 if err := enc.Close(); err != nil {
2235 return err
2236 }
2237 }
2238```
2239
2240In `verifyBackup`, change the signature to `func verifyBackup(path, identity string) error` and replace `gz, err := gzip.NewReader(f)` with:
2241
2242```go
2243 plain, err := archiveReader(f, path, identity)
2244 if err != nil {
2245 return err
2246 }
2247 gz, err := gzip.NewReader(plain)
2248```
2249
2250New function (import `"bufio"` and `"filippo.io/age"`):
2251
2252```go
2253const ageHeader = "age-encryption.org/v1\n"
2254
2255// archiveReader returns the archive's gzip stream, decrypting it first
2256// when it is an age file.
2257func archiveReader(f io.Reader, path, identity string) (io.Reader, error) {
2258 br := bufio.NewReader(f)
2259 head, _ := br.Peek(len(ageHeader))
2260 if string(head) != ageHeader {
2261 return br, nil
2262 }
2263 if identity == "" {
2264 return nil, fmt.Errorf("%s is encrypted; pass --identity <file> with the private key for one of its recipients", path)
2265 }
2266 idf, err := os.Open(identity)
2267 if err != nil {
2268 return nil, err
2269 }
2270 defer idf.Close()
2271 ids, err := age.ParseIdentities(idf)
2272 if err != nil {
2273 return nil, fmt.Errorf("%s: %w", identity, err)
2274 }
2275 r, err := age.Decrypt(br, ids...)
2276 if err != nil {
2277 return nil, fmt.Errorf("%s: decrypting: %w", path, err)
2278 }
2279 return r, nil
2280}
2281```
2282
2283Update the verify doc comment's first sentence to "verifyBackup reads an archive back, decrypting it with identity when it is encrypted:".
2284
2285- [ ] **Step 4: Run the package tests**
2286
2287Run: `go test ./cmd/gitbayd/ -count=1 && go vet ./cmd/gitbayd/`
2288Expected: PASS.
2289
2290- [ ] **Step 5: Commit**
2291
2292```bash
2293git add cmd/gitbayd
2294git commit -S -m "backup: encrypt archives to [backup] age_recipients; --verify --identity
2295
2296Ref #274"
2297```
2298
2299### Task 2.3: scripts, wiki, changelog
2300
2301**Files:**
2302- Modify: `deploy/cloud-init.yaml:96-100` (`age_h`), `:258-259`, `:276-277` (prune globs)
2303- Modify: `.gitbay/wiki/Admin.org` (Configuration reference: new `** [backup]` after `** [push]`; `* Backup and restore` 373-393)
2304- Modify: `.gitbay/wiki/Architecture/06-Data-and-Cryptography.org` (At rest, Backups row), `Architecture/08-Operations.org` (Backup table), `Architecture/09-Controls.org:61`, `Architecture/10-Known-Gaps.org` (#274 row)
2305- Modify: `CHANGELOG.org`
2306
2307- [ ] **Step 1: cloud-init**
2308
2309`age_h`:
2310
2311```sh
2312 age_h() {
2313 f=$(ls -t "$1"/*.tar.gz "$1"/*.tar.gz.age 2>/dev/null | head -1)
2314```
2315
2316Full backup prune (line 259):
2317
2318```sh
2319 ls -1t "$dir"/gitbay-*.tar.gz* | tail -n +8 | xargs -r rm --
2320```
2321
2322Database backup prune (line 277):
2323
2324```sh
2325 ls -1t "$dir"/gitbay-db-*.tar.gz* | tail -n +49 | xargs -r rm --
2326```
2327
2328- [ ] **Step 2: Admin.org**
2329
2330New `** [backup]` section after `** [push]`:
2331
2332```org
2333** [backup]
2334- =age_recipients= (optional) — age public keys (=age1...=). When set,
2335 =admin backup= encrypts every archive to them and appends =.age= to
2336 its name. Generate the pair off the host with =age-keygen=; only the
2337 public key goes here, so the host writes archives it cannot read.
2338 The restic copy is unaffected: the offsite job stages its own
2339 =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
2340 not the archives.
2341```
2342
2343In `* Backup and restore`, after the `--verify` paragraph:
2344
2345```org
2346With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
2347and =--verify= needs the private key:
2348
2349#+begin_src sh
2350gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
2351age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
2352#+end_src
2353
2354The identity lives off the host (with the secret key file and the
2355restic credentials), so verifying an encrypted archive happens there
2356or on a restore host.
2357```
2358
2359- [ ] **Step 3: Architecture pages**
2360
2361`06-Data-and-Cryptography.org`, At rest, Backups row:
2362
2363```org
2364| Backups | local archives age-encrypted when =[backup] age_recipients= is set; restic encrypts the offsite copy; neither carries the secret key file, whose offsite copy is a separate keys repository |
2365```
2366
2367`08-Operations.org`, Full archive and Database only rows: append `; age-encrypted when =[backup] age_recipients= is set` to Contents.
2368
2369`09-Controls.org:61`:
2370
2371```org
2372| Local backups encrypted | in place | age to =[backup] age_recipients= (=cmd/gitbayd/backup.go=); offsite copy by restic |
2373```
2374
2375`10-Known-Gaps.org`: delete the `#274` row.
2376
2377- [ ] **Step 4: CHANGELOG.org** under `* Unreleased`:
2378
2379```org
2380- =gitbayd admin backup= encrypts archives to =[backup] age_recipients=
2381 when set (#274); =--verify= takes =--identity <file>=. Archive names
2382 gain =.age=; the shipped backup scripts and monitor match both.
2383```
2384
2385- [ ] **Step 5: Verify and commit**
2386
2387Run: `go build ./... && go vet ./...`
2388Expected: no output.
2389
2390```bash
2391git add deploy/cloud-init.yaml .gitbay/wiki CHANGELOG.org
2392git commit -S -m "deploy, wiki: encrypted archives in the backup scripts and docs
2393
2394Closes #274"
2395git push -u origin backup-age
2396gitbay mr create --source backup-age --target main --title "Encrypt backup archives to an age recipient"
2397```
2398
2399Merge with `--strategy ff` after CI, delete the branch both places.
2400
2401---
2402
2403# MR 3: verify connectivity, hold moves during a backup (branch `backup-verify-lock`, ref #259)
2404
2405### Task 3.1: `gitutil.FsckConnectivity`
2406
2407**Files:**
2408- Modify: `internal/gitutil/merge.go` (after `PruneNow`, line 68)
2409- Test: `internal/gitutil/fsck_test.go` (create)
2410
2411**Interfaces:**
2412- Produces: `func FsckConnectivity(dir string) error`.
2413
2414- [ ] **Step 1: Failing test**
2415
2416```go
2417package gitutil
2418
2419import (
2420 "os"
2421 "os/exec"
2422 "path/filepath"
2423 "strings"
2424 "testing"
2425)
2426
2427func TestFsckConnectivityFindsAMissingObject(t *testing.T) {
2428 dir := t.TempDir()
2429 git(t, dir, "init", "-q", "-b", "main")
2430 write(t, dir, "a.txt", "a\n")
2431 git(t, dir, "add", "a.txt")
2432 git(t, dir, "commit", "-q", "-m", "one")
2433 if err := FsckConnectivity(dir); err != nil {
2434 t.Fatalf("intact repository: %v", err)
2435 }
2436 out, err := exec.Command("git", "-C", dir, "rev-parse", "HEAD:a.txt").Output()
2437 if err != nil {
2438 t.Fatal(err)
2439 }
2440 blob := strings.TrimSpace(string(out))
2441 if err := os.Remove(filepath.Join(dir, ".git", "objects", blob[:2], blob[2:])); err != nil {
2442 t.Fatal(err)
2443 }
2444 if err := FsckConnectivity(dir); err == nil {
2445 t.Fatal("a repository missing a blob passed")
2446 }
2447}
2448```
2449
2450- [ ] **Step 2: Run it**
2451
2452Run: `go test ./internal/gitutil/ -run TestFsckConnectivity -count=1`
2453Expected: FAIL, `undefined: FsckConnectivity`.
2454
2455- [ ] **Step 3: Implement** (in `merge.go`, after `PruneNow`)
2456
2457```go
2458// FsckConnectivity checks that every object reachable from the
2459// repository's refs is present, without reading blob contents. A backup
2460// verify runs it on each archived repository (#259).
2461func FsckConnectivity(dir string) error {
2462 cmd := exec.Command(toolpath.Look("git"), "-C", dir, "fsck", "--connectivity-only", "--no-progress", "--no-dangling")
2463 if out, err := cmd.CombinedOutput(); err != nil {
2464 return fmt.Errorf("fsck --connectivity-only: %v\n%s", err, out)
2465 }
2466 return nil
2467}
2468```
2469
2470- [ ] **Step 4: Run and commit**
2471
2472Run: `go test ./internal/gitutil/ -count=1`
2473Expected: PASS.
2474
2475```bash
2476git add internal/gitutil
2477git commit -S -m "gitutil: FsckConnectivity
2478
2479Ref #259"
2480```
2481
2482### Task 3.2: `internal/backuplock`
2483
2484**Files:**
2485- Create: `internal/backuplock/backuplock.go`, `internal/backuplock/backuplock_test.go`
2486
2487**Interfaces:**
2488- Produces: `const Name = "backup.lock"`, `var ErrBusy error`, `func Hold(root string) (func(), error)`, `func TryShared(root string) (func(), error)`.
2489
2490- [ ] **Step 1: Failing tests**
2491
2492```go
2493package backuplock
2494
2495import (
2496 "errors"
2497 "testing"
2498 "time"
2499)
2500
2501func TestTrySharedRefusedWhileHeld(t *testing.T) {
2502 root := t.TempDir()
2503 release, err := Hold(root)
2504 if err != nil {
2505 t.Fatal(err)
2506 }
2507 if _, err := TryShared(root); !errors.Is(err, ErrBusy) {
2508 t.Fatalf("TryShared during a backup: %v", err)
2509 }
2510 release()
2511 r, err := TryShared(root)
2512 if err != nil {
2513 t.Fatalf("TryShared after the backup: %v", err)
2514 }
2515 r()
2516}
2517
2518func TestSharedHoldersCoexist(t *testing.T) {
2519 root := t.TempDir()
2520 a, err := TryShared(root)
2521 if err != nil {
2522 t.Fatal(err)
2523 }
2524 defer a()
2525 b, err := TryShared(root)
2526 if err != nil {
2527 t.Fatalf("second shared holder: %v", err)
2528 }
2529 b()
2530}
2531
2532// A backup waits for a delete already under way.
2533func TestHoldWaitsForSharedHolder(t *testing.T) {
2534 root := t.TempDir()
2535 shared, err := TryShared(root)
2536 if err != nil {
2537 t.Fatal(err)
2538 }
2539 got := make(chan struct{})
2540 go func() {
2541 release, err := Hold(root)
2542 if err != nil {
2543 t.Error(err)
2544 close(got)
2545 return
2546 }
2547 close(got)
2548 release()
2549 }()
2550 select {
2551 case <-got:
2552 t.Fatal("Hold returned while a shared holder was in")
2553 case <-time.After(100 * time.Millisecond):
2554 }
2555 shared()
2556 select {
2557 case <-got:
2558 case <-time.After(5 * time.Second):
2559 t.Fatal("Hold never returned after the shared holder left")
2560 }
2561}
2562```
2563
2564- [ ] **Step 2: Run them**
2565
2566Run: `go test ./internal/backuplock/ -count=1`
2567Expected: FAIL to compile.
2568
2569- [ ] **Step 3: Implement**
2570
2571```go
2572// Package backuplock keeps repository deletes, renames and transfers
2573// out of a full backup's way (#259). The backup runs in its own process
2574// (gitbayd admin backup) and a delete in the daemon's, so the lock is
2575// flock(2) on a file under server.root: the backup holds it exclusively
2576// from its database snapshot until the last repository is archived, and
2577// each delete or move holds it shared while it runs.
2578package backuplock
2579
2580import (
2581 "errors"
2582 "os"
2583 "path/filepath"
2584 "syscall"
2585)
2586
2587// Name is the lock file under server.root. Backups skip it.
2588const Name = "backup.lock"
2589
2590// ErrBusy is TryShared's answer while a backup holds the lock.
2591var ErrBusy = errors.New("a backup is running; repositories cannot be deleted, renamed or moved until it finishes, usually within minutes")
2592
2593// open opens the lock file read-only, which is all flock needs, so the
2594// daemon's user can lock a file a root-run backup created.
2595func open(root string) (*os.File, error) {
2596 return os.OpenFile(filepath.Join(root, Name), os.O_RDONLY|os.O_CREATE, 0o644)
2597}
2598
2599// Hold takes the lock exclusively, waiting for deletes and moves under
2600// way to finish. Closing the file releases it.
2601func Hold(root string) (func(), error) {
2602 f, err := open(root)
2603 if err != nil {
2604 return nil, err
2605 }
2606 if err := syscall.Flock(int(f.Fd()), syscall.LOCK_EX); err != nil {
2607 f.Close()
2608 return nil, err
2609 }
2610 return func() { f.Close() }, nil
2611}
2612
2613// TryShared takes the lock shared without waiting: ErrBusy while a
2614// backup holds it.
2615func TryShared(root string) (func(), error) {
2616 f, err := open(root)
2617 if err != nil {
2618 return nil, err
2619 }
2620 if err := syscall.Flock(int(f.Fd()), syscall.LOCK_SH|syscall.LOCK_NB); err != nil {
2621 f.Close()
2622 if errors.Is(err, syscall.EWOULDBLOCK) {
2623 return nil, ErrBusy
2624 }
2625 return nil, err
2626 }
2627 return func() { f.Close() }, nil
2628}
2629```
2630
2631- [ ] **Step 4: Run and commit**
2632
2633Run: `go test ./internal/backuplock/ -count=1 -race`
2634Expected: PASS.
2635
2636```bash
2637git add internal/backuplock
2638git commit -S -m "backuplock: flock between a full backup and repository moves
2639
2640Ref #259"
2641```
2642
2643### Task 3.3: deletes and moves refuse during a full backup
2644
2645**Files:**
2646- Modify: `internal/control/repo.go` (`runRepoTransfer` 481-538, `runRepoRename` 540-575, `deleteRepo` 611-626; new `holdOffBackup`)
2647- Modify: `internal/control/org.go` (`runOrgRename` 152-184)
2648- Test: `internal/control/backuplock_test.go` (create)
2649
2650**Interfaces:**
2651- Consumes: `backuplock.TryShared`, `backuplock.Hold`, `backuplock.ErrBusy`.
2652- Produces: `func holdOffBackup(c *Ctx) (func(), int)` in package control.
2653
2654- [ ] **Step 1: Failing test**
2655
2656```go
2657package control
2658
2659import (
2660 "strings"
2661 "testing"
2662
2663 "gitbay.org/gitbay/internal/backuplock"
2664 "gitbay.org/gitbay/internal/protocol"
2665)
2666
2667func TestRepoDeleteAndRenameRefusedDuringBackup(t *testing.T) {
2668 st, repo, uid := newQueueTestRepo(t)
2669 owner, err := st.UserByID(uid)
2670 if err != nil {
2671 t.Fatal(err)
2672 }
2673 root := t.TempDir()
2674 release, err := backuplock.Hold(root)
2675 if err != nil {
2676 t.Fatal(err)
2677 }
2678 for _, argv := range [][]string{
2679 {"repo", "rename", repo.Path(), "renamed"},
2680 {"repo", "delete", repo.Path(), "--yes"},
2681 } {
2682 c, errOut := pruneCtx(st, root, owner)
2683 if code := Dispatch(c, argv); code != protocol.ExitFailure || !strings.Contains(errOut.String(), "a backup is running") {
2684 t.Fatalf("%v during a backup: exit %d, %s", argv, code, errOut)
2685 }
2686 }
2687 if got, err := st.RepoByID(repo.ID); err != nil || got.Name != repo.Name {
2688 t.Fatalf("repository changed during a backup: %+v, %v", got, err)
2689 }
2690 release()
2691
2692 c, errOut := pruneCtx(st, root, owner)
2693 if code := Dispatch(c, []string{"repo", "delete", repo.Path(), "--yes"}); code != protocol.ExitOK {
2694 t.Fatalf("delete after the backup: exit %d, %s", code, errOut)
2695 }
2696}
2697```
2698
2699Transfer and org rename get the same call; the test covers rename and
2700delete because a transfer target needs an org fixture this test does
2701not build, and the call is identical.
2702
2703- [ ] **Step 2: Run it**
2704
2705Run: `go test ./internal/control/ -run TestRepoDeleteAndRenameRefusedDuringBackup -count=1`
2706Expected: FAIL, rename exits 0 during the backup.
2707
2708- [ ] **Step 3: Implement**
2709
2710`repo.go`, import `"gitbay.org/gitbay/internal/backuplock"` and add after `deleteRepo`:
2711
2712```go
2713// holdOffBackup keeps a full backup from starting while a repository
2714// directory moves or goes, and refuses while one runs: the backup's
2715// database snapshot names every repository its walk then archives
2716// (#259). The caller defers the returned release.
2717func holdOffBackup(c *Ctx) (func(), int) {
2718 release, err := backuplock.TryShared(c.Cfg.Server.Root)
2719 if err != nil {
2720 return nil, c.fail(protocol.ExitFailure, "%v", err)
2721 }
2722 return release, -1
2723}
2724```
2725
2726Call it with this block:
2727
2728```go
2729 release, code := holdOffBackup(c)
2730 if code >= 0 {
2731 return code
2732 }
2733 defer release()
2734```
2735
2736- `deleteRepo`: first statement of the function.
2737- `runRepoRename`: after the `os.Stat(newDir)` check, before `c.Store.RenameRepo` (line 560). `code` is already declared there by `resolveRepo`, so this call uses `lockCode`:
2738
2739```go
2740 release, lockCode := holdOffBackup(c)
2741 if lockCode >= 0 {
2742 return lockCode
2743 }
2744 defer release()
2745```
2746
2747- `runRepoTransfer`: the same `lockCode` block after the `os.Stat(newDir)` check, before `os.MkdirAll` (line 522).
2748- `org.go` `runOrgRename`: the same `lockCode` block after its `os.Stat(newDir)` check, before `c.Store.RenameOrg` (line 170).
2749
2750Repository creation, forks and imports are not held: a repository the
2751snapshot does not name is reported by `--verify` as extra, which is
2752harmless.
2753
2754- [ ] **Step 4: Run and commit**
2755
2756Run: `go test ./internal/control/ -count=1 && go vet ./internal/control/`
2757Expected: PASS.
2758
2759```bash
2760git add internal/control
2761git commit -S -m "control: repository delete, rename, transfer and org rename wait out a full backup
2762
2763Ref #259"
2764```
2765
2766### Task 3.4: the backup holds the lock; `--verify` checks connectivity
2767
2768**Files:**
2769- Modify: `cmd/gitbayd/backup.go` (`runBackup`, `verifyBackup`)
2770- Test: `cmd/gitbayd/backup_test.go`
2771
2772**Interfaces:**
2773- Consumes: `backuplock.Hold`, `backuplock.Name`, `gitutil.FsckConnectivity`, `archiveReader` (Task 2.2).
2774- Produces: `verifyBackup(path, identity string) error` now also runs `FsckConnectivity` per repository.
2775
2776- [ ] **Step 1: Failing tests**
2777
2778```go
2779func gitIn(t *testing.T, dir string, args ...string) string {
2780 t.Helper()
2781 cmd := exec.Command("git", append([]string{"-C", dir}, args...)...)
2782 cmd.Env = append(os.Environ(),
2783 "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@e",
2784 "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@e")
2785 out, err := cmd.CombinedOutput()
2786 if err != nil {
2787 t.Fatalf("git %v: %v\n%s", args, err, out)
2788 }
2789 return strings.TrimSpace(string(out))
2790}
2791
2792// verify runs git's connectivity check on every repository the
2793// database names: a repository missing an object fails it.
2794func TestVerifyChecksConnectivity(t *testing.T) {
2795 cfg := testConfig(t)
2796 st, err := openStore(cfg)
2797 if err != nil {
2798 t.Fatal(err)
2799 }
2800 uid, err := st.CreateUser("krz", false)
2801 if err != nil {
2802 t.Fatal(err)
2803 }
2804 if _, err := st.CreateRepo("user", uid, "thing", "public"); err != nil {
2805 t.Fatal(err)
2806 }
2807 st.Close()
2808
2809 work := t.TempDir()
2810 gitIn(t, work, "init", "-q", "-b", "main")
2811 if err := os.WriteFile(filepath.Join(work, "a.txt"), []byte("a\n"), 0o644); err != nil {
2812 t.Fatal(err)
2813 }
2814 gitIn(t, work, "add", "a.txt")
2815 gitIn(t, work, "commit", "-q", "-m", "one")
2816 dir := filepath.Join(cfg.Server.Root, "repos", "krz", "thing.git")
2817 gitIn(t, work, "clone", "-q", "--bare", work, dir)
2818
2819 good := filepath.Join(t.TempDir(), "good.tar.gz")
2820 if err := runBackup(cfg, good, false); err != nil {
2821 t.Fatal(err)
2822 }
2823 if err := verifyBackup(good, ""); err != nil {
2824 t.Fatalf("intact archive: %v", err)
2825 }
2826
2827 blob := gitIn(t, dir, "rev-parse", "HEAD:a.txt")
2828 if err := os.Remove(filepath.Join(dir, "objects", blob[:2], blob[2:])); err != nil {
2829 t.Fatal(err)
2830 }
2831 bad := filepath.Join(t.TempDir(), "bad.tar.gz")
2832 if err := runBackup(cfg, bad, false); err != nil {
2833 t.Fatal(err)
2834 }
2835 err = verifyBackup(bad, "")
2836 if err == nil || !strings.Contains(err.Error(), "krz/thing") || !strings.Contains(err.Error(), "connectivity") {
2837 t.Fatalf("archive with a missing blob: %v", err)
2838 }
2839}
2840
2841// A full backup waits for a delete under way, and does not archive its
2842// own lock file.
2843func TestFullBackupWaitsForRepositoryMoves(t *testing.T) {
2844 cfg := testConfig(t)
2845 s, err := openStore(cfg)
2846 if err != nil {
2847 t.Fatal(err)
2848 }
2849 s.Close()
2850 inFlight, err := backuplock.TryShared(cfg.Server.Root)
2851 if err != nil {
2852 t.Fatal(err)
2853 }
2854 out := filepath.Join(t.TempDir(), "b.tar.gz")
2855 done := make(chan error, 1)
2856 go func() { done <- runBackup(cfg, out, false) }()
2857 select {
2858 case err := <-done:
2859 t.Fatalf("backup finished while a delete held the lock: %v", err)
2860 case <-time.After(200 * time.Millisecond):
2861 }
2862 inFlight()
2863 select {
2864 case err := <-done:
2865 if err != nil {
2866 t.Fatal(err)
2867 }
2868 case <-time.After(10 * time.Second):
2869 t.Fatal("backup never started after the delete finished")
2870 }
2871 for _, n := range members(t, out) {
2872 if n == backuplock.Name {
2873 t.Fatalf("archive carries %s", n)
2874 }
2875 }
2876}
2877```
2878
2879(`backup_test.go` imports gain `"os/exec"` and `"gitbay.org/gitbay/internal/backuplock"`; `strings` and `time` came with MR 2.)
2880
2881`TestVerifyChecksConnectivity` relies on a local `git clone --bare`
2882hardlinking loose objects into `dir`, so removing the blob there leaves
2883`work` intact. If git packs instead, the `os.Remove` fails and the test
2884says so.
2885
2886- [ ] **Step 2: Run them**
2887
2888Run: `go test ./cmd/gitbayd/ -run 'TestVerifyChecksConnectivity|TestFullBackupWaitsForRepositoryMoves' -count=1`
2889Expected: FAIL: the bad archive verifies; the backup finishes while the lock is held.
2890
2891- [ ] **Step 3: `runBackup` holds the lock**
2892
2893At the top of `runBackup`, before `openStore`:
2894
2895```go
2896 // Deletes, renames and transfers wait until the walk finishes, so
2897 // every repository the snapshot names is still on disk when the walk
2898 // reaches it (#259). A database-only archive reads no repository.
2899 if !dbOnly {
2900 release, err := backuplock.Hold(cfg.Server.Root)
2901 if err != nil {
2902 return fmt.Errorf("backup lock: %w", err)
2903 }
2904 defer release()
2905 }
2906```
2907
2908Add `backuplock.Name: true` to the `skip` map.
2909
2910- [ ] **Step 4: `verifyBackup` extracts and checks repositories**
2911
2912Replace the function (keeping `archiveReader` from Task 2.2):
2913
2914```go
2915// verifyBackup reads an archive back, decrypting it with identity when
2916// it is encrypted: the database snapshot must pass SQLite's integrity
2917// check, every repository it names must be in the archive, and each of
2918// those must pass git fsck --connectivity-only. Repositories are
2919// extracted to a temporary directory for the check, so it needs free
2920// space for them. A database-only archive is checked for integrity
2921// alone and says so.
2922func verifyBackup(path, identity string) error {
2923 f, err := os.Open(path)
2924 if err != nil {
2925 return err
2926 }
2927 defer f.Close()
2928 plain, err := archiveReader(f, path, identity)
2929 if err != nil {
2930 return err
2931 }
2932 gz, err := gzip.NewReader(plain)
2933 if err != nil {
2934 return fmt.Errorf("%s: not a gzip archive: %w", path, err)
2935 }
2936 tr := tar.NewReader(gz)
2937 tmp, err := os.MkdirTemp("", "gitbay-verify-")
2938 if err != nil {
2939 return err
2940 }
2941 defer os.RemoveAll(tmp)
2942 dbPath := ""
2943 inArchive := map[string]bool{}
2944 members := 0
2945 for {
2946 h, err := tr.Next()
2947 if err == io.EOF {
2948 break
2949 }
2950 if err != nil {
2951 return fmt.Errorf("%s: archive damaged after %d members: %w", path, members, err)
2952 }
2953 members++
2954 switch {
2955 case h.Name == "gitbay.db":
2956 dbPath = filepath.Join(tmp, "gitbay.db")
2957 if err := extractTo(tr, dbPath); err != nil {
2958 return fmt.Errorf("%s: extracting the database: %w", path, err)
2959 }
2960 case strings.HasPrefix(h.Name, "repos/"):
2961 // repos/<owner>/<name>.git/HEAD marks one repository present.
2962 parts := strings.Split(h.Name, "/")
2963 if len(parts) == 4 && parts[3] == "HEAD" && strings.HasSuffix(parts[2], ".git") {
2964 inArchive[parts[1]+"/"+strings.TrimSuffix(parts[2], ".git")] = true
2965 }
2966 if h.Typeflag != tar.TypeReg {
2967 continue
2968 }
2969 if !filepath.IsLocal(h.Name) {
2970 return fmt.Errorf("%s: member %q leaves the archive root", path, h.Name)
2971 }
2972 if err := extractTo(tr, filepath.Join(tmp, filepath.FromSlash(h.Name))); err != nil {
2973 return fmt.Errorf("%s: extracting %s: %w", path, h.Name, err)
2974 }
2975 }
2976 }
2977 if dbPath == "" {
2978 return fmt.Errorf("%s: no gitbay.db in the archive", path)
2979 }
2980 st, err := store.Open(dbPath)
2981 if err != nil {
2982 return fmt.Errorf("%s: database does not open: %w", path, err)
2983 }
2984 defer st.Close()
2985 var integrity string
2986 if err := st.DB.QueryRow("PRAGMA integrity_check").Scan(&integrity); err != nil {
2987 return fmt.Errorf("%s: integrity check: %w", path, err)
2988 }
2989 if integrity != "ok" {
2990 return fmt.Errorf("%s: database integrity: %s", path, integrity)
2991 }
2992 repos, err := st.ListAllRepos()
2993 if err != nil {
2994 return err
2995 }
2996 if len(inArchive) == 0 {
2997 fmt.Printf("%s: database only; integrity ok, %d repositories in the database, none in the archive\n", path, len(repos))
2998 return nil
2999 }
3000 var missing []string
3001 for _, r := range repos {
3002 if !inArchive[r.Path()] {
3003 missing = append(missing, r.Path())
3004 }
3005 }
3006 extra := len(inArchive) - (len(repos) - len(missing))
3007 fmt.Printf("%s: integrity ok, %d repositories in the database, %d in the archive\n", path, len(repos), len(inArchive))
3008 if len(missing) > 0 {
3009 return fmt.Errorf("%s: %d repositories the database names are not in the archive: %s", path, len(missing), strings.Join(missing, ", "))
3010 }
3011 if extra > 0 {
3012 fmt.Printf("%d repositories in the archive that the database does not name (created after the snapshot)\n", extra)
3013 }
3014 var broken []string
3015 for _, r := range repos {
3016 dir := filepath.Join(tmp, "repos", r.OwnerName, r.Name+".git")
3017 if err := gitutil.FsckConnectivity(dir); err != nil {
3018 fmt.Fprintf(os.Stderr, "%s: %v\n", r.Path(), err)
3019 broken = append(broken, r.Path())
3020 }
3021 }
3022 if len(broken) > 0 {
3023 return fmt.Errorf("%s: %d repositories fail the connectivity check: %s", path, len(broken), strings.Join(broken, ", "))
3024 }
3025 fmt.Printf("connectivity ok on %d repositories\n", len(repos))
3026 return nil
3027}
3028
3029// extractTo writes one archive member to dest, owner-only.
3030func extractTo(r io.Reader, dest string) error {
3031 if err := os.MkdirAll(filepath.Dir(dest), 0o700); err != nil {
3032 return err
3033 }
3034 w, err := os.OpenFile(dest, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
3035 if err != nil {
3036 return err
3037 }
3038 if _, err := io.Copy(w, r); err != nil {
3039 w.Close()
3040 return err
3041 }
3042 return w.Close()
3043}
3044```
3045
3046Imports gain `"gitbay.org/gitbay/internal/backuplock"` and
3047`"gitbay.org/gitbay/internal/gitutil"`. The "extra" message changes
3048from "deleted after the snapshot" to "created after the snapshot",
3049since a delete can no longer land mid-backup.
3050
3051Update `backupCmd`'s `--verify` flag text:
3052
3053```go
3054 cmd.Flags().StringVar(&verify, "verify", "", "check an archive instead of writing one: database integrity, its repositories against the archive's, and git connectivity of each")
3055```
3056
3057- [ ] **Step 5: Run the package tests**
3058
3059Run: `go test ./cmd/gitbayd/ -count=1 && go vet ./cmd/gitbayd/`
3060Expected: PASS, including `TestBackupDBOnlyOmitsRepositories` (its repository is not in the database, so no fsck runs on its HEAD-only directory).
3061
3062- [ ] **Step 6: e2e**
3063
3064In `e2e/backup_test.go`, after the transient-state checks:
3065
3066```go
3067 if out := inst.admin(t, "admin", "backup", "--verify", archive); !strings.Contains(out, "connectivity ok on 1 repositories") {
3068 t.Fatalf("verify: %s", out)
3069 }
3070```
3071
3072Run: `go test ./e2e -run TestAdminBackup -count=1`
3073Expected: PASS.
3074
3075- [ ] **Step 7: Commit**
3076
3077```bash
3078git add cmd/gitbayd e2e/backup_test.go
3079git commit -S -m "backup: hold repository moves off during a full backup; verify git connectivity
3080
3081Ref #259"
3082```
3083
3084### Task 3.5: wiki: behaviour and the restore drill procedure
3085
3086**Files:**
3087- Modify: `.gitbay/wiki/Admin.org` (`* Backup and restore`: the `--verify` paragraph; new `** Restore drill`)
3088- Modify: `.gitbay/wiki/Architecture/08-Operations.org:55-70`
3089- Modify: `.gitbay/wiki/Threat-Model.org:218-220`
3090
3091- [ ] **Step 1: Admin.org**
3092
3093Replace the `--verify` paragraph with:
3094
3095```org
3096=--verify= reads an archive back: the snapshot must pass SQLite's
3097integrity check, every repository the snapshot names must be in the
3098archive, and each must pass =git fsck --connectivity-only=. It
3099extracts the repositories to a temporary directory for that, so it
3100needs free space the size of the repositories. A database-only archive
3101is checked for integrity and says so. Exit is non-zero on damage, a
3102missing repository or a missing object.
3103
3104A full backup holds =<root>/backup.lock= from its database snapshot to
3105its last repository. While it runs, =repo delete=, =repo rename=,
3106=repo transfer=, =admin repo delete= and =org rename= refuse with "a
3107backup is running"; retry when it finishes. Database-only backups take
3108no lock.
3109```
3110
3111Add a new subsection after `** Secret key`:
3112
3113```org
3114** Restore drill
3115
3116A restore onto a clean host, run quarterly and after any change to the
3117backup code (=cmd/gitbayd/backup.go=, the offsite job), and recorded
3118below. The disaster it rehearses is losing bay1, so the local archives
3119are gone with it and the sources are the main offsite restic
3120repository (repositories, LFS, the staged database, =config.toml=),
3121the keys repository (=secret.key=, =apns.p8=), and the operator's
3122password manager (=offsite.env=, the keys repository's password and
3123token, =backup-identity.txt=). The steps are in the data-at-rest
3124plan's operator runbook
3125(=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
3126
3127Time to service runs from the clean host's first root login to the
3128first successful =git clone= over SSH from it. The recovery point is
3129the time of the newest restic snapshot restored.
3130
3131| Date | Host | Snapshot restored (UTC) | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
3132|------+------+-------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
3133```
3134
3135- [ ] **Step 2: Architecture/08, Known-Gaps and Threat-Model**
3136
3137`10-Known-Gaps.org:17`, the `#259` row's description becomes
3138`No restore has been exercised; the drill is written (Admin wiki) and not yet run`.
3139The row stays until the drill is recorded.
3140
3141`08-Operations.org`: replace the `--verify` bullet (lines 60-62) with:
3142
3143```org
3144- =gitbayd admin backup --verify= checks SQLite integrity, that every
3145 repository the database names is present, and =git fsck
3146 --connectivity-only= on each (=backup.go=).
3147- Repository deletes, renames and transfers refuse while a full backup
3148 runs (=internal/backuplock=), so the snapshot and the walk agree.
3149```
3150
3151Replace the "Recovery time" bullet with:
3152
3153```org
3154- Recovery time: see the Admin wiki's Restore drill table.
3155```
3156
3157`Threat-Model.org:218-220` becomes:
3158
3159```org
3160- Backups are consistent per the DB-snapshot-first ordering, and
3161 repository deletes and moves wait out a full backup; a push during
3162 one leaves only unreferenced objects (see [[Admin]]).
3163```
3164
3165- [ ] **Step 3: Commit and MR**
3166
3167```bash
3168git add .gitbay/wiki
3169git commit -S -m "wiki: backup verify, backup lock, restore drill procedure
3170
3171Ref #259"
3172git push -u origin backup-verify-lock
3173gitbay mr create --source backup-verify-lock --target main --title "Backup verify checks git connectivity; moves wait out a backup"
3174```
3175
3176Merge with `--strategy ff` after CI, delete the branch both places.
3177#259 stays open after this MR: every commit in it says `Ref #259`, and
3178only the drill record (runbook C, deferred) closes it.
3179
3180---
3181
3182# Operator runbook (cmc)
3183
3184Run on bay1, the laptop and (for C) a clean host. Nothing here is
3185automated by the MRs. One forge write per shell call; bay1 root is
3186`ssh -p 2222 root@gitbay.org`.
3187
3188D does not depend on any MR and can run first; A.3 and C use the keys
3189repository it creates.
3190
3191## A. After MR 1 deploys
3192
31931. `make deploy` runs `deploy/install.sh`, which creates
3194 `/etc/gitbay/secret.key` (missing on bay1) before the restart.
3195 Confirm: `ssh -p 2222 root@gitbay.org 'ls -l /etc/gitbay/secret.key; journalctl -u gitbayd -n 50 | grep "sealed secret values"'`
3196 → mode `-rw-------`, owner `gitbay gitbay`, one log line with a count.
31972. `ssh -p 2222 root@gitbay.org 'gitbayd --config /etc/gitbay/config.toml admin secrets check'`
3198 → one `key <id>: N sealed` line, no `clear:` line.
31993. Back the key up to the keys repository (D.9) with `secret.key` as
3200 the file and `secret-key` as the tag. If D has not run yet, do D.2,
3201 D.3 (keys bucket), D.4 (keys token) and D.9 first. After every
3202 `admin secrets rotate`, repeat this step.
32034. Check CI builds that use secrets (blotter, hutch, orgo) still run,
3204 and a webhook delivery still verifies.
3205
3206## B. After MR 2 deploys
3207
32081. On the laptop: `age-keygen -o ~/.config/gitbay/backup-identity.txt`
3209 (mode 600). Note the printed `age1...` public key. Put a copy of the
3210 identity in the password manager.
32112. On bay1, add to `/etc/gitbay/config.toml`:
3212 ```toml
3213 [backup]
3214 age_recipients = ["age1..."]
3215 ```
3216 then `gitbayd --config /etc/gitbay/config.toml check-config --no-host-checks`.
32173. bay1's installed `/usr/local/bin/gitbay-backup.sh`,
3218 `/usr/local/bin/gitbay-db-backup.sh` and `/usr/local/bin/gitbay-monitor.sh`
3219 predate the cloud-init change (cloud-init runs once). Apply the same
3220 glob edits as Task 2.3 Step 1 by hand.
32214. The offsite job needs no change. `/usr/local/bin/gitbay-offsite`
3222 builds `/var/lib/gitbay-stage` from live data (`sqlite3 gitbay.db
3223 "VACUUM INTO ..."` and `cp -a /etc/gitbay/config.toml`), not from the
3224 local archives, so encrypting the archives does not reach restic.
32255. After the next hourly run: `ls -l /var/backups/gitbay/db | tail -2`
3226 shows `.tar.gz.age`; the monitor's `db_snapshot_h` stays under 2.
3227 Copy one archive to the laptop and run
3228 `gitbayd admin backup --verify <file> --identity ~/.config/gitbay/backup-identity.txt`
3229 (a local gitbayd build; `--verify` reads no config).
32306. Old unencrypted archives age out of the 7/48 rotation on their own.
3231
3232## C. Restore drill (deferred; closes #259)
3233
3234Deferred by the operator. Run it after MR 3 deploys and D is complete,
3235then quarterly, and after any change to `cmd/gitbayd/backup.go` or the
3236offsite job (`/usr/local/bin/gitbay-offsite`, its env file, the R2 lock
3237rules). Each run adds a row to the Admin page's table; the first one
3238closes #259.
3239
3240Record every timestamp as you go. Start the clock at step 2.
3241
32421. On the laptop, export the `gitbay r2-prune` variables from the
3243 password manager (D.5), then:
3244 `restic snapshots --tag gitbay --latest 1`. Note its time (recovery
3245 point). `restic ls latest /var/lib/gitbay-stage` shows the staged
3246 database's file name and `config.toml`.
32472. Provision a clean Ubuntu 24.04 host (throwaway VPS or local VM) with
3248 `deploy/cloud-init.yaml`. First root login: **clock starts**.
32493. Before gitbayd ever starts, block outbound traffic so the restored
3250 instance cannot send mail, deliver webhooks, push mirrors or call
3251 APNs: `ufw default deny outgoing; ufw allow out 53; ufw allow out to <account-id>.r2.cloudflarestorage.com port 443; ufw reload`.
3252 (ufw resolves the name once; allow R2 only for the restore, then
3253 remove the rule.)
32544. Install restic, and with the same variables exported on the drill
3255 host: `restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage`.
3256 The snapshot excludes `gitbay.db`, `gitbay.db-wal`, `gitbay.db-shm`
3257 and `hook.sock`: copy the staged database from
3258 `/var/lib/gitbay-stage/` to `/var/lib/gitbay/gitbay.db`.
3259 `chown -R gitbay:gitbay /var/lib/gitbay`. LFS objects are under
3260 `/var/lib/gitbay/lfs` (inside `server.root`) and come back with it.
32615. Config and keys: copy `/var/lib/gitbay-stage/config.toml` to
3262 `/etc/gitbay/config.toml`. From the laptop, with the `gitbay r2-keys`
3263 variables exported (D.5):
3264 ```sh
3265 restic dump --tag secret-key latest /secret.key | ssh root@<drill-ip> -p 2222 'umask 077; cat > /etc/gitbay/secret.key; chown gitbay:gitbay /etc/gitbay/secret.key'
3266 restic dump --tag apns latest /apns.p8 | ssh root@<drill-ip> -p 2222 'umask 077; cat > /etc/gitbay/apns.p8; chown gitbay:gitbay /etc/gitbay/apns.p8'
3267 ```
3268 In the config for the drill only: `site_url` to `http://<drill-ip>:8080`,
3269 `[http] addr = ":8080"`, `tls = "off"`, remove `[mail]`, set
3270 `registration.mode = "closed"`, `[push] enabled = false`, remove
3271 `[backup]` (step 7's archive is local and read back at once).
32726. Install the gitbayd binary of the tag bay1 runs (`/healthz` names the
3273 commit) with `deploy/install.sh <drill-ip> 2222`; it will not create
3274 a key because one is present.
32757. Checks, each recorded in the table:
3276 - Database integrity and connectivity: as `gitbay`,
3277 `gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz && gitbayd admin backup --verify /tmp/drill.tar.gz`
3278 → `integrity ok`, `connectivity ok on N repositories`; N equals
3279 `gitbayd admin stats --json` repository count on bay1 at the
3280 snapshot.
3281 - Secrets: `gitbayd --config /etc/gitbay/config.toml admin secrets check`
3282 → every value under one key, no error. The journal shows no
3283 `sealing secrets` failure. `sha256sum /etc/gitbay/apns.p8` equals
3284 bay1's.
3285 - LFS: `cd /var/lib/gitbay/lfs && find . -type f | while read f; do [ "$(sha256sum < "$f" | cut -c1-64)" = "$(basename "$f")" ] || echo "BAD $f"; done` → no output; object count against bay1's.
3286 - Release assets: every row's file exists with its digest:
3287 ```sh
3288 sqlite3 /var/lib/gitbay/gitbay.db "SELECT COALESCE(u.username, o.name) || '/' || r.name || '.git/gitbay-releases/' || a.release_id || '/' || a.name, a.sha256 FROM release_assets a JOIN releases rl ON rl.id = a.release_id JOIN repos r ON r.id = rl.repo_id LEFT JOIN users u ON r.owner_kind = 'user' AND u.id = r.owner_id LEFT JOIN orgs o ON r.owner_kind = 'org' AND o.id = r.owner_id" |
3289 while IFS='|' read p sum; do [ "$(sha256sum < "/var/lib/gitbay/repos/$p" | cut -c1-64)" = "$sum" ] || echo "BAD $p"; done
3290 ```
3291 → no output.
3292 - Host key: `ssh-keyscan -p 22 <drill-ip>` fingerprint equals
3293 `ssh-keyscan -p 22 gitbay.org`'s.
3294 - Config: `gitbayd --config /etc/gitbay/config.toml check-config` → `config ok`.
32958. Service: from the laptop, `ssh -p 22 git@<drill-ip> whoami` →
3296 `cmc`; `git clone ssh://git@<drill-ip>/krz/gitbay.git` succeeds.
3297 **Clock stops** at the clone.
32989. Record on `.gitbay/wiki/Admin.org`, Restore drill table: date, host,
3299 snapshot time, time to service (step 2 → step 8), each check's
3300 result, and anything that needed a manual fix in Notes. Update:
3301 - `Architecture/10-Known-Gaps.org`: delete the `#259` row; the
3302 "measured recovery time" question row reads the measured figure
3303 and the date.
3304 - `Architecture/09-Controls.org:102`:
3305 `| Restore tested | in place | drill <date>, Admin wiki "Restore drill" |`.
3306 - `Architecture/08-Operations.org`: the "Recovery time" bullet names
3307 the figure.
3308 Branch `restore-drill-record`, one signed commit ending
3309 `Closes #259`, MR, ff merge, delete the branch. Later drills add a
3310 row with `Ref #259`.
331110. Destroy the drill host.
3312
3313## D. Offsite backup to Cloudflare R2; keys repository
3314
3315Today: `gitbay-offsite.service` (timer nightly, about 00:19 UTC) runs
3316`/usr/local/bin/gitbay-offsite`, which sources `/etc/gitbay/offsite.env`
3317(`RESTIC_REPOSITORY` and the rest), builds `/var/lib/gitbay-stage`
3318with `sqlite3 gitbay.db "VACUUM INTO ..."` and `cp -a
3319/etc/gitbay/config.toml`, runs `restic backup --tag gitbay` of the stage
3320and `/var/lib/gitbay` excluding `gitbay.db`, `gitbay.db-wal`,
3321`gitbay.db-shm` and `hook.sock`, then `restic check` up to three
3322attempts. The target is Scaleway `fr-par` with append-only credentials;
3323forget and prune run from the laptop. `/etc/gitbay/apns.p8` and
3324`/etc/gitbay/offsite.env` are in no restic repository.
3325
3326After D:
3327
3328- The main repository is on R2, bucket `gitbay-offsite`. `config.toml`
3329 stays in it through the stage, as today.
3330- `apns.p8` and `secret.key` are in a separate restic repository,
3331 bucket `gitbay-keys`, with its own password and token. Neither the
3332 password nor the token is ever on bay1: the laptop streams the files
3333 out of bay1 into it, so a leak of the main backup or of bay1's
3334 `offsite.env` does not reach the key that opens the sealed secrets.
3335- `offsite.env` stays on bay1 because the nightly job sources it. Its
3336 only copy off bay1 is the password manager; neither repository
3337 carries it. The keys repository's password and token exist only in
3338 the password manager.
3339
3340What protects history changes. R2 has no append-only token (token
3341permissions are Admin R/W, Admin R, Object R/W, Object R, optionally
3342scoped to buckets), and restic needs to write and delete under
3343`locks/`, so the host holds Object R/W on its bucket. Bucket lock
3344rules refuse deletion and overwrite of matching objects for a period or
3345indefinitely, whatever the token, and changing them needs an Admin
3346token or the dashboard, neither of which is on bay1. So a compromised
3347host cannot delete or overwrite anything younger than the retention
3348period `RET`; unlike the Scaleway key, it can delete snapshots and data
3349older than `RET`. Prune from the laptop can likewise only remove what
3350is past `RET`.
3351
3352`RET` is 90 days in the commands below (open question 1). The lock
3353rules:
3354
3355| Prefix | Rule | Why |
3356|---|---|---|
3357| `data/` | `RET` days | pack files; prune deletes unused ones once past `RET` |
3358| `index/` | `RET` days | prune replaces index files (see D.6) |
3359| `snapshots/` | `RET` days | forget deletes snapshot files once past `RET` |
3360| `keys/` | indefinite | restic deletes a key file only on `restic key remove` |
3361| `config` | indefinite | written once at `restic init`; the repository cannot be opened without it |
3362| `locks/` | none | restic creates and deletes a lock on every command |
3363
3364`config` is not in the four prefixes named when this was decided; it
3365is one object that restic never deletes and without which nothing in
3366the repository opens, so it is locked too. With `keys/` indefinite, a
3367repository password change adds a key but cannot remove the old one.
3368
3369The keys bucket uses `--retention-indefinite` on every prefix except
3370`locks/` and is never forgotten or pruned: a retired secret key still
3371opens the database in every main snapshot sealed under it, and the
3372snapshots are a few hundred bytes.
3373
33741. File the issue the wiki MR (D.14) references:
3375 `gitbay issue create krz/gitbay --title "Offsite backup: move from Scaleway to Cloudflare R2" --body "Runbook D of docs/plans/2026-09-27-data-at-rest-and-backup.md: R2 with bucket lock rules, a separate keys repository, Scaleway retired after 30 nights in parallel."`
3376 Note the number.
33772. Buckets, from the laptop (`wrangler login` as the account owner):
3378 ```sh
3379 npx wrangler r2 bucket create gitbay-offsite-scratch --location weur
3380 npx wrangler r2 bucket create gitbay-offsite --location weur
3381 npx wrangler r2 bucket create gitbay-keys --location weur
3382 ```
3383 `weur` is a location hint that keeps the data in western Europe as
3384 Scaleway `fr-par` did.
33853. Lock rules. Scratch uses one day so D.6 can see a rule expire;
3386 `gitbay-offsite` uses `RET`:
3387 ```sh
3388 lock() { # bucket retention-flag...
3389 b=$1; shift
3390 for p in data index snapshots; do
3391 npx wrangler r2 bucket lock add "$b" --name "restic-$p" --prefix "$p/" "$@"
3392 done
3393 npx wrangler r2 bucket lock add "$b" --name restic-keys --prefix keys/ --retention-indefinite
3394 npx wrangler r2 bucket lock add "$b" --name restic-config --prefix config --retention-indefinite
3395 npx wrangler r2 bucket lock list "$b"
3396 }
3397 lock gitbay-offsite-scratch --retention-days 1
3398 lock gitbay-offsite --retention-days 90
3399 lock gitbay-keys --retention-indefinite
3400 ```
3401 → five rules on each bucket, none covering `locks/`. No other object
3402 in a restic repository starts with `config`, so that prefix matches
3403 the one object.
34044. Tokens, in the dashboard (R2 → Manage API tokens), each "Object Read
3405 & Write" and scoped to one bucket:
3406 - `gitbay-host` → `gitbay-offsite`. Goes to bay1.
3407 - `gitbay-prune` → `gitbay-offsite`. Laptop only: forget, prune,
3408 check, restore.
3409 - `gitbay-keys` → `gitbay-keys`. Laptop only.
3410 - `gitbay-scratch` → `gitbay-offsite-scratch`. Laptop only; revoke
3411 after D.6.
3412 `gitbay-host` and `gitbay-prune` have the same rights; they are
3413 separate so either can be revoked alone. The lock rules, not the
3414 token, protect history. Store each access key id and secret in the
3415 password manager.
34165. One password manager entry per repository, each holding the
3417 variables restic reads:
3418 ```sh
3419 RESTIC_REPOSITORY=s3:https://<account-id>.r2.cloudflarestorage.com/<bucket>
3420 RESTIC_PASSWORD=<openssl rand -base64 32, generated once per repository>
3421 AWS_ACCESS_KEY_ID=<token access key id>
3422 AWS_SECRET_ACCESS_KEY=<token secret access key>
3423 AWS_DEFAULT_REGION=auto
3424 ```
3425 Entries: `gitbay r2-host` (bucket `gitbay-offsite`, token
3426 `gitbay-host`), `gitbay r2-prune` (same bucket and password, token
3427 `gitbay-prune`), `gitbay r2-keys` (bucket `gitbay-keys`, its own
3428 password, token `gitbay-keys`), `gitbay r2-scratch`. If the current
3429 `offsite.env` sets anything else (`RESTIC_OPTS`, which the Admin
3430 wiki's commands use), carry it into `r2-host` and `r2-prune`. The
3431 commands below assume the named entry's variables are exported in
3432 the shell.
34336. Validate on the scratch bucket (`gitbay r2-scratch`), from the
3434 laptop, before anything real depends on it. Record each result in
3435 the issue from D.1.
3436 ```sh
3437 restic init
3438 mkdir -p /tmp/r2t && head -c 50M /dev/urandom > /tmp/r2t/a
3439 restic backup --tag gitbay /tmp/r2t # snapshot 1
3440 head -c 50M /dev/urandom > /tmp/r2t/b
3441 restic backup --tag gitbay /tmp/r2t # snapshot 2
3442 rm /tmp/r2t/a
3443 restic backup --tag gitbay /tmp/r2t # snapshot 3
3444 restic check # passes; lock files come and go
3445 restic forget --keep-last 1 # expect a refusal: snapshots 1-2 are younger than a day
3446 restic check
3447 restic prune --max-unused unlimited # record what it deletes or is refused
3448 restic check
3449 ```
3450 Expected on day 0: backup and check succeed (locks/ is unlocked);
3451 forget fails to delete the two snapshot files and says so; check
3452 still passes afterwards. Record whether prune tries to delete an
3453 index file and fails. After 24 hours:
3454 ```sh
3455 restic forget --keep-last 1
3456 restic prune --max-unused unlimited
3457 restic check --read-data
3458 ```
3459 → forget removes snapshots 1 and 2, prune removes the pack holding
3460 only `a`'s data and the superseded index files, and `check
3461 --read-data` passes. Then `restic unlock` succeeds on a stale lock
3462 left by interrupting a backup with Ctrl-C.
3463 If prune on day 0 or day 1 fails on an index file younger than the
3464 rule, restic's index rewrite is deleting files the lock keeps, and
3465 prune would fail every run on the real bucket. Stop there and bring
3466 the result back: the choice is between an index rule shorter than
3467 the data rule, running prune only when no index file is younger
3468 than `RET`, or leaving `index/` unlocked (it can be rebuilt from
3469 `data/` with `restic repair index`). Do not continue to D.7 until
3470 one is chosen.
3471 `--max-unused unlimited` keeps prune from repacking partly used
3472 packs, since a repack deletes the old pack and that pack may be
3473 younger than `RET`.
34747. Initialise the real repositories from the laptop:
3475 `restic init` with `gitbay r2-prune` exported, and again with
3476 `gitbay r2-keys`.
34778. Install the host env file and run both targets. On bay1, paste the
3478 `gitbay r2-host` entry into the file over stdin (Ctrl-D ends it):
3479 ```sh
3480 ssh -p 2222 root@gitbay.org 'umask 077; cat > /etc/gitbay/offsite-r2.env'
3481 ```
3482 Keep the current script as `/usr/local/bin/gitbay-offsite.scaleway`
3483 (`cp -a`). Edit `/usr/local/bin/gitbay-offsite`: leave the staging
3484 lines (`VACUUM INTO`, `cp -a config.toml`) as they are and run once;
3485 move the `. /etc/gitbay/offsite.env`, the `restic backup` and the
3486 `restic check` retry loop into a function called once per env file,
3487 each call in a subshell so one target's variables cannot reach the
3488 other. With the flags the job uses today, the section after staging
3489 reads:
3490 ```sh
3491 target() {
3492 set -a; . "$1"; set +a
3493 restic backup --tag gitbay \
3494 --exclude /var/lib/gitbay/gitbay.db \
3495 --exclude /var/lib/gitbay/gitbay.db-wal \
3496 --exclude /var/lib/gitbay/gitbay.db-shm \
3497 --exclude /var/lib/gitbay/hook.sock \
3498 /var/lib/gitbay-stage /var/lib/gitbay || return 1
3499 for i in 1 2 3; do
3500 restic check && return 0
3501 done
3502 return 1
3503 }
3504 rc=0
3505 (target /etc/gitbay/offsite.env) || rc=1
3506 (target /etc/gitbay/offsite-r2.env) || rc=1
3507 exit $rc
3508 ```
3509 Compare with `diff -u /usr/local/bin/gitbay-offsite.scaleway /usr/local/bin/gitbay-offsite`
3510 and keep anything the current restic lines carry that this section
3511 does not (`$RESTIC_OPTS`, a sleep between check attempts, the exact
3512 exclude spelling). A failure on one target no longer stops the
3513 other; the unit still fails if either did. The job now uploads
3514 twice: check `TimeoutStartSec` in `systemctl cat gitbay-offsite.service`
3515 against twice the last run's duration
3516 (`journalctl -u gitbay-offsite -n 200`).
35179. Keys repository, from the laptop with `gitbay r2-keys` exported.
3518 The file goes from bay1 to R2 through a pipe and is never written on
3519 the laptop:
3520 ```sh
3521 ssh -p 2222 root@gitbay.org cat /etc/gitbay/apns.p8 | restic backup --stdin --stdin-filename apns.p8 --tag apns
3522 restic dump --tag apns latest /apns.p8 | sha256sum
3523 ssh -p 2222 root@gitbay.org sha256sum /etc/gitbay/apns.p8
3524 ```
3525 → the two digests match. After MR 1 deploys (A.3) and after every
3526 rotation, the same with `secret.key` and `--tag secret-key`. After
3527 an APNs key change, the same with `apns.p8`.
352810. First full backup: `ssh -p 2222 root@gitbay.org 'systemctl start gitbay-offsite.service; journalctl -u gitbay-offsite -n 50 --no-pager'`
3529 → both targets back up and check. From the laptop with
3530 `gitbay r2-prune`: `restic snapshots` shows one `gitbay` snapshot
3531 with the stage and `/var/lib/gitbay`; `restic ls latest /var/lib/gitbay-stage`
3532 lists the database copy and `config.toml`.
353311. Run both targets for 30 nights. Each week, from the laptop with
3534 `gitbay r2-prune`: `restic snapshots --tag gitbay --latest 7` (one
3535 per night) and `restic check --read-data-subset 1/4`, a different
3536 quarter each week. At the end: `restic check --read-data` on R2
3537 (R2 does not charge egress).
353812. Laptop forget and prune for R2, from the laptop with
3539 `gitbay r2-prune`: the forget policy used for Scaleway today plus
3540 `--keep-within 90d`, so no snapshot younger than `RET` is ever
3541 forgotten, then `restic prune --max-unused unlimited` (or what D.6
3542 settled on). Nothing is removable before day 90; the first prune
3543 that deletes anything is after that.
354413. Retire Scaleway after the 30 nights and a clean `check --read-data`:
3545 - On bay1: `mv /etc/gitbay/offsite-r2.env /etc/gitbay/offsite.env`
3546 (the Scaleway file is replaced) and drop the second `target` call
3547 from the script, so the job runs `(target /etc/gitbay/offsite.env)`
3548 alone. Run the unit once and check the journal.
3549 - Replace the `offsite.env` copy in the password manager with
3550 `gitbay r2-host`; remove the Scaleway entries once the bucket is
3551 gone.
3552 - Revoke the Scaleway append-only key. Keep the Scaleway bucket
3553 until R2 holds 90 days of snapshots, then delete it with the
3554 Scaleway owner credentials from the laptop.
3555 - Revoke `gitbay-scratch`. The scratch bucket's `keys/` and
3556 `config` rules are indefinite: remove its five rules
3557 (`npx wrangler r2 bucket lock remove gitbay-offsite-scratch --name <rule>`),
3558 then empty and delete the bucket in the dashboard.
355914. Wiki MR, branch `offsite-r2-wiki`, commit ending `Ref #<D.1 issue>`
3560 (and `Closes #<D.1 issue>` if the move is finished):
3561 - `Admin.org`, `** Offsite copies`, the first paragraph becomes:
3562 ```org
3563 bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
3564 =/var/lib/gitbay-stage= (a =VACUUM INTO= copy of the database and
3565 =config.toml=) to a Cloudflare R2 bucket. R2 has no append-only
3566 token, so the host's token can write and delete objects; bucket
3567 lock rules refuse deletion and overwrite of =data/=, =index/= and
3568 =snapshots/= for 90 days, and of =keys/= and =config= for good,
3569 whatever the token. Changing the rules needs an Admin token or
3570 the dashboard, neither of which is on bay1. A compromised host
3571 cannot remove anything younger than 90 days; it can remove older
3572 snapshots. Forgetting, pruning and rewriting run from the
3573 operator's machine, and can likewise only remove what is past 90
3574 days.
3575
3576 =apns.p8= and =secret.key= are in a separate restic repository in
3577 its own bucket, with its own password and token, neither of which
3578 is on bay1; the operator streams the files into it from bay1. A
3579 leak of the main backup therefore does not carry the key that
3580 opens its secrets. =offsite.env= is in neither repository.
3581 ```
3582 - `Admin.org`, `*** Removing a repository's history from every
3583 snapshot`: after the sentence ending "rather than forgetting the
3584 snapshots: everything else in them stays restorable.", add
3585 "Snapshots and packs younger than 90 days are locked:
3586 =rewrite --forget= and =prune= cannot remove them until they
3587 age out." The `~/.config/gitbay/offsite.env` sourcing line
3588 becomes "export the =gitbay r2-prune= entry from the password
3589 manager".
3590 - `Architecture/08-Operations.org:53`: "to object storage" → "to
3591 Cloudflare R2"; lines 63-65 become "The host's R2 token can
3592 delete, but bucket lock rules keep everything younger than 90
3593 days; the lock rules are changed only off the host
3594 (documented: Admin wiki)."
3595 - `Architecture/09-Controls.org:101`:
3596 `| Backups offsite and delete-locked | in place | restic to R2; bucket lock rules, 90 days on data, index and snapshots (documented) |`
3597 - `Architecture/04-Trust-Boundaries.org:30`: "append-only offsite
3598 backup credentials" → "offsite backup under R2 bucket locks".
3599 - `Threat-Model.org:195`: "restic append-only credentials" →
3600 "restic under R2 bucket locks".
3601 - `Architecture/diagrams/diagrams.py:165`: `"restic · append-only key"`
3602 → `"restic · R2 bucket locks"`; regenerate with
3603 `python3 .gitbay/wiki/Architecture/diagrams/diagrams.py .gitbay/wiki/Architecture/diagrams`
3604 and commit the SVGs.
3605 - If MR 1 landed without the keys-repository sentences (Task 1.6),
3606 add them to `Admin.org` `** Secret key` and
3607 `Architecture/06-Data-and-Cryptography.org` here.
3608
3609---
3610
3611## Open questions
3612
36131. The lock retention `RET`: the runbook uses 90 days. It is the
3614 window a compromised host cannot touch and the minimum age of
3615 anything prune can remove, so it should be at least as long as the
3616 shortest period the current forget policy keeps (that policy is on
3617 the laptop, not in the repository).
36182. Out of scope, noted while reading: `webhook add` takes `--secret`
3619 on argv (`internal/control/webhook.go:16-23`), against the
3620 stdin-only rule for secrets.
36213. Out of scope: the offsite job (`/usr/local/bin/gitbay-offsite`, its
3622 unit and timer) is not in the repository and `deploy/cloud-init.yaml`
3623 does not template it, so a host built from cloud-init has no offsite
3624 backup until the operator installs one by hand.
3625
3626## Self-review
3627
3628- #273: encryption of the four columns (Task 1.3), key file outside the
3629 database and root (1.2), key id prefix (1.1), rotation (1.4
3630 `rotate`), existing clear rows sealed at startup (1.4 `serve`,
3631 `ResealSecrets`), missing key behaviour (1.4 `openStore`, documented
3632 1.6), backups do not carry it (validation 1.2, e2e 1.5), install
3633 provisioning (1.6), offsite copy only in the separate keys repository
3634 (runbook A.3, D.9).
3635- #274: age recipients config (2.1), encryption (2.2), `--verify
3636 --identity` (2.2), restic path unaffected by the archives because the
3637 offsite job stages from live data (runbook B.4), scripts (2.3).
3638- #259: `--verify` connectivity (3.1, 3.4), delete/rename/transfer held
3639 (3.2, 3.3, 3.4), drill covering database integrity, connectivity,
3640 LFS, release assets, config, host keys, secrets and the keys, with time
3641 to service on the Admin page (3.5, runbook C). The drill is deferred;
3642 MR 3 says `Ref #259` and #259 stays open until the first drill is
3643 recorded. Cadence: quarterly and after any backup code change.
3644- Offsite move: R2 buckets, bucket-scoped tokens for host, prune and
3645 keys, lock rules on `data/`, `index/`, `snapshots/`, `keys/` (and
3646 `config`) with `locks/` open, validation on a scratch bucket with the
3647 same rules, both targets in parallel for 30 nights, Scaleway retired,
3648 wiki MR (runbook D). No code MR: the offsite job is not in the
3649 repository or in `deploy/cloud-init.yaml`.
3650- Names used across tasks: `seal.Keyring`/`Load`/`Seal`/`Open`/
3651 `CurrentID`/`KeyID`/`IsSealed`/`NewKey`/`ReadKeys`/`WriteKeys`;
3652 `Store.SetKeyring`/`ResealSecrets`/`SecretKeyUse`; `testConfig`;
3653 `archivePath`/`archiveReader`/`verifyBackup(path, identity)`;
3654 `backuplock.Hold`/`TryShared`/`ErrBusy`/`Name`; `holdOffBackup`;
3655 `gitutil.FsckConnectivity`. Consistent.