.gitbay/wiki/Architecture/08-Operations.org

 1#+title: Operations
 2
 3* Logging
 4
 5- The daemon logs with Go's =log/slog= default handler to stderr, which
 6  systemd sends to the journal. Retention is the journal's.
 7- Logged: listener start-up, schema version, worker failures (webhook,
 8  mail, push, mirror), sweeps and reaps with counts, SSH lookup errors.
 9- Not logged: request bodies, tokens, secrets. Mail errors are logged
10  with addresses redacted.
11
12* Audit log
13
14Table =audit_log=: actor, action, JSON data, time
15(=internal/store/audit.go=). Readable by admins with =audit=.
16
17| Recorded                                     | How                                                  |
18|----------------------------------------------+------------------------------------------------------|
19| Every successful mutating command, every surface | =Dispatch= writes =cmd <path>= with pruned argv and the source: key fingerprint, =web=, =api= or =host= (=internal/control/control.go=) |
20| SSH authentication failures and throttling   | =auth.failed= (IP, fingerprint), =auth.throttled= (IP) |
21| Registration                                  | =auth.registered=, =pending.expired=                 |
22| Administration                                | =admin user.*=, =admin email.*=, =admin invite.issued=, =admin repo.*=, =admin mr.prune=, =admin runners.forget= |
23| Repository events of security interest        | =push.forced=, =repo.runner.add/remove=, =pages.domain_verified= |
24
25Failed commands and reads are not audited. The separate =events= table is
26the product activity feed, not an audit trail.
27
28* Monitoring
29
30- =/healthz= returns the serving commit and a database check.
31- =deploy/cloud-init.yaml= installs an hourly heartbeat that checks the
32  service, disk, certificate expiry and backup age, and can POST to an
33  external monitor URL.
34- =admin runners= reports the build queue: pending builds, claims and
35  average and worst claim wait over 24 hours, reaped builds, and each
36  runner key's last poll.
37
38* Patching
39
40- Host: =unattended-upgrades= with automatic security updates and a
41  04:30 reboot (=deploy/cloud-init.yaml=).
42- Application: =govulncheck= nightly in CI; a module update is a
43  normal merge request and deploy.
44- CI image: rebuilt by the operator when =deploy/Containerfile.ci=
45  changes; weekly =podman image prune= removes old images.
46
47* Backup and recovery
48
49| Item            | Schedule | Kept | Contents                                                        |
50|-----------------+----------+------+-----------------------------------------------------------------|
51| Full archive    | nightly  | 7    | SQLite snapshot (=VACUUM INTO=), all repositories, LFS, SSH host keys; age-encrypted when =[backup] age_recipients= is set |
52| Database only   | hourly   | 48   | SQLite snapshot; age-encrypted when =[backup] age_recipients= is set |
53| Offsite (restic)| nightly  | per prune policy | =/var/lib/gitbay= and a staged database copy, to object storage |
54
55- The database snapshot is taken before repositories are read, and
56  each repository's HEAD, refs/ and packed-refs are archived before its
57  objects, so every archived ref finds the objects it reaches, unless
58  git's own automatic gc after a push repacks during the walk; the
59  archive can then miss objects, and =--verify= reports it. A push
60  during the backup is missing or present as unreferenced objects
61  (=cmd/gitbayd/backup.go=).
62- Excluded: WAL files, the hook socket, askpass scripts, generated
63  hooks.
64- =gitbayd admin backup --verify= checks SQLite integrity, that every
65  repository the database names is present, =git fsck
66  --connectivity-only= on each, release assets against their recorded
67  sha256, and LFS objects against their names (=backup.go=).
68  =gitbayd admin restore-drill= runs the same checks on a full
69  extraction and reports elapsed time and the newest recovered
70  activity (=restoredrill.go=).
71- Repository deletes, renames and transfers refuse while a full backup
72  runs (=internal/backuplock=), so the snapshot and the walk agree.
73- The host's restic credentials are append-only; the key that can
74  delete or prune snapshots is held off the host, so a compromised host
75  cannot destroy its own history (documented: Admin wiki).
76- Recovery point: about one hour for database-only data (issues, merge
77  requests, reviews), one day for repositories.
78- Recovery time: see the Admin wiki's Restore drill table.
79
80Restore procedure: extract the archive into an empty directory, point
81=server.root= at it, start =gitbayd=; hooks regenerate and the host key
82is preserved.
83
84* Operator levers during an incident
85
86| Need                               | Command                                          |
87|------------------------------------+--------------------------------------------------|
88| Stop a user                        | =admin user disable <name>=                      |
89| Remove a key                       | =keys remove= (own) or =admin user= commands     |
90| Kill a user's browser sessions     | =web sessions revoke --all= (as that user)       |
91| Revoke a token                     | =token revoke <name>=                            |
92| Stop a runner key claiming         | =repo runner remove=, =admin runners forget <fingerprint>= |
93| Hide a repository                  | =admin repo visibility <repo> private=           |
94| Close registration                 | =registration.mode = "closed"= and restart       |
95| See what happened                  | =audit= (filter by actor, action, time)                |