.gitbay/wiki/Architecture/08-Operations.org

 1#+title: Operations
 2
 3* Logging
 4
 5- The daemon logs with Go's =log/slog= default handler to stderr, which
 6  systemd sends to the journal. Retention is the journal's.
 7- Logged: listener start-up, schema version, worker failures (webhook,
 8  mail, push, mirror), sweeps and reaps with counts, SSH lookup errors.
 9- Not logged: request bodies, tokens, secrets. Mail errors are logged
10  with addresses redacted.
11
12* Audit log
13
14Table =audit_log=: actor, action, JSON data, time
15(=internal/store/audit.go=). Readable by admins with =audit=.
16
17| Recorded                                     | How                                                  |
18|----------------------------------------------+------------------------------------------------------|
19| Every successful mutating command, every surface | =Dispatch= writes =cmd <path>= with pruned argv and the source: key fingerprint, =web=, =api= or =host= (=internal/control/control.go=) |
20| SSH authentication failures and throttling   | =auth.failed= (IP, fingerprint), =auth.throttled= (IP) |
21| Registration                                  | =auth.registered=, =pending.expired=                 |
22| Administration                                | =admin user.*=, =admin email.*=, =admin invite.issued=, =admin repo.*=, =admin mr.prune=, =admin runners.forget= |
23| Repository events of security interest        | =push.forced=, =repo.runner.add/remove=, =pages.domain_verified= |
24
25Failed commands and reads are not audited. The separate =events= table is
26the product activity feed, not an audit trail.
27
28* Monitoring
29
30- =/healthz= returns the serving commit and a database check.
31- =deploy/cloud-init.yaml= installs an hourly heartbeat that checks the
32  service, disk, certificate expiry and backup age, and can POST to an
33  external monitor URL.
34- =admin runners= reports the build queue: pending builds, claims and
35  average and worst claim wait over 24 hours, reaped builds, and each
36  runner key's last poll.
37
38* Patching
39
40- Host: =unattended-upgrades= with automatic security updates and a
41  04:30 reboot (=deploy/cloud-init.yaml=).
42- Application: =govulncheck= nightly in CI; a module update is a
43  normal merge request and deploy.
44- CI image: rebuilt by the operator when =deploy/Containerfile.ci=
45  changes; weekly =podman image prune= removes old images.
46
47* Backup and recovery
48
49| Item            | Schedule | Kept | Contents                                                        |
50|-----------------+----------+------+-----------------------------------------------------------------|
51| Full archive    | nightly  | 7    | SQLite snapshot (=VACUUM INTO=), all repositories, LFS, SSH host keys; age-encrypted when =[backup] age_recipients= is set |
52| Database only   | hourly   | 48   | SQLite snapshot; age-encrypted when =[backup] age_recipients= is set |
53| Offsite (restic)| nightly  | per prune policy | =/var/lib/gitbay= and a staged database copy, to object storage |
54
55- The database snapshot is taken before repositories are read, so a
56  push during the backup leaves only unreferenced objects
57  (=cmd/gitbayd/backup.go=).
58- Excluded: WAL files, the hook socket, askpass scripts, generated
59  hooks.
60- =gitbayd admin backup --verify= checks SQLite integrity and that every
61  repository the database names is present (=backup.go=). It does
62  not check git object connectivity.
63- The host's restic credentials are append-only; the key that can
64  delete or prune snapshots is held off the host, so a compromised host
65  cannot destroy its own history (documented: Admin wiki).
66- Recovery point: about one hour for database-only data (issues, merge
67  requests, reviews), one day for repositories.
68- Recovery time: not measured. No restore onto a clean host has been
69  recorded (#259).
70
71Restore procedure: extract the archive into an empty directory, point
72=server.root= at it, start =gitbayd=; hooks regenerate and the host key
73is preserved.
74
75* Operator levers during an incident
76
77| Need                               | Command                                          |
78|------------------------------------+--------------------------------------------------|
79| Stop a user                        | =admin user disable <name>=                      |
80| Remove a key                       | =keys remove= (own) or =admin user= commands     |
81| Kill a user's browser sessions     | =web sessions revoke --all= (as that user)       |
82| Revoke a token                     | =token revoke <name>=                            |
83| Stop a runner key claiming         | =repo runner remove=, =admin runners forget <fingerprint>= |
84| Hide a repository                  | =admin repo visibility <repo> private=           |
85| Close registration                 | =registration.mode = "closed"= and restart       |
86| See what happened                  | =audit= (filter by actor, action, time)                |