.gitbay/wiki/Admin.org

   1#+title: gitbay admin guide
   2
   3One static binary (=gitbayd=), one SQLite file, bare repositories on
   4disk, and the system =git=. Schema migrations run automatically on
   5startup and on every admin command.
   6
   7* Install
   8
   9Build from source (=go build ./cmd/gitbayd=), install via the vanity
  10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
  11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
  12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
  13manifest (CGO off, trimpath, stripped — byte-identical per commit and
  14toolchain).
  15
  16#+begin_src sh
  17install -m 755 gitbayd /usr/local/bin/
  18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
  19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
  20gitbayd --config /etc/gitbay/config.toml check-config
  21gitbayd --config /etc/gitbay/config.toml admin secrets init
  22chown gitbay:gitbay /etc/gitbay/secret.key
  23#+end_src
  24
  25=admin secrets init= refuses when the key file already exists, so a
  26reinstall on the same host should skip it — =deploy/install.sh= does
  27this with a file check before running it.
  28
  29=deploy/= in the source tree has a cloud-init file, a hardened systemd
  30unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
  31the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
  3222/80/443 without root.
  33
  34** The SSH port decision
  35
  36- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
  37  22 — move the host's admin sshd to another port. Remotes read
  38  =git@host:owner/repo= with no port gymnastics.
  39- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
  40  =AuthorizedKeysCommand=:
  41  #+begin_example
  42  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
  43  AuthorizedKeysCommandUser gitbay
  44  #+end_example
  45  sshd requires that binary to be root-owned and not group/world
  46  writable. Unknown keys fail authentication inside sshd, so system mode
  47  requires =registration.mode = "closed"= (check-config enforces this).
  48  Under the forced command the CLI's leading =--term=<cols>[,color]=
  49  argument works as is; a client that sets =GITBAY_TERM= instead needs
  50  =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
  51  handled by the host's sshd, not gitbayd.
  52
  53* Configuration reference
  54
  55=/etc/gitbay/config.toml=. =check-config= validates and names every
  56contradiction; =--no-host-checks= skips port/path probes.
  57=gitbayd admin config show= prints the configuration in effect as TOML,
  58every default filled in and =smtp_pass= redacted. A file that fails
  59validation still prints, followed by the contradiction.
  60
  61** [server]
  62- =root= (default =/var/lib/gitbay=) — repositories, database, host
  63  keys, ACME cache all live here.
  64- =site_url= (required) — canonical =https://host=; drives ACME, clone
  65  URLs, mail links.
  66- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
  67  seal CI secrets, webhook secrets, mirror tokens and push device
  68  tokens in the database. Must be outside =root=, mode 0600, readable
  69  by the daemon's user. See "Secret key" below.
  70- =source_repo= (optional, =owner/name=) — the repository this instance
  71  develops itself in. Startup warns when the running build's commit is
  72  not on that repository's default branch, which is how a binary built
  73  from an unmerged branch stops being invisible. Leave it unset unless
  74  the instance hosts its own source.
  75
  76** [ssh]
  77- =mode= — =embedded= | =system= (above).
  78- =port= (22) — embedded listener port.
  79- =host_keys= — list of private key paths; empty generates an ed25519
  80  key at =<root>/ssh/host_ed25519=.
  81
  82** [http]
  83- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
  84- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
  85  =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
  86  (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
  87  failing to bind it is a warning, not fatal. Requires an =https://=
  88  site_url with a public DNS name.
  89- =files=: =cert_file= + =key_file=.
  90- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
  91- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
  92  =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
  93  release the binary was built with. =openssl s_client -connect
  94  <host>:443 -tls1_1= fails the handshake.
  95
  96=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
  97front of the daemon. A request from one of them is attributed, for API
  98rate limiting, to the last =X-Forwarded-For= hop that is not itself a
  99trusted proxy; from anyone else the header is ignored. Empty, the
 100default, is right when gitbayd terminates TLS itself.
 101
 102A reverse proxy in front of gitbay must not buffer responses, or the
 103build page's live log arrives only when the build ends;
 104=X-Accel-Buffering: no= covers nginx.
 105
 106** [web]
 107- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
 108  web routes are never registered; in accounts, browser sessions are
 109  minted over SSH (=web login=), and users with write access can create
 110  repos, comment, and make simple file edits (which commit unsigned,
 111  honestly). =password_auth= is reserved and currently rejected.
 112- =title= — the instance's display name in the rail and page titles;
 113  empty falls back to the site host. Lower case is the convention for
 114  gitbay itself.
 115- =privacy_notice= — operator text shown on =/privacy= under the fixed
 116  statement.
 117
 118** [registration]
 119- =mode= — =closed= (default) | =invite= | =open=. invite/open require
 120  [mail]. See the user guide for the flows.
 121
 122- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
 123  self-registered account still unverified after that long is removed,
 124  hourly and at start, audited as =pending.expired=.
 125
 126- =notify_admin= (false) — mail every instance admin when an account
 127  becomes active: an invite redeemed, or an open-mode signup that
 128  verified its address. The unverified row an open signup creates is
 129  not reported, because anyone can post the form and mailing on that
 130  would point a flood at the admins. Recipients are the verified
 131  primary addresses of active admins who have activity mail on, the
 132  same rule any other notice follows, so an admin with no verified
 133  address hears nothing. The notice is queued, so a dead SMTP host
 134  shows up in the admin page's Mail table instead of failing the
 135  registration. Requires [mail].
 136
 137** [mail]
 138- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
 139  =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
 140  registration and self-service =email add=; in closed mode you may omit
 141  it entirely and assert addresses by hand (below).
 142- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
 143  for relays on 465).
 144- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
 145  gets no mail: delivery fails and retries, and the admin page's Mail
 146  table shows the error. Unset, it is on for every relay except
 147  =localhost= and loopback addresses; set =false= to allow plaintext
 148  to a remote relay.
 149
 150** [push]
 151Push notifications to Apple devices, delivered by gitbayd talking to
 152APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
 153operator-supplied provider key. Off unless configured.
 154
 155- =enabled= (false).
 156- =key_file= — path to the =.p8= provider key from Apple's developer
 157  portal (Certificates, Identifiers & Profiles → Keys). It belongs at
 158  =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
 159  as. Read and validated at startup: it must parse as a PEM-wrapped
 160  PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
 161  than fill a queue nobody is watching.
 162- =key_id=, =team_id= — the key's id and your Apple developer team id,
 163  both from the same portal page.
 164- =topic= — the app's bundle identifier. *An APNs key belongs to a
 165  bundle ID.* gitbay.org pushes to the App Store build under its own
 166  bundle id; a self-hoster who wants push ships their own iOS build
 167  under their own bundle id, with its own =.p8= key from their own
 168  developer account, and points =topic= at that id. There is no way to
 169  push to someone else's build, by design — this is Apple's model, not
 170  gitbay's.
 171- =environment= — =production= or =sandbox=, naming the APNs host
 172  rather than taking a URL, so a typo cannot aim the key at a host that
 173  is not Apple's.
 174
 175All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
 176are required when =enabled= is true; validation runs at config load,
 177so a misconfigured =[push]= is caught before the daemon serves
 178anything. The delivery queue (a device's undelivered and attempted
 179pushes) is capped the same way the mail queue is, by =[retention]
 180push=.
 181
 182** [backup]
 183- =age_recipients= (optional) — age public keys (=age1...=). When set,
 184  =admin backup= encrypts every archive to them and appends =.age= to
 185  its name. Generate the pair off the host with =age-keygen=; only the
 186  public key goes here, so the host writes archives it cannot read.
 187  The restic copy is unaffected: the offsite job stages its own
 188  =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
 189  not the archives.
 190
 191** [api]
 192- =enabled= (false) — the JSON API surface; see [[API]]. Off
 193  means no credential-bearing HTTP endpoint exists at all.
 194
 195** [webhooks]
 196- =allow_local= (false) — permit webhook and mirror targets on
 197  loopback, private, shared (100.64.0.0/10), link-local or multicast
 198  addresses. Leave off unless you know why you need it (SSRF).
 199
 200** [limits]
 201- =clone_timeout= (3600s) — cap on =repo import= fetches.
 202- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
 203- =max_asset_bytes= (512MB) — cap per uploaded release asset.
 204- =max_snippet_bytes= (1MB) — cap per snippet file.
 205- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
 206- =max_repos_per_user= (0, unlimited) — repositories an account may own
 207  directly; =repo create=, =fork= and =import= refuse past it.
 208  Organizations are not capped.
 209- =max_bytes_per_user= (0, unlimited) — disk the account's own
 210  repositories may take; a push may be no larger than what is left.
 211- =pack_concurrency= (3), =pack_per_principal= (2), =pack_queue= (32),
 212  =pack_queue_wait= (="60s"=) — git pack generation (clones, fetches,
 213  =git archive --remote=, web archive downloads) over SSH, smart HTTP
 214  and git:// shares one
 215  budget: this many at once, this many per account (per client
 216  address when anonymous: an IPv4 address, or an IPv6 /64), and this
 217  many waiting for at most the wait. Anonymous clients together hold
 218  at most =pack_concurrency= − 1 slots when it is above 1, so a signed-in
 219  client (SSH key, bearer token or web session) can always get the last.
 220  Past that an SSH client gets "the server is busy…" and exit 1, HTTP
 221  gets 503 with =Retry-After: 30=, git:// an =ERR= line. A queued
 222  client that disconnects leaves the queue; a running clone whose
 223  client disconnects is killed. Ref listings (info/refs, protocol v2
 224  =ls-refs=), pushes and =repo download= are outside the budget. For the
 225  three counts 0 means the default and a negative value turns that
 226  bound off. The defaults suit a four-core host; see [[Performance]].
 227  With =ssh.mode = "system"= each SSH session is its own process and
 228  SSH clones are not counted.
 229- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
 230
 231** [git_daemon]
 232- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
 233  Serves only public repositories that additionally ran
 234  =repo settings git-daemon <repo> on=.
 235
 236** [mirrors]
 237- =pull_interval_minutes= (15) — how often pull mirrors fetch their
 238  upstream. Push mirrors sync shortly after each local ref update.
 239  Mirror URLs pass the same SSRF rules as webhook targets, when saved
 240  and again before every sync; git then connects only to the addresses
 241  that were checked (=http.curloptResolve=) and does not follow
 242  redirects, so a mirror of a renamed repository fails until its URL
 243  is updated. Needs git 2.37 or later on the server; with an older git
 244  the worker logs an error at start and syncs no mirror, recording the
 245  reason on each. Sync ignores the system and global gitconfig.
 246
 247** [go_import]
 248Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
 249Requests with =?go-get=1= at or under the module path answer with the
 250go-import meta tag pointing at the repository's HTTPS clone URL, so
 251=go install host/module/cmd/...@latest= resolves. The repository should
 252be public (the module path itself confirms it exists).
 253
 254* Users, email, invites
 255
 256Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
 257backfills is a wrapper that dispatches the registry command of the same
 258name as the host: an admin context with no account behind it, so its
 259audit rows carry no actor and =source: host=. The same commands run in an
 260instance admin's SSH session (=ssh git@<host> admin ...=) and write the
 261same rows with the key fingerprint as source. One implementation, two
 262credentials.
 263
 264#+begin_src sh
 265gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
 266gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
 267gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP
 268#+end_src
 269
 270"Verified" means SMTP-confirmed or host-admin-asserted; the database
 271records which. Verified emails are what make commit signatures
 272meaningful — an unverified address never produces a =verified= badge.
 273
 274* Audit and account control
 275
 276The audit log is the security feed (events are the product feed): every
 277successful mutating command with its argv and source credential (SSH key
 278fingerprint or API), every refused one (exit 3 or 4) as =refused
 279<command>=, refused pushes as =refused git-receive-pack=, registrations,
 280admin actions, force-pushes, and auth failures/throttling. A refusal row
 281keeps the flag names and the first positional, not the values.
 282Refusals are recorded up to ten a minute per account and 600 a minute
 283across the instance; past either, one =refused.throttled= row stands
 284for the rest of that minute. The caps bound the embedded listener, the
 285web and the API; under =ssh.mode = "system"= each =gitbayd shell=
 286connection counts separately. Secrets never appear — they travel on
 287stdin, never in argv.
 288
 289Each row carries the SHA-256 of the row before it. =gitbayd admin audit
 290verify= opens the store as other admin commands do, applying pending
 291migrations, so run it with the binary that matches the daemon. It
 292recomputes the chain and exits 1 naming the first row that was
 293edited or whose predecessor was removed. Retention removing the oldest
 294rows is not a break. Rows written before the chain existed are counted
 295and skipped; when every row is such a row, verify warns and exits 1,
 296since clearing the hash columns looks the same. After an upgrade that
 297clears with the first new audit row.
 298
 299Removing the newest rows leaves no break, and neither do rows written
 300afterwards under the freed ids. The database cannot show either. The
 301daemon logs every row it writes to its journal, outside the database
 302(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
 303id and hash: compare them with the newest journal line. Rows written by
 304host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
 305"system"=, are not copied to the journal.
 306
 307#+begin_src sh
 308gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
 309gitbayd admin audit verify           # check the hash chain; exit 1 names the first bad row
 310ssh git@<host> audit ...             # the same, from an admin session
 311ssh git@<host> admin user list [--state active|pending|disabled|admin]
 312ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
 313ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default]   # per-account caps
 314ssh git@<host> admin user promote <name>   # grant instance admin
 315ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
 316gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
 317gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
 318gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
 319gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
 320                                     #   refused (with each blocker named) while
 321                                     #   the account owns repos, authored
 322                                     #   issues/MRs/comments/reviews, or is an
 323                                     #   org's only admin
 324#+end_src
 325
 326=--actor= takes a username, or =-= for rows with no actor: host commands
 327and auth failures. =--action= is a prefix, so =cmd repo= catches every
 328repository command and =admin= every host or admin-session action.
 329=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
 330
 331=admin user list= pages by username (=--limit=, =--cursor=) and carries
 332each account's state and =last_seen=, the newest use of any of its SSH
 333keys or API tokens. =admin user show= adds the keys with their last use,
 334each address with how it was verified, PGP keys, org roles, the owned
 335repository count, API token names, and live browser sessions. Both are
 336Both are refused to non-admins, like =audit=, on every surface.
 337
 338=/admin/users= is the same list in a browser, linked from the admin
 339page: the state filter the command takes, keyset paging on its cursor,
 340and a row per account with promote, demote, disable and enable, each
 341dispatching the command. Demote and disable ask for the username to be
 342typed, since both take someone's access away. Creating and deleting an
 343account, issuing an invite and asserting an address stay on the command
 344line: each takes a key, mints a credential, or cannot be undone. A
 345non-admin gets the 404 a missing page would, so the URL confirms
 346nothing.
 347
 348Promotion needs an active account: a pending or disabled one is refused.
 349Demotion is refused when it would leave no admin, over SSH, in the
 350browser and on the host alike, so the host-local =promote= is the way
 351back in when the only admin key is lost.
 352
 353Instance admin carries no right on anyone's repository: policy does not
 354consult it, and a private repository still answers not-found to an
 355admin. Moderation goes through explicit overrides that skip the access
 356check and write their own audit row:
 357
 358#+begin_src sh
 359ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
 360ssh git@<host> admin repo archive|unarchive <owner/name>
 361ssh git@<host> admin repo visibility <owner/name> public|private
 362ssh git@<host> admin repo delete <owner/name> --yes
 363#+end_src
 364
 365Each lands in the audit log as =admin repo.<action>= naming the
 366repository, on top of the =cmd= row every mutating command gets.
 367
 368=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
 369per minute — successful auths never count and clear the slate.
 370=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
 371push.
 372
 373=limits.write_rate= (60) bounds *mutating commands per account per
 374minute*. It is counted in the dispatcher, so SSH, the JSON API and the
 375web spend one budget and a caller cannot refresh it by changing surface;
 376=limits.api_rate= stays in front of it, bounding a network source rather
 377than an account. A command is one token whatever it writes, so a bundle
 378import costs one and only a loop of separate commands spends the budget.
 379Read-only commands, the runner protocol (a build streams its log in many
 380small writes) and the host CLI are exempt. Refusals exit 4 and say when
 381to retry. A negative value turns the limit off; it matters most with
 382=registration = "open"=, where every write also queues notification mail
 383and webhook deliveries.
 384
 385* Queues
 386
 387Every background worker keeps a backlog and a failure state. An instance
 388admin reads them all in one place:
 389
 390#+begin_src sh
 391gitbay dashboard --json | jq .queues   # webhooks, mail, push, mirrors, builds, deps
 392#+end_src
 393
 394Per worker: pending, retrying (pending with a failed attempt) and
 395dead-lettered counts with the oldest pending age, and the retrying or
 396failed rows themselves, capped at twenty each. Builds list what is
 397running and then what is pending, each since when, so a build no runner
 398is scoped to claim is visible here rather than only in its repository;
 399mirrors list the ones whose last sync failed;
 400dependency checks list the ones whose last check errored. Non-admins get
 401no =queues= key at all.
 402
 403Push rows name the device id, never the token. Watch this one after
 404configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
 405config validation, which can only check that the =.p8= parses, and then
 406every send comes back =403 InvalidProviderToken= and dead-letters on its
 407first attempt.
 408
 409In accounts mode the same read renders at =/admin=, linked from the rail
 410for admins. Anyone else gets a 404 there.
 411
 412A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
 413with the address redacted out of the relay's error. The id is the queue
 414row: find it in the Mail table on =/admin=, or in =dashboard --json=,
 415where the recipient and the unredacted error are. That is deliberate —
 416see the Threat-Model page.
 417
 418* Maintenance
 419
 420#+begin_src sh
 421gitbayd admin stats [--json]         # counts, database size, per-repo disk
 422ssh git@<host> admin stats [--json]  # the same, from an admin session
 423gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
 424gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
 425gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)
 426#+end_src
 427
 428A history rewrite leaves the commits it removed reachable through
 429=refs/merge-requests/N/head= of the merge requests that landed them, so
 430they stay fetchable by anyone who can read the repository. Nothing drops
 431a head ref on its own — an open or source-gone MR is merged through it,
 432and a merged or closed one keeps its diff readable through it — so the
 433cleanup is a command an instance admin runs, naming the MRs:
 434
 435#+begin_src sh
 436ssh git@<host> admin mr prune owner/name 1 2 3 --yes
 437#+end_src
 438
 439It refuses an open or source-gone MR, deletes the named refs, runs
 440=git gc --prune=now= on that one repository so the objects go at once
 441rather than after git's two-week grace, leaves a system comment on each
 442MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
 443reviews and head sha; =mr diff= and the MR page say the head is gone.
 444Run it when nothing is pushing to that repository: without the grace, a
 445push caught between leaving quarantine and writing its ref loses its
 446objects. Objects also survive in offsite backups until those are
 447rewritten; see "Removing a repository's history from every snapshot".
 448
 449=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
 450weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
 451layout the source sent, so a first manual =admin gc= after a bulk
 452import is worthwhile.
 453
 454* Backup and restore
 455
 456#+begin_src sh
 457gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
 458gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back
 459#+end_src
 460
 461One archive: a consistent SQLite snapshot (taken *before* the
 462repositories are read, so the database never references objects the
 463archive missed), every repository, and the SSH host keys. Excluded:
 464hook socket, regenerated hook scripts, WAL files. Safe to run against a
 465live daemon. =--out= must be outside =server.root=, or the next full
 466backup would carry the archive. Each run removes the snapshot
 467directories (=.gitbay-snap-*=) and temporary archives (=.*.tmp-*=) a
 468killed run left beside its archive once they are a day old, and prints
 469each one it removes.
 470
 471=--verify= reads an archive back: the snapshot must pass SQLite's
 472integrity check, every repository the snapshot names must be in the
 473archive, and each must pass =git fsck --connectivity-only=. It
 474extracts the repositories to a temporary directory for that, so it
 475needs free space the size of the repositories. A database-only archive
 476is checked for integrity and says so. Exit is non-zero on damage, a
 477missing repository or a missing object.
 478
 479A full backup holds =<root>/backup.lock= from its database snapshot to
 480its last repository. While it runs, =repo delete=, =repo rename=,
 481=repo transfer=, =admin repo delete=, =org rename=, =admin mr prune=
 482and =gitbayd admin gc= refuse with "a backup is running"; retry when it
 483finishes. Database-only backups take no lock. A pack that git's own
 484automatic gc removes during the walk is skipped; each repository's refs
 485are archived before its objects, so the refs still find their objects,
 486and =--verify= reports it if one does not.
 487
 488Verifying an archive of unknown origin: run it as an unprivileged
 489user. The connectivity check runs git with =--git-dir= on each
 490extracted repository, so a directory that is not a repository fails
 491rather than git checking an enclosing one. =objects/info/alternates=
 492and a =commondir= directly in a =*.git= directory are not extracted,
 493so an archive cannot use either to have git read another repository's
 494objects or refs on the host. Git still reads each archived
 495repository's own =config=.
 496
 497Archives carry a directory entry for every directory, including an
 498empty one, so a bare repository whose refs are all packed restores as
 499a repository. Archives written before this release do not: extracting
 500one can leave a repository's =refs/= directory missing, which stops
 501git from recognizing it as a repository at all. =gitbayd admin backup
 502--verify <archive>= names the repositories this affects; the fix is
 503=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
 504which it opens normally.
 505
 506With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
 507and =--verify= needs the private key:
 508
 509#+begin_src sh
 510gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
 511age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
 512#+end_src
 513
 514The identity lives off the host (with the secret key file and the
 515restic credentials), so verifying an encrypted archive happens there
 516or on a restore host.
 517
 518Restore: extract into an empty directory, point =server.root= at it,
 519restore =server.secret_key_file= from its own copy (mode 0600, owned
 520by the account gitbayd runs as), then start gitbayd. No archive carries
 521the key file, and without it gitbayd refuses to start. Host keys are
 522preserved, so clients keep their known_hosts entries; hooks regenerate
 523at startup.
 524
 525** Schedule and recovery point
 526
 527Two timers, because the two halves of the data have different exposure.
 528
 529- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
 530- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
 531  writes the SQLite snapshot alone, last 48 kept. A few MB against the
 532  full archive's hundreds, which is what makes the frequency affordable.
 533
 534The split follows what a loss would actually cost. Repositories are git,
 535so a mirror or any clone is a second copy; the database is the only copy
 536of issues, merge requests, comments and review state. So the recovery
 537point is about an hour for the data that exists nowhere else, and a day
 538for the data that does.
 539
 540Continuous replication (litestream and similar) was considered and not
 541adopted. It would take the database's recovery point to seconds, but the
 542repositories would still be on the nightly archive, so a restore could
 543produce a database referencing commits the repository backup does not
 544have. Consistency between the two halves is worth more here than latency
 545on one of them. Revisit if repository replication becomes continuous
 546too.
 547
 548** Offsite copies
 549
 550bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
 551=/var/lib/gitbay-stage= (a database copy and =config.toml=, staged
 552there) to an S3 bucket at Scaleway, with a key that can only add
 553snapshots. Nothing else under =/etc/gitbay= is in it: not the secret
 554key file, not =apns.p8=. The key that can
 555remove them lives on the operator's machine, in
 556=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
 557cannot destroy its own history. Forgetting, pruning and rewriting all
 558run from there.
 559
 560*** Removing a repository's history from every snapshot
 561
 562A history rewrite plus =admin mr prune= takes commits off the server,
 563but every snapshot taken before it still holds them, and the retention
 564window is the only thing that ages them out. To remove them now,
 565rewrite the snapshots without that repository rather than forgetting
 566the snapshots: everything else in them stays restorable. The next
 567nightly run adds the repository back in its current state.
 568
 569Repositories are stored under the name they had on disk when each
 570snapshot was taken, so a renamed repository needs every name it has
 571carried. Check what an older snapshot holds before choosing the paths:
 572
 573#+begin_src sh
 574set -a; . ~/.config/gitbay/offsite.env; set +a
 575restic $RESTIC_OPTS snapshots
 576restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
 577#+end_src
 578
 579Then dry-run, apply, prune, and confirm nothing matches:
 580
 581#+begin_src sh
 582EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
 583restic $RESTIC_OPTS rewrite --dry-run $EXCL     # "would modify N snapshots"
 584restic $RESTIC_OPTS rewrite --forget $EXCL      # new snapshots replace the originals
 585restic $RESTIC_OPTS prune                       # drops the data nothing references
 586restic $RESTIC_OPTS find <name>.git <old-name>.git   # expect no output
 587#+end_src
 588
 589=--forget= is what makes the originals go; without it the rewritten
 590snapshots sit beside them and the data stays referenced. Snapshot IDs
 591change; their times do not. Done for krz/keycask (formerly rust-pass)
 592on 2026-09-18, across 22 snapshots.
 593
 594** Secret key
 595
 596CI secrets, webhook secrets, mirror tokens and APNs device tokens are
 597stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
 598each value prefixed with the id of the key that sealed it
 599(=gbs1:<id>:=). The key file is not in the database, not under
 600=server.root=, and therefore in neither the local archives nor the
 601main restic repository. It must be copied off the host separately;
 602without it a restored database's secrets cannot be opened, and
 603gitbayd refuses to start against them. A separate restic repository
 604for it and =apns.p8= is planned (runbook D of the data-at-rest plan)
 605and not yet in place, so today the only off-host copy is one the
 606operator makes by hand after =init= and after every =rotate=.
 607
 608#+begin_src sh
 609gitbayd admin secrets init     # once; deploy/install.sh does it on first install
 610gitbayd admin secrets check    # open every value, count by key
 611gitbayd admin secrets rotate   # new key, reseal, retire the old one (as root)
 612#+end_src
 613
 614- Missing file: every gitbayd process that opens the database refuses
 615  to run and names the path, including =serve= and, in system mode,
 616  =authorized-keys=. =migrate= does not need it.
 617- Backups: =admin backup= and =admin backup --verify= do not need the
 618  key file; a restore does.
 619- Wrong key: =serve= stops at startup naming the first row that does
 620  not open; =secrets check= does the same without starting anything.
 621- Upgrade: the first start after the upgrade seals every value still
 622  in clear and logs =sealed secret values=.
 623- Rotation: =rotate= adds a key, reseals every value under it in one
 624  transaction, then removes the old keys. Run it as root, since it
 625  replaces the key file in =/etc/gitbay=; the file keeps its owner.
 626  The daemon re-reads the file when it changes, so it needs no
 627  restart. Copy the new file off the host afterwards. =init= and
 628  =rotate= hold an flock on =<key file>.lock= while they run, so a
 629  second run waits for the first.
 630- Push devices are looked up by the SHA-256 of their token
 631  (=push_devices.token_hash=), since two seals of one token differ.
 632
 633** Restore drill
 634
 635A restore onto a clean host, run quarterly and after any change to the
 636backup code (=cmd/gitbayd/backup.go=, the offsite job), and recorded
 637below. The disaster it rehearses is losing bay1, so the local archives
 638are gone with it and the sources are the main offsite restic
 639repository (repositories, LFS, the staged database, =config.toml=),
 640the off-host copy of =secret.key= and =apns.p8= (a keys repository once
 641runbook D creates it; until then the operator's hand-made copy), and
 642the operator's password manager (=offsite.env=, the keys repository's
 643password and token once it exists, =backup-identity.txt=). The steps are in the data-at-rest
 644plan's operator runbook
 645(=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
 646
 647Time to service runs from the clean host's first root login to the
 648first successful =git clone= over SSH from it. The recovery point is
 649the time of the newest restic snapshot restored.
 650
 651No drill has been run yet; the procedure above is written but
 652unexercised, and #259 stays open until the first row below is
 653recorded.
 654
 655| Date | Host | Snapshot restored (UTC) | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
 656|------+------+-------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
 657
 658* Upgrades
 659
 660Replace the binary, restart the unit. Migrations apply automatically and
 661are transactional; hook scripts under =<root>/hooks= are rewritten at
 662startup to point at the current binary path.
 663
 664* CI runner
 665
 666=gitbay-runner= executes builds queued by pushes and merge requests. It
 667polls over SSH with a key of scope =runner=, which reaches only the
 668runner protocol and read-only git (a runner executes arbitrary
 669repository code, so the key it holds must not do more). A runner key
 670claims builds only for the repositories it is attached to, by =repo
 671runner add= from a repository admin or an instance admin; an admin key
 672claims any. Users attach their own runners: see the Users page. For an
 673instance runner, run it as a dedicated unprivileged user on a non-admin
 674account. =admin user create --key= registers a full-scope key, so the
 675runner key is added afterwards through a bootstrap key that is then
 676removed, and attached to each repository it should build:
 677
 678#+begin_src sh
 679useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
 680sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
 681ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
 682gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
 683ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
 684ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
 685rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
 686gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
 687#+end_src
 688
 689#+begin_src sh
 690gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
 691#+end_src
 692
 693=-jobs N= runs N builds at once. Claiming is one transaction that
 694selects and updates, and each build works in its own =build-<id>=
 695directory, so workers do not collide; idle polls are staggered across
 696the interval so N of them do not wake together. The drop-in's weights
 697below are per service, not per build, so raising =-jobs= divides them
 698rather than multiplying the host's load.
 699
 700=admin runners= shows which account each runner polls as, and what each
 701may claim. A runner key claims builds only for the repositories it is
 702attached to: with none attached it claims nothing, and =-repos= may only
 703narrow within them. An admin's full-scope key claims any repository —
 704that is what =-repos= was for — and still works for the protocol during
 705a rotation. A merge request head from a fork is built in the target
 706repository as untrusted: the claim carries no secrets, and only a runner
 707started with =-untrusted= takes it. Same-repository heads were built by
 708their branch push and are not built again.
 709
 710=make deploy-runner= also installs
 711=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
 712=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
 713the daemon or the backup timers, and =NoNewPrivileges=,
 714=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
 715and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
 716and the runner's home. The e2e suite alone starts sixty
 717daemon instances; without the drop-in a deploy's copy over the admin
 718sshd stalled. Both deploy targets copy with =rsync --partial=, which
 719resumes a stalled transfer.
 720
 721Under =-isolation podman=, the default and what bay1 runs, each build
 722is confined to a container (see Container isolation below). Under
 723=-isolation none= steps run directly on the host as the runner's user,
 724so treat that machine as executing whatever your users push, and
 725install the toolchains your builds need on it.
 726
 727A runner claims the oldest pending build among the repositories its key
 728is attached to — for an admin key, the oldest in the instance. =-repos=
 729narrows within that set, which is what makes a runner outside the server
 730practical: one on a machine that should build a single project, or that
 731holds credentials for one deployment, stays on it.
 732
 733Oldest-first is across everything the key may claim, so a repository
 734with a deep queue holds every other repository the same runner serves;
 735bay1 measured a 15-minute average wait on a day of merge request
 736stacks from one repository. A runner attached to one repository cannot
 737be starved. That is the rule, decided in krz/gitbay#207: a runner
 738serving several repositories takes them oldest-first, and an operator
 739who wants one repository never to wait on another runs a second
 740runner attached to it alone. Nothing caps what an account queues,
 741and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
 742while the job's last build is pending or running, so a repository
 743with no runner holds one row per scheduled job rather than one per
 744tick, and a build runs only on a runner its owner attaches, so a busy
 745schedule spends the owner's compute. Pushes are bounded by what an
 746account can push.
 747
 748#+begin_src sh
 749gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
 750  -workdir /var/lib/gitbay-runner/work
 751#+end_src
 752
 753Add =-untrusted= only with =-isolation podman=.
 754
 755gitbay.org's runner is attached to the forge's own repositories and
 756the isolation canary, nothing else, because it shares the host with
 757the forge; its unit names no =-repos=, the attachments are the
 758boundary. Any other repository builds on a runner its owner attaches.
 759
 760=-repos= narrows an admin runner; for a runner key the attachments are
 761the boundary, held by the server, and =-repos= may only name
 762repositories among them. =-untrusted= makes a runner claim merge
 763request heads from forks; the bay1 unit sets it because it isolates in
 764podman. A runner without it builds trusted commits only.
 765
 766=gitbay dashboard= and =ssh git@<host> admin runners= list every key
 767that has polled as a runner: the account, the key's fingerprint, when it
 768last polled, the repositories it may claim — its attachments for a runner
 769key, the =-repos= it asked for or =any= for an admin key — and the build
 770it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
 771that polled by mistake, the key itself untouched. =admin runners= also
 772heads the list with the queue: builds
 773pending now, and over the last day how many were claimed, how long they
 774waited to be claimed (average and worst), and
 775how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
 776reported is failed by the scheduler's minute tick, whether or not any
 777runner is still alive: within about two minutes of its log stream ending
 778with no outcome reported — the runner reports right after closing the
 779stream, retrying for half a minute if gitbayd is unreachable — or, if no
 780stream was ever seen, at the deadline.
 781
 782Instance admin on the runner account only authorizes the claim/report
 783protocol; it grants no repo access. A build that pushes back — a pages
 784deploy, an archive publish, an automated MR branch — needs an explicit
 785grant on that repo: =repo access grant <owner/name> ci write=. Private
 786repos likewise need at least read for the clone.
 787
 788** Container isolation
 789
 790Builds run in a rootless podman container, one per job, with the
 791workspace bind mounted and nothing else: the clone happens outside with
 792the runner's key, so a step cannot read it. =-isolation none= keeps the
 793old behaviour — steps on the host as the runner's user — for an instance
 794where every repository is trusted. There is no automatic fallback: a
 795runner started with =-isolation podman= that cannot find a working
 796podman exits rather than running a build unsandboxed.
 797
 798gitbay's own jobs name =localhost/gitbay-ci:2=, built from
 799=deploy/Containerfile.ci= on the runner host. A job's image must carry
 800what its steps need: the suite drives real git, git-lfs, gpg and sshd and
 801asserts they exist before running, so the stock runner default would fail
 802it immediately. Build or rebuild it with:
 803
 804#+begin_src sh
 805ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
 806ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
 807  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
 808#+end_src
 809
 810The tag is deliberate rather than =:latest=: changing the file means
 811bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
 812change under it.
 813
 814=-image= names the image a job runs in when it declares none, and is
 815required under =-isolation podman=: there is no built-in default,
 816because an image this host does not have would fail every build. A job
 817overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
 818so a config file cannot turn it into podman arguments.
 819
 820=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
 821-memory 4g=); unset means uncapped. The runner applies them itself: it
 822creates a cgroup per build under its own delegated service cgroup,
 823writes the limits there, and starts every podman process for the build
 824inside it, with podman's cgroup handling off. Podman's own =--memory=
 825and =--cpus= never applied under rootless cgroupfs, which is what a
 826system service gets (krz/gitbay#188). The unit therefore needs
 827=Delegate=yes=, which the drop-in sets; without it the runner refuses
 828to start when a limit is set, and logs that builds run unconfined when
 829none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
 830and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
 831swap: the memory cap is what keeps the forge alive when a build
 832allocates without bound, and it sits above the e2e suite's 5GB peak
 833rather than at a fair share. =OOMPolicy=continue= keeps systemd from
 834stopping the runner when a build is OOM-killed.
 835
 836A trusted build's home is its repository's, under
 837=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
 838=HOME=: caches persist between trusted builds of one repository and are
 839never read by another's. An untrusted build — a merge request head from
 840a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
 841the build ends. Homes under =<workdir>/home= are from runners before
 842krz/gitbay#255, which shared them with untrusted builds; nothing reads
 843them any more, and they can be deleted.
 844
 845*Images are provisioned, never pulled by a build.* The runner passes
 846=--pull=never=. Two reasons, and the second is the better one: the
 847service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
 848holding a setuid file, which is nearly every distribution image; and on
 849an instance where anyone can push a =ci.yml=, =image:= would otherwise
 850mean "fetch and run anything from the internet". An operator pulls or
 851builds what is allowed and a build picks among those. A job naming an
 852image the host does not have fails with a message saying so.
 853
 854#+begin_src sh
 855su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
 856su - ci-runner -s /bin/sh -c "podman images"
 857#+end_src
 858
 859Prepare a host before pointing an isolating runner at it:
 860
 861#+begin_src sh
 862ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
 863make deploy-runner
 864#+end_src
 865
 866The script installs podman, delegates a subuid/subgid range to
 867=ci-runner=, checks that user namespaces are enabled rather than
 868assuming, enables lingering, and verifies rootless podman actually runs
 869as that user. It is idempotent.
 870
 871It also installs nftables. =make deploy-runner= ships
 872=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
 873with =gitbay-runner-egress.service=, which loads it and which the
 874runner's unit requires; it checks the file with =nft -c=, reloads the
 875unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
 876restarting the runner: =127.0.0.1:22= and the public 22 must answer,
 8772222 must not. The table limits the runner's user to =127.0.0.1:22=,
 878DNS on loopback, and 22, 80 and 443 on the host's public address; the
 879Threat-Model page says why. A restart of =nftables.service= flushes it;
 880=systemctl reload gitbay-runner-egress= restores it.
 881
 882The drop-in sets =NoNewPrivileges=no=, without which rootless podman
 883cannot call =newuidmap= and the runner refuses to start. That is a
 884considered trade, explained in the file and in the Threat-Model; if you
 885run with =-isolation none=, set it back to =yes=.
 886
 887*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
 888the build in flight, reports it, and exits; the drop-in's
 889=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
 890is what makes the signal reach the runner alone — under systemd's default
 891the build's container and the log session are signalled with it, and the
 892runner drains a build that is already dead. So =make deploy-runner=
 893waits for a running build rather than orphaning it, and a build's result
 894is retried for half a minute if gitbayd is restarting at that moment. A
 895second SIGTERM ends the runner at once, abandoning the build to the
 896reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
 897the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
 898=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
 899the runner as a transient user unit under =systemd-run= and stops it
 900under both kill modes. That last one needs a systemd user manager, so it
 901skips in the container CI runs in; run it on a Linux host with
 902=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
 903
 904*Validate podman mode on a scratch repository before pointing the runner
 905at real ones.* Every deploy that switched the whole instance to
 906containers and failed took CI down with it. Instead: create a throwaway
 907repository the runner account can read (public, or granted read — a
 908private one is "not found" to the runner and the build stays pending),
 909give it one job that names the CI image, and deploy the runner with
 910=-repos= naming only that repository. The production unit, with its real
 911hardening, then claims nothing else; other repositories' builds queue
 912until =-repos= is switched back, which is a pause, not an outage.
 913
 914#+begin_src sh
 915gitbay repo create cmc/ci-smoke          # then push a .gitbay/ci.yml naming the image
 916sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
 917systemctl daemon-reload && systemctl restart gitbay-runner
 918gitbay build log cmc/ci-smoke 1         # green: switch -repos back, redeploy
 919#+end_src
 920
 921*Do not deploy an isolating runner to a host that has not been
 922prepared.* The runner is specified to refuse to start without a working
 923podman rather than fall back to running builds unsandboxed — a fallback
 924that silently drops isolation is worse than a stopped runner, because
 925nothing surfaces it. On an unprepared host that refusal stops every
 926build on the instance.
 927
 928The service drop-in carries =Delegate=yes= for rootless cgroup
 929management and =ReadWritePaths= for podman's store under
 930=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
 931make read-only. Those paths are prefixed =-= so they are ignored when
 932absent: the drop-in installs on unprepared hosts too, and a unit that
 933refused to start would stop every build.
 934The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
 935names that repository too; a scoped runner claims nothing else.
 936=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
 937user: rootless storage belongs to that user, and root's prune would not
 938see it. An unpruned image store on a 40GB host is a slow outage.
 939
 940* LFS storage
 941
 942Objects live content-addressed under =[lfs] root= (default
 943=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
 944(512MB default). Storage sits behind a small interface — an
 945S3-compatible backend is a drop-in with the server proxying, and
 946presigned URLs a later optimization. LFS objects do not travel with
 947push mirrors (mirrors move git refs only), and gc does not yet collect
 948orphaned objects.
 949
 950* Pages
 951
 952=[pages] domain = "example.site"= serves public repos' =pages= branches
 953on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
 954server; ACME issues per-subdomain certificates on demand (only for
 955owners that exist). The domain must not be the site host or a parent of
 956it — pages content runs its own scripts and must stay off the forge's
 957origin.
 958
 959Users with repo admin claim custom domains with =repo domain add=.
 960Claims activate only after a DNS TXT challenge proves control of the
 961domain (=repo domain verify=, audit-logged); pending claims serve
 962nothing, get no certificates, and expire after 7 days. ACME issues
 963certificates only for verified hosts, so stray DNS pointed at the
 964server gets nothing.
 965
 966* Security
 967
 968The [[Threat-Model]] file is the reference for what the forge
 969trusts and refuses to do. Operational checklist:
 970
 971- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
 972  (the module list is deliberately short — review it on each release),
 973  and a short fuzz pass over every attacker-facing parser (pkt-line,
 974  commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
 975  tagging a release. CI's own =vuln= job runs =govulncheck= nightly
 976  against main rather than per push, because =@latest= scans today's
 977  advisory database and an advisory lands without anyone pushing;
 978  =build trigger krz/gitbay vuln= runs it on demand.
 979- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
 980  DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
 981  configuration needed.
 982- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
 983  gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
 984  =LockPersonality=, =MemoryDenyWriteExecute=,
 985  =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
 986  INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
 987- *OS patches* apply via =unattended-upgrades= (security origins,
 988  auto-reboot 04:30 if required).
 989- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
 990  watched by =fail2ban=; gitbayd's own port 22 is throttled by
 991  =limits.ssh_auth_rate= (auth failures per IP per minute), and every
 992  account's writes by =limits.write_rate=.
 993- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
 994journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
 995webhook: disk, service, the daemon's own =/healthz= answer, certificate
 996expiry, and the age of the newest full backup and database snapshot.
 997It exits non-zero on an alert so the unit shows in =systemctl
 998--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
 999a certificate under 21 days, a full backup over 25 hours old, or a
1000database snapshot over 2 hours old.
1001
1002=GET /healthz= is unauthenticated and cache-free: whether the database
1003answers and which commit serves, 503 when it does not.
1004- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
1005  0750, owned by =gitbay=). The nightly archive plus provider snapshots
1006  are the recovery path; for tighter RPO, add continuous replication
1007  (litestream) against the same file — it coexists with the WAL.
1008
1009* Odds and ends
1010
1011- deleting a fork marks MRs sourced from it =source_gone=; their diffs
1012  remain viewable and mergeable because the target repo owns the
1013  objects.
1014- =refs/merge-requests/*= is server-owned and unpushable by clients;
1015  only =admin mr prune= removes one (see Maintenance).
1016- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
1017  the =events= table, which also feeds webhooks.
1018- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.