.gitbay/wiki/Admin.org

   1#+title: gitbay admin guide
   2
   3One static binary (=gitbayd=), one SQLite file, bare repositories on
   4disk, and the system =git=. Schema migrations run automatically on
   5startup and on every admin command.
   6
   7* Install
   8
   9Build from source (=go build ./cmd/gitbayd=), install via the vanity
  10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
  11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
  12linux/amd64, linux/arm64, and darwin/arm64 binaries, each gzipped, with
  13a SHA256SUMS manifest over the =.gz= files (CGO off, trimpath, stripped —
  14the decompressed binary is byte-identical per commit and toolchain).
  15
  16#+begin_src sh
  17gunzip -c gitbayd-<tag>-linux-amd64.gz > gitbayd
  18install -m 755 gitbayd /usr/local/bin/
  19adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
  20install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
  21gitbayd --config /etc/gitbay/config.toml check-config
  22gitbayd --config /etc/gitbay/config.toml admin secrets init
  23chown gitbay:gitbay /etc/gitbay/secret.key
  24#+end_src
  25
  26=admin secrets init= refuses when the key file already exists, so a
  27reinstall on the same host should skip it — =deploy/install.sh= does
  28this with a file check before running it.
  29
  30=deploy/= in the source tree has a cloud-init file, a hardened systemd
  31unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
  32the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
  3322/80/443 without root.
  34
  35** The SSH port decision
  36
  37- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
  38  22 — move the host's admin sshd to another port. Remotes read
  39  =git@host:owner/repo= with no port gymnastics.
  40- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
  41  =AuthorizedKeysCommand=:
  42  #+begin_example
  43  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
  44  AuthorizedKeysCommandUser gitbay
  45  #+end_example
  46  sshd requires that binary to be root-owned and not group/world
  47  writable. Unknown keys fail authentication inside sshd, so system mode
  48  requires =registration.mode = "closed"= (check-config enforces this).
  49  Under the forced command the CLI's leading =--term=<cols>[,color]=
  50  argument works as is; a client that sets =GITBAY_TERM= instead needs
  51  =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
  52  handled by the host's sshd, not gitbayd.
  53
  54* Configuration reference
  55
  56=/etc/gitbay/config.toml=. =check-config= validates and names every
  57contradiction; =--no-host-checks= skips port/path probes.
  58=gitbayd admin config show= prints the configuration in effect as TOML,
  59every default filled in and =smtp_pass= redacted. A file that fails
  60validation still prints, followed by the contradiction.
  61
  62** [server]
  63- =root= (default =/var/lib/gitbay=) — repositories, database, host
  64  keys, ACME cache all live here.
  65- =site_url= (required) — canonical =https://host=; drives ACME, clone
  66  URLs, mail links.
  67- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
  68  seal CI secrets, webhook secrets, mirror tokens and push device
  69  tokens in the database. Must be outside =root=, mode 0600, readable
  70  by the daemon's user. See "Secret key" below.
  71- =source_repo= (optional, =owner/name=) — the repository this instance
  72  develops itself in. Startup warns when the running build's commit is
  73  not on that repository's default branch, which is how a binary built
  74  from an unmerged branch stops being invisible. Leave it unset unless
  75  the instance hosts its own source.
  76
  77** [ssh]
  78- =mode= — =embedded= | =system= (above).
  79- =port= (22) — embedded listener port.
  80- =host_keys= — list of private key paths; empty generates an ed25519
  81  key at =<root>/ssh/host_ed25519=.
  82
  83** [http]
  84- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
  85- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
  86  =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
  87  (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
  88  failing to bind it is a warning, not fatal. Requires an =https://=
  89  site_url with a public DNS name.
  90- =files=: =cert_file= + =key_file=.
  91- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
  92- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
  93  =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
  94  release the binary was built with. =openssl s_client -connect
  95  <host>:443 -tls1_1= fails the handshake.
  96
  97=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
  98front of the daemon. A request from one of them is attributed, for API
  99rate limiting, to the last =X-Forwarded-For= hop that is not itself a
 100trusted proxy; from anyone else the header is ignored. Empty, the
 101default, is right when gitbayd terminates TLS itself.
 102
 103A reverse proxy in front of gitbay must not buffer responses, or the
 104build page's live log arrives only when the build ends;
 105=X-Accel-Buffering: no= covers nginx.
 106
 107** [web]
 108- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
 109  web routes are never registered; in accounts, browser sessions are
 110  minted over SSH (=web login=), and users with write access can create
 111  repos, comment, and make simple file edits (which commit unsigned,
 112  honestly). =password_auth= is reserved and currently rejected.
 113- =title= — the instance's display name in the rail and page titles;
 114  empty falls back to the site host. Lower case is the convention for
 115  gitbay itself.
 116- =privacy_notice= — operator text shown on =/privacy= under the fixed
 117  statement.
 118- =abuse_url=, =terms_url= (empty) — linked from every page's footer as
 119  "Report abuse" and "Terms"; no link when empty. gitbay.org points them
 120  at the [[Abuse][Abuse]] and [[Terms][Terms]] wiki pages.
 121- =apple_app_ids= (empty) — iOS app IDs (=TEAMID.bundle.id=) allowed to
 122  open this instance's links, served at
 123  =/.well-known/apple-app-site-association=. Empty leaves the path a
 124  404. The gitbay iOS app is =ZCNAX3VL9D.org.gitbay.gitbay=; its
 125  entitlement names gitbay.org only, so listing it elsewhere does
 126  nothing.
 127
 128** [registration]
 129- =mode= — =closed= (default) | =invite= | =open=. invite/open require
 130  [mail]. See the user guide for the flows.
 131
 132- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
 133  self-registered account still unverified after that long is removed,
 134  hourly and at start, audited as =pending.expired=.
 135
 136- =notify_admin= (false) — mail every instance admin when an account
 137  becomes active: an invite redeemed, or an open-mode signup that
 138  verified its address. The unverified row an open signup creates is
 139  not reported, because anyone can post the form and mailing on that
 140  would point a flood at the admins. Recipients are the verified
 141  primary addresses of active admins who have activity mail on, the
 142  same rule any other notice follows, so an admin with no verified
 143  address hears nothing. The notice is queued, so a dead SMTP host
 144  shows up in the admin page's Mail table instead of failing the
 145  registration. Requires [mail].
 146
 147** [mail]
 148- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
 149  =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
 150  registration and self-service =email add=; in closed mode you may omit
 151  it entirely and assert addresses by hand (below).
 152- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
 153  for relays on 465).
 154- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
 155  gets no mail: delivery fails and retries, and the admin page's Mail
 156  table shows the error. Unset, it is on for every relay except
 157  =localhost= and loopback addresses; set =false= to allow plaintext
 158  to a remote relay.
 159
 160** [mail.inbound]
 161Reply by mail: a reply to issue or merge request mail posts a comment
 162(#295). gitbayd polls a mailbox over IMAP; no port is opened for it.
 163Off unless =enabled=, and it requires =[mail] smtp_host=, since the
 164replies answer mail gitbayd sends.
 165
 166#+begin_src toml
 167[mail.inbound]
 168enabled       = true
 169imap_host     = "imap.example.org"       # host:port; 993, or 143 with tls = "starttls"
 170tls           = "implicit"               # or "starttls"; there is no plaintext setting
 171user          = "reply@gitbay.example"
 172password_file = "/etc/gitbay/imap.pass"  # one line, mode 0600, owned by gitbayd's user
 173mailbox       = "INBOX"                  # default
 174poll_interval = "1m"                     # default; at least 10s
 175reply_address = "reply@gitbay.example"
 176require_dkim  = true                     # recommended; see below
 177trusted_authserv_id = "mx.example.org"   # the mail host's Authentication-Results id
 178#+end_src
 179
 180- The password is read from =password_file= at every connection and
 181  never appears in the config, argv or the log. An unreadable file, or
 182  one readable by group or others, stops the daemon at start.
 183- =reply_address= is what each Reply-To is built from:
 184  =reply@gitbay.example= becomes =reply+<token>@gitbay.example=. The
 185  mailbox must receive that. Hosts that deliver subaddresses to the
 186  mailbox need nothing more; others need a wildcard rule. Migadu, for
 187  one, sent =threads+x@gitbay.org= nowhere until a rewrite from
 188  =threads+*= to =threads@gitbay.org= was added. A domain catch-all only
 189  works if it points at this mailbox, which then also holds everything
 190  else sent to the domain. Send a test to =<reply_address>+test= and
 191  expect a =refused mail reply= audit row with "malformed reply token"
 192  within a poll interval (=gitbay audit --action 'refused mail'=). Point
 193  the domain's MX at the mail host as for any other mailbox; nothing
 194  about it involves gitbayd.
 195- Use a mailbox that holds nothing else. gitbayd reads every unseen
 196  message in it and marks each one =\Seen= when it is handled, posted
 197  or refused. A message that fails for a reason that may pass (the
 198  database busy, the server refusing that one fetch) stays unseen and is
 199  tried again on the next poll, up to five times, then is marked seen
 200  and audited. A dropped connection or a timeout ends the poll and
 201  counts against no message. A
 202  message over 10 MiB is refused by its size without being fetched. A
 203  server that sends more than about 11 MiB, or more than a thousand
 204  untagged responses, for one command has its connection closed; one
 205  poll handles at most ten thousand messages.
 206- Whatever the settings, a message is refused when a header field name
 207  is not RFC 5322 =ftext= (=From : x=, a space or a non-ASCII byte in a
 208  name), when it does not have exactly one =From=, or when it has more
 209  than one =To=, =Cc=, =Message-ID=, =Content-Type= or
 210  =Content-Transfer-Encoding=.
 211- =require_dkim= (default false) should be on for any instance
 212  reachable from the internet. With it, a reply is posted only when
 213  gitbayd itself verifies one of its DKIM signatures (RFC 6376) with a
 214  =d== in relaxed alignment with the From domain (the same
 215  organizational domain by the public suffix list; =d=github.io= aligns
 216  with nothing) and an =h== that covers =From=, the =To= or =Cc= holding
 217  the reply address, =Content-Type=, and =Message-ID= when the message
 218  has one. A =Content-Transfer-Encoding= outside =h== is accepted only
 219  when it is =7bit=, =8bit= or =binary=, which leave the decoded body as
 220  it is; Thunderbird, for one, does not sign it. An unsigned
 221  =quoted-printable= or =base64= is refused. The reply
 222  address is read only from =To= or =Cc=: a reply that reached the
 223  mailbox by Bcc, with the address only in =Delivered-To=, is refused
 224  ("reply address not in To or Cc"). It needs nothing from the mail
 225  host, so it works where the host adds no =Authentication-Results=.
 226  rsa-sha256 (keys of 1024 bits or more) and ed25519-sha256 are
 227  accepted, with simple or relaxed canonicalization; rsa-sha1, a body
 228  length tag (=l==), an expired =x== and a =t== more than fifteen
 229  minutes ahead are refused. Only the first five signatures are
 230  checked. The key is looked up at =<s>._domainkey.<d>= with a
 231  five-second timeout and cached for fifteen minutes (the resolver does
 232  not report the record's TTL); a lookup that fails for a reason that
 233  may pass (a timeout, SERVFAIL) leaves the message for the next poll,
 234  up to the five tries above, while a missing key refuses it.
 235  Signatures are checked on the message as fetched. Each passing
 236  signature is recorded with the =Message-ID=, so a copy of the same
 237  signed message posts once even with unsigned fields changed.
 238- Either =require_dkim= or =trusted_authserv_id= passing is enough to
 239  authenticate =From=; when both are set, a reply needs only one of
 240  them, and a refusal names both reasons. With only
 241  =trusted_authserv_id=, the reply address may come from any recipient
 242  field (=Delivered-To=, =X-Original-To=, =Envelope-To=, =To=, =Cc=),
 243  since the mail host vouches for the sender and not for the fields.
 244  Set both when the mail host adds =Authentication-Results= for most
 245  senders but not all.
 246- =trusted_authserv_id= names the authserv-id the mail host writes at
 247  the start of its =Authentication-Results= header (Gmail's is =mx.google.com=).
 248  With it set, a reply is posted only when the topmost header with that
 249  id shows =dmarc=pass= with =header.from= equal to the From domain, or
 250  =dkim=pass= with a =header.d= in relaxed alignment with it (the same
 251  organizational domain by the public suffix list; a public suffix such
 252  as =github.io= aligns with nothing). The header is parsed per RFC
 253  8601, so text inside a quoted string or a comment (a quoted MAIL FROM
 254  local part, a reason) is never read as a result. Lower headers
 255  claiming the same id are the sender's and are not read. This is only safe when the mail host
 256  removes incoming =Authentication-Results= headers that claim its id,
 257  as RFC 8601 asks; Gmail, Fastmail and Migadu do. Check yours before
 258  relying on it. With neither this nor =require_dkim= set, the daemon
 259  logs a warning at start and =admin mail inbound check= repeats it:
 260  =From= is then whatever the sender wrote. Mail between two addresses
 261  at the same host may carry no =Authentication-Results= at all: at
 262  Migadu, mail from another
 263  Migadu-hosted domain is delivered through its outbound path and gets
 264  none, so setting the id alone refuses every reply from such users.
 265  That mail does carry a DKIM signature aligned with =From= (for
 266  example =d=cleberg.net; s=key1; a=rsa-sha256; c=simple/simple;
 267  h=from:to:subject:date:message-id:mime-version:content-type=), which
 268  is why gitbay.org sets =require_dkim= instead.
 269- =gitbay admin mail inbound check= logs in, opens the mailbox
 270  read-only (EXAMINE) and reports the message and unseen counts and how
 271  =From= is authenticated (=require_dkim=, =trusted_authserv_id=), so a
 272  check never marks a reply seen before the poller reads it. It warns
 273  when neither is set. With inbound off it says so and exits 0. Poll
 274  failures are logged as =mail reply: poll failed= with the server and
 275  the IMAP error.
 276
 277A reply is posted when all of these hold, checked when it is read:
 278
 2791. It is not an automatic reply (=Auto-Submitted=, =Precedence: bulk=
 280   and the like).
 2812. A recipient header carries a reply address whose token verifies
 282   and has not expired (thirty days from the mail it came on).
 2833. The token's account exists, is active and not disabled, still has
 284   reply by mail on, and was created before the token was: an id freed
 285   by a delete and reused is not the account the token named. The same
 286   holds for the repository.
 2874. =From= is one of that account's verified addresses, and, with
 288   =require_dkim= or =trusted_authserv_id= set, a DKIM signature
 289   gitbayd verified or the mail host's result authenticated it.
 2905. The message has a =text/plain= part (HTML-only mail is refused, not
 291   converted), and what is left after quoted text and the signature
 292   are removed is not empty and fits a comment (64 KiB).
 2936. No earlier copy of the message (same =Message-ID=, account and
 294   thread) posted.
 2957. =issue comment= or =mr comment=, dispatched as the account, accepts
 296   it: the account can still read the repository, the thread exists,
 297   the repository is not archived, the write budget is not spent.
 298
 299The comment's audit row is =cmd issue comment= (or =mr comment=) with
 300=source: mail=. A refusal writes a =refused mail reply= row with the
 301reason and the =Message-ID=, never the message's content, at most sixty
 302a minute, and sends nothing back. See Threat-Model for why the token
 303and the sender address are both required.
 304
 305** [push]
 306Push notifications to Apple devices, delivered by gitbayd talking to
 307APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
 308operator-supplied provider key. Off unless configured.
 309
 310- =enabled= (false).
 311- =key_file= — path to the =.p8= provider key from Apple's developer
 312  portal (Certificates, Identifiers & Profiles → Keys). It belongs at
 313  =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
 314  as. Read and validated at startup: it must parse as a PEM-wrapped
 315  PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
 316  than fill a queue nobody is watching.
 317- =key_id=, =team_id= — the key's id and your Apple developer team id,
 318  both from the same portal page.
 319- =topic= — the app's bundle identifier. *An APNs key belongs to a
 320  bundle ID.* gitbay.org pushes to the App Store build under its own
 321  bundle id; a self-hoster who wants push ships their own iOS build
 322  under their own bundle id, with its own =.p8= key from their own
 323  developer account, and points =topic= at that id. There is no way to
 324  push to someone else's build, by design — this is Apple's model, not
 325  gitbay's.
 326- =environment= — =production= or =sandbox=, naming the APNs host
 327  rather than taking a URL, so a typo cannot aim the key at a host that
 328  is not Apple's.
 329
 330All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
 331are required when =enabled= is true; validation runs at config load,
 332so a misconfigured =[push]= is caught before the daemon serves
 333anything. The delivery queue (a device's undelivered and attempted
 334pushes) is capped the same way the mail queue is, by =[retention]
 335push=.
 336
 337** [backup]
 338- =age_recipients= (optional) — age public keys (=age1...=). When set,
 339  =admin backup= encrypts every archive to them and appends =.age= to
 340  its name. Generate the pair off the host with =age-keygen=; only the
 341  public key goes here, so the host writes archives it cannot read.
 342  The restic copy is unaffected: the offsite job stages its own
 343  =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
 344  not the archives.
 345
 346** [api]
 347- =enabled= (false) — the JSON API surface; see [[API]]. Off
 348  means no credential-bearing HTTP endpoint exists at all.
 349
 350** [webhooks]
 351- =allow_local= (false) — permit webhook, mirror and import targets on
 352  loopback, private, shared (100.64.0.0/10), link-local or multicast
 353  addresses. Leave off unless you know why you need it (SSRF).
 354
 355** [limits]
 356- =clone_timeout= (3600s) — cap on =repo import= fetches. An import
 357  takes http and https URLs only, passes the same address check as a
 358  mirror sync and is pinned the same way, so it needs git 2.37 too.
 359- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
 360- =max_asset_bytes= (512MB) — cap per uploaded release asset.
 361- =max_snippet_bytes= (1MB) — cap per snippet file.
 362- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
 363- =max_repos_per_user= (0, unlimited) — repositories an account may own
 364  directly; =repo create=, =fork=, =import= and =repo transfer= refuse
 365  past it.
 366- =max_bytes_per_user= (0, unlimited) — disk the account's own
 367  repositories may take; a push may be no larger than what is left, and
 368  a transfer in must fit.
 369- =max_orgs_per_user= (0, unlimited) — organizations an account may
 370  create. Membership in an org someone else created does not count.
 371- =max_repos_per_org=, =max_bytes_per_org= (0, unlimited) — the same two
 372  caps for each organization.
 373- =pack_concurrency= (3), =pack_per_principal= (2), =pack_queue= (32),
 374  =pack_queue_wait= (="60s"=) — git pack generation (clones, fetches,
 375  =git archive --remote=, web archive downloads, =repo download= over
 376  SSH and the API) over SSH, smart HTTP and git:// shares one
 377  budget: this many at once, this many per account (per client
 378  address when anonymous: an IPv4 address, or an IPv6 /64), and this
 379  many waiting for at most the wait. Anonymous clients together hold
 380  at most =pack_concurrency= − 1 slots when it is above 1: an account
 381  (SSH key, bearer token or web session) can take the last slot when it
 382  is free, and anonymous clients cannot hold it.
 383  Past that an SSH client gets "the server is busy…" and exit 1, HTTP
 384  gets 503 with =Retry-After: 30=, git:// an =ERR= line; the daemon
 385  logs a =pack limit= warning naming the transport and whether the
 386  client was signed in, at most once a minute per transport. An SSH client
 387  that disconnects while queued leaves the queue; an HTTP or git:// one
 388  keeps its place until the wait runs out. A running clone is killed
 389  when its client disconnects, or when no write to the client completes
 390  for two minutes: a client reading below about 550 B/s, or an HTTP
 391  request body that takes over two minutes with nothing written back,
 392  is cut. Ref listings (info/refs, protocol v2
 393  =ls-refs=) and pushes are outside the budget. A busy =repo download=
 394  exits 1 with the same message; over the API it is a 503 with
 395  =Retry-After: 60=. For the
 396  three counts 0 means the default and a negative value turns that
 397  bound off. The defaults suit a four-core host; see [[Performance]].
 398  With =ssh.mode = "system"= each SSH session is its own process and
 399  SSH clones are not counted.
 400- =push_concurrency= (2), =push_per_principal= (1), =push_queue= (16),
 401  =push_queue_wait= (="60s"=) — =git receive-pack= over SSH, on a
 402  budget of its own so clones cannot starve pushes or the reverse:
 403  this many at once, this many per account or deploy key, and this
 404  many waiting for at most the wait. One principal may have up to half
 405  of =push_queue= (at least one) waiting, so parallel pushes from one
 406  account queue rather than being refused, but cannot fill the queue.
 407  A deploy key is its own principal, apart from the account that
 408  registered it, so an account with deploy keys can hold
 409  =push_per_principal= slots per key;
 410  the global cap still holds. A bot account such as a runner's counts
 411  like any other account. The slot is taken after the access checks and held
 412  until receive-pack exits, which is after =post-receive= (merge
 413  detection, CI queueing, webhooks) has run. Past that the client gets
 414  "the server is busy: it is at its limit of concurrent pushes…" and
 415  exit 1, and the daemon logs a =push limit= warning at most once a
 416  minute. A client that disconnects while queued leaves the queue; one
 417  that disconnects mid-push ends receive-pack and frees its slot, and
 418  a revoked key kills it. 0 and negative values read as for =pack_*=.
 419  HTTP and git:// carry no pushes. With =ssh.mode = "system"= pushes
 420  are not counted and the two timeouts below do not apply.
 421- =push_idle= (="60s"=), =push_receive_timeout= (="15m"=) — a push
 422  holding a slot is killed, its process group with it, when its
 423  pre-receive has not started =push_receive_timeout= after it took the
 424  slot, or when no byte has been read from the client and none written
 425  to it for =push_idle=. The idle rule starts when the pack signature
 426  arrives from the client or pre-receive starts, whichever is first
 427  (a delete-only push sends no pack): before that the client may be
 428  silent while its =pack-objects= counts and compresses, and only
 429  =push_receive_timeout= applies. receive-pack runs with
 430  =receive.keepAlive= set to a quarter of =push_idle= (1 to 5 seconds),
 431  so the keepalives it sends to side-band clients while it indexes and
 432  runs hooks keep a working push alive. Once pre-receive has started
 433  only =push_idle= applies: post-receive is never cut short by the
 434  clock. A push killed before pre-receive answers updates no refs.
 435- =max_pack_bytes= (2 GiB) — the largest pack one push may send,
 436  enforced as =receive.maxInputSize= and lowered to what an owner's
 437  storage quota has left.
 438- =ssh_auth_rate= (10) — SSH authentication failures per client address
 439  per minute on the embedded listener; see below.
 440
 441** [git_daemon]
 442- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
 443  Serves only public repositories that additionally ran
 444  =repo settings git-daemon <repo> on=.
 445
 446** [mirrors]
 447- =pull_interval_minutes= (15) — how often pull mirrors fetch their
 448  upstream. Push mirrors sync shortly after each local ref update.
 449  Mirror URLs pass the same SSRF rules as webhook targets, when saved
 450  and again before every sync; git then connects only to the addresses
 451  that were checked (=http.curloptResolve=) and does not follow
 452  redirects, so a mirror of a renamed repository fails until its URL
 453  is updated. Needs git 2.37 or later on the server; with an older git
 454  the worker logs an error at start and syncs no mirror, recording the
 455  reason on each. Sync ignores the system and global gitconfig.
 456
 457** [go_import]
 458Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
 459Requests with =?go-get=1= at or under the module path answer with the
 460go-import meta tag pointing at the repository's HTTPS clone URL, so
 461=go install host/module/cmd/...@latest= resolves. The repository should
 462be public (the module path itself confirms it exists).
 463
 464* Users, email, invites
 465
 466Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
 467backfills is a wrapper that dispatches the registry command of the same
 468name as the host: an admin context with no account behind it, so its
 469audit rows carry no actor and =source: host=. The same commands run in an
 470instance admin's SSH session (=ssh git@<host> admin ...=) and write the
 471same rows with the key fingerprint as source. One implementation, two
 472credentials.
 473
 474#+begin_src sh
 475gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
 476gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
 477gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP
 478#+end_src
 479
 480"Verified" means SMTP-confirmed or host-admin-asserted; the database
 481records which. Verified emails are what make commit signatures
 482meaningful — an unverified address never produces a =verified= badge.
 483
 484* Audit and account control
 485
 486The audit log is the security feed (events are the product feed): every
 487successful mutating command with its argv and source credential (SSH key
 488fingerprint or API), every refused one (exit 3 or 4) as =refused
 489<command>=, refused pushes as =refused git-receive-pack= (the access
 490check) or =refused push= (a branch or tag rule, a release anchor or an
 491unsigned commit, with the repository and ref names), hook socket
 492requests failing the peer or push-token check as =refused hook= (never
 493the token), registrations, admin actions, force-pushes, and auth
 494failures/throttling. A refusal row
 495keeps the flag names and the first positional, not the values.
 496Refusals are recorded up to ten a minute per account and 600 a minute
 497across the instance; past either, one =refused.throttled= row stands
 498for the rest of that minute. The caps bound the embedded listener, the
 499web and the API; under =ssh.mode = "system"= each =gitbayd shell=
 500connection counts separately. Secrets never appear — they travel on
 501stdin, never in argv.
 502
 503Each row carries the SHA-256 of the row before it. =gitbayd admin audit
 504verify= opens the store as other admin commands do, applying pending
 505migrations, so run it with the binary that matches the daemon. It
 506recomputes the chain and exits 1 naming the first row that was
 507edited or whose predecessor was removed. The chain is unkeyed: whoever
 508can write the database can recompute every hash after an edit, and
 509verify then finds nothing. It catches an edit only when the later
 510hashes were not recomputed. Retention removing the oldest
 511rows is not a break. Rows written before the chain existed are counted
 512and skipped; when every row is such a row, verify warns and exits 1,
 513since clearing the hash columns looks the same. After an upgrade that
 514clears with the first new audit row.
 515
 516Removing the newest rows leaves no break, and neither do rows written
 517afterwards under the freed ids. The database cannot show either. The
 518daemon logs every row it writes to its journal, outside the database
 519(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
 520id and hash: comparing them with the newest journal line is the check
 521for any change, recomputed hashes included. Rows written by
 522host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
 523"system"=, are not copied to the journal.
 524
 525#+begin_src sh
 526gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
 527gitbayd admin audit verify           # check the hash chain; exit 1 names the first bad row
 528ssh git@<host> audit ...             # the same, from an admin session
 529ssh git@<host> admin user list [--state active|pending|disabled|admin]
 530ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
 531ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] [--orgs n|default]   # per-account caps
 532ssh git@<host> admin org limits <org> [--repos n|default] [--bytes n|default]   # per-org caps
 533ssh git@<host> admin user promote <name>   # grant instance admin
 534ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
 535gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
 536gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
 537gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
 538gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
 539                                     #   refused (with each blocker named) while
 540                                     #   the account owns repos, authored
 541                                     #   issues/MRs/comments/reviews, or is an
 542                                     #   org's only admin
 543#+end_src
 544
 545=--actor= takes a username, or =-= for rows with no actor: host commands
 546and auth failures. =--action= is a prefix, so =cmd repo= catches every
 547repository command and =admin= every host or admin-session action.
 548=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
 549
 550=admin user list= pages by username (=--limit=, =--cursor=) and carries
 551each account's state and =last_seen=, the newest use of any of its SSH
 552keys or API tokens. =admin user show= adds the keys with their last use,
 553each address with how it was verified, PGP keys, org roles, the owned
 554repository count, API token names, and live browser sessions. Both are
 555Both are refused to non-admins, like =audit=, on every surface.
 556
 557=/admin/users= is the same list in a browser, linked from the admin
 558page: the state filter the command takes, keyset paging on its cursor,
 559and a row per account with promote, demote, disable and enable, each
 560dispatching the command. Demote and disable ask for the username to be
 561typed, since both take someone's access away. Creating and deleting an
 562account, issuing an invite and asserting an address stay on the command
 563line: each takes a key, mints a credential, or cannot be undone. A
 564non-admin gets the 404 a missing page would, so the URL confirms
 565nothing.
 566
 567Promotion needs an active account: a pending or disabled one is refused.
 568Demotion is refused when it would leave no admin, over SSH, in the
 569browser and on the host alike, so the host-local =promote= is the way
 570back in when the only admin key is lost.
 571
 572Instance admin carries no right on anyone's repository: policy does not
 573consult it, and a private repository still answers not-found to an
 574admin. Moderation goes through explicit overrides that skip the access
 575check and write their own audit row:
 576
 577#+begin_src sh
 578ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
 579ssh git@<host> admin repo archive|unarchive <owner/name>
 580ssh git@<host> admin repo visibility <owner/name> public|private
 581ssh git@<host> admin repo delete <owner/name> --yes
 582#+end_src
 583
 584Each lands in the audit log as =admin repo.<action>= naming the
 585repository, on top of the =cmd= row every mutating command gets.
 586
 587=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
 588per minute — successful auths never count and clear the slate.
 589=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
 590push.
 591
 592=limits.write_rate= (60) bounds *mutating commands per account per
 593minute*. It is counted in the dispatcher, so SSH, the JSON API and the
 594web spend one budget and a caller cannot refresh it by changing surface;
 595=limits.api_rate= stays in front of it, bounding a network source rather
 596than an account. A command is one token whatever it writes, so a bundle
 597import costs one and only a loop of separate commands spends the budget.
 598Read-only commands, the runner protocol (a build streams its log in many
 599small writes) and the host CLI are exempt. Refusals exit 4 and say when
 600to retry. A negative value turns the limit off; it matters most with
 601=registration = "open"=, where every write also queues notification mail
 602and webhook deliveries.
 603
 604* Queues
 605
 606Every background worker keeps a backlog and a failure state. An instance
 607admin reads them all in one place:
 608
 609#+begin_src sh
 610gitbay dashboard --json | jq .queues   # webhooks, mail, push, mirrors, builds, deps
 611#+end_src
 612
 613Per worker: pending, retrying (pending with a failed attempt) and
 614dead-lettered counts with the oldest pending age, and the retrying or
 615failed rows themselves, capped at twenty each. Builds list what is
 616running and then what is pending, each since when, so a build no runner
 617is scoped to claim is visible here rather than only in its repository;
 618mirrors list the ones whose last sync failed;
 619dependency checks list the ones whose last check errored. Non-admins get
 620no =queues= key at all.
 621
 622Push rows name the device id, never the token. Watch this one after
 623configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
 624config validation, which can only check that the =.p8= parses, and then
 625every send comes back =403 InvalidProviderToken= and dead-letters on its
 626first attempt.
 627
 628In accounts mode the same read renders at =/admin=, linked from the rail
 629for admins. Anyone else gets a 404 there.
 630
 631A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
 632with the address redacted out of the relay's error. The id is the queue
 633row: find it in the Mail table on =/admin=, or in =dashboard --json=,
 634where the recipient and the unredacted error are. That is deliberate —
 635see the Threat-Model page.
 636
 637* Maintenance
 638
 639#+begin_src sh
 640gitbayd admin stats [--json]         # counts, database size, per-repo disk
 641ssh git@<host> admin stats [--json]  # the same, from an admin session
 642gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
 643gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
 644gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)
 645#+end_src
 646
 647A history rewrite leaves the commits it removed reachable through
 648=refs/merge-requests/N/head= of the merge requests that landed them, so
 649they stay fetchable by anyone who can read the repository. Nothing drops
 650a head ref on its own — an open or source-gone MR is merged through it,
 651and a merged or closed one keeps its diff readable through it — so the
 652cleanup is a command an instance admin runs, naming the MRs:
 653
 654#+begin_src sh
 655ssh git@<host> admin mr prune owner/name 1 2 3 --yes
 656#+end_src
 657
 658It refuses an open or source-gone MR, deletes the named refs, runs
 659=git gc --prune=now= on that one repository so the objects go at once
 660rather than after git's two-week grace, leaves a system comment on each
 661MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
 662reviews and head sha; =mr diff= and the MR page say the head is gone.
 663Run it when nothing is pushing to that repository: without the grace, a
 664push caught between leaving quarantine and writing its ref loses its
 665objects. Objects also survive in offsite backups until those are
 666rewritten; see "Removing a repository's history from every snapshot".
 667
 668=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
 669weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
 670layout the source sent, so a first manual =admin gc= after a bulk
 671import is worthwhile.
 672
 673* Backup and restore
 674
 675#+begin_src sh
 676gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
 677gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back
 678#+end_src
 679
 680One archive: a consistent SQLite snapshot (taken *before* the
 681repositories are read, so the database never references objects the
 682archive missed), every repository, and the SSH host keys. Excluded:
 683hook socket, regenerated hook scripts, WAL files. Safe to run against a
 684live daemon. =--out= must be outside =server.root=, or the next full
 685backup would carry the archive. Each run removes the snapshot
 686directories (=.gitbay-snap-*=) and temporary archives (=.*.tmp-*=) a
 687killed run left beside its archive once they are a day old, and prints
 688each one it removes.
 689
 690=--verify= reads an archive back: the snapshot must pass SQLite's
 691integrity check, every repository the snapshot names must be in the
 692archive, and each must pass =git fsck --connectivity-only=. Every
 693release asset the snapshot names must be in its repository with its
 694recorded size and sha256, and every LFS object in the archive must
 695hash to its name. LFS objects are named by pointer files in git
 696history rather than by the database, so a pointer whose object was
 697never uploaded is not caught, and with =[lfs] root= outside
 698=server.root= the archive holds no LFS objects at all. It extracts the
 699repositories to a temporary directory for the checks, so it needs free
 700space the size of the repositories. A database-only archive is checked
 701for integrity and says so. Exit is non-zero on damage, a missing
 702repository, a missing object, or a missing or altered asset or LFS
 703object.
 704
 705A full backup holds =<root>/backup.lock= from its database snapshot to
 706its last repository. While it runs, =repo delete=, =repo rename=,
 707=repo transfer=, =admin repo delete=, =org rename=, =admin mr prune=
 708and =gitbayd admin gc= refuse with "a backup is running"; retry when it
 709finishes. Database-only backups take no lock. A pack that git's own
 710automatic gc removes during the walk is skipped; each repository's refs
 711are archived before its objects, so the refs still find their objects,
 712and =--verify= reports it if one does not.
 713
 714Verifying an archive of unknown origin: run it as an unprivileged
 715user. The connectivity check runs git with =--git-dir= on each
 716extracted repository, so a directory that is not a repository fails
 717rather than git checking an enclosing one. =objects/info/alternates=
 718and a =commondir= directly in a =*.git= directory are not extracted,
 719so an archive cannot use either to have git read another repository's
 720objects or refs on the host. Git still reads each archived
 721repository's own =config=.
 722
 723Archives carry a directory entry for every directory, including an
 724empty one, so a bare repository whose refs are all packed restores as
 725a repository. Archives written before this release do not: extracting
 726one can leave a repository's =refs/= directory missing, which stops
 727git from recognizing it as a repository at all. =gitbayd admin backup
 728--verify <archive>= names the repositories this affects; the fix is
 729=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
 730which it opens normally.
 731
 732With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
 733and =--verify= needs the private key:
 734
 735#+begin_src sh
 736gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
 737age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
 738#+end_src
 739
 740The identity lives off the host (with the secret key file and the
 741restic credentials), so verifying an encrypted archive happens there
 742or on a restore host.
 743
 744Restore: extract into an empty directory, point =server.root= at it,
 745restore =server.secret_key_file= from its own copy (mode 0600, owned
 746by the account gitbayd runs as), then start gitbayd. No archive carries
 747the key file, and without it gitbayd refuses to start. Host keys are
 748preserved, so clients keep their known_hosts entries; hooks regenerate
 749at startup.
 750
 751** Schedule and recovery point
 752
 753Two timers, because the two halves of the data have different exposure.
 754
 755- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
 756- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
 757  writes the SQLite snapshot alone, last 48 kept. A few MB against the
 758  full archive's hundreds, which is what makes the frequency affordable.
 759
 760The split follows what a loss would actually cost. Repositories are git,
 761so a mirror or any clone is a second copy; the database is the only copy
 762of issues, merge requests, comments and review state. So the recovery
 763point is about an hour for the data that exists nowhere else, and a day
 764for the data that does.
 765
 766Continuous replication (litestream and similar) was considered and not
 767adopted. It would take the database's recovery point to seconds, but the
 768repositories would still be on the nightly archive, so a restore could
 769produce a database referencing commits the repository backup does not
 770have. Consistency between the two halves is worth more here than latency
 771on one of them. Revisit if repository replication becomes continuous
 772too.
 773
 774** Offsite copies
 775
 776bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
 777=/var/lib/gitbay-stage= (a database copy and =config.toml=, staged
 778there) to an S3 bucket at Scaleway, with a key that can only add
 779snapshots. Nothing else under =/etc/gitbay= is in it: not the secret
 780key file, not =apns.p8=. The key that can
 781remove them lives on the operator's machine, in
 782=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
 783cannot destroy its own history. Forgetting, pruning and rewriting all
 784run from there.
 785
 786*** Removing a repository's history from every snapshot
 787
 788A history rewrite plus =admin mr prune= takes commits off the server,
 789but every snapshot taken before it still holds them, and the retention
 790window is the only thing that ages them out. To remove them now,
 791rewrite the snapshots without that repository rather than forgetting
 792the snapshots: everything else in them stays restorable. The next
 793nightly run adds the repository back in its current state.
 794
 795Repositories are stored under the name they had on disk when each
 796snapshot was taken, so a renamed repository needs every name it has
 797carried. Check what an older snapshot holds before choosing the paths:
 798
 799#+begin_src sh
 800set -a; . ~/.config/gitbay/offsite.env; set +a
 801restic $RESTIC_OPTS snapshots
 802restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
 803#+end_src
 804
 805Then dry-run, apply, prune, and confirm nothing matches:
 806
 807#+begin_src sh
 808EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
 809restic $RESTIC_OPTS rewrite --dry-run $EXCL     # "would modify N snapshots"
 810restic $RESTIC_OPTS rewrite --forget $EXCL      # new snapshots replace the originals
 811restic $RESTIC_OPTS prune                       # drops the data nothing references
 812restic $RESTIC_OPTS find <name>.git <old-name>.git   # expect no output
 813#+end_src
 814
 815=--forget= is what makes the originals go; without it the rewritten
 816snapshots sit beside them and the data stays referenced. Snapshot IDs
 817change; their times do not. Done for krz/keycask (formerly rust-pass)
 818on 2026-09-18, across 22 snapshots.
 819
 820** Secret key
 821
 822CI secrets, webhook secrets, mirror tokens and APNs device tokens are
 823stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
 824each value prefixed with the id of the key that sealed it
 825(=gbs1:<id>:=). The key file is not in the database, not under
 826=server.root=, and therefore in neither the local archives nor the
 827main restic repository. It must be copied off the host separately;
 828without it a restored database's secrets cannot be opened, and
 829gitbayd refuses to start against them. A separate restic repository
 830for it and =apns.p8= is planned (runbook D of the data-at-rest plan)
 831and not yet in place, so today the only off-host copy is one the
 832operator makes by hand after =init= and after every =rotate=.
 833
 834#+begin_src sh
 835gitbayd admin secrets init     # once; deploy/install.sh does it on first install
 836gitbayd admin secrets check    # open every value, count by key
 837gitbayd admin secrets rotate   # new key, reseal, retire the old one (as root)
 838#+end_src
 839
 840- Missing file: every gitbayd process that opens the database refuses
 841  to run and names the path, including =serve= and, in system mode,
 842  =authorized-keys=. =migrate= does not need it.
 843- Backups: =admin backup= and =admin backup --verify= do not need the
 844  key file; a restore does.
 845- Wrong key: =serve= stops at startup naming the first row that does
 846  not open; =secrets check= does the same without starting anything.
 847- Upgrade: the first start after the upgrade seals every value still
 848  in clear and logs =sealed secret values=.
 849- Rotation: =rotate= adds a key, reseals every value under it in one
 850  transaction, then removes the old keys. Run it as root, since it
 851  replaces the key file in =/etc/gitbay=; the file keeps its owner.
 852  The daemon re-reads the file when it changes, so it needs no
 853  restart. Copy the new file off the host afterwards. =init= and
 854  =rotate= hold an flock on =<key file>.lock= while they run, so a
 855  second run waits for the first.
 856- Push devices are looked up by the SHA-256 of their token
 857  (=push_devices.token_hash=), since two seals of one token differ.
 858
 859** Restore drill
 860
 861A restore onto a clean host, run quarterly (January, April, July,
 862October) and after any change to the backup code
 863(=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite
 864job), and recorded below. The disaster it rehearses is losing bay1, so
 865the local archives are gone with it and the sources are the main
 866offsite restic repository (repositories, LFS, the staged database,
 867=config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys
 868repository once runbook D creates it; until then the operator's
 869hand-made copy), and the operator's password manager (=offsite.env=,
 870the keys repository's password and token once it exists,
 871=backup-identity.txt=). The full steps are runbook C of the
 872data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
 873
 874=gitbayd admin restore-drill= does the archive half. It extracts a
 875full archive into an empty or absent directory, runs every =--verify=
 876check on the extracted copy, and prints what was restored, the newest
 877issue, issue comment, merge request comment and push in the restored
 878database, and the elapsed time. Exit is non-zero if any check fails.
 879
 880#+begin_src sh
 881gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
 882gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill
 883#+end_src
 884
 885=admin backup --verify= extracts the archive under =$TMPDIR=. A host
 886whose =/tmp= is a tmpfs smaller than a full archive (bay1's is 3.9 GB
 887of memory) fails with "no space left on device"; point =TMPDIR= at a
 888directory on disk that the backup's user owns:
 889
 890#+begin_src sh
 891install -d -o gitbay -g gitbay -m 700 /var/backups/gitbay/verify-tmp
 892cd /var/backups/gitbay
 893sudo -u gitbay env TMPDIR=/var/backups/gitbay/verify-tmp gitbayd admin backup --verify <archive>
 894#+end_src
 895
 896What to restore, and from where:
 897
 898| Item                          | Source                                                      | Path on the drill host                         |
 899|-------------------------------+-------------------------------------------------------------+------------------------------------------------|
 900| Database                      | staged copy in =/var/lib/gitbay-stage= (restic), or an archive | =<root>/gitbay.db=                          |
 901| Repositories                  | =/var/lib/gitbay/repos= (restic), or an archive              | =<root>/repos=                                 |
 902| LFS objects                   | =/var/lib/gitbay/lfs= (restic), or an archive                | =<root>/lfs= (or =[lfs] root=)                 |
 903| Release assets                | inside each repository (=gitbay-releases/=)                  | with the repositories                          |
 904| Host keys                     | =/var/lib/gitbay/ssh= (restic), or an archive                | =<root>/ssh= (or =[ssh] host_keys=)            |
 905| =config.toml=                 | =/var/lib/gitbay-stage/config.toml= (restic)                 | =/etc/gitbay/config.toml=                      |
 906| =secret.key=, =apns.p8=       | the off-host copy; no archive or main snapshot carries them | =/etc/gitbay/=, mode 0600, owned by =gitbay=   |
 907
 908From the offsite path (restic is not run by any gitbay command):
 909
 910#+begin_src sh
 911restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
 912cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
 913gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
 914gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root
 915#+end_src
 916
 917The second archive is how the restic tree gets the same checks and
 918timestamps; =/tmp/drill-root= is discarded afterwards.
 919
 920What to check, each a column below:
 921
 922- DB integrity, connectivity, release assets, LFS: =restore-drill=
 923  prints =integrity ok=, =connectivity ok on N repositories=, =release
 924  assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin
 925  stats --json= on the source at the snapshot time.
 926- Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets
 927  check= opens every value.
 928- Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's
 929  fingerprint.
 930- Config: =gitbayd --config /etc/gitbay/config.toml check-config=.
 931- Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over
 932  SSH succeed.
 933
 934Time to service runs from the clean host's first root login to the
 935first successful =git clone= over SSH from it. The recovery point is
 936the time of the newest restic snapshot restored; record beside it the
 937newest issue, comment and push =restore-drill= printed, which show how
 938much activity the restore carries.
 939
 940The first drill ran on the operator's laptop rather than a provisioned
 941host, so its time to service has no provisioning in it, and it could
 942not check secrets: no copy of =secret.key= was on the machine, and
 943gitbayd refuses to start while any sealed value does not open. The
 944sealed values were cleared in the drill copy to reach service. The
 945off-host copy of the key is confirmed to exist (#305); the next drill
 946restores it with the snapshot and runs =admin secrets check= on the
 947restored copy.
 948
 949| Date | Host | Snapshot restored (UTC) | Newest issue / comment / push | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
 950|------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
 951| 2026-09-29 | laptop (macOS), restic from offsite | 2026-09-29 00:19:15 (ccd646cf) | 00:05:17 / 00:09:44 / 00:15:31 | 8m43s (restore 1m49s, checks 33s) | ok | ok, 69/69 | ok, 2 | ok, 529 | matches | not checked (#305) | 4.99 GiB restored; 71 sealed values cleared in the drill copy to start |
 952
 953* Upgrades
 954
 955Replace the binary, restart the unit. Migrations apply automatically and
 956are transactional; hook scripts under =<root>/hooks= are rewritten at
 957startup to point at the current binary path.
 958
 959* CI runner
 960
 961=gitbay-runner= executes builds queued by pushes and merge requests. It
 962polls over SSH with a key of scope =runner=, which reaches only the
 963runner protocol and read-only git (a runner executes arbitrary
 964repository code, so the key it holds must not do more). A runner key
 965claims builds only for the repositories it is attached to, by =repo
 966runner add= from a repository admin or an instance admin; an admin key
 967claims any. Users attach their own runners: see the Users page. For an
 968instance runner, run it as a dedicated unprivileged user on a non-admin
 969account. =admin user create --key= registers a full-scope key, so the
 970runner key is added afterwards through a bootstrap key that is then
 971removed, and attached to each repository it should build:
 972
 973#+begin_src sh
 974useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
 975sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
 976ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
 977gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
 978ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
 979ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
 980rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
 981gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
 982#+end_src
 983
 984#+begin_src sh
 985gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
 986#+end_src
 987
 988=-jobs N= runs N builds at once. Claiming is one transaction that
 989selects and updates, and each build works in its own =build-<id>=
 990directory, so workers do not collide; idle polls are staggered across
 991the interval so N of them do not wake together. The drop-in's weights
 992below are per service, not per build, so raising =-jobs= divides them
 993rather than multiplying the host's load.
 994
 995=admin runners= shows which account each runner polls as, and what each
 996may claim. A runner key claims builds only for the repositories it is
 997attached to: with none attached it claims nothing, and =-repos= may only
 998narrow within them. An admin's full-scope key claims any repository —
 999that is what =-repos= was for — and still works for the protocol during
1000a rotation. A merge request head from a fork is built in the target
1001repository as untrusted: the claim carries no secrets, and only a runner
1002started with =-untrusted= takes it. Same-repository heads were built by
1003their branch push and are not built again.
1004
1005=make deploy-runner= also installs
1006=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
1007=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
1008the daemon or the backup timers, and =NoNewPrivileges=,
1009=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
1010and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
1011and the runner's home. The e2e suite alone starts sixty
1012daemon instances; without the drop-in a deploy's copy over the admin
1013sshd stalled. Both deploy targets copy with =rsync --partial=, which
1014resumes a stalled transfer.
1015
1016Under =-isolation podman=, the default and what bay1 runs, each build
1017is confined to a container (see Container isolation below). Under
1018=-isolation none= steps run directly on the host as the runner's user,
1019so treat that machine as executing whatever your users push, and
1020install the toolchains your builds need on it.
1021
1022A runner claims the oldest pending build among the repositories its key
1023is attached to — for an admin key, the oldest in the instance. =-repos=
1024narrows within that set, which is what makes a runner outside the server
1025practical: one on a machine that should build a single project, or that
1026holds credentials for one deployment, stays on it.
1027
1028Oldest-first is across everything the key may claim, so a repository
1029with a deep queue holds every other repository the same runner serves;
1030bay1 measured a 15-minute average wait on a day of merge request
1031stacks from one repository. A runner attached to one repository cannot
1032be starved. That is the rule, decided in krz/gitbay#207: a runner
1033serving several repositories takes them oldest-first, and an operator
1034who wants one repository never to wait on another runs a second
1035runner attached to it alone. Nothing caps what an account queues,
1036and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
1037while the job's last build is pending or running, so a repository
1038with no runner holds one row per scheduled job rather than one per
1039tick, and a build runs only on a runner its owner attaches, so a busy
1040schedule spends the owner's compute. Pushes are bounded by what an
1041account can push.
1042
1043#+begin_src sh
1044gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
1045  -workdir /var/lib/gitbay-runner/work
1046#+end_src
1047
1048Add =-untrusted= only with =-isolation podman=.
1049
1050gitbay.org's runner is attached to the forge's own repositories and
1051the isolation canary, nothing else, because it shares the host with
1052the forge; its unit names no =-repos=, the attachments are the
1053boundary. Any other repository builds on a runner its owner attaches.
1054
1055=-repos= narrows an admin runner; for a runner key the attachments are
1056the boundary, held by the server, and =-repos= may only name
1057repositories among them. =-untrusted= makes a runner claim merge
1058request heads from forks; the bay1 unit sets it because it isolates in
1059podman. A runner without it builds trusted commits only.
1060
1061=gitbay dashboard= and =ssh git@<host> admin runners= list every key
1062that has polled as a runner: the account, the key's fingerprint, when it
1063last polled, the repositories it may claim — its attachments for a runner
1064key, the =-repos= it asked for or =any= for an admin key — and the build
1065it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
1066that polled by mistake, the key itself untouched. =admin runners= also
1067heads the list with the queue: builds
1068pending now, and over the last day how many were claimed, how long they
1069waited to be claimed (average and worst), and
1070how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
1071reported is failed by the scheduler's minute tick, whether or not any
1072runner is still alive: within about two minutes of its log stream ending
1073with no outcome reported — the runner reports right after closing the
1074stream, retrying for half a minute if gitbayd is unreachable — or, if no
1075stream was ever seen, at the deadline.
1076
1077Instance admin on the runner account only authorizes the claim/report
1078protocol; it grants no repo access. A build that pushes back — a pages
1079deploy, an archive publish, an automated MR branch — needs an explicit
1080grant on that repo: =repo access grant <owner/name> ci write=. Private
1081repos likewise need at least read for the clone.
1082
1083** Container isolation
1084
1085Builds run in a rootless podman container, one per job, with the
1086workspace bind mounted and nothing else: the clone happens outside with
1087the runner's key, so a step cannot read it. =-isolation none= keeps the
1088old behaviour — steps on the host as the runner's user — for an instance
1089where every repository is trusted. There is no automatic fallback: a
1090runner started with =-isolation podman= that cannot find a working
1091podman exits rather than running a build unsandboxed.
1092
1093gitbay's own jobs name =localhost/gitbay-ci:2=, built from
1094=deploy/Containerfile.ci= on the runner host. A job's image must carry
1095what its steps need: the suite drives real git, git-lfs, gpg and sshd and
1096asserts they exist before running, so the stock runner default would fail
1097it immediately. Build or rebuild it with:
1098
1099#+begin_src sh
1100ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
1101ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
1102  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
1103#+end_src
1104
1105The tag is deliberate rather than =:latest=: changing the file means
1106bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
1107change under it.
1108
1109=-image= names the image a job runs in when it declares none, and is
1110required under =-isolation podman=: there is no built-in default,
1111because an image this host does not have would fail every build. A job
1112overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
1113so a config file cannot turn it into podman arguments.
1114
1115=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
1116-memory 4g=); unset means uncapped. The runner applies them itself: it
1117creates a cgroup per build under its own delegated service cgroup,
1118writes the limits there, and starts every podman process for the build
1119inside it, with podman's cgroup handling off. Podman's own =--memory=
1120and =--cpus= never applied under rootless cgroupfs, which is what a
1121system service gets (krz/gitbay#188). The unit therefore needs
1122=Delegate=yes=, which the drop-in sets; without it the runner refuses
1123to start when a limit is set, and logs that builds run unconfined when
1124none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
1125and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
1126swap: the memory cap is what keeps the forge alive when a build
1127allocates without bound, and it sits above the e2e suite's 5GB peak
1128rather than at a fair share. =OOMPolicy=continue= keeps systemd from
1129stopping the runner when a build is OOM-killed.
1130
1131A trusted build's home is its repository's, under
1132=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
1133=HOME=: caches persist between trusted builds of one repository and are
1134never read by another's. An untrusted build — a merge request head from
1135a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
1136the build ends. Homes under =<workdir>/home= are from runners before
1137krz/gitbay#255, which shared them with untrusted builds; nothing reads
1138them any more, and they can be deleted.
1139
1140*Images are provisioned, never pulled by a build.* The runner passes
1141=--pull=never=. Two reasons, and the second is the better one: the
1142service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
1143holding a setuid file, which is nearly every distribution image; and on
1144an instance where anyone can push a =ci.yml=, =image:= would otherwise
1145mean "fetch and run anything from the internet". An operator pulls or
1146builds what is allowed and a build picks among those. A job naming an
1147image the host does not have fails with a message saying so.
1148
1149#+begin_src sh
1150su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
1151su - ci-runner -s /bin/sh -c "podman images"
1152#+end_src
1153
1154Prepare a host before pointing an isolating runner at it:
1155
1156#+begin_src sh
1157ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
1158make deploy-runner
1159#+end_src
1160
1161The script installs podman, delegates a subuid/subgid range to
1162=ci-runner=, checks that user namespaces are enabled rather than
1163assuming, enables lingering, and verifies rootless podman actually runs
1164as that user. It is idempotent.
1165
1166It also installs nftables. =make deploy-runner= ships
1167=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
1168with =gitbay-runner-egress.service=, which loads it and which the
1169runner's unit requires; it checks the file with =nft -c=, reloads the
1170unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
1171restarting the runner: =127.0.0.1:22= and the public 22 must answer,
11722222 must not. The table limits the runner's user to =127.0.0.1:22=,
1173DNS on loopback, and 22, 80 and 443 on the host's public address; the
1174Threat-Model page says why.
1175
1176That table cannot tell a build from the runner, since both run as
1177=ci-runner=. =deploy/gitbay-runner-builds.nft=, shipped as
1178=/etc/gitbay-runner/builds.nft=, matches by cgroup instead: the runner
1179starts each build under =builds/trusted= or =builds/untrusted= in its
1180service cgroup, and the table closes the host's loopback to every
1181build, limits an untrusted build to TCP 80 and 443 and DNS, and closes
1182private ranges to both (the CI page has the table). nftables resolves a
1183cgroup path to its id when the table loads, and the service cgroup is
1184new on every start, so the drop-in's =ExecStartPre= creates the two
1185cgroups, hands them to =ci-runner= and loads the table before the
1186runner starts; a table that does not load stops the start. =make
1187deploy-runner= checks the file's syntax before the restart and that the
1188table is loaded after it. If =/etc/resolv.conf= names a nameserver in a
1189private range, allow it in the file first or builds resolve nothing.
1190
1191A restart of =nftables.service= flushes both tables;
1192=systemctl reload gitbay-runner-egress= restores both.
1193
1194The egress unit's stop and the rollback below use =nft destroy=,
1195which needs nftables 1.0.8 or later; check =nft --version= on a new
1196host.
1197
1198Checking the tables on a running host, during a build (the flood test
1199below waits a minute before its first probe, for this):
1200
1201#+begin_src sh
1202nft list table inet gitbay_builds          # counters on the reject rules
1203for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
1204# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>
1205#+end_src
1206
1207A pasta process anywhere else — the runner's own =runner= cgroup, a
1208user slice — means the builds table does not see that build's traffic
1209and only the first table applies. Run
1210=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
1211once as a push and once with =--untrusted=. It fails if the runner was
1212locked out or if the build log lacks the lines only a working table
1213produces. The authoritative proof that pasta's sockets are in the
1214build's cgroup is the untrusted run: =169.254.1.2:22= and
1215=github.com:22= refused, all twelve logins refused, and the counters on
1216the =untrusted= chain's rejects rising. The first table lets
1217=ci-runner= reach both of those addresses.
1218
1219To take the builds table out, on the host:
1220
1221#+begin_src sh
1222sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
1223rm /etc/gitbay-runner/builds.nft
1224systemctl daemon-reload
1225nft destroy table inet gitbay_builds
1226#+end_src
1227
1228The runner needs no restart. It still places builds under
1229=builds/trusted= and =builds/untrusted=, but with the file gone neither
1230a runner start nor =systemctl reload gitbay-runner-egress= loads the
1231table again, and builds keep the first table and =--no-map-gw=. A
1232runner that takes =-untrusted= or polls over loopback refuses to start
1233without build cgroups at all, since the table would match nothing. The
1234next =make deploy-runner= installs the file and the lines again.
1235
1236The drop-in sets =NoNewPrivileges=no=, without which rootless podman
1237cannot call =newuidmap= and the runner refuses to start. That is a
1238considered trade, explained in the file and in the Threat-Model; if you
1239run with =-isolation none=, set it back to =yes=.
1240
1241*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
1242the build in flight, reports it, and exits; the drop-in's
1243=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
1244is what makes the signal reach the runner alone — under systemd's default
1245the build's container and the log session are signalled with it, and the
1246runner drains a build that is already dead. So =make deploy-runner=
1247waits for a running build rather than orphaning it, and a build's result
1248is retried for half a minute if gitbayd is restarting at that moment. A
1249second SIGTERM ends the runner at once, abandoning the build to the
1250reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
1251the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
1252=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
1253the runner as a transient user unit under =systemd-run= and stops it
1254under both kill modes. That last one needs a systemd user manager, so it
1255skips in the container CI runs in; run it on a Linux host with
1256=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
1257
1258*Validate podman mode on a scratch repository before pointing the runner
1259at real ones.* Every deploy that switched the whole instance to
1260containers and failed took CI down with it. Instead: create a throwaway
1261repository the runner account can read (public, or granted read — a
1262private one is "not found" to the runner and the build stays pending),
1263give it one job that names the CI image, attach the runner's key to it,
1264and deploy the runner with =-repos= naming only that repository. The
1265production unit, with its real hardening, then claims nothing else;
1266other repositories' builds queue until =-repos= is removed again, which
1267is a pause, not an outage.
1268
1269#+begin_src sh
1270# on a machine with an admin gitbay identity (root on the host has none),
1271# with the runner's public key copied from the host:
1272gitbay repo create cmc/runner-scratch   # then push a .gitbay/ci.yml naming the image
1273scp -P 2222 root@<host>:/var/lib/gitbay-runner/.ssh/id_ed25519.pub runner.pub
1274gitbay repo runner add cmc/runner-scratch < runner.pub
1275# on the host:
1276sed -i 's#^ExecStart=/usr/local/bin/gitbay-runner #&-repos cmc/runner-scratch #' /etc/systemd/system/gitbay-runner.service.d/override.conf
1277systemctl daemon-reload && systemctl restart gitbay-runner
1278# back on the admin machine:
1279gitbay build log cmc/runner-scratch 1   # green: remove -repos, redeploy, delete the scratch repository
1280#+end_src
1281
1282*Do not deploy an isolating runner to a host that has not been
1283prepared.* The runner is specified to refuse to start without a working
1284podman rather than fall back to running builds unsandboxed — a fallback
1285that silently drops isolation is worse than a stopped runner, because
1286nothing surfaces it. On an unprepared host that refusal stops every
1287build on the instance.
1288
1289The service drop-in carries =Delegate=yes= for rootless cgroup
1290management and =ReadWritePaths= for podman's store under
1291=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
1292make read-only. Those paths are prefixed =-= so they are ignored when
1293absent: the drop-in installs on unprepared hosts too, and a unit that
1294refused to start would stop every build.
1295=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
1296user: rootless storage belongs to that user, and root's prune would not
1297see it. An unpruned image store on a 40GB host is a slow outage.
1298Images labelled =org.gitbay.keep=true= are skipped, since a locally
1299built =localhost/= image cannot be pulled back; =deploy/Containerfile.ci=
1300sets it.
1301
1302* LFS storage
1303
1304Objects live content-addressed under =[lfs] root= (default
1305=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
1306(512MB default). Storage sits behind a small interface — an
1307S3-compatible backend is a drop-in with the server proxying, and
1308presigned URLs a later optimization. LFS objects do not travel with
1309push mirrors (mirrors move git refs only), and gc does not yet collect
1310orphaned objects.
1311
1312* Pages
1313
1314=[pages] domain = "example.site"= serves public repos' =pages= branches
1315on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
1316server; ACME issues per-subdomain certificates on demand (only for
1317owners that exist). The domain must not be the site host or a parent of
1318it — pages content runs its own scripts and must stay off the forge's
1319origin.
1320
1321Users with repo admin claim custom domains with =repo domain add=.
1322Claims activate only after a DNS TXT challenge proves control of the
1323domain (=repo domain verify=, audit-logged); pending claims serve
1324nothing, get no certificates, and expire after 7 days. ACME issues
1325certificates only for verified hosts, so stray DNS pointed at the
1326server gets nothing.
1327
1328* Security
1329
1330The [[Threat-Model]] file is the reference for what the forge
1331trusts and refuses to do. Operational checklist:
1332
1333- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
1334  (the module list is deliberately short — review it on each release),
1335  and a short fuzz pass over every attacker-facing parser (pkt-line,
1336  commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
1337  tagging a release. CI's own =vuln= job runs =govulncheck= nightly
1338  against main rather than per push, because =@latest= scans today's
1339  advisory database and an advisory lands without anyone pushing;
1340  =build trigger krz/gitbay vuln= runs it on demand.
1341- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
1342  DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
1343  configuration needed.
1344- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
1345  gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
1346  =LockPersonality=, =MemoryDenyWriteExecute=,
1347  =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
1348  INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
1349- *OS patches* apply via =unattended-upgrades= (security origins,
1350  auto-reboot 04:30 if required).
1351- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
1352  watched by =fail2ban=; gitbayd's own port 22 is throttled by
1353  =limits.ssh_auth_rate= (auth failures per IP per minute), and every
1354  account's writes by =limits.write_rate=.
1355- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
1356journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
1357webhook: disk, service, the daemon's own =/healthz= answer, certificate
1358expiry, and the age of the newest full backup and database snapshot.
1359It exits non-zero on an alert so the unit shows in =systemctl
1360--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
1361a certificate under 21 days, a full backup over 25 hours old, or a
1362database snapshot over 2 hours old.
1363
1364=GET /healthz= is unauthenticated and cache-free: whether the database
1365answers and which commit serves, 503 when it does not.
1366- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
1367  0750, owned by =gitbay=). The nightly archive plus provider snapshots
1368  are the recovery path; for tighter RPO, add continuous replication
1369  (litestream) against the same file — it coexists with the WAL.
1370
1371* Odds and ends
1372
1373- deleting a fork marks MRs sourced from it =source_gone=; their diffs
1374  remain viewable and mergeable because the target repo owns the
1375  objects.
1376- =refs/merge-requests/*= is server-owned and unpushable by clients;
1377  only =admin mr prune= removes one (see Maintenance).
1378- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
1379  the =events= table, which also feeds webhooks.
1380- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.