.gitbay/wiki/Admin.org

  1#+title: gitbay admin guide
  2
  3One static binary (=gitbayd=), one SQLite file, bare repositories on
  4disk, and the system =git=. Schema migrations run automatically on
  5startup and on every admin command.
  6
  7* Install
  8
  9Build from source (=go build ./cmd/gitbayd=), install via the vanity
 10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
 11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
 12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
 13manifest (CGO off, trimpath, stripped — byte-identical per commit and
 14toolchain).
 15
 16#+begin_src sh
 17install -m 755 gitbayd /usr/local/bin/
 18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
 19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
 20gitbayd --config /etc/gitbay/config.toml check-config
 21gitbayd --config /etc/gitbay/config.toml admin secrets init
 22chown gitbay:gitbay /etc/gitbay/secret.key
 23#+end_src
 24
 25=admin secrets init= refuses when the key file already exists, so a
 26reinstall on the same host should skip it — =deploy/install.sh= does
 27this with a file check before running it.
 28
 29=deploy/= in the source tree has a cloud-init file, a hardened systemd
 30unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
 31the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
 3222/80/443 without root.
 33
 34** The SSH port decision
 35
 36- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
 37  22 — move the host's admin sshd to another port. Remotes read
 38  =git@host:owner/repo= with no port gymnastics.
 39- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
 40  =AuthorizedKeysCommand=:
 41  #+begin_example
 42  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
 43  AuthorizedKeysCommandUser gitbay
 44  #+end_example
 45  sshd requires that binary to be root-owned and not group/world
 46  writable. Unknown keys fail authentication inside sshd, so system mode
 47  requires =registration.mode = "closed"= (check-config enforces this).
 48  Under the forced command the CLI's leading =--term=<cols>[,color]=
 49  argument works as is; a client that sets =GITBAY_TERM= instead needs
 50  =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
 51  handled by the host's sshd, not gitbayd.
 52
 53* Configuration reference
 54
 55=/etc/gitbay/config.toml=. =check-config= validates and names every
 56contradiction; =--no-host-checks= skips port/path probes.
 57=gitbayd admin config show= prints the configuration in effect as TOML,
 58every default filled in and =smtp_pass= redacted. A file that fails
 59validation still prints, followed by the contradiction.
 60
 61** [server]
 62- =root= (default =/var/lib/gitbay=) — repositories, database, host
 63  keys, ACME cache all live here.
 64- =site_url= (required) — canonical =https://host=; drives ACME, clone
 65  URLs, mail links.
 66- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
 67  seal CI secrets, webhook secrets, mirror tokens and push device
 68  tokens in the database. Must be outside =root=, mode 0600, readable
 69  by the daemon's user. See "Secret key" below.
 70- =source_repo= (optional, =owner/name=) — the repository this instance
 71  develops itself in. Startup warns when the running build's commit is
 72  not on that repository's default branch, which is how a binary built
 73  from an unmerged branch stops being invisible. Leave it unset unless
 74  the instance hosts its own source.
 75
 76** [ssh]
 77- =mode= — =embedded= | =system= (above).
 78- =port= (22) — embedded listener port.
 79- =host_keys= — list of private key paths; empty generates an ed25519
 80  key at =<root>/ssh/host_ed25519=.
 81
 82** [http]
 83- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
 84- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
 85  =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
 86  (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
 87  failing to bind it is a warning, not fatal. Requires an =https://=
 88  site_url with a public DNS name.
 89- =files=: =cert_file= + =key_file=.
 90- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
 91- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
 92  =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
 93  release the binary was built with. =openssl s_client -connect
 94  <host>:443 -tls1_1= fails the handshake.
 95
 96=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
 97front of the daemon. A request from one of them is attributed, for API
 98rate limiting, to the last =X-Forwarded-For= hop that is not itself a
 99trusted proxy; from anyone else the header is ignored. Empty, the
100default, is right when gitbayd terminates TLS itself.
101
102A reverse proxy in front of gitbay must not buffer responses, or the
103build page's live log arrives only when the build ends;
104=X-Accel-Buffering: no= covers nginx.
105
106** [web]
107- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
108  web routes are never registered; in accounts, browser sessions are
109  minted over SSH (=web login=), and users with write access can create
110  repos, comment, and make simple file edits (which commit unsigned,
111  honestly). =password_auth= is reserved and currently rejected.
112- =title= — the instance's display name in the rail and page titles;
113  empty falls back to the site host. Lower case is the convention for
114  gitbay itself.
115- =privacy_notice= — operator text shown on =/privacy= under the fixed
116  statement.
117
118** [registration]
119- =mode= — =closed= (default) | =invite= | =open=. invite/open require
120  [mail]. See the user guide for the flows.
121
122- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
123  self-registered account still unverified after that long is removed,
124  hourly and at start, audited as =pending.expired=.
125
126- =notify_admin= (false) — mail every instance admin when an account
127  becomes active: an invite redeemed, or an open-mode signup that
128  verified its address. The unverified row an open signup creates is
129  not reported, because anyone can post the form and mailing on that
130  would point a flood at the admins. Recipients are the verified
131  primary addresses of active admins who have activity mail on, the
132  same rule any other notice follows, so an admin with no verified
133  address hears nothing. The notice is queued, so a dead SMTP host
134  shows up in the admin page's Mail table instead of failing the
135  registration. Requires [mail].
136
137** [mail]
138- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
139  =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
140  registration and self-service =email add=; in closed mode you may omit
141  it entirely and assert addresses by hand (below).
142- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
143  for relays on 465).
144- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
145  gets no mail: delivery fails and retries, and the admin page's Mail
146  table shows the error. Unset, it is on for every relay except
147  =localhost= and loopback addresses; set =false= to allow plaintext
148  to a remote relay.
149
150** [push]
151Push notifications to Apple devices, delivered by gitbayd talking to
152APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
153operator-supplied provider key. Off unless configured.
154
155- =enabled= (false).
156- =key_file= — path to the =.p8= provider key from Apple's developer
157  portal (Certificates, Identifiers & Profiles → Keys). It belongs at
158  =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
159  as. Read and validated at startup: it must parse as a PEM-wrapped
160  PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
161  than fill a queue nobody is watching.
162- =key_id=, =team_id= — the key's id and your Apple developer team id,
163  both from the same portal page.
164- =topic= — the app's bundle identifier. *An APNs key belongs to a
165  bundle ID.* gitbay.org pushes to the App Store build under its own
166  bundle id; a self-hoster who wants push ships their own iOS build
167  under their own bundle id, with its own =.p8= key from their own
168  developer account, and points =topic= at that id. There is no way to
169  push to someone else's build, by design — this is Apple's model, not
170  gitbay's.
171- =environment= — =production= or =sandbox=, naming the APNs host
172  rather than taking a URL, so a typo cannot aim the key at a host that
173  is not Apple's.
174
175All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
176are required when =enabled= is true; validation runs at config load,
177so a misconfigured =[push]= is caught before the daemon serves
178anything. The delivery queue (a device's undelivered and attempted
179pushes) is capped the same way the mail queue is, by =[retention]
180push=.
181
182** [backup]
183- =age_recipients= (optional) — age public keys (=age1...=). When set,
184  =admin backup= encrypts every archive to them and appends =.age= to
185  its name. Generate the pair off the host with =age-keygen=; only the
186  public key goes here, so the host writes archives it cannot read.
187  The restic copy is unaffected: the offsite job stages its own
188  =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
189  not the archives.
190
191** [api]
192- =enabled= (false) — the JSON API surface; see [[API]]. Off
193  means no credential-bearing HTTP endpoint exists at all.
194
195** [webhooks]
196- =allow_local= (false) — permit webhook and mirror targets on
197  loopback, private, shared (100.64.0.0/10), link-local or multicast
198  addresses. Leave off unless you know why you need it (SSRF).
199
200** [limits]
201- =clone_timeout= (3600s) — cap on =repo import= fetches.
202- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
203- =max_asset_bytes= (512MB) — cap per uploaded release asset.
204- =max_snippet_bytes= (1MB) — cap per snippet file.
205- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
206- =max_repos_per_user= (0, unlimited) — repositories an account may own
207  directly; =repo create=, =fork= and =import= refuse past it.
208  Organizations are not capped.
209- =max_bytes_per_user= (0, unlimited) — disk the account's own
210  repositories may take; a push may be no larger than what is left.
211- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
212
213** [git_daemon]
214- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
215  Serves only public repositories that additionally ran
216  =repo settings git-daemon <repo> on=.
217
218** [mirrors]
219- =pull_interval_minutes= (15) — how often pull mirrors fetch their
220  upstream. Push mirrors sync shortly after each local ref update.
221  Mirror URLs pass the same SSRF rules as webhook targets, when saved
222  and again before every sync; git then connects only to the addresses
223  that were checked (=http.curloptResolve=) and does not follow
224  redirects, so a mirror of a renamed repository fails until its URL
225  is updated. Needs git 2.37 or later on the server; with an older git
226  the worker logs an error at start and syncs no mirror, recording the
227  reason on each. Sync ignores the system and global gitconfig.
228
229** [go_import]
230Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
231Requests with =?go-get=1= at or under the module path answer with the
232go-import meta tag pointing at the repository's HTTPS clone URL, so
233=go install host/module/cmd/...@latest= resolves. The repository should
234be public (the module path itself confirms it exists).
235
236* Users, email, invites
237
238Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
239backfills is a wrapper that dispatches the registry command of the same
240name as the host: an admin context with no account behind it, so its
241audit rows carry no actor and =source: host=. The same commands run in an
242instance admin's SSH session (=ssh git@<host> admin ...=) and write the
243same rows with the key fingerprint as source. One implementation, two
244credentials.
245
246#+begin_src sh
247gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
248gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
249gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP
250#+end_src
251
252"Verified" means SMTP-confirmed or host-admin-asserted; the database
253records which. Verified emails are what make commit signatures
254meaningful — an unverified address never produces a =verified= badge.
255
256* Audit and account control
257
258The audit log is the security feed (events are the product feed): every
259successful mutating command with its argv and source credential (SSH key
260fingerprint or API), every refused one (exit 3 or 4) as =refused
261<command>=, refused pushes as =refused git-receive-pack=, registrations,
262admin actions, force-pushes, and auth failures/throttling. A refusal row
263keeps the flag names and the first positional, not the values.
264Refusals are recorded up to ten a minute per account and 600 a minute
265across the instance; past either, one =refused.throttled= row stands
266for the rest of that minute. The caps bound the embedded listener, the
267web and the API; under =ssh.mode = "system"= each =gitbayd shell=
268connection counts separately. Secrets never appear — they travel on
269stdin, never in argv.
270
271Each row carries the SHA-256 of the row before it. =gitbayd admin audit
272verify= opens the store as other admin commands do, applying pending
273migrations, so run it with the binary that matches the daemon. It
274recomputes the chain and exits 1 naming the first row that was
275edited or whose predecessor was removed. Retention removing the oldest
276rows is not a break. Rows written before the chain existed are counted
277and skipped; when every row is such a row, verify warns and exits 1,
278since clearing the hash columns looks the same. After an upgrade that
279clears with the first new audit row.
280
281Removing the newest rows leaves no break, and neither do rows written
282afterwards under the freed ids. The database cannot show either. The
283daemon logs every row it writes to its journal, outside the database
284(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
285id and hash: compare them with the newest journal line. Rows written by
286host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
287"system"=, are not copied to the journal.
288
289#+begin_src sh
290gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
291gitbayd admin audit verify           # check the hash chain; exit 1 names the first bad row
292ssh git@<host> audit ...             # the same, from an admin session
293ssh git@<host> admin user list [--state active|pending|disabled|admin]
294ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
295ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default]   # per-account caps
296ssh git@<host> admin user promote <name>   # grant instance admin
297ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
298gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
299gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
300gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
301gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
302                                     #   refused (with each blocker named) while
303                                     #   the account owns repos, authored
304                                     #   issues/MRs/comments/reviews, or is an
305                                     #   org's only admin
306#+end_src
307
308=--actor= takes a username, or =-= for rows with no actor: host commands
309and auth failures. =--action= is a prefix, so =cmd repo= catches every
310repository command and =admin= every host or admin-session action.
311=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
312
313=admin user list= pages by username (=--limit=, =--cursor=) and carries
314each account's state and =last_seen=, the newest use of any of its SSH
315keys or API tokens. =admin user show= adds the keys with their last use,
316each address with how it was verified, PGP keys, org roles, the owned
317repository count, API token names, and live browser sessions. Both are
318Both are refused to non-admins, like =audit=, on every surface.
319
320=/admin/users= is the same list in a browser, linked from the admin
321page: the state filter the command takes, keyset paging on its cursor,
322and a row per account with promote, demote, disable and enable, each
323dispatching the command. Demote and disable ask for the username to be
324typed, since both take someone's access away. Creating and deleting an
325account, issuing an invite and asserting an address stay on the command
326line: each takes a key, mints a credential, or cannot be undone. A
327non-admin gets the 404 a missing page would, so the URL confirms
328nothing.
329
330Promotion needs an active account: a pending or disabled one is refused.
331Demotion is refused when it would leave no admin, over SSH, in the
332browser and on the host alike, so the host-local =promote= is the way
333back in when the only admin key is lost.
334
335Instance admin carries no right on anyone's repository: policy does not
336consult it, and a private repository still answers not-found to an
337admin. Moderation goes through explicit overrides that skip the access
338check and write their own audit row:
339
340#+begin_src sh
341ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
342ssh git@<host> admin repo archive|unarchive <owner/name>
343ssh git@<host> admin repo visibility <owner/name> public|private
344ssh git@<host> admin repo delete <owner/name> --yes
345#+end_src
346
347Each lands in the audit log as =admin repo.<action>= naming the
348repository, on top of the =cmd= row every mutating command gets.
349
350=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
351per minute — successful auths never count and clear the slate.
352=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
353push.
354
355=limits.write_rate= (60) bounds *mutating commands per account per
356minute*. It is counted in the dispatcher, so SSH, the JSON API and the
357web spend one budget and a caller cannot refresh it by changing surface;
358=limits.api_rate= stays in front of it, bounding a network source rather
359than an account. A command is one token whatever it writes, so a bundle
360import costs one and only a loop of separate commands spends the budget.
361Read-only commands, the runner protocol (a build streams its log in many
362small writes) and the host CLI are exempt. Refusals exit 4 and say when
363to retry. A negative value turns the limit off; it matters most with
364=registration = "open"=, where every write also queues notification mail
365and webhook deliveries.
366
367* Queues
368
369Every background worker keeps a backlog and a failure state. An instance
370admin reads them all in one place:
371
372#+begin_src sh
373gitbay dashboard --json | jq .queues   # webhooks, mail, push, mirrors, builds, deps
374#+end_src
375
376Per worker: pending, retrying (pending with a failed attempt) and
377dead-lettered counts with the oldest pending age, and the retrying or
378failed rows themselves, capped at twenty each. Builds list what is
379running and then what is pending, each since when, so a build no runner
380is scoped to claim is visible here rather than only in its repository;
381mirrors list the ones whose last sync failed;
382dependency checks list the ones whose last check errored. Non-admins get
383no =queues= key at all.
384
385Push rows name the device id, never the token. Watch this one after
386configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
387config validation, which can only check that the =.p8= parses, and then
388every send comes back =403 InvalidProviderToken= and dead-letters on its
389first attempt.
390
391In accounts mode the same read renders at =/admin=, linked from the rail
392for admins. Anyone else gets a 404 there.
393
394A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
395with the address redacted out of the relay's error. The id is the queue
396row: find it in the Mail table on =/admin=, or in =dashboard --json=,
397where the recipient and the unredacted error are. That is deliberate —
398see the Threat-Model page.
399
400* Maintenance
401
402#+begin_src sh
403gitbayd admin stats [--json]         # counts, database size, per-repo disk
404ssh git@<host> admin stats [--json]  # the same, from an admin session
405gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
406gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
407gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)
408#+end_src
409
410A history rewrite leaves the commits it removed reachable through
411=refs/merge-requests/N/head= of the merge requests that landed them, so
412they stay fetchable by anyone who can read the repository. Nothing drops
413a head ref on its own — an open or source-gone MR is merged through it,
414and a merged or closed one keeps its diff readable through it — so the
415cleanup is a command an instance admin runs, naming the MRs:
416
417#+begin_src sh
418ssh git@<host> admin mr prune owner/name 1 2 3 --yes
419#+end_src
420
421It refuses an open or source-gone MR, deletes the named refs, runs
422=git gc --prune=now= on that one repository so the objects go at once
423rather than after git's two-week grace, leaves a system comment on each
424MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
425reviews and head sha; =mr diff= and the MR page say the head is gone.
426Run it when nothing is pushing to that repository: without the grace, a
427push caught between leaving quarantine and writing its ref loses its
428objects. Objects also survive in offsite backups until those are
429rewritten; see "Removing a repository's history from every snapshot".
430
431=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
432weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
433layout the source sent, so a first manual =admin gc= after a bulk
434import is worthwhile.
435
436* Backup and restore
437
438#+begin_src sh
439gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
440gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back
441#+end_src
442
443One archive: a consistent SQLite snapshot (taken *before* the
444repositories are read, so the database never references objects the
445archive missed), every repository, and the SSH host keys. Excluded:
446hook socket, regenerated hook scripts, WAL files. Safe to run against a
447live daemon.
448
449=--verify= reads an archive back: the snapshot must pass SQLite's
450integrity check, and every repository the snapshot names must be in the
451archive. A database-only archive is checked for integrity and says so.
452Exit is non-zero on damage or a missing repository.
453
454With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
455and =--verify= needs the private key:
456
457#+begin_src sh
458gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
459age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
460#+end_src
461
462The identity lives off the host (with the secret key file and the
463restic credentials), so verifying an encrypted archive happens there
464or on a restore host.
465
466Restore: extract into an empty directory, point =server.root= at it,
467start gitbayd. Host keys are preserved, so clients keep their
468known_hosts entries; hooks regenerate at startup.
469
470** Schedule and recovery point
471
472Two timers, because the two halves of the data have different exposure.
473
474- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
475- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
476  writes the SQLite snapshot alone, last 48 kept. A few MB against the
477  full archive's hundreds, which is what makes the frequency affordable.
478
479The split follows what a loss would actually cost. Repositories are git,
480so a mirror or any clone is a second copy; the database is the only copy
481of issues, merge requests, comments and review state. So the recovery
482point is about an hour for the data that exists nowhere else, and a day
483for the data that does.
484
485Continuous replication (litestream and similar) was considered and not
486adopted. It would take the database's recovery point to seconds, but the
487repositories would still be on the nightly archive, so a restore could
488produce a database referencing commits the repository backup does not
489have. Consistency between the two halves is worth more here than latency
490on one of them. Revisit if repository replication becomes continuous
491too.
492
493** Offsite copies
494
495bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
496=/var/lib/gitbay-stage= (the staged database copy) to an S3 bucket at
497Scaleway, with a key that can only add snapshots. The key that can
498remove them lives on the operator's machine, in
499=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
500cannot destroy its own history. Forgetting, pruning and rewriting all
501run from there.
502
503*** Removing a repository's history from every snapshot
504
505A history rewrite plus =admin mr prune= takes commits off the server,
506but every snapshot taken before it still holds them, and the retention
507window is the only thing that ages them out. To remove them now,
508rewrite the snapshots without that repository rather than forgetting
509the snapshots: everything else in them stays restorable. The next
510nightly run adds the repository back in its current state.
511
512Repositories are stored under the name they had on disk when each
513snapshot was taken, so a renamed repository needs every name it has
514carried. Check what an older snapshot holds before choosing the paths:
515
516#+begin_src sh
517set -a; . ~/.config/gitbay/offsite.env; set +a
518restic $RESTIC_OPTS snapshots
519restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
520#+end_src
521
522Then dry-run, apply, prune, and confirm nothing matches:
523
524#+begin_src sh
525EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
526restic $RESTIC_OPTS rewrite --dry-run $EXCL     # "would modify N snapshots"
527restic $RESTIC_OPTS rewrite --forget $EXCL      # new snapshots replace the originals
528restic $RESTIC_OPTS prune                       # drops the data nothing references
529restic $RESTIC_OPTS find <name>.git <old-name>.git   # expect no output
530#+end_src
531
532=--forget= is what makes the originals go; without it the rewritten
533snapshots sit beside them and the data stays referenced. Snapshot IDs
534change; their times do not. Done for krz/keycask (formerly rust-pass)
535on 2026-09-18, across 22 snapshots.
536
537** Secret key
538
539CI secrets, webhook secrets, mirror tokens and APNs device tokens are
540stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
541each value prefixed with the id of the key that sealed it
542(=gbs1:<id>:=). The key file is not in the database, not under
543=server.root=, and therefore in neither the local archives nor the
544main restic repository. It must be copied off the host separately;
545without it a restored database's secrets cannot be opened, and
546gitbayd refuses to start against them.
547
548#+begin_src sh
549gitbayd admin secrets init     # once; deploy/install.sh does it on first install
550gitbayd admin secrets check    # open every value, count by key
551gitbayd admin secrets rotate   # new key, reseal, retire the old one (as root)
552#+end_src
553
554- Missing file: every gitbayd process that opens the database refuses
555  to run and names the path, including =serve= and, in system mode,
556  =authorized-keys=. =migrate= does not need it.
557- Wrong key: =serve= stops at startup naming the first row that does
558  not open; =secrets check= does the same without starting anything.
559- Upgrade: the first start after the upgrade seals every value still
560  in clear and logs =sealed secret values=.
561- Rotation: =rotate= adds a key, reseals every value under it in one
562  transaction, then removes the old keys. Run it as root, since it
563  replaces the key file in =/etc/gitbay=; the file keeps its owner.
564  The daemon re-reads the file when it changes, so it needs no
565  restart. Copy the new file off the host afterwards.
566- Push devices are looked up by the SHA-256 of their token
567  (=push_devices.token_hash=), since two seals of one token differ.
568
569* Upgrades
570
571Replace the binary, restart the unit. Migrations apply automatically and
572are transactional; hook scripts under =<root>/hooks= are rewritten at
573startup to point at the current binary path.
574
575* CI runner
576
577=gitbay-runner= executes builds queued by pushes and merge requests. It
578polls over SSH with a key of scope =runner=, which reaches only the
579runner protocol and read-only git (a runner executes arbitrary
580repository code, so the key it holds must not do more). A runner key
581claims builds only for the repositories it is attached to, by =repo
582runner add= from a repository admin or an instance admin; an admin key
583claims any. Users attach their own runners: see the Users page. For an
584instance runner, run it as a dedicated unprivileged user on a non-admin
585account. =admin user create --key= registers a full-scope key, so the
586runner key is added afterwards through a bootstrap key that is then
587removed, and attached to each repository it should build:
588
589#+begin_src sh
590useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
591sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
592ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
593gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
594ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
595ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
596rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
597gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
598#+end_src
599
600#+begin_src sh
601gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
602#+end_src
603
604=-jobs N= runs N builds at once. Claiming is one transaction that
605selects and updates, and each build works in its own =build-<id>=
606directory, so workers do not collide; idle polls are staggered across
607the interval so N of them do not wake together. The drop-in's weights
608below are per service, not per build, so raising =-jobs= divides them
609rather than multiplying the host's load.
610
611=admin runners= shows which account each runner polls as, and what each
612may claim. A runner key claims builds only for the repositories it is
613attached to: with none attached it claims nothing, and =-repos= may only
614narrow within them. An admin's full-scope key claims any repository —
615that is what =-repos= was for — and still works for the protocol during
616a rotation. A merge request head from a fork is built in the target
617repository as untrusted: the claim carries no secrets, and only a runner
618started with =-untrusted= takes it. Same-repository heads were built by
619their branch push and are not built again.
620
621=make deploy-runner= also installs
622=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
623=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
624the daemon or the backup timers, and =NoNewPrivileges=,
625=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
626and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
627and the runner's home. The e2e suite alone starts sixty
628daemon instances; without the drop-in a deploy's copy over the admin
629sshd stalled. Both deploy targets copy with =rsync --partial=, which
630resumes a stalled transfer.
631
632Under =-isolation podman=, the default and what bay1 runs, each build
633is confined to a container (see Container isolation below). Under
634=-isolation none= steps run directly on the host as the runner's user,
635so treat that machine as executing whatever your users push, and
636install the toolchains your builds need on it.
637
638A runner claims the oldest pending build among the repositories its key
639is attached to — for an admin key, the oldest in the instance. =-repos=
640narrows within that set, which is what makes a runner outside the server
641practical: one on a machine that should build a single project, or that
642holds credentials for one deployment, stays on it.
643
644Oldest-first is across everything the key may claim, so a repository
645with a deep queue holds every other repository the same runner serves;
646bay1 measured a 15-minute average wait on a day of merge request
647stacks from one repository. A runner attached to one repository cannot
648be starved. That is the rule, decided in krz/gitbay#207: a runner
649serving several repositories takes them oldest-first, and an operator
650who wants one repository never to wait on another runs a second
651runner attached to it alone. Nothing caps what an account queues,
652and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
653while the job's last build is pending or running, so a repository
654with no runner holds one row per scheduled job rather than one per
655tick, and a build runs only on a runner its owner attaches, so a busy
656schedule spends the owner's compute. Pushes are bounded by what an
657account can push.
658
659#+begin_src sh
660gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
661  -workdir /var/lib/gitbay-runner/work
662#+end_src
663
664Add =-untrusted= only with =-isolation podman=.
665
666gitbay.org's runner is attached to the forge's own repositories and
667the isolation canary, nothing else, because it shares the host with
668the forge; its unit names no =-repos=, the attachments are the
669boundary. Any other repository builds on a runner its owner attaches.
670
671=-repos= narrows an admin runner; for a runner key the attachments are
672the boundary, held by the server, and =-repos= may only name
673repositories among them. =-untrusted= makes a runner claim merge
674request heads from forks; the bay1 unit sets it because it isolates in
675podman. A runner without it builds trusted commits only.
676
677=gitbay dashboard= and =ssh git@<host> admin runners= list every key
678that has polled as a runner: the account, the key's fingerprint, when it
679last polled, the repositories it may claim — its attachments for a runner
680key, the =-repos= it asked for or =any= for an admin key — and the build
681it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
682that polled by mistake, the key itself untouched. =admin runners= also
683heads the list with the queue: builds
684pending now, and over the last day how many were claimed, how long they
685waited to be claimed (average and worst), and
686how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
687reported is failed by the scheduler's minute tick, whether or not any
688runner is still alive: within about two minutes of its log stream ending
689with no outcome reported — the runner reports right after closing the
690stream, retrying for half a minute if gitbayd is unreachable — or, if no
691stream was ever seen, at the deadline.
692
693Instance admin on the runner account only authorizes the claim/report
694protocol; it grants no repo access. A build that pushes back — a pages
695deploy, an archive publish, an automated MR branch — needs an explicit
696grant on that repo: =repo access grant <owner/name> ci write=. Private
697repos likewise need at least read for the clone.
698
699** Container isolation
700
701Builds run in a rootless podman container, one per job, with the
702workspace bind mounted and nothing else: the clone happens outside with
703the runner's key, so a step cannot read it. =-isolation none= keeps the
704old behaviour — steps on the host as the runner's user — for an instance
705where every repository is trusted. There is no automatic fallback: a
706runner started with =-isolation podman= that cannot find a working
707podman exits rather than running a build unsandboxed.
708
709gitbay's own jobs name =localhost/gitbay-ci:2=, built from
710=deploy/Containerfile.ci= on the runner host. A job's image must carry
711what its steps need: the suite drives real git, git-lfs, gpg and sshd and
712asserts they exist before running, so the stock runner default would fail
713it immediately. Build or rebuild it with:
714
715#+begin_src sh
716ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
717ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
718  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
719#+end_src
720
721The tag is deliberate rather than =:latest=: changing the file means
722bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
723change under it.
724
725=-image= names the image a job runs in when it declares none, and is
726required under =-isolation podman=: there is no built-in default,
727because an image this host does not have would fail every build. A job
728overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
729so a config file cannot turn it into podman arguments.
730
731=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
732-memory 4g=); unset means uncapped. The runner applies them itself: it
733creates a cgroup per build under its own delegated service cgroup,
734writes the limits there, and starts every podman process for the build
735inside it, with podman's cgroup handling off. Podman's own =--memory=
736and =--cpus= never applied under rootless cgroupfs, which is what a
737system service gets (krz/gitbay#188). The unit therefore needs
738=Delegate=yes=, which the drop-in sets; without it the runner refuses
739to start when a limit is set, and logs that builds run unconfined when
740none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
741and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
742swap: the memory cap is what keeps the forge alive when a build
743allocates without bound, and it sits above the e2e suite's 5GB peak
744rather than at a fair share. =OOMPolicy=continue= keeps systemd from
745stopping the runner when a build is OOM-killed.
746
747A trusted build's home is its repository's, under
748=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
749=HOME=: caches persist between trusted builds of one repository and are
750never read by another's. An untrusted build — a merge request head from
751a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
752the build ends. Homes under =<workdir>/home= are from runners before
753krz/gitbay#255, which shared them with untrusted builds; nothing reads
754them any more, and they can be deleted.
755
756*Images are provisioned, never pulled by a build.* The runner passes
757=--pull=never=. Two reasons, and the second is the better one: the
758service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
759holding a setuid file, which is nearly every distribution image; and on
760an instance where anyone can push a =ci.yml=, =image:= would otherwise
761mean "fetch and run anything from the internet". An operator pulls or
762builds what is allowed and a build picks among those. A job naming an
763image the host does not have fails with a message saying so.
764
765#+begin_src sh
766su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
767su - ci-runner -s /bin/sh -c "podman images"
768#+end_src
769
770Prepare a host before pointing an isolating runner at it:
771
772#+begin_src sh
773ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
774make deploy-runner
775#+end_src
776
777The script installs podman, delegates a subuid/subgid range to
778=ci-runner=, checks that user namespaces are enabled rather than
779assuming, enables lingering, and verifies rootless podman actually runs
780as that user. It is idempotent.
781
782It also installs nftables. =make deploy-runner= ships
783=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
784with =gitbay-runner-egress.service=, which loads it and which the
785runner's unit requires; it checks the file with =nft -c=, reloads the
786unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
787restarting the runner: =127.0.0.1:22= and the public 22 must answer,
7882222 must not. The table limits the runner's user to =127.0.0.1:22=,
789DNS on loopback, and 22, 80 and 443 on the host's public address; the
790Threat-Model page says why. A restart of =nftables.service= flushes it;
791=systemctl reload gitbay-runner-egress= restores it.
792
793The drop-in sets =NoNewPrivileges=no=, without which rootless podman
794cannot call =newuidmap= and the runner refuses to start. That is a
795considered trade, explained in the file and in the Threat-Model; if you
796run with =-isolation none=, set it back to =yes=.
797
798*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
799the build in flight, reports it, and exits; the drop-in's
800=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
801is what makes the signal reach the runner alone — under systemd's default
802the build's container and the log session are signalled with it, and the
803runner drains a build that is already dead. So =make deploy-runner=
804waits for a running build rather than orphaning it, and a build's result
805is retried for half a minute if gitbayd is restarting at that moment. A
806second SIGTERM ends the runner at once, abandoning the build to the
807reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
808the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
809=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
810the runner as a transient user unit under =systemd-run= and stops it
811under both kill modes. That last one needs a systemd user manager, so it
812skips in the container CI runs in; run it on a Linux host with
813=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
814
815*Validate podman mode on a scratch repository before pointing the runner
816at real ones.* Every deploy that switched the whole instance to
817containers and failed took CI down with it. Instead: create a throwaway
818repository the runner account can read (public, or granted read — a
819private one is "not found" to the runner and the build stays pending),
820give it one job that names the CI image, and deploy the runner with
821=-repos= naming only that repository. The production unit, with its real
822hardening, then claims nothing else; other repositories' builds queue
823until =-repos= is switched back, which is a pause, not an outage.
824
825#+begin_src sh
826gitbay repo create cmc/ci-smoke          # then push a .gitbay/ci.yml naming the image
827sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
828systemctl daemon-reload && systemctl restart gitbay-runner
829gitbay build log cmc/ci-smoke 1         # green: switch -repos back, redeploy
830#+end_src
831
832*Do not deploy an isolating runner to a host that has not been
833prepared.* The runner is specified to refuse to start without a working
834podman rather than fall back to running builds unsandboxed — a fallback
835that silently drops isolation is worse than a stopped runner, because
836nothing surfaces it. On an unprepared host that refusal stops every
837build on the instance.
838
839The service drop-in carries =Delegate=yes= for rootless cgroup
840management and =ReadWritePaths= for podman's store under
841=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
842make read-only. Those paths are prefixed =-= so they are ignored when
843absent: the drop-in installs on unprepared hosts too, and a unit that
844refused to start would stop every build.
845The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
846names that repository too; a scoped runner claims nothing else.
847=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
848user: rootless storage belongs to that user, and root's prune would not
849see it. An unpruned image store on a 40GB host is a slow outage.
850
851* LFS storage
852
853Objects live content-addressed under =[lfs] root= (default
854=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
855(512MB default). Storage sits behind a small interface — an
856S3-compatible backend is a drop-in with the server proxying, and
857presigned URLs a later optimization. LFS objects do not travel with
858push mirrors (mirrors move git refs only), and gc does not yet collect
859orphaned objects.
860
861* Pages
862
863=[pages] domain = "example.site"= serves public repos' =pages= branches
864on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
865server; ACME issues per-subdomain certificates on demand (only for
866owners that exist). The domain must not be the site host or a parent of
867it — pages content runs its own scripts and must stay off the forge's
868origin.
869
870Users with repo admin claim custom domains with =repo domain add=.
871Claims activate only after a DNS TXT challenge proves control of the
872domain (=repo domain verify=, audit-logged); pending claims serve
873nothing, get no certificates, and expire after 7 days. ACME issues
874certificates only for verified hosts, so stray DNS pointed at the
875server gets nothing.
876
877* Security
878
879The [[Threat-Model]] file is the reference for what the forge
880trusts and refuses to do. Operational checklist:
881
882- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
883  (the module list is deliberately short — review it on each release),
884  and a short fuzz pass over every attacker-facing parser (pkt-line,
885  commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
886  tagging a release. CI's own =vuln= job runs =govulncheck= nightly
887  against main rather than per push, because =@latest= scans today's
888  advisory database and an advisory lands without anyone pushing;
889  =build trigger krz/gitbay vuln= runs it on demand.
890- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
891  DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
892  configuration needed.
893- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
894  gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
895  =LockPersonality=, =MemoryDenyWriteExecute=,
896  =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
897  INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
898- *OS patches* apply via =unattended-upgrades= (security origins,
899  auto-reboot 04:30 if required).
900- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
901  watched by =fail2ban=; gitbayd's own port 22 is throttled by
902  =limits.ssh_auth_rate= (auth failures per IP per minute), and every
903  account's writes by =limits.write_rate=.
904- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
905journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
906webhook: disk, service, the daemon's own =/healthz= answer, certificate
907expiry, and the age of the newest full backup and database snapshot.
908It exits non-zero on an alert so the unit shows in =systemctl
909--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
910a certificate under 21 days, a full backup over 25 hours old, or a
911database snapshot over 2 hours old.
912
913=GET /healthz= is unauthenticated and cache-free: whether the database
914answers and which commit serves, 503 when it does not.
915- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
916  0750, owned by =gitbay=). The nightly archive plus provider snapshots
917  are the recovery path; for tighter RPO, add continuous replication
918  (litestream) against the same file — it coexists with the WAL.
919
920* Odds and ends
921
922- deleting a fork marks MRs sourced from it =source_gone=; their diffs
923  remain viewable and mergeable because the target repo owns the
924  objects.
925- =refs/merge-requests/*= is server-owned and unpushable by clients;
926  only =admin mr prune= removes one (see Maintenance).
927- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
928  the =events= table, which also feeds webhooks.
929- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.