.gitbay/wiki/Admin.org
1162 lines · 60999 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21gitbayd --config /etc/gitbay/config.toml admin secrets init
22chown gitbay:gitbay /etc/gitbay/secret.key
23#+end_src
24
25=admin secrets init= refuses when the key file already exists, so a
26reinstall on the same host should skip it — =deploy/install.sh= does
27this with a file check before running it.
28
29=deploy/= in the source tree has a cloud-init file, a hardened systemd
30unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
31the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
3222/80/443 without root.
33
34** The SSH port decision
35
36- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
37 22 — move the host's admin sshd to another port. Remotes read
38 =git@host:owner/repo= with no port gymnastics.
39- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
40 =AuthorizedKeysCommand=:
41 #+begin_example
42 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
43 AuthorizedKeysCommandUser gitbay
44 #+end_example
45 sshd requires that binary to be root-owned and not group/world
46 writable. Unknown keys fail authentication inside sshd, so system mode
47 requires =registration.mode = "closed"= (check-config enforces this).
48 Under the forced command the CLI's leading =--term=<cols>[,color]=
49 argument works as is; a client that sets =GITBAY_TERM= instead needs
50 =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
51 handled by the host's sshd, not gitbayd.
52
53* Configuration reference
54
55=/etc/gitbay/config.toml=. =check-config= validates and names every
56contradiction; =--no-host-checks= skips port/path probes.
57=gitbayd admin config show= prints the configuration in effect as TOML,
58every default filled in and =smtp_pass= redacted. A file that fails
59validation still prints, followed by the contradiction.
60
61** [server]
62- =root= (default =/var/lib/gitbay=) — repositories, database, host
63 keys, ACME cache all live here.
64- =site_url= (required) — canonical =https://host=; drives ACME, clone
65 URLs, mail links.
66- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
67 seal CI secrets, webhook secrets, mirror tokens and push device
68 tokens in the database. Must be outside =root=, mode 0600, readable
69 by the daemon's user. See "Secret key" below.
70- =source_repo= (optional, =owner/name=) — the repository this instance
71 develops itself in. Startup warns when the running build's commit is
72 not on that repository's default branch, which is how a binary built
73 from an unmerged branch stops being invisible. Leave it unset unless
74 the instance hosts its own source.
75
76** [ssh]
77- =mode= — =embedded= | =system= (above).
78- =port= (22) — embedded listener port.
79- =host_keys= — list of private key paths; empty generates an ed25519
80 key at =<root>/ssh/host_ed25519=.
81
82** [http]
83- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
84- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
85 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
86 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
87 failing to bind it is a warning, not fatal. Requires an =https://=
88 site_url with a public DNS name.
89- =files=: =cert_file= + =key_file=.
90- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
91- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
92 =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
93 release the binary was built with. =openssl s_client -connect
94 <host>:443 -tls1_1= fails the handshake.
95
96=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
97front of the daemon. A request from one of them is attributed, for API
98rate limiting, to the last =X-Forwarded-For= hop that is not itself a
99trusted proxy; from anyone else the header is ignored. Empty, the
100default, is right when gitbayd terminates TLS itself.
101
102A reverse proxy in front of gitbay must not buffer responses, or the
103build page's live log arrives only when the build ends;
104=X-Accel-Buffering: no= covers nginx.
105
106** [web]
107- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
108 web routes are never registered; in accounts, browser sessions are
109 minted over SSH (=web login=), and users with write access can create
110 repos, comment, and make simple file edits (which commit unsigned,
111 honestly). =password_auth= is reserved and currently rejected.
112- =title= — the instance's display name in the rail and page titles;
113 empty falls back to the site host. Lower case is the convention for
114 gitbay itself.
115- =privacy_notice= — operator text shown on =/privacy= under the fixed
116 statement.
117
118** [registration]
119- =mode= — =closed= (default) | =invite= | =open=. invite/open require
120 [mail]. See the user guide for the flows.
121
122- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
123 self-registered account still unverified after that long is removed,
124 hourly and at start, audited as =pending.expired=.
125
126- =notify_admin= (false) — mail every instance admin when an account
127 becomes active: an invite redeemed, or an open-mode signup that
128 verified its address. The unverified row an open signup creates is
129 not reported, because anyone can post the form and mailing on that
130 would point a flood at the admins. Recipients are the verified
131 primary addresses of active admins who have activity mail on, the
132 same rule any other notice follows, so an admin with no verified
133 address hears nothing. The notice is queued, so a dead SMTP host
134 shows up in the admin page's Mail table instead of failing the
135 registration. Requires [mail].
136
137** [mail]
138- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
139 =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
140 registration and self-service =email add=; in closed mode you may omit
141 it entirely and assert addresses by hand (below).
142- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
143 for relays on 465).
144- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
145 gets no mail: delivery fails and retries, and the admin page's Mail
146 table shows the error. Unset, it is on for every relay except
147 =localhost= and loopback addresses; set =false= to allow plaintext
148 to a remote relay.
149
150** [push]
151Push notifications to Apple devices, delivered by gitbayd talking to
152APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
153operator-supplied provider key. Off unless configured.
154
155- =enabled= (false).
156- =key_file= — path to the =.p8= provider key from Apple's developer
157 portal (Certificates, Identifiers & Profiles → Keys). It belongs at
158 =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
159 as. Read and validated at startup: it must parse as a PEM-wrapped
160 PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
161 than fill a queue nobody is watching.
162- =key_id=, =team_id= — the key's id and your Apple developer team id,
163 both from the same portal page.
164- =topic= — the app's bundle identifier. *An APNs key belongs to a
165 bundle ID.* gitbay.org pushes to the App Store build under its own
166 bundle id; a self-hoster who wants push ships their own iOS build
167 under their own bundle id, with its own =.p8= key from their own
168 developer account, and points =topic= at that id. There is no way to
169 push to someone else's build, by design — this is Apple's model, not
170 gitbay's.
171- =environment= — =production= or =sandbox=, naming the APNs host
172 rather than taking a URL, so a typo cannot aim the key at a host that
173 is not Apple's.
174
175All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
176are required when =enabled= is true; validation runs at config load,
177so a misconfigured =[push]= is caught before the daemon serves
178anything. The delivery queue (a device's undelivered and attempted
179pushes) is capped the same way the mail queue is, by =[retention]
180push=.
181
182** [backup]
183- =age_recipients= (optional) — age public keys (=age1...=). When set,
184 =admin backup= encrypts every archive to them and appends =.age= to
185 its name. Generate the pair off the host with =age-keygen=; only the
186 public key goes here, so the host writes archives it cannot read.
187 The restic copy is unaffected: the offsite job stages its own
188 =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
189 not the archives.
190
191** [api]
192- =enabled= (false) — the JSON API surface; see [[API]]. Off
193 means no credential-bearing HTTP endpoint exists at all.
194
195** [webhooks]
196- =allow_local= (false) — permit webhook, mirror and import targets on
197 loopback, private, shared (100.64.0.0/10), link-local or multicast
198 addresses. Leave off unless you know why you need it (SSRF).
199
200** [limits]
201- =clone_timeout= (3600s) — cap on =repo import= fetches. An import
202 takes http and https URLs only, passes the same address check as a
203 mirror sync and is pinned the same way, so it needs git 2.37 too.
204- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
205- =max_asset_bytes= (512MB) — cap per uploaded release asset.
206- =max_snippet_bytes= (1MB) — cap per snippet file.
207- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
208- =max_repos_per_user= (0, unlimited) — repositories an account may own
209 directly; =repo create=, =fork= and =import= refuse past it.
210 Organizations are not capped.
211- =max_bytes_per_user= (0, unlimited) — disk the account's own
212 repositories may take; a push may be no larger than what is left.
213- =pack_concurrency= (3), =pack_per_principal= (2), =pack_queue= (32),
214 =pack_queue_wait= (="60s"=) — git pack generation (clones, fetches,
215 =git archive --remote=, web archive downloads) over SSH, smart HTTP
216 and git:// shares one
217 budget: this many at once, this many per account (per client
218 address when anonymous: an IPv4 address, or an IPv6 /64), and this
219 many waiting for at most the wait. Anonymous clients together hold
220 at most =pack_concurrency= − 1 slots when it is above 1: an account
221 (SSH key, bearer token or web session) can take the last slot when it
222 is free, and anonymous clients cannot hold it.
223 Past that an SSH client gets "the server is busy…" and exit 1, HTTP
224 gets 503 with =Retry-After: 30=, git:// an =ERR= line; the daemon
225 logs a =pack limit= warning naming the transport and whether the
226 client was signed in, at most once a minute per transport. An SSH client
227 that disconnects while queued leaves the queue; an HTTP or git:// one
228 keeps its place until the wait runs out. A running clone is killed
229 when its client disconnects, or when no write to the client completes
230 for two minutes: a client reading below about 550 B/s, or an HTTP
231 request body that takes over two minutes with nothing written back,
232 is cut. Ref listings (info/refs, protocol v2
233 =ls-refs=), pushes and =repo download= are outside the budget. For the
234 three counts 0 means the default and a negative value turns that
235 bound off. The defaults suit a four-core host; see [[Performance]].
236 With =ssh.mode = "system"= each SSH session is its own process and
237 SSH clones are not counted.
238- =max_pack_bytes= (2 GiB) — the largest pack one push may send,
239 enforced as =receive.maxInputSize= and lowered to what an owner's
240 storage quota has left.
241- =ssh_auth_rate= (10) — SSH authentication failures per client address
242 per minute on the embedded listener; see below.
243
244** [git_daemon]
245- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
246 Serves only public repositories that additionally ran
247 =repo settings git-daemon <repo> on=.
248
249** [mirrors]
250- =pull_interval_minutes= (15) — how often pull mirrors fetch their
251 upstream. Push mirrors sync shortly after each local ref update.
252 Mirror URLs pass the same SSRF rules as webhook targets, when saved
253 and again before every sync; git then connects only to the addresses
254 that were checked (=http.curloptResolve=) and does not follow
255 redirects, so a mirror of a renamed repository fails until its URL
256 is updated. Needs git 2.37 or later on the server; with an older git
257 the worker logs an error at start and syncs no mirror, recording the
258 reason on each. Sync ignores the system and global gitconfig.
259
260** [go_import]
261Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
262Requests with =?go-get=1= at or under the module path answer with the
263go-import meta tag pointing at the repository's HTTPS clone URL, so
264=go install host/module/cmd/...@latest= resolves. The repository should
265be public (the module path itself confirms it exists).
266
267* Users, email, invites
268
269Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
270backfills is a wrapper that dispatches the registry command of the same
271name as the host: an admin context with no account behind it, so its
272audit rows carry no actor and =source: host=. The same commands run in an
273instance admin's SSH session (=ssh git@<host> admin ...=) and write the
274same rows with the key fingerprint as source. One implementation, two
275credentials.
276
277#+begin_src sh
278gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
279gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
280gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
281#+end_src
282
283"Verified" means SMTP-confirmed or host-admin-asserted; the database
284records which. Verified emails are what make commit signatures
285meaningful — an unverified address never produces a =verified= badge.
286
287* Audit and account control
288
289The audit log is the security feed (events are the product feed): every
290successful mutating command with its argv and source credential (SSH key
291fingerprint or API), every refused one (exit 3 or 4) as =refused
292<command>=, refused pushes as =refused git-receive-pack= (the access
293check) or =refused push= (a branch or tag rule, a release anchor or an
294unsigned commit, with the repository and ref names), hook socket
295requests failing the peer or push-token check as =refused hook= (never
296the token), registrations, admin actions, force-pushes, and auth
297failures/throttling. A refusal row
298keeps the flag names and the first positional, not the values.
299Refusals are recorded up to ten a minute per account and 600 a minute
300across the instance; past either, one =refused.throttled= row stands
301for the rest of that minute. The caps bound the embedded listener, the
302web and the API; under =ssh.mode = "system"= each =gitbayd shell=
303connection counts separately. Secrets never appear — they travel on
304stdin, never in argv.
305
306Each row carries the SHA-256 of the row before it. =gitbayd admin audit
307verify= opens the store as other admin commands do, applying pending
308migrations, so run it with the binary that matches the daemon. It
309recomputes the chain and exits 1 naming the first row that was
310edited or whose predecessor was removed. The chain is unkeyed: whoever
311can write the database can recompute every hash after an edit, and
312verify then finds nothing. It catches an edit only when the later
313hashes were not recomputed. Retention removing the oldest
314rows is not a break. Rows written before the chain existed are counted
315and skipped; when every row is such a row, verify warns and exits 1,
316since clearing the hash columns looks the same. After an upgrade that
317clears with the first new audit row.
318
319Removing the newest rows leaves no break, and neither do rows written
320afterwards under the freed ids. The database cannot show either. The
321daemon logs every row it writes to its journal, outside the database
322(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
323id and hash: comparing them with the newest journal line is the check
324for any change, recomputed hashes included. Rows written by
325host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
326"system"=, are not copied to the journal.
327
328#+begin_src sh
329gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
330gitbayd admin audit verify # check the hash chain; exit 1 names the first bad row
331ssh git@<host> audit ... # the same, from an admin session
332ssh git@<host> admin user list [--state active|pending|disabled|admin]
333ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
334ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
335ssh git@<host> admin user promote <name> # grant instance admin
336ssh git@<host> admin user demote <name> # remove it; the last admin is refused
337gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
338gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
339gitbayd admin user enable <name> # sessions dropped, nothing deleted
340gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
341 # refused (with each blocker named) while
342 # the account owns repos, authored
343 # issues/MRs/comments/reviews, or is an
344 # org's only admin
345#+end_src
346
347=--actor= takes a username, or =-= for rows with no actor: host commands
348and auth failures. =--action= is a prefix, so =cmd repo= catches every
349repository command and =admin= every host or admin-session action.
350=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
351
352=admin user list= pages by username (=--limit=, =--cursor=) and carries
353each account's state and =last_seen=, the newest use of any of its SSH
354keys or API tokens. =admin user show= adds the keys with their last use,
355each address with how it was verified, PGP keys, org roles, the owned
356repository count, API token names, and live browser sessions. Both are
357Both are refused to non-admins, like =audit=, on every surface.
358
359=/admin/users= is the same list in a browser, linked from the admin
360page: the state filter the command takes, keyset paging on its cursor,
361and a row per account with promote, demote, disable and enable, each
362dispatching the command. Demote and disable ask for the username to be
363typed, since both take someone's access away. Creating and deleting an
364account, issuing an invite and asserting an address stay on the command
365line: each takes a key, mints a credential, or cannot be undone. A
366non-admin gets the 404 a missing page would, so the URL confirms
367nothing.
368
369Promotion needs an active account: a pending or disabled one is refused.
370Demotion is refused when it would leave no admin, over SSH, in the
371browser and on the host alike, so the host-local =promote= is the way
372back in when the only admin key is lost.
373
374Instance admin carries no right on anyone's repository: policy does not
375consult it, and a private repository still answers not-found to an
376admin. Moderation goes through explicit overrides that skip the access
377check and write their own audit row:
378
379#+begin_src sh
380ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
381ssh git@<host> admin repo archive|unarchive <owner/name>
382ssh git@<host> admin repo visibility <owner/name> public|private
383ssh git@<host> admin repo delete <owner/name> --yes
384#+end_src
385
386Each lands in the audit log as =admin repo.<action>= naming the
387repository, on top of the =cmd= row every mutating command gets.
388
389=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
390per minute — successful auths never count and clear the slate.
391=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
392push.
393
394=limits.write_rate= (60) bounds *mutating commands per account per
395minute*. It is counted in the dispatcher, so SSH, the JSON API and the
396web spend one budget and a caller cannot refresh it by changing surface;
397=limits.api_rate= stays in front of it, bounding a network source rather
398than an account. A command is one token whatever it writes, so a bundle
399import costs one and only a loop of separate commands spends the budget.
400Read-only commands, the runner protocol (a build streams its log in many
401small writes) and the host CLI are exempt. Refusals exit 4 and say when
402to retry. A negative value turns the limit off; it matters most with
403=registration = "open"=, where every write also queues notification mail
404and webhook deliveries.
405
406* Queues
407
408Every background worker keeps a backlog and a failure state. An instance
409admin reads them all in one place:
410
411#+begin_src sh
412gitbay dashboard --json | jq .queues # webhooks, mail, push, mirrors, builds, deps
413#+end_src
414
415Per worker: pending, retrying (pending with a failed attempt) and
416dead-lettered counts with the oldest pending age, and the retrying or
417failed rows themselves, capped at twenty each. Builds list what is
418running and then what is pending, each since when, so a build no runner
419is scoped to claim is visible here rather than only in its repository;
420mirrors list the ones whose last sync failed;
421dependency checks list the ones whose last check errored. Non-admins get
422no =queues= key at all.
423
424Push rows name the device id, never the token. Watch this one after
425configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
426config validation, which can only check that the =.p8= parses, and then
427every send comes back =403 InvalidProviderToken= and dead-letters on its
428first attempt.
429
430In accounts mode the same read renders at =/admin=, linked from the rail
431for admins. Anyone else gets a 404 there.
432
433A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
434with the address redacted out of the relay's error. The id is the queue
435row: find it in the Mail table on =/admin=, or in =dashboard --json=,
436where the recipient and the unredacted error are. That is deliberate —
437see the Threat-Model page.
438
439* Maintenance
440
441#+begin_src sh
442gitbayd admin stats [--json] # counts, database size, per-repo disk
443ssh git@<host> admin stats [--json] # the same, from an admin session
444gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
445gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
446gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
447#+end_src
448
449A history rewrite leaves the commits it removed reachable through
450=refs/merge-requests/N/head= of the merge requests that landed them, so
451they stay fetchable by anyone who can read the repository. Nothing drops
452a head ref on its own — an open or source-gone MR is merged through it,
453and a merged or closed one keeps its diff readable through it — so the
454cleanup is a command an instance admin runs, naming the MRs:
455
456#+begin_src sh
457ssh git@<host> admin mr prune owner/name 1 2 3 --yes
458#+end_src
459
460It refuses an open or source-gone MR, deletes the named refs, runs
461=git gc --prune=now= on that one repository so the objects go at once
462rather than after git's two-week grace, leaves a system comment on each
463MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
464reviews and head sha; =mr diff= and the MR page say the head is gone.
465Run it when nothing is pushing to that repository: without the grace, a
466push caught between leaving quarantine and writing its ref loses its
467objects. Objects also survive in offsite backups until those are
468rewritten; see "Removing a repository's history from every snapshot".
469
470=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
471weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
472layout the source sent, so a first manual =admin gc= after a bulk
473import is worthwhile.
474
475* Backup and restore
476
477#+begin_src sh
478gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
479gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
480#+end_src
481
482One archive: a consistent SQLite snapshot (taken *before* the
483repositories are read, so the database never references objects the
484archive missed), every repository, and the SSH host keys. Excluded:
485hook socket, regenerated hook scripts, WAL files. Safe to run against a
486live daemon. =--out= must be outside =server.root=, or the next full
487backup would carry the archive. Each run removes the snapshot
488directories (=.gitbay-snap-*=) and temporary archives (=.*.tmp-*=) a
489killed run left beside its archive once they are a day old, and prints
490each one it removes.
491
492=--verify= reads an archive back: the snapshot must pass SQLite's
493integrity check, every repository the snapshot names must be in the
494archive, and each must pass =git fsck --connectivity-only=. Every
495release asset the snapshot names must be in its repository with its
496recorded size and sha256, and every LFS object in the archive must
497hash to its name. LFS objects are named by pointer files in git
498history rather than by the database, so a pointer whose object was
499never uploaded is not caught, and with =[lfs] root= outside
500=server.root= the archive holds no LFS objects at all. It extracts the
501repositories to a temporary directory for the checks, so it needs free
502space the size of the repositories. A database-only archive is checked
503for integrity and says so. Exit is non-zero on damage, a missing
504repository, a missing object, or a missing or altered asset or LFS
505object.
506
507A full backup holds =<root>/backup.lock= from its database snapshot to
508its last repository. While it runs, =repo delete=, =repo rename=,
509=repo transfer=, =admin repo delete=, =org rename=, =admin mr prune=
510and =gitbayd admin gc= refuse with "a backup is running"; retry when it
511finishes. Database-only backups take no lock. A pack that git's own
512automatic gc removes during the walk is skipped; each repository's refs
513are archived before its objects, so the refs still find their objects,
514and =--verify= reports it if one does not.
515
516Verifying an archive of unknown origin: run it as an unprivileged
517user. The connectivity check runs git with =--git-dir= on each
518extracted repository, so a directory that is not a repository fails
519rather than git checking an enclosing one. =objects/info/alternates=
520and a =commondir= directly in a =*.git= directory are not extracted,
521so an archive cannot use either to have git read another repository's
522objects or refs on the host. Git still reads each archived
523repository's own =config=.
524
525Archives carry a directory entry for every directory, including an
526empty one, so a bare repository whose refs are all packed restores as
527a repository. Archives written before this release do not: extracting
528one can leave a repository's =refs/= directory missing, which stops
529git from recognizing it as a repository at all. =gitbayd admin backup
530--verify <archive>= names the repositories this affects; the fix is
531=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
532which it opens normally.
533
534With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
535and =--verify= needs the private key:
536
537#+begin_src sh
538gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
539age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
540#+end_src
541
542The identity lives off the host (with the secret key file and the
543restic credentials), so verifying an encrypted archive happens there
544or on a restore host.
545
546Restore: extract into an empty directory, point =server.root= at it,
547restore =server.secret_key_file= from its own copy (mode 0600, owned
548by the account gitbayd runs as), then start gitbayd. No archive carries
549the key file, and without it gitbayd refuses to start. Host keys are
550preserved, so clients keep their known_hosts entries; hooks regenerate
551at startup.
552
553** Schedule and recovery point
554
555Two timers, because the two halves of the data have different exposure.
556
557- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
558- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
559 writes the SQLite snapshot alone, last 48 kept. A few MB against the
560 full archive's hundreds, which is what makes the frequency affordable.
561
562The split follows what a loss would actually cost. Repositories are git,
563so a mirror or any clone is a second copy; the database is the only copy
564of issues, merge requests, comments and review state. So the recovery
565point is about an hour for the data that exists nowhere else, and a day
566for the data that does.
567
568Continuous replication (litestream and similar) was considered and not
569adopted. It would take the database's recovery point to seconds, but the
570repositories would still be on the nightly archive, so a restore could
571produce a database referencing commits the repository backup does not
572have. Consistency between the two halves is worth more here than latency
573on one of them. Revisit if repository replication becomes continuous
574too.
575
576** Offsite copies
577
578bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
579=/var/lib/gitbay-stage= (a database copy and =config.toml=, staged
580there) to an S3 bucket at Scaleway, with a key that can only add
581snapshots. Nothing else under =/etc/gitbay= is in it: not the secret
582key file, not =apns.p8=. The key that can
583remove them lives on the operator's machine, in
584=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
585cannot destroy its own history. Forgetting, pruning and rewriting all
586run from there.
587
588*** Removing a repository's history from every snapshot
589
590A history rewrite plus =admin mr prune= takes commits off the server,
591but every snapshot taken before it still holds them, and the retention
592window is the only thing that ages them out. To remove them now,
593rewrite the snapshots without that repository rather than forgetting
594the snapshots: everything else in them stays restorable. The next
595nightly run adds the repository back in its current state.
596
597Repositories are stored under the name they had on disk when each
598snapshot was taken, so a renamed repository needs every name it has
599carried. Check what an older snapshot holds before choosing the paths:
600
601#+begin_src sh
602set -a; . ~/.config/gitbay/offsite.env; set +a
603restic $RESTIC_OPTS snapshots
604restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
605#+end_src
606
607Then dry-run, apply, prune, and confirm nothing matches:
608
609#+begin_src sh
610EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
611restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
612restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
613restic $RESTIC_OPTS prune # drops the data nothing references
614restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
615#+end_src
616
617=--forget= is what makes the originals go; without it the rewritten
618snapshots sit beside them and the data stays referenced. Snapshot IDs
619change; their times do not. Done for krz/keycask (formerly rust-pass)
620on 2026-09-18, across 22 snapshots.
621
622** Secret key
623
624CI secrets, webhook secrets, mirror tokens and APNs device tokens are
625stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
626each value prefixed with the id of the key that sealed it
627(=gbs1:<id>:=). The key file is not in the database, not under
628=server.root=, and therefore in neither the local archives nor the
629main restic repository. It must be copied off the host separately;
630without it a restored database's secrets cannot be opened, and
631gitbayd refuses to start against them. A separate restic repository
632for it and =apns.p8= is planned (runbook D of the data-at-rest plan)
633and not yet in place, so today the only off-host copy is one the
634operator makes by hand after =init= and after every =rotate=.
635
636#+begin_src sh
637gitbayd admin secrets init # once; deploy/install.sh does it on first install
638gitbayd admin secrets check # open every value, count by key
639gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
640#+end_src
641
642- Missing file: every gitbayd process that opens the database refuses
643 to run and names the path, including =serve= and, in system mode,
644 =authorized-keys=. =migrate= does not need it.
645- Backups: =admin backup= and =admin backup --verify= do not need the
646 key file; a restore does.
647- Wrong key: =serve= stops at startup naming the first row that does
648 not open; =secrets check= does the same without starting anything.
649- Upgrade: the first start after the upgrade seals every value still
650 in clear and logs =sealed secret values=.
651- Rotation: =rotate= adds a key, reseals every value under it in one
652 transaction, then removes the old keys. Run it as root, since it
653 replaces the key file in =/etc/gitbay=; the file keeps its owner.
654 The daemon re-reads the file when it changes, so it needs no
655 restart. Copy the new file off the host afterwards. =init= and
656 =rotate= hold an flock on =<key file>.lock= while they run, so a
657 second run waits for the first.
658- Push devices are looked up by the SHA-256 of their token
659 (=push_devices.token_hash=), since two seals of one token differ.
660
661** Restore drill
662
663A restore onto a clean host, run quarterly (January, April, July,
664October) and after any change to the backup code
665(=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite
666job), and recorded below. The disaster it rehearses is losing bay1, so
667the local archives are gone with it and the sources are the main
668offsite restic repository (repositories, LFS, the staged database,
669=config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys
670repository once runbook D creates it; until then the operator's
671hand-made copy), and the operator's password manager (=offsite.env=,
672the keys repository's password and token once it exists,
673=backup-identity.txt=). The full steps are runbook C of the
674data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
675
676=gitbayd admin restore-drill= does the archive half. It extracts a
677full archive into an empty or absent directory, runs every =--verify=
678check on the extracted copy, and prints what was restored, the newest
679issue, issue comment, merge request comment and push in the restored
680database, and the elapsed time. Exit is non-zero if any check fails.
681
682#+begin_src sh
683gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
684gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill
685#+end_src
686
687What to restore, and from where:
688
689| Item | Source | Path on the drill host |
690|-------------------------------+-------------------------------------------------------------+------------------------------------------------|
691| Database | staged copy in =/var/lib/gitbay-stage= (restic), or an archive | =<root>/gitbay.db= |
692| Repositories | =/var/lib/gitbay/repos= (restic), or an archive | =<root>/repos= |
693| LFS objects | =/var/lib/gitbay/lfs= (restic), or an archive | =<root>/lfs= (or =[lfs] root=) |
694| Release assets | inside each repository (=gitbay-releases/=) | with the repositories |
695| Host keys | =/var/lib/gitbay/ssh= (restic), or an archive | =<root>/ssh= (or =[ssh] host_keys=) |
696| =config.toml= | =/var/lib/gitbay-stage/config.toml= (restic) | =/etc/gitbay/config.toml= |
697| =secret.key=, =apns.p8= | the off-host copy; no archive or main snapshot carries them | =/etc/gitbay/=, mode 0600, owned by =gitbay= |
698
699From the offsite path (restic is not run by any gitbay command):
700
701#+begin_src sh
702restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
703cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
704gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
705gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root
706#+end_src
707
708The second archive is how the restic tree gets the same checks and
709timestamps; =/tmp/drill-root= is discarded afterwards.
710
711What to check, each a column below:
712
713- DB integrity, connectivity, release assets, LFS: =restore-drill=
714 prints =integrity ok=, =connectivity ok on N repositories=, =release
715 assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin
716 stats --json= on the source at the snapshot time.
717- Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets
718 check= opens every value.
719- Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's
720 fingerprint.
721- Config: =gitbayd --config /etc/gitbay/config.toml check-config=.
722- Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over
723 SSH succeed.
724
725Time to service runs from the clean host's first root login to the
726first successful =git clone= over SSH from it. The recovery point is
727the time of the newest restic snapshot restored; record beside it the
728newest issue, comment and push =restore-drill= printed, which show how
729much activity the restore carries.
730
731No drill has been run yet; the procedure above is written but
732unexercised, and #259 stays open until the first row below is
733recorded.
734
735| Date | Host | Snapshot restored (UTC) | Newest issue / comment / push | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
736|------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
737
738* Upgrades
739
740Replace the binary, restart the unit. Migrations apply automatically and
741are transactional; hook scripts under =<root>/hooks= are rewritten at
742startup to point at the current binary path.
743
744* CI runner
745
746=gitbay-runner= executes builds queued by pushes and merge requests. It
747polls over SSH with a key of scope =runner=, which reaches only the
748runner protocol and read-only git (a runner executes arbitrary
749repository code, so the key it holds must not do more). A runner key
750claims builds only for the repositories it is attached to, by =repo
751runner add= from a repository admin or an instance admin; an admin key
752claims any. Users attach their own runners: see the Users page. For an
753instance runner, run it as a dedicated unprivileged user on a non-admin
754account. =admin user create --key= registers a full-scope key, so the
755runner key is added afterwards through a bootstrap key that is then
756removed, and attached to each repository it should build:
757
758#+begin_src sh
759useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
760sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
761ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
762gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
763ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
764ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
765rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
766gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
767#+end_src
768
769#+begin_src sh
770gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
771#+end_src
772
773=-jobs N= runs N builds at once. Claiming is one transaction that
774selects and updates, and each build works in its own =build-<id>=
775directory, so workers do not collide; idle polls are staggered across
776the interval so N of them do not wake together. The drop-in's weights
777below are per service, not per build, so raising =-jobs= divides them
778rather than multiplying the host's load.
779
780=admin runners= shows which account each runner polls as, and what each
781may claim. A runner key claims builds only for the repositories it is
782attached to: with none attached it claims nothing, and =-repos= may only
783narrow within them. An admin's full-scope key claims any repository —
784that is what =-repos= was for — and still works for the protocol during
785a rotation. A merge request head from a fork is built in the target
786repository as untrusted: the claim carries no secrets, and only a runner
787started with =-untrusted= takes it. Same-repository heads were built by
788their branch push and are not built again.
789
790=make deploy-runner= also installs
791=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
792=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
793the daemon or the backup timers, and =NoNewPrivileges=,
794=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
795and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
796and the runner's home. The e2e suite alone starts sixty
797daemon instances; without the drop-in a deploy's copy over the admin
798sshd stalled. Both deploy targets copy with =rsync --partial=, which
799resumes a stalled transfer.
800
801Under =-isolation podman=, the default and what bay1 runs, each build
802is confined to a container (see Container isolation below). Under
803=-isolation none= steps run directly on the host as the runner's user,
804so treat that machine as executing whatever your users push, and
805install the toolchains your builds need on it.
806
807A runner claims the oldest pending build among the repositories its key
808is attached to — for an admin key, the oldest in the instance. =-repos=
809narrows within that set, which is what makes a runner outside the server
810practical: one on a machine that should build a single project, or that
811holds credentials for one deployment, stays on it.
812
813Oldest-first is across everything the key may claim, so a repository
814with a deep queue holds every other repository the same runner serves;
815bay1 measured a 15-minute average wait on a day of merge request
816stacks from one repository. A runner attached to one repository cannot
817be starved. That is the rule, decided in krz/gitbay#207: a runner
818serving several repositories takes them oldest-first, and an operator
819who wants one repository never to wait on another runs a second
820runner attached to it alone. Nothing caps what an account queues,
821and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
822while the job's last build is pending or running, so a repository
823with no runner holds one row per scheduled job rather than one per
824tick, and a build runs only on a runner its owner attaches, so a busy
825schedule spends the owner's compute. Pushes are bounded by what an
826account can push.
827
828#+begin_src sh
829gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
830 -workdir /var/lib/gitbay-runner/work
831#+end_src
832
833Add =-untrusted= only with =-isolation podman=.
834
835gitbay.org's runner is attached to the forge's own repositories and
836the isolation canary, nothing else, because it shares the host with
837the forge; its unit names no =-repos=, the attachments are the
838boundary. Any other repository builds on a runner its owner attaches.
839
840=-repos= narrows an admin runner; for a runner key the attachments are
841the boundary, held by the server, and =-repos= may only name
842repositories among them. =-untrusted= makes a runner claim merge
843request heads from forks; the bay1 unit sets it because it isolates in
844podman. A runner without it builds trusted commits only.
845
846=gitbay dashboard= and =ssh git@<host> admin runners= list every key
847that has polled as a runner: the account, the key's fingerprint, when it
848last polled, the repositories it may claim — its attachments for a runner
849key, the =-repos= it asked for or =any= for an admin key — and the build
850it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
851that polled by mistake, the key itself untouched. =admin runners= also
852heads the list with the queue: builds
853pending now, and over the last day how many were claimed, how long they
854waited to be claimed (average and worst), and
855how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
856reported is failed by the scheduler's minute tick, whether or not any
857runner is still alive: within about two minutes of its log stream ending
858with no outcome reported — the runner reports right after closing the
859stream, retrying for half a minute if gitbayd is unreachable — or, if no
860stream was ever seen, at the deadline.
861
862Instance admin on the runner account only authorizes the claim/report
863protocol; it grants no repo access. A build that pushes back — a pages
864deploy, an archive publish, an automated MR branch — needs an explicit
865grant on that repo: =repo access grant <owner/name> ci write=. Private
866repos likewise need at least read for the clone.
867
868** Container isolation
869
870Builds run in a rootless podman container, one per job, with the
871workspace bind mounted and nothing else: the clone happens outside with
872the runner's key, so a step cannot read it. =-isolation none= keeps the
873old behaviour — steps on the host as the runner's user — for an instance
874where every repository is trusted. There is no automatic fallback: a
875runner started with =-isolation podman= that cannot find a working
876podman exits rather than running a build unsandboxed.
877
878gitbay's own jobs name =localhost/gitbay-ci:2=, built from
879=deploy/Containerfile.ci= on the runner host. A job's image must carry
880what its steps need: the suite drives real git, git-lfs, gpg and sshd and
881asserts they exist before running, so the stock runner default would fail
882it immediately. Build or rebuild it with:
883
884#+begin_src sh
885ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
886ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
887 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
888#+end_src
889
890The tag is deliberate rather than =:latest=: changing the file means
891bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
892change under it.
893
894=-image= names the image a job runs in when it declares none, and is
895required under =-isolation podman=: there is no built-in default,
896because an image this host does not have would fail every build. A job
897overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
898so a config file cannot turn it into podman arguments.
899
900=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
901-memory 4g=); unset means uncapped. The runner applies them itself: it
902creates a cgroup per build under its own delegated service cgroup,
903writes the limits there, and starts every podman process for the build
904inside it, with podman's cgroup handling off. Podman's own =--memory=
905and =--cpus= never applied under rootless cgroupfs, which is what a
906system service gets (krz/gitbay#188). The unit therefore needs
907=Delegate=yes=, which the drop-in sets; without it the runner refuses
908to start when a limit is set, and logs that builds run unconfined when
909none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
910and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
911swap: the memory cap is what keeps the forge alive when a build
912allocates without bound, and it sits above the e2e suite's 5GB peak
913rather than at a fair share. =OOMPolicy=continue= keeps systemd from
914stopping the runner when a build is OOM-killed.
915
916A trusted build's home is its repository's, under
917=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
918=HOME=: caches persist between trusted builds of one repository and are
919never read by another's. An untrusted build — a merge request head from
920a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
921the build ends. Homes under =<workdir>/home= are from runners before
922krz/gitbay#255, which shared them with untrusted builds; nothing reads
923them any more, and they can be deleted.
924
925*Images are provisioned, never pulled by a build.* The runner passes
926=--pull=never=. Two reasons, and the second is the better one: the
927service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
928holding a setuid file, which is nearly every distribution image; and on
929an instance where anyone can push a =ci.yml=, =image:= would otherwise
930mean "fetch and run anything from the internet". An operator pulls or
931builds what is allowed and a build picks among those. A job naming an
932image the host does not have fails with a message saying so.
933
934#+begin_src sh
935su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
936su - ci-runner -s /bin/sh -c "podman images"
937#+end_src
938
939Prepare a host before pointing an isolating runner at it:
940
941#+begin_src sh
942ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
943make deploy-runner
944#+end_src
945
946The script installs podman, delegates a subuid/subgid range to
947=ci-runner=, checks that user namespaces are enabled rather than
948assuming, enables lingering, and verifies rootless podman actually runs
949as that user. It is idempotent.
950
951It also installs nftables. =make deploy-runner= ships
952=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
953with =gitbay-runner-egress.service=, which loads it and which the
954runner's unit requires; it checks the file with =nft -c=, reloads the
955unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
956restarting the runner: =127.0.0.1:22= and the public 22 must answer,
9572222 must not. The table limits the runner's user to =127.0.0.1:22=,
958DNS on loopback, and 22, 80 and 443 on the host's public address; the
959Threat-Model page says why.
960
961That table cannot tell a build from the runner, since both run as
962=ci-runner=. =deploy/gitbay-runner-builds.nft=, shipped as
963=/etc/gitbay-runner/builds.nft=, matches by cgroup instead: the runner
964starts each build under =builds/trusted= or =builds/untrusted= in its
965service cgroup, and the table closes the host's loopback to every
966build, limits an untrusted build to TCP 80 and 443 and DNS, and closes
967private ranges to both (the CI page has the table). nftables resolves a
968cgroup path to its id when the table loads, and the service cgroup is
969new on every start, so the drop-in's =ExecStartPre= creates the two
970cgroups, hands them to =ci-runner= and loads the table before the
971runner starts; a table that does not load stops the start. =make
972deploy-runner= checks the file's syntax before the restart and that the
973table is loaded after it. If =/etc/resolv.conf= names a nameserver in a
974private range, allow it in the file first or builds resolve nothing.
975
976A restart of =nftables.service= flushes both tables;
977=systemctl reload gitbay-runner-egress= restores both.
978
979The egress unit's stop and the rollback below use =nft destroy=,
980which needs nftables 1.0.8 or later; check =nft --version= on a new
981host.
982
983Checking the tables on a running host, during a build (the flood test
984below waits a minute before its first probe, for this):
985
986#+begin_src sh
987nft list table inet gitbay_builds # counters on the reject rules
988for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
989# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>
990#+end_src
991
992A pasta process anywhere else — the runner's own =runner= cgroup, a
993user slice — means the builds table does not see that build's traffic
994and only the first table applies. Run
995=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
996once as a push and once with =--untrusted=. It fails if the runner was
997locked out or if the build log lacks the lines only a working table
998produces. The authoritative proof that pasta's sockets are in the
999build's cgroup is the untrusted run: =169.254.1.2:22= and
1000=github.com:22= refused, all twelve logins refused, and the counters on
1001the =untrusted= chain's rejects rising. The first table lets
1002=ci-runner= reach both of those addresses.
1003
1004To take the builds table out, on the host:
1005
1006#+begin_src sh
1007sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
1008rm /etc/gitbay-runner/builds.nft
1009systemctl daemon-reload
1010nft destroy table inet gitbay_builds
1011#+end_src
1012
1013The runner needs no restart. It still places builds under
1014=builds/trusted= and =builds/untrusted=, but with the file gone neither
1015a runner start nor =systemctl reload gitbay-runner-egress= loads the
1016table again, and builds keep the first table and =--no-map-gw=. A
1017runner that takes =-untrusted= or polls over loopback refuses to start
1018without build cgroups at all, since the table would match nothing. The
1019next =make deploy-runner= installs the file and the lines again.
1020
1021The drop-in sets =NoNewPrivileges=no=, without which rootless podman
1022cannot call =newuidmap= and the runner refuses to start. That is a
1023considered trade, explained in the file and in the Threat-Model; if you
1024run with =-isolation none=, set it back to =yes=.
1025
1026*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
1027the build in flight, reports it, and exits; the drop-in's
1028=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
1029is what makes the signal reach the runner alone — under systemd's default
1030the build's container and the log session are signalled with it, and the
1031runner drains a build that is already dead. So =make deploy-runner=
1032waits for a running build rather than orphaning it, and a build's result
1033is retried for half a minute if gitbayd is restarting at that moment. A
1034second SIGTERM ends the runner at once, abandoning the build to the
1035reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
1036the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
1037=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
1038the runner as a transient user unit under =systemd-run= and stops it
1039under both kill modes. That last one needs a systemd user manager, so it
1040skips in the container CI runs in; run it on a Linux host with
1041=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
1042
1043*Validate podman mode on a scratch repository before pointing the runner
1044at real ones.* Every deploy that switched the whole instance to
1045containers and failed took CI down with it. Instead: create a throwaway
1046repository the runner account can read (public, or granted read — a
1047private one is "not found" to the runner and the build stays pending),
1048give it one job that names the CI image, attach the runner's key to it,
1049and deploy the runner with =-repos= naming only that repository. The
1050production unit, with its real hardening, then claims nothing else;
1051other repositories' builds queue until =-repos= is removed again, which
1052is a pause, not an outage.
1053
1054#+begin_src sh
1055# on a machine with an admin gitbay identity (root on the host has none),
1056# with the runner's public key copied from the host:
1057gitbay repo create cmc/runner-scratch # then push a .gitbay/ci.yml naming the image
1058scp -P 2222 root@<host>:/var/lib/gitbay-runner/.ssh/id_ed25519.pub runner.pub
1059gitbay repo runner add cmc/runner-scratch < runner.pub
1060# on the host:
1061sed -i 's#^ExecStart=/usr/local/bin/gitbay-runner #&-repos cmc/runner-scratch #' /etc/systemd/system/gitbay-runner.service.d/override.conf
1062systemctl daemon-reload && systemctl restart gitbay-runner
1063# back on the admin machine:
1064gitbay build log cmc/runner-scratch 1 # green: remove -repos, redeploy, delete the scratch repository
1065#+end_src
1066
1067*Do not deploy an isolating runner to a host that has not been
1068prepared.* The runner is specified to refuse to start without a working
1069podman rather than fall back to running builds unsandboxed — a fallback
1070that silently drops isolation is worse than a stopped runner, because
1071nothing surfaces it. On an unprepared host that refusal stops every
1072build on the instance.
1073
1074The service drop-in carries =Delegate=yes= for rootless cgroup
1075management and =ReadWritePaths= for podman's store under
1076=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
1077make read-only. Those paths are prefixed =-= so they are ignored when
1078absent: the drop-in installs on unprepared hosts too, and a unit that
1079refused to start would stop every build.
1080=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
1081user: rootless storage belongs to that user, and root's prune would not
1082see it. An unpruned image store on a 40GB host is a slow outage.
1083
1084* LFS storage
1085
1086Objects live content-addressed under =[lfs] root= (default
1087=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
1088(512MB default). Storage sits behind a small interface — an
1089S3-compatible backend is a drop-in with the server proxying, and
1090presigned URLs a later optimization. LFS objects do not travel with
1091push mirrors (mirrors move git refs only), and gc does not yet collect
1092orphaned objects.
1093
1094* Pages
1095
1096=[pages] domain = "example.site"= serves public repos' =pages= branches
1097on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
1098server; ACME issues per-subdomain certificates on demand (only for
1099owners that exist). The domain must not be the site host or a parent of
1100it — pages content runs its own scripts and must stay off the forge's
1101origin.
1102
1103Users with repo admin claim custom domains with =repo domain add=.
1104Claims activate only after a DNS TXT challenge proves control of the
1105domain (=repo domain verify=, audit-logged); pending claims serve
1106nothing, get no certificates, and expire after 7 days. ACME issues
1107certificates only for verified hosts, so stray DNS pointed at the
1108server gets nothing.
1109
1110* Security
1111
1112The [[Threat-Model]] file is the reference for what the forge
1113trusts and refuses to do. Operational checklist:
1114
1115- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
1116 (the module list is deliberately short — review it on each release),
1117 and a short fuzz pass over every attacker-facing parser (pkt-line,
1118 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
1119 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
1120 against main rather than per push, because =@latest= scans today's
1121 advisory database and an advisory lands without anyone pushing;
1122 =build trigger krz/gitbay vuln= runs it on demand.
1123- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
1124 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
1125 configuration needed.
1126- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
1127 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
1128 =LockPersonality=, =MemoryDenyWriteExecute=,
1129 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
1130 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
1131- *OS patches* apply via =unattended-upgrades= (security origins,
1132 auto-reboot 04:30 if required).
1133- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
1134 watched by =fail2ban=; gitbayd's own port 22 is throttled by
1135 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
1136 account's writes by =limits.write_rate=.
1137- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
1138journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
1139webhook: disk, service, the daemon's own =/healthz= answer, certificate
1140expiry, and the age of the newest full backup and database snapshot.
1141It exits non-zero on an alert so the unit shows in =systemctl
1142--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
1143a certificate under 21 days, a full backup over 25 hours old, or a
1144database snapshot over 2 hours old.
1145
1146=GET /healthz= is unauthenticated and cache-free: whether the database
1147answers and which commit serves, 503 when it does not.
1148- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
1149 0750, owned by =gitbay=). The nightly archive plus provider snapshots
1150 are the recovery path; for tighter RPO, add continuous replication
1151 (litestream) against the same file — it coexists with the WAL.
1152
1153* Odds and ends
1154
1155- deleting a fork marks MRs sourced from it =source_gone=; their diffs
1156 remain viewable and mergeable because the target repo owns the
1157 objects.
1158- =refs/merge-requests/*= is server-owned and unpushable by clients;
1159 only =admin mr prune= removes one (see Maintenance).
1160- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
1161 the =events= table, which also feeds webhooks.
1162- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.