.gitbay/wiki/Admin.org
866 lines · 44437 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21#+end_src
22
23=deploy/= in the source tree has a cloud-init file, a hardened systemd
24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
2622/80/443 without root.
27
28** The SSH port decision
29
30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
31 22 — move the host's admin sshd to another port. Remotes read
32 =git@host:owner/repo= with no port gymnastics.
33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
34 =AuthorizedKeysCommand=:
35 #+begin_example
36 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
37 AuthorizedKeysCommandUser gitbay
38 #+end_example
39 sshd requires that binary to be root-owned and not group/world
40 writable. Unknown keys fail authentication inside sshd, so system mode
41 requires =registration.mode = "closed"= (check-config enforces this).
42 Under the forced command the CLI's leading =--term=<cols>[,color]=
43 argument works as is; a client that sets =GITBAY_TERM= instead needs
44 =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
45 handled by the host's sshd, not gitbayd.
46
47* Configuration reference
48
49=/etc/gitbay/config.toml=. =check-config= validates and names every
50contradiction; =--no-host-checks= skips port/path probes.
51=gitbayd admin config show= prints the configuration in effect as TOML,
52every default filled in and =smtp_pass= redacted. A file that fails
53validation still prints, followed by the contradiction.
54
55** [server]
56- =root= (default =/var/lib/gitbay=) — repositories, database, host
57 keys, ACME cache all live here.
58- =site_url= (required) — canonical =https://host=; drives ACME, clone
59 URLs, mail links.
60- =source_repo= (optional, =owner/name=) — the repository this instance
61 develops itself in. Startup warns when the running build's commit is
62 not on that repository's default branch, which is how a binary built
63 from an unmerged branch stops being invisible. Leave it unset unless
64 the instance hosts its own source.
65
66** [ssh]
67- =mode= — =embedded= | =system= (above).
68- =port= (22) — embedded listener port.
69- =host_keys= — list of private key paths; empty generates an ed25519
70 key at =<root>/ssh/host_ed25519=.
71
72** [http]
73- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
74- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
75 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
76 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
77 failing to bind it is a warning, not fatal. Requires an =https://=
78 site_url with a public DNS name.
79- =files=: =cert_file= + =key_file=.
80- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
81- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
82 =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
83 release the binary was built with. =openssl s_client -connect
84 <host>:443 -tls1_1= fails the handshake.
85
86=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
87front of the daemon. A request from one of them is attributed, for API
88rate limiting, to the last =X-Forwarded-For= hop that is not itself a
89trusted proxy; from anyone else the header is ignored. Empty, the
90default, is right when gitbayd terminates TLS itself.
91
92A reverse proxy in front of gitbay must not buffer responses, or the
93build page's live log arrives only when the build ends;
94=X-Accel-Buffering: no= covers nginx.
95
96** [web]
97- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
98 web routes are never registered; in accounts, browser sessions are
99 minted over SSH (=web login=), and users with write access can create
100 repos, comment, and make simple file edits (which commit unsigned,
101 honestly). =password_auth= is reserved and currently rejected.
102- =title= — the instance's display name in the rail and page titles;
103 empty falls back to the site host. Lower case is the convention for
104 gitbay itself.
105- =privacy_notice= — operator text shown on =/privacy= under the fixed
106 statement.
107
108** [registration]
109- =mode= — =closed= (default) | =invite= | =open=. invite/open require
110 [mail]. See the user guide for the flows.
111
112- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
113 self-registered account still unverified after that long is removed,
114 hourly and at start, audited as =pending.expired=.
115
116- =notify_admin= (false) — mail every instance admin when an account
117 becomes active: an invite redeemed, or an open-mode signup that
118 verified its address. The unverified row an open signup creates is
119 not reported, because anyone can post the form and mailing on that
120 would point a flood at the admins. Recipients are the verified
121 primary addresses of active admins who have activity mail on, the
122 same rule any other notice follows, so an admin with no verified
123 address hears nothing. The notice is queued, so a dead SMTP host
124 shows up in the admin page's Mail table instead of failing the
125 registration. Requires [mail].
126
127** [mail]
128- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
129 =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
130 registration and self-service =email add=; in closed mode you may omit
131 it entirely and assert addresses by hand (below).
132- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
133 for relays on 465).
134- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
135 gets no mail: delivery fails and retries, and the admin page's Mail
136 table shows the error. Unset, it is on for every relay except
137 =localhost= and loopback addresses; set =false= to allow plaintext
138 to a remote relay.
139
140** [push]
141Push notifications to Apple devices, delivered by gitbayd talking to
142APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
143operator-supplied provider key. Off unless configured.
144
145- =enabled= (false).
146- =key_file= — path to the =.p8= provider key from Apple's developer
147 portal (Certificates, Identifiers & Profiles → Keys). It belongs at
148 =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
149 as. Read and validated at startup: it must parse as a PEM-wrapped
150 PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
151 than fill a queue nobody is watching.
152- =key_id=, =team_id= — the key's id and your Apple developer team id,
153 both from the same portal page.
154- =topic= — the app's bundle identifier. *An APNs key belongs to a
155 bundle ID.* gitbay.org pushes to the App Store build under its own
156 bundle id; a self-hoster who wants push ships their own iOS build
157 under their own bundle id, with its own =.p8= key from their own
158 developer account, and points =topic= at that id. There is no way to
159 push to someone else's build, by design — this is Apple's model, not
160 gitbay's.
161- =environment= — =production= or =sandbox=, naming the APNs host
162 rather than taking a URL, so a typo cannot aim the key at a host that
163 is not Apple's.
164
165All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
166are required when =enabled= is true; validation runs at config load,
167so a misconfigured =[push]= is caught before the daemon serves
168anything. The delivery queue (a device's undelivered and attempted
169pushes) is capped the same way the mail queue is, by =[retention]
170push=.
171
172** [api]
173- =enabled= (false) — the JSON API surface; see [[API]]. Off
174 means no credential-bearing HTTP endpoint exists at all.
175
176** [webhooks]
177- =allow_local= (false) — permit webhook and mirror targets on
178 loopback, private, shared (100.64.0.0/10), link-local or multicast
179 addresses. Leave off unless you know why you need it (SSRF).
180
181** [limits]
182- =clone_timeout= (3600s) — cap on =repo import= fetches.
183- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
184- =max_asset_bytes= (512MB) — cap per uploaded release asset.
185- =max_snippet_bytes= (1MB) — cap per snippet file.
186- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
187- =max_repos_per_user= (0, unlimited) — repositories an account may own
188 directly; =repo create=, =fork= and =import= refuse past it.
189 Organizations are not capped.
190- =max_bytes_per_user= (0, unlimited) — disk the account's own
191 repositories may take; a push may be no larger than what is left.
192- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
193
194** [git_daemon]
195- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
196 Serves only public repositories that additionally ran
197 =repo settings git-daemon <repo> on=.
198
199** [mirrors]
200- =pull_interval_minutes= (15) — how often pull mirrors fetch their
201 upstream. Push mirrors sync shortly after each local ref update.
202 Mirror URLs pass the same SSRF rules as webhook targets, when saved
203 and again before every sync; git then connects only to the addresses
204 that were checked (=http.curloptResolve=) and does not follow
205 redirects, so a mirror of a renamed repository fails until its URL
206 is updated. Needs git 2.37 or later on the server; with an older git
207 the worker logs an error at start and syncs no mirror, recording the
208 reason on each. Sync ignores the system and global gitconfig.
209
210** [go_import]
211Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
212Requests with =?go-get=1= at or under the module path answer with the
213go-import meta tag pointing at the repository's HTTPS clone URL, so
214=go install host/module/cmd/...@latest= resolves. The repository should
215be public (the module path itself confirms it exists).
216
217* Users, email, invites
218
219Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
220backfills is a wrapper that dispatches the registry command of the same
221name as the host: an admin context with no account behind it, so its
222audit rows carry no actor and =source: host=. The same commands run in an
223instance admin's SSH session (=ssh git@<host> admin ...=) and write the
224same rows with the key fingerprint as source. One implementation, two
225credentials.
226
227#+begin_src sh
228gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
229gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
230gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
231#+end_src
232
233"Verified" means SMTP-confirmed or host-admin-asserted; the database
234records which. Verified emails are what make commit signatures
235meaningful — an unverified address never produces a =verified= badge.
236
237* Audit and account control
238
239The audit log is the security feed (events are the product feed): every
240successful mutating command with its argv and source credential (SSH key
241fingerprint or API), every refused one (exit 3 or 4) as =refused
242<command>=, refused pushes as =refused git-receive-pack=, registrations,
243admin actions, force-pushes, and auth failures/throttling. A refusal row
244keeps the flag names and the first positional, not the values.
245Refusals are recorded up to ten a minute per account and 600 a minute
246across the instance; past either, one =refused.throttled= row stands
247for the rest of that minute. The caps bound the embedded listener, the
248web and the API; under =ssh.mode = "system"= each =gitbayd shell=
249connection counts separately. Secrets never appear — they travel on
250stdin, never in argv.
251
252Each row carries the SHA-256 of the row before it. =gitbayd admin audit
253verify= opens the store as other admin commands do, applying pending
254migrations, so run it with the binary that matches the daemon. It
255recomputes the chain and exits 1 naming the first row that was
256edited or whose predecessor was removed. Retention removing the oldest
257rows is not a break. Rows written before the chain existed are counted
258and skipped; when every row is such a row, verify warns and exits 1,
259since clearing the hash columns looks the same. After an upgrade that
260clears with the first new audit row.
261
262Removing the newest rows leaves no break, and neither do rows written
263afterwards under the freed ids. The database cannot show either. The
264daemon logs every row it writes to its journal, outside the database
265(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
266id and hash: compare them with the newest journal line. Rows written by
267host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
268"system"=, are not copied to the journal.
269
270#+begin_src sh
271gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
272gitbayd admin audit verify # check the hash chain; exit 1 names the first bad row
273ssh git@<host> audit ... # the same, from an admin session
274ssh git@<host> admin user list [--state active|pending|disabled|admin]
275ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
276ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
277ssh git@<host> admin user promote <name> # grant instance admin
278ssh git@<host> admin user demote <name> # remove it; the last admin is refused
279gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
280gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
281gitbayd admin user enable <name> # sessions dropped, nothing deleted
282gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
283 # refused (with each blocker named) while
284 # the account owns repos, authored
285 # issues/MRs/comments/reviews, or is an
286 # org's only admin
287#+end_src
288
289=--actor= takes a username, or =-= for rows with no actor: host commands
290and auth failures. =--action= is a prefix, so =cmd repo= catches every
291repository command and =admin= every host or admin-session action.
292=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
293
294=admin user list= pages by username (=--limit=, =--cursor=) and carries
295each account's state and =last_seen=, the newest use of any of its SSH
296keys or API tokens. =admin user show= adds the keys with their last use,
297each address with how it was verified, PGP keys, org roles, the owned
298repository count, API token names, and live browser sessions. Both are
299Both are refused to non-admins, like =audit=, on every surface.
300
301=/admin/users= is the same list in a browser, linked from the admin
302page: the state filter the command takes, keyset paging on its cursor,
303and a row per account with promote, demote, disable and enable, each
304dispatching the command. Demote and disable ask for the username to be
305typed, since both take someone's access away. Creating and deleting an
306account, issuing an invite and asserting an address stay on the command
307line: each takes a key, mints a credential, or cannot be undone. A
308non-admin gets the 404 a missing page would, so the URL confirms
309nothing.
310
311Promotion needs an active account: a pending or disabled one is refused.
312Demotion is refused when it would leave no admin, over SSH, in the
313browser and on the host alike, so the host-local =promote= is the way
314back in when the only admin key is lost.
315
316Instance admin carries no right on anyone's repository: policy does not
317consult it, and a private repository still answers not-found to an
318admin. Moderation goes through explicit overrides that skip the access
319check and write their own audit row:
320
321#+begin_src sh
322ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
323ssh git@<host> admin repo archive|unarchive <owner/name>
324ssh git@<host> admin repo visibility <owner/name> public|private
325ssh git@<host> admin repo delete <owner/name> --yes
326#+end_src
327
328Each lands in the audit log as =admin repo.<action>= naming the
329repository, on top of the =cmd= row every mutating command gets.
330
331=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
332per minute — successful auths never count and clear the slate.
333=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
334push.
335
336=limits.write_rate= (60) bounds *mutating commands per account per
337minute*. It is counted in the dispatcher, so SSH, the JSON API and the
338web spend one budget and a caller cannot refresh it by changing surface;
339=limits.api_rate= stays in front of it, bounding a network source rather
340than an account. A command is one token whatever it writes, so a bundle
341import costs one and only a loop of separate commands spends the budget.
342Read-only commands, the runner protocol (a build streams its log in many
343small writes) and the host CLI are exempt. Refusals exit 4 and say when
344to retry. A negative value turns the limit off; it matters most with
345=registration = "open"=, where every write also queues notification mail
346and webhook deliveries.
347
348* Queues
349
350Every background worker keeps a backlog and a failure state. An instance
351admin reads them all in one place:
352
353#+begin_src sh
354gitbay dashboard --json | jq .queues # webhooks, mail, push, mirrors, builds, deps
355#+end_src
356
357Per worker: pending, retrying (pending with a failed attempt) and
358dead-lettered counts with the oldest pending age, and the retrying or
359failed rows themselves, capped at twenty each. Builds list what is
360running and then what is pending, each since when, so a build no runner
361is scoped to claim is visible here rather than only in its repository;
362mirrors list the ones whose last sync failed;
363dependency checks list the ones whose last check errored. Non-admins get
364no =queues= key at all.
365
366Push rows name the device id, never the token. Watch this one after
367configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
368config validation, which can only check that the =.p8= parses, and then
369every send comes back =403 InvalidProviderToken= and dead-letters on its
370first attempt.
371
372In accounts mode the same read renders at =/admin=, linked from the rail
373for admins. Anyone else gets a 404 there.
374
375A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
376with the address redacted out of the relay's error. The id is the queue
377row: find it in the Mail table on =/admin=, or in =dashboard --json=,
378where the recipient and the unredacted error are. That is deliberate —
379see the Threat-Model page.
380
381* Maintenance
382
383#+begin_src sh
384gitbayd admin stats [--json] # counts, database size, per-repo disk
385ssh git@<host> admin stats [--json] # the same, from an admin session
386gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
387gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
388gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
389#+end_src
390
391A history rewrite leaves the commits it removed reachable through
392=refs/merge-requests/N/head= of the merge requests that landed them, so
393they stay fetchable by anyone who can read the repository. Nothing drops
394a head ref on its own — an open or source-gone MR is merged through it,
395and a merged or closed one keeps its diff readable through it — so the
396cleanup is a command an instance admin runs, naming the MRs:
397
398#+begin_src sh
399ssh git@<host> admin mr prune owner/name 1 2 3 --yes
400#+end_src
401
402It refuses an open or source-gone MR, deletes the named refs, runs
403=git gc --prune=now= on that one repository so the objects go at once
404rather than after git's two-week grace, leaves a system comment on each
405MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
406reviews and head sha; =mr diff= and the MR page say the head is gone.
407Run it when nothing is pushing to that repository: without the grace, a
408push caught between leaving quarantine and writing its ref loses its
409objects. Objects also survive in offsite backups until those are
410rewritten; see "Removing a repository's history from every snapshot".
411
412=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
413weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
414layout the source sent, so a first manual =admin gc= after a bulk
415import is worthwhile.
416
417* Backup and restore
418
419#+begin_src sh
420gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
421gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
422#+end_src
423
424One archive: a consistent SQLite snapshot (taken *before* the
425repositories are read, so the database never references objects the
426archive missed), every repository, and the SSH host keys. Excluded:
427hook socket, regenerated hook scripts, WAL files. Safe to run against a
428live daemon.
429
430=--verify= reads an archive back: the snapshot must pass SQLite's
431integrity check, and every repository the snapshot names must be in the
432archive. A database-only archive is checked for integrity and says so.
433Exit is non-zero on damage or a missing repository.
434
435Restore: extract into an empty directory, point =server.root= at it,
436start gitbayd. Host keys are preserved, so clients keep their
437known_hosts entries; hooks regenerate at startup.
438
439** Schedule and recovery point
440
441Two timers, because the two halves of the data have different exposure.
442
443- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
444- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
445 writes the SQLite snapshot alone, last 48 kept. A few MB against the
446 full archive's hundreds, which is what makes the frequency affordable.
447
448The split follows what a loss would actually cost. Repositories are git,
449so a mirror or any clone is a second copy; the database is the only copy
450of issues, merge requests, comments and review state. So the recovery
451point is about an hour for the data that exists nowhere else, and a day
452for the data that does.
453
454Continuous replication (litestream and similar) was considered and not
455adopted. It would take the database's recovery point to seconds, but the
456repositories would still be on the nightly archive, so a restore could
457produce a database referencing commits the repository backup does not
458have. Consistency between the two halves is worth more here than latency
459on one of them. Revisit if repository replication becomes continuous
460too.
461
462** Offsite copies
463
464bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
465=/var/lib/gitbay-stage= (the staged database copy) to an S3 bucket at
466Scaleway, with a key that can only add snapshots. The key that can
467remove them lives on the operator's machine, in
468=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
469cannot destroy its own history. Forgetting, pruning and rewriting all
470run from there.
471
472*** Removing a repository's history from every snapshot
473
474A history rewrite plus =admin mr prune= takes commits off the server,
475but every snapshot taken before it still holds them, and the retention
476window is the only thing that ages them out. To remove them now,
477rewrite the snapshots without that repository rather than forgetting
478the snapshots: everything else in them stays restorable. The next
479nightly run adds the repository back in its current state.
480
481Repositories are stored under the name they had on disk when each
482snapshot was taken, so a renamed repository needs every name it has
483carried. Check what an older snapshot holds before choosing the paths:
484
485#+begin_src sh
486set -a; . ~/.config/gitbay/offsite.env; set +a
487restic $RESTIC_OPTS snapshots
488restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
489#+end_src
490
491Then dry-run, apply, prune, and confirm nothing matches:
492
493#+begin_src sh
494EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
495restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
496restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
497restic $RESTIC_OPTS prune # drops the data nothing references
498restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
499#+end_src
500
501=--forget= is what makes the originals go; without it the rewritten
502snapshots sit beside them and the data stays referenced. Snapshot IDs
503change; their times do not. Done for krz/keycask (formerly rust-pass)
504on 2026-09-18, across 22 snapshots.
505
506* Upgrades
507
508Replace the binary, restart the unit. Migrations apply automatically and
509are transactional; hook scripts under =<root>/hooks= are rewritten at
510startup to point at the current binary path.
511
512* CI runner
513
514=gitbay-runner= executes builds queued by pushes and merge requests. It
515polls over SSH with a key of scope =runner=, which reaches only the
516runner protocol and read-only git (a runner executes arbitrary
517repository code, so the key it holds must not do more). A runner key
518claims builds only for the repositories it is attached to, by =repo
519runner add= from a repository admin or an instance admin; an admin key
520claims any. Users attach their own runners: see the Users page. For an
521instance runner, run it as a dedicated unprivileged user on a non-admin
522account. =admin user create --key= registers a full-scope key, so the
523runner key is added afterwards through a bootstrap key that is then
524removed, and attached to each repository it should build:
525
526#+begin_src sh
527useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
528sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
529ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
530gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
531ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
532ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
533rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
534gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
535#+end_src
536
537#+begin_src sh
538gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
539#+end_src
540
541=-jobs N= runs N builds at once. Claiming is one transaction that
542selects and updates, and each build works in its own =build-<id>=
543directory, so workers do not collide; idle polls are staggered across
544the interval so N of them do not wake together. The drop-in's weights
545below are per service, not per build, so raising =-jobs= divides them
546rather than multiplying the host's load.
547
548=admin runners= shows which account each runner polls as, and what each
549may claim. A runner key claims builds only for the repositories it is
550attached to: with none attached it claims nothing, and =-repos= may only
551narrow within them. An admin's full-scope key claims any repository —
552that is what =-repos= was for — and still works for the protocol during
553a rotation. A merge request head from a fork is built in the target
554repository as untrusted: the claim carries no secrets, and only a runner
555started with =-untrusted= takes it. Same-repository heads were built by
556their branch push and are not built again.
557
558=make deploy-runner= also installs
559=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
560=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
561the daemon or the backup timers, and =NoNewPrivileges=,
562=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
563and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
564and the runner's home. The e2e suite alone starts sixty
565daemon instances; without the drop-in a deploy's copy over the admin
566sshd stalled. Both deploy targets copy with =rsync --partial=, which
567resumes a stalled transfer.
568
569Under =-isolation podman=, the default and what bay1 runs, each build
570is confined to a container (see Container isolation below). Under
571=-isolation none= steps run directly on the host as the runner's user,
572so treat that machine as executing whatever your users push, and
573install the toolchains your builds need on it.
574
575A runner claims the oldest pending build among the repositories its key
576is attached to — for an admin key, the oldest in the instance. =-repos=
577narrows within that set, which is what makes a runner outside the server
578practical: one on a machine that should build a single project, or that
579holds credentials for one deployment, stays on it.
580
581Oldest-first is across everything the key may claim, so a repository
582with a deep queue holds every other repository the same runner serves;
583bay1 measured a 15-minute average wait on a day of merge request
584stacks from one repository. A runner attached to one repository cannot
585be starved. That is the rule, decided in krz/gitbay#207: a runner
586serving several repositories takes them oldest-first, and an operator
587who wants one repository never to wait on another runs a second
588runner attached to it alone. Nothing caps what an account queues,
589and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
590while the job's last build is pending or running, so a repository
591with no runner holds one row per scheduled job rather than one per
592tick, and a build runs only on a runner its owner attaches, so a busy
593schedule spends the owner's compute. Pushes are bounded by what an
594account can push.
595
596#+begin_src sh
597gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
598 -workdir /var/lib/gitbay-runner/work
599#+end_src
600
601Add =-untrusted= only with =-isolation podman=.
602
603gitbay.org's runner is attached to the forge's own repositories and
604the isolation canary, nothing else, because it shares the host with
605the forge; its unit names no =-repos=, the attachments are the
606boundary. Any other repository builds on a runner its owner attaches.
607
608=-repos= narrows an admin runner; for a runner key the attachments are
609the boundary, held by the server, and =-repos= may only name
610repositories among them. =-untrusted= makes a runner claim merge
611request heads from forks; the bay1 unit sets it because it isolates in
612podman. A runner without it builds trusted commits only.
613
614=gitbay dashboard= and =ssh git@<host> admin runners= list every key
615that has polled as a runner: the account, the key's fingerprint, when it
616last polled, the repositories it may claim — its attachments for a runner
617key, the =-repos= it asked for or =any= for an admin key — and the build
618it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
619that polled by mistake, the key itself untouched. =admin runners= also
620heads the list with the queue: builds
621pending now, and over the last day how many were claimed, how long they
622waited to be claimed (average and worst), and
623how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
624reported is failed by the scheduler's minute tick, whether or not any
625runner is still alive: within about two minutes of its log stream ending
626with no outcome reported — the runner reports right after closing the
627stream, retrying for half a minute if gitbayd is unreachable — or, if no
628stream was ever seen, at the deadline.
629
630Instance admin on the runner account only authorizes the claim/report
631protocol; it grants no repo access. A build that pushes back — a pages
632deploy, an archive publish, an automated MR branch — needs an explicit
633grant on that repo: =repo access grant <owner/name> ci write=. Private
634repos likewise need at least read for the clone.
635
636** Container isolation
637
638Builds run in a rootless podman container, one per job, with the
639workspace bind mounted and nothing else: the clone happens outside with
640the runner's key, so a step cannot read it. =-isolation none= keeps the
641old behaviour — steps on the host as the runner's user — for an instance
642where every repository is trusted. There is no automatic fallback: a
643runner started with =-isolation podman= that cannot find a working
644podman exits rather than running a build unsandboxed.
645
646gitbay's own jobs name =localhost/gitbay-ci:2=, built from
647=deploy/Containerfile.ci= on the runner host. A job's image must carry
648what its steps need: the suite drives real git, git-lfs, gpg and sshd and
649asserts they exist before running, so the stock runner default would fail
650it immediately. Build or rebuild it with:
651
652#+begin_src sh
653ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
654ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
655 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
656#+end_src
657
658The tag is deliberate rather than =:latest=: changing the file means
659bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
660change under it.
661
662=-image= names the image a job runs in when it declares none, and is
663required under =-isolation podman=: there is no built-in default,
664because an image this host does not have would fail every build. A job
665overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
666so a config file cannot turn it into podman arguments.
667
668=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
669-memory 4g=); unset means uncapped. The runner applies them itself: it
670creates a cgroup per build under its own delegated service cgroup,
671writes the limits there, and starts every podman process for the build
672inside it, with podman's cgroup handling off. Podman's own =--memory=
673and =--cpus= never applied under rootless cgroupfs, which is what a
674system service gets (krz/gitbay#188). The unit therefore needs
675=Delegate=yes=, which the drop-in sets; without it the runner refuses
676to start when a limit is set, and logs that builds run unconfined when
677none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
678and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
679swap: the memory cap is what keeps the forge alive when a build
680allocates without bound, and it sits above the e2e suite's 5GB peak
681rather than at a fair share. =OOMPolicy=continue= keeps systemd from
682stopping the runner when a build is OOM-killed.
683
684A trusted build's home is its repository's, under
685=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
686=HOME=: caches persist between trusted builds of one repository and are
687never read by another's. An untrusted build — a merge request head from
688a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
689the build ends. Homes under =<workdir>/home= are from runners before
690krz/gitbay#255, which shared them with untrusted builds; nothing reads
691them any more, and they can be deleted.
692
693*Images are provisioned, never pulled by a build.* The runner passes
694=--pull=never=. Two reasons, and the second is the better one: the
695service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
696holding a setuid file, which is nearly every distribution image; and on
697an instance where anyone can push a =ci.yml=, =image:= would otherwise
698mean "fetch and run anything from the internet". An operator pulls or
699builds what is allowed and a build picks among those. A job naming an
700image the host does not have fails with a message saying so.
701
702#+begin_src sh
703su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
704su - ci-runner -s /bin/sh -c "podman images"
705#+end_src
706
707Prepare a host before pointing an isolating runner at it:
708
709#+begin_src sh
710ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
711make deploy-runner
712#+end_src
713
714The script installs podman, delegates a subuid/subgid range to
715=ci-runner=, checks that user namespaces are enabled rather than
716assuming, enables lingering, and verifies rootless podman actually runs
717as that user. It is idempotent.
718
719It also installs nftables. =make deploy-runner= ships
720=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
721with =gitbay-runner-egress.service=, which loads it and which the
722runner's unit requires; it checks the file with =nft -c=, reloads the
723unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
724restarting the runner: =127.0.0.1:22= and the public 22 must answer,
7252222 must not. The table limits the runner's user to =127.0.0.1:22=,
726DNS on loopback, and 22, 80 and 443 on the host's public address; the
727Threat-Model page says why. A restart of =nftables.service= flushes it;
728=systemctl reload gitbay-runner-egress= restores it.
729
730The drop-in sets =NoNewPrivileges=no=, without which rootless podman
731cannot call =newuidmap= and the runner refuses to start. That is a
732considered trade, explained in the file and in the Threat-Model; if you
733run with =-isolation none=, set it back to =yes=.
734
735*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
736the build in flight, reports it, and exits; the drop-in's
737=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
738is what makes the signal reach the runner alone — under systemd's default
739the build's container and the log session are signalled with it, and the
740runner drains a build that is already dead. So =make deploy-runner=
741waits for a running build rather than orphaning it, and a build's result
742is retried for half a minute if gitbayd is restarting at that moment. A
743second SIGTERM ends the runner at once, abandoning the build to the
744reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
745the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
746=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
747the runner as a transient user unit under =systemd-run= and stops it
748under both kill modes. That last one needs a systemd user manager, so it
749skips in the container CI runs in; run it on a Linux host with
750=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
751
752*Validate podman mode on a scratch repository before pointing the runner
753at real ones.* Every deploy that switched the whole instance to
754containers and failed took CI down with it. Instead: create a throwaway
755repository the runner account can read (public, or granted read — a
756private one is "not found" to the runner and the build stays pending),
757give it one job that names the CI image, and deploy the runner with
758=-repos= naming only that repository. The production unit, with its real
759hardening, then claims nothing else; other repositories' builds queue
760until =-repos= is switched back, which is a pause, not an outage.
761
762#+begin_src sh
763gitbay repo create cmc/ci-smoke # then push a .gitbay/ci.yml naming the image
764sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
765systemctl daemon-reload && systemctl restart gitbay-runner
766gitbay build log cmc/ci-smoke 1 # green: switch -repos back, redeploy
767#+end_src
768
769*Do not deploy an isolating runner to a host that has not been
770prepared.* The runner is specified to refuse to start without a working
771podman rather than fall back to running builds unsandboxed — a fallback
772that silently drops isolation is worse than a stopped runner, because
773nothing surfaces it. On an unprepared host that refusal stops every
774build on the instance.
775
776The service drop-in carries =Delegate=yes= for rootless cgroup
777management and =ReadWritePaths= for podman's store under
778=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
779make read-only. Those paths are prefixed =-= so they are ignored when
780absent: the drop-in installs on unprepared hosts too, and a unit that
781refused to start would stop every build.
782The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
783names that repository too; a scoped runner claims nothing else.
784=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
785user: rootless storage belongs to that user, and root's prune would not
786see it. An unpruned image store on a 40GB host is a slow outage.
787
788* LFS storage
789
790Objects live content-addressed under =[lfs] root= (default
791=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
792(512MB default). Storage sits behind a small interface — an
793S3-compatible backend is a drop-in with the server proxying, and
794presigned URLs a later optimization. LFS objects do not travel with
795push mirrors (mirrors move git refs only), and gc does not yet collect
796orphaned objects.
797
798* Pages
799
800=[pages] domain = "example.site"= serves public repos' =pages= branches
801on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
802server; ACME issues per-subdomain certificates on demand (only for
803owners that exist). The domain must not be the site host or a parent of
804it — pages content runs its own scripts and must stay off the forge's
805origin.
806
807Users with repo admin claim custom domains with =repo domain add=.
808Claims activate only after a DNS TXT challenge proves control of the
809domain (=repo domain verify=, audit-logged); pending claims serve
810nothing, get no certificates, and expire after 7 days. ACME issues
811certificates only for verified hosts, so stray DNS pointed at the
812server gets nothing.
813
814* Security
815
816The [[Threat-Model]] file is the reference for what the forge
817trusts and refuses to do. Operational checklist:
818
819- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
820 (the module list is deliberately short — review it on each release),
821 and a short fuzz pass over every attacker-facing parser (pkt-line,
822 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
823 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
824 against main rather than per push, because =@latest= scans today's
825 advisory database and an advisory lands without anyone pushing;
826 =build trigger krz/gitbay vuln= runs it on demand.
827- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
828 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
829 configuration needed.
830- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
831 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
832 =LockPersonality=, =MemoryDenyWriteExecute=,
833 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
834 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
835- *OS patches* apply via =unattended-upgrades= (security origins,
836 auto-reboot 04:30 if required).
837- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
838 watched by =fail2ban=; gitbayd's own port 22 is throttled by
839 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
840 account's writes by =limits.write_rate=.
841- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
842journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
843webhook: disk, service, the daemon's own =/healthz= answer, certificate
844expiry, and the age of the newest full backup and database snapshot.
845It exits non-zero on an alert so the unit shows in =systemctl
846--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
847a certificate under 21 days, a full backup over 25 hours old, or a
848database snapshot over 2 hours old.
849
850=GET /healthz= is unauthenticated and cache-free: whether the database
851answers and which commit serves, 503 when it does not.
852- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
853 0750, owned by =gitbay=). The nightly archive plus provider snapshots
854 are the recovery path; for tighter RPO, add continuous replication
855 (litestream) against the same file — it coexists with the WAL.
856
857* Odds and ends
858
859- deleting a fork marks MRs sourced from it =source_gone=; their diffs
860 remain viewable and mergeable because the target repo owns the
861 objects.
862- =refs/merge-requests/*= is server-owned and unpushable by clients;
863 only =admin mr prune= removes one (see Maintenance).
864- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
865 the =events= table, which also feeds webhooks.
866- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.