.gitbay/wiki/Admin.org
802 lines · 40436 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21#+end_src
22
23=deploy/= in the source tree has a cloud-init file, a hardened systemd
24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
2622/80/443 without root.
27
28** The SSH port decision
29
30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
31 22 — move the host's admin sshd to another port. Remotes read
32 =git@host:owner/repo= with no port gymnastics.
33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
34 =AuthorizedKeysCommand=:
35 #+begin_example
36 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
37 AuthorizedKeysCommandUser gitbay
38 #+end_example
39 sshd requires that binary to be root-owned and not group/world
40 writable. Unknown keys fail authentication inside sshd, so system mode
41 requires =registration.mode = "closed"= (check-config enforces this).
42
43* Configuration reference
44
45=/etc/gitbay/config.toml=. =check-config= validates and names every
46contradiction; =--no-host-checks= skips port/path probes.
47=gitbayd admin config show= prints the configuration in effect as TOML,
48every default filled in and =smtp_pass= redacted. A file that fails
49validation still prints, followed by the contradiction.
50
51** [server]
52- =root= (default =/var/lib/gitbay=) — repositories, database, host
53 keys, ACME cache all live here.
54- =site_url= (required) — canonical =https://host=; drives ACME, clone
55 URLs, mail links.
56- =source_repo= (optional, =owner/name=) — the repository this instance
57 develops itself in. Startup warns when the running build's commit is
58 not on that repository's default branch, which is how a binary built
59 from an unmerged branch stops being invisible. Leave it unset unless
60 the instance hosts its own source.
61
62** [ssh]
63- =mode= — =embedded= | =system= (above).
64- =port= (22) — embedded listener port.
65- =host_keys= — list of private key paths; empty generates an ed25519
66 key at =<root>/ssh/host_ed25519=.
67
68** [http]
69- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
70- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
71 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
72 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
73 failing to bind it is a warning, not fatal. Requires an =https://=
74 site_url with a public DNS name.
75- =files=: =cert_file= + =key_file=.
76- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
77
78=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
79front of the daemon. A request from one of them is attributed, for API
80rate limiting, to the last =X-Forwarded-For= hop that is not itself a
81trusted proxy; from anyone else the header is ignored. Empty, the
82default, is right when gitbayd terminates TLS itself.
83
84A reverse proxy in front of gitbay must not buffer responses, or the
85build page's live log arrives only when the build ends;
86=X-Accel-Buffering: no= covers nginx.
87
88** [web]
89- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
90 web routes are never registered; in accounts, browser sessions are
91 minted over SSH (=web login=), and users with write access can create
92 repos, comment, and make simple file edits (which commit unsigned,
93 honestly). =password_auth= is reserved and currently rejected.
94- =title= — the instance's display name in the rail and page titles;
95 empty falls back to the site host. Lower case is the convention for
96 gitbay itself.
97- =privacy_notice= — operator text shown on =/privacy= under the fixed
98 statement.
99
100** [registration]
101- =mode= — =closed= (default) | =invite= | =open=. invite/open require
102 [mail]. See the user guide for the flows.
103
104- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
105 self-registered account still unverified after that long is removed,
106 hourly and at start, audited as =pending.expired=.
107
108- =notify_admin= (false) — mail every instance admin when an account
109 becomes active: an invite redeemed, or an open-mode signup that
110 verified its address. The unverified row an open signup creates is
111 not reported, because anyone can post the form and mailing on that
112 would point a flood at the admins. Recipients are the verified
113 primary addresses of active admins who have activity mail on, the
114 same rule any other notice follows, so an admin with no verified
115 address hears nothing. The notice is queued, so a dead SMTP host
116 shows up in the admin page's Mail table instead of failing the
117 registration. Requires [mail].
118
119** [mail]
120- =smtp_host= (host:port, 587 assumed), =from=, optional =smtp_user= /
121 =smtp_pass=. STARTTLS when offered. Required for invite/open
122 registration and self-service =email add=; in closed mode you may omit
123 it entirely and assert addresses by hand (below).
124
125** [push]
126Push notifications to Apple devices, delivered by gitbayd talking to
127APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
128operator-supplied provider key. Off unless configured.
129
130- =enabled= (false).
131- =key_file= — path to the =.p8= provider key from Apple's developer
132 portal (Certificates, Identifiers & Profiles → Keys). It belongs at
133 =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
134 as. Read and validated at startup: it must parse as a PEM-wrapped
135 PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
136 than fill a queue nobody is watching.
137- =key_id=, =team_id= — the key's id and your Apple developer team id,
138 both from the same portal page.
139- =topic= — the app's bundle identifier. *An APNs key belongs to a
140 bundle ID.* gitbay.org pushes to the App Store build under its own
141 bundle id; a self-hoster who wants push ships their own iOS build
142 under their own bundle id, with its own =.p8= key from their own
143 developer account, and points =topic= at that id. There is no way to
144 push to someone else's build, by design — this is Apple's model, not
145 gitbay's.
146- =environment= — =production= or =sandbox=, naming the APNs host
147 rather than taking a URL, so a typo cannot aim the key at a host that
148 is not Apple's.
149
150All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
151are required when =enabled= is true; validation runs at config load,
152so a misconfigured =[push]= is caught before the daemon serves
153anything. The delivery queue (a device's undelivered and attempted
154pushes) is capped the same way the mail queue is, by =[retention]
155push=.
156
157** [api]
158- =enabled= (false) — the JSON API surface; see [[API]]. Off
159 means no credential-bearing HTTP endpoint exists at all.
160
161** [webhooks]
162- =allow_local= (false) — permit webhook targets on loopback/private
163 addresses. Leave off unless you know why you need it (SSRF).
164
165** [limits]
166- =clone_timeout= (3600s) — cap on =repo import= fetches.
167- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
168- =max_asset_bytes= (512MB) — cap per uploaded release asset.
169- =max_snippet_bytes= (1MB) — cap per snippet file.
170- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
171- =max_repos_per_user= (0, unlimited) — repositories an account may own
172 directly; =repo create=, =fork= and =import= refuse past it.
173 Organizations are not capped.
174- =max_bytes_per_user= (0, unlimited) — disk the account's own
175 repositories may take; a push may be no larger than what is left.
176- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
177
178** [git_daemon]
179- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
180 Serves only public repositories that additionally ran
181 =repo settings git-daemon <repo> on=.
182
183** [mirrors]
184- =pull_interval_minutes= (15) — how often pull mirrors fetch their
185 upstream. Push mirrors sync shortly after each local ref update.
186 Mirror URLs pass the same SSRF rules as webhook targets.
187
188** [go_import]
189Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
190Requests with =?go-get=1= at or under the module path answer with the
191go-import meta tag pointing at the repository's HTTPS clone URL, so
192=go install host/module/cmd/...@latest= resolves. The repository should
193be public (the module path itself confirms it exists).
194
195* Users, email, invites
196
197Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
198backfills is a wrapper that dispatches the registry command of the same
199name as the host: an admin context with no account behind it, so its
200audit rows carry no actor and =source: host=. The same commands run in an
201instance admin's SSH session (=ssh git@<host> admin ...=) and write the
202same rows with the key fingerprint as source. One implementation, two
203credentials.
204
205#+begin_src sh
206gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
207gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
208gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
209#+end_src
210
211"Verified" means SMTP-confirmed or host-admin-asserted; the database
212records which. Verified emails are what make commit signatures
213meaningful — an unverified address never produces a =verified= badge.
214
215* Audit and account control
216
217The audit log is the security feed (events are the product feed): every
218successful mutating command with its argv and source credential (SSH key
219fingerprint or API), registrations, admin actions, force-pushes, and
220auth failures/throttling. Secrets never appear — they travel on stdin,
221never in argv.
222
223#+begin_src sh
224gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
225ssh git@<host> audit ... # the same, from an admin session
226ssh git@<host> admin user list [--state active|pending|disabled|admin]
227ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
228ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
229ssh git@<host> admin user promote <name> # grant instance admin
230ssh git@<host> admin user demote <name> # remove it; the last admin is refused
231gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
232gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
233gitbayd admin user enable <name> # sessions dropped, nothing deleted
234gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
235 # refused (with each blocker named) while
236 # the account owns repos, authored
237 # issues/MRs/comments/reviews, or is an
238 # org's only admin
239#+end_src
240
241=--actor= takes a username, or =-= for rows with no actor: host commands
242and auth failures. =--action= is a prefix, so =cmd repo= catches every
243repository command and =admin= every host or admin-session action.
244=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
245
246=admin user list= pages by username (=--limit=, =--cursor=) and carries
247each account's state and =last_seen=, the newest use of any of its SSH
248keys or API tokens. =admin user show= adds the keys with their last use,
249each address with how it was verified, PGP keys, org roles, the owned
250repository count, API token names, and live browser sessions. Both are
251Both are refused to non-admins, like =audit=, on every surface.
252
253=/admin/users= is the same list in a browser, linked from the admin
254page: the state filter the command takes, keyset paging on its cursor,
255and a row per account with promote, demote, disable and enable, each
256dispatching the command. Demote and disable ask for the username to be
257typed, since both take someone's access away. Creating and deleting an
258account, issuing an invite and asserting an address stay on the command
259line: each takes a key, mints a credential, or cannot be undone. A
260non-admin gets the 404 a missing page would, so the URL confirms
261nothing.
262
263Promotion needs an active account: a pending or disabled one is refused.
264Demotion is refused when it would leave no admin, over SSH, in the
265browser and on the host alike, so the host-local =promote= is the way
266back in when the only admin key is lost.
267
268Instance admin carries no right on anyone's repository: policy does not
269consult it, and a private repository still answers not-found to an
270admin. Moderation goes through explicit overrides that skip the access
271check and write their own audit row:
272
273#+begin_src sh
274ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
275ssh git@<host> admin repo archive|unarchive <owner/name>
276ssh git@<host> admin repo visibility <owner/name> public|private
277ssh git@<host> admin repo delete <owner/name> --yes
278#+end_src
279
280Each lands in the audit log as =admin repo.<action>= naming the
281repository, on top of the =cmd= row every mutating command gets.
282
283=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
284per minute — successful auths never count and clear the slate.
285=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
286push.
287
288=limits.write_rate= (60) bounds *mutating commands per account per
289minute*. It is counted in the dispatcher, so SSH, the JSON API and the
290web spend one budget and a caller cannot refresh it by changing surface;
291=limits.api_rate= stays in front of it, bounding a network source rather
292than an account. A command is one token whatever it writes, so a bundle
293import costs one and only a loop of separate commands spends the budget.
294Read-only commands, the runner protocol (a build streams its log in many
295small writes) and the host CLI are exempt. Refusals exit 4 and say when
296to retry. A negative value turns the limit off; it matters most with
297=registration = "open"=, where every write also queues notification mail
298and webhook deliveries.
299
300* Queues
301
302Every background worker keeps a backlog and a failure state. An instance
303admin reads them all in one place:
304
305#+begin_src sh
306gitbay dashboard --json | jq .queues # webhooks, mail, push, mirrors, builds, deps
307#+end_src
308
309Per worker: pending, retrying (pending with a failed attempt) and
310dead-lettered counts with the oldest pending age, and the retrying or
311failed rows themselves, capped at twenty each. Builds list what is
312running and then what is pending, each since when, so a build no runner
313is scoped to claim is visible here rather than only in its repository;
314mirrors list the ones whose last sync failed;
315dependency checks list the ones whose last check errored. Non-admins get
316no =queues= key at all.
317
318Push rows name the device id, never the token. Watch this one after
319configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
320config validation, which can only check that the =.p8= parses, and then
321every send comes back =403 InvalidProviderToken= and dead-letters on its
322first attempt.
323
324In accounts mode the same read renders at =/admin=, linked from the rail
325for admins. Anyone else gets a 404 there.
326
327A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
328with the address redacted out of the relay's error. The id is the queue
329row: find it in the Mail table on =/admin=, or in =dashboard --json=,
330where the recipient and the unredacted error are. That is deliberate —
331see the Threat-Model page.
332
333* Maintenance
334
335#+begin_src sh
336gitbayd admin stats [--json] # counts, database size, per-repo disk
337ssh git@<host> admin stats [--json] # the same, from an admin session
338gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
339gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
340gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
341#+end_src
342
343A history rewrite leaves the commits it removed reachable through
344=refs/merge-requests/N/head= of the merge requests that landed them, so
345they stay fetchable by anyone who can read the repository. Nothing drops
346a head ref on its own — an open or source-gone MR is merged through it,
347and a merged or closed one keeps its diff readable through it — so the
348cleanup is a command an instance admin runs, naming the MRs:
349
350#+begin_src sh
351ssh git@<host> admin mr prune owner/name 1 2 3 --yes
352#+end_src
353
354It refuses an open or source-gone MR, deletes the named refs, runs
355=git gc --prune=now= on that one repository so the objects go at once
356rather than after git's two-week grace, leaves a system comment on each
357MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
358reviews and head sha; =mr diff= and the MR page say the head is gone.
359Run it when nothing is pushing to that repository: without the grace, a
360push caught between leaving quarantine and writing its ref loses its
361objects. Objects also survive in offsite backups until those are
362rewritten; see "Removing a repository's history from every snapshot".
363
364=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
365weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
366layout the source sent, so a first manual =admin gc= after a bulk
367import is worthwhile.
368
369* Backup and restore
370
371#+begin_src sh
372gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
373gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
374#+end_src
375
376One archive: a consistent SQLite snapshot (taken *before* the
377repositories are read, so the database never references objects the
378archive missed), every repository, and the SSH host keys. Excluded:
379hook socket, regenerated hook scripts, WAL files. Safe to run against a
380live daemon.
381
382=--verify= reads an archive back: the snapshot must pass SQLite's
383integrity check, and every repository the snapshot names must be in the
384archive. A database-only archive is checked for integrity and says so.
385Exit is non-zero on damage or a missing repository.
386
387Restore: extract into an empty directory, point =server.root= at it,
388start gitbayd. Host keys are preserved, so clients keep their
389known_hosts entries; hooks regenerate at startup.
390
391** Schedule and recovery point
392
393Two timers, because the two halves of the data have different exposure.
394
395- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
396- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
397 writes the SQLite snapshot alone, last 48 kept. A few MB against the
398 full archive's hundreds, which is what makes the frequency affordable.
399
400The split follows what a loss would actually cost. Repositories are git,
401so a mirror or any clone is a second copy; the database is the only copy
402of issues, merge requests, comments and review state. So the recovery
403point is about an hour for the data that exists nowhere else, and a day
404for the data that does.
405
406Continuous replication (litestream and similar) was considered and not
407adopted. It would take the database's recovery point to seconds, but the
408repositories would still be on the nightly archive, so a restore could
409produce a database referencing commits the repository backup does not
410have. Consistency between the two halves is worth more here than latency
411on one of them. Revisit if repository replication becomes continuous
412too.
413
414** Offsite copies
415
416bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
417=/var/lib/gitbay-stage= (the staged database copy) to an S3 bucket at
418Scaleway, with a key that can only add snapshots. The key that can
419remove them lives on the operator's machine, in
420=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
421cannot destroy its own history. Forgetting, pruning and rewriting all
422run from there.
423
424*** Removing a repository's history from every snapshot
425
426A history rewrite plus =admin mr prune= takes commits off the server,
427but every snapshot taken before it still holds them, and the retention
428window is the only thing that ages them out. To remove them now,
429rewrite the snapshots without that repository rather than forgetting
430the snapshots: everything else in them stays restorable. The next
431nightly run adds the repository back in its current state.
432
433Repositories are stored under the name they had on disk when each
434snapshot was taken, so a renamed repository needs every name it has
435carried. Check what an older snapshot holds before choosing the paths:
436
437#+begin_src sh
438set -a; . ~/.config/gitbay/offsite.env; set +a
439restic $RESTIC_OPTS snapshots
440restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
441#+end_src
442
443Then dry-run, apply, prune, and confirm nothing matches:
444
445#+begin_src sh
446EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
447restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
448restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
449restic $RESTIC_OPTS prune # drops the data nothing references
450restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
451#+end_src
452
453=--forget= is what makes the originals go; without it the rewritten
454snapshots sit beside them and the data stays referenced. Snapshot IDs
455change; their times do not. Done for krz/keycask (formerly rust-pass)
456on 2026-09-18, across 22 snapshots.
457
458* Upgrades
459
460Replace the binary, restart the unit. Migrations apply automatically and
461are transactional; hook scripts under =<root>/hooks= are rewritten at
462startup to point at the current binary path.
463
464* CI runner
465
466=gitbay-runner= executes builds queued by pushes and merge requests. It
467polls over SSH with a key of scope =runner=, which reaches only the
468runner protocol and read-only git (a runner executes arbitrary
469repository code, so the key it holds must not do more). A runner key
470claims builds only for the repositories it is attached to, by =repo
471runner add= from a repository admin or an instance admin; an admin key
472claims any. Users attach their own runners: see the Users page. For an
473instance runner, run it as a dedicated unprivileged user on a non-admin
474account. =admin user create --key= registers a full-scope key, so the
475runner key is added afterwards through a bootstrap key that is then
476removed, and attached to each repository it should build:
477
478#+begin_src sh
479useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
480sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
481ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
482gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
483ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
484ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
485rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
486gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
487#+end_src
488
489#+begin_src sh
490gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
491#+end_src
492
493=-jobs N= runs N builds at once. Claiming is one transaction that
494selects and updates, and each build works in its own =build-<id>=
495directory, so workers do not collide; idle polls are staggered across
496the interval so N of them do not wake together. The drop-in's weights
497below are per service, not per build, so raising =-jobs= divides them
498rather than multiplying the host's load.
499
500=admin runners= shows which account each runner polls as, and what each
501may claim. A runner key claims builds only for the repositories it is
502attached to: with none attached it claims nothing, and =-repos= may only
503narrow within them. An admin's full-scope key claims any repository —
504that is what =-repos= was for — and still works for the protocol during
505a rotation. A merge request head from a fork is built in the target
506repository as untrusted: the claim carries no secrets, and only a runner
507started with =-untrusted= takes it. Same-repository heads were built by
508their branch push and are not built again.
509
510=make deploy-runner= also installs
511=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
512=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
513the daemon or the backup timers, and =NoNewPrivileges=,
514=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
515and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
516and the runner's home. The e2e suite alone starts sixty
517daemon instances; without the drop-in a deploy's copy over the admin
518sshd stalled. Both deploy targets copy with =rsync --partial=, which
519resumes a stalled transfer.
520
521Under =-isolation podman=, the default and what bay1 runs, each build
522is confined to a container (see Container isolation below). Under
523=-isolation none= steps run directly on the host as the runner's user,
524so treat that machine as executing whatever your users push, and
525install the toolchains your builds need on it.
526
527A runner claims the oldest pending build among the repositories its key
528is attached to — for an admin key, the oldest in the instance. =-repos=
529narrows within that set, which is what makes a runner outside the server
530practical: one on a machine that should build a single project, or that
531holds credentials for one deployment, stays on it.
532
533Oldest-first is across everything the key may claim, so a repository
534with a deep queue holds every other repository the same runner serves;
535bay1 measured a 15-minute average wait on a day of merge request
536stacks from one repository. A runner attached to one repository cannot
537be starved. That is the rule, decided in krz/gitbay#207: a runner
538serving several repositories takes them oldest-first, and an operator
539who wants one repository never to wait on another runs a second
540runner attached to it alone. Nothing caps what an account queues,
541and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
542while the job's last build is pending or running, so a repository
543with no runner holds one row per scheduled job rather than one per
544tick, and a build runs only on a runner its owner attaches, so a busy
545schedule spends the owner's compute. Pushes are bounded by what an
546account can push.
547
548#+begin_src sh
549gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
550 -workdir /var/lib/gitbay-runner/work
551#+end_src
552
553Add =-untrusted= only with =-isolation podman=.
554
555gitbay.org's runner is attached to the forge's own repositories and
556the isolation canary, nothing else, because it shares the host with
557the forge; its unit names no =-repos=, the attachments are the
558boundary. Any other repository builds on a runner its owner attaches.
559
560=-repos= narrows an admin runner; for a runner key the attachments are
561the boundary, held by the server, and =-repos= may only name
562repositories among them. =-untrusted= makes a runner claim merge
563request heads from forks; the bay1 unit sets it because it isolates in
564podman. A runner without it builds trusted commits only.
565
566=gitbay dashboard= and =ssh git@<host> admin runners= list every key
567that has polled as a runner: the account, the key's fingerprint, when it
568last polled, the repositories it may claim — its attachments for a runner
569key, the =-repos= it asked for or =any= for an admin key — and the build
570it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
571that polled by mistake, the key itself untouched. =admin runners= also
572heads the list with the queue: builds
573pending now, and over the last day how many were claimed, how long they
574waited to be claimed (average and worst), and
575how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
576reported is failed by the scheduler's minute tick, whether or not any
577runner is still alive: within about two minutes of its log stream ending
578with no outcome reported — the runner reports right after closing the
579stream, retrying for half a minute if gitbayd is unreachable — or, if no
580stream was ever seen, at the deadline.
581
582Instance admin on the runner account only authorizes the claim/report
583protocol; it grants no repo access. A build that pushes back — a pages
584deploy, an archive publish, an automated MR branch — needs an explicit
585grant on that repo: =repo access grant <owner/name> ci write=. Private
586repos likewise need at least read for the clone.
587
588** Container isolation
589
590Builds run in a rootless podman container, one per job, with the
591workspace bind mounted and nothing else: the clone happens outside with
592the runner's key, so a step cannot read it. =-isolation none= keeps the
593old behaviour — steps on the host as the runner's user — for an instance
594where every repository is trusted. There is no automatic fallback: a
595runner started with =-isolation podman= that cannot find a working
596podman exits rather than running a build unsandboxed.
597
598gitbay's own jobs name =localhost/gitbay-ci:2=, built from
599=deploy/Containerfile.ci= on the runner host. A job's image must carry
600what its steps need: the suite drives real git, git-lfs, gpg and sshd and
601asserts they exist before running, so the stock runner default would fail
602it immediately. Build or rebuild it with:
603
604#+begin_src sh
605ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
606ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
607 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
608#+end_src
609
610The tag is deliberate rather than =:latest=: changing the file means
611bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
612change under it.
613
614=-image= names the image a job runs in when it declares none, and is
615required under =-isolation podman=: there is no built-in default,
616because an image this host does not have would fail every build. A job
617overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
618so a config file cannot turn it into podman arguments.
619
620=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
621-memory 4g=); unset means uncapped. The runner applies them itself: it
622creates a cgroup per build under its own delegated service cgroup,
623writes the limits there, and starts every podman process for the build
624inside it, with podman's cgroup handling off. Podman's own =--memory=
625and =--cpus= never applied under rootless cgroupfs, which is what a
626system service gets (krz/gitbay#188). The unit therefore needs
627=Delegate=yes=, which the drop-in sets; without it the runner refuses
628to start when a limit is set, and logs that builds run unconfined when
629none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
630and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
631swap: the memory cap is what keeps the forge alive when a build
632allocates without bound, and it sits above the e2e suite's 5GB peak
633rather than at a fair share. =OOMPolicy=continue= keeps systemd from
634stopping the runner when a build is OOM-killed.
635
636Each repository gets its own build home under the runner's workdir,
637mounted into its containers as =HOME=. Caches persist between builds of
638one repository and are never read by another's.
639
640*Images are provisioned, never pulled by a build.* The runner passes
641=--pull=never=. Two reasons, and the second is the better one: the
642service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
643holding a setuid file, which is nearly every distribution image; and on
644an instance where anyone can push a =ci.yml=, =image:= would otherwise
645mean "fetch and run anything from the internet". An operator pulls or
646builds what is allowed and a build picks among those. A job naming an
647image the host does not have fails with a message saying so.
648
649#+begin_src sh
650su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
651su - ci-runner -s /bin/sh -c "podman images"
652#+end_src
653
654Prepare a host before pointing an isolating runner at it:
655
656#+begin_src sh
657ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
658make deploy-runner
659#+end_src
660
661The script installs podman, delegates a subuid/subgid range to
662=ci-runner=, checks that user namespaces are enabled rather than
663assuming, enables lingering, and verifies rootless podman actually runs
664as that user. It is idempotent.
665
666The drop-in sets =NoNewPrivileges=no=, without which rootless podman
667cannot call =newuidmap= and the runner refuses to start. That is a
668considered trade, explained in the file and in the Threat-Model; if you
669run with =-isolation none=, set it back to =yes=.
670
671*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
672the build in flight, reports it, and exits; the drop-in's
673=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
674is what makes the signal reach the runner alone — under systemd's default
675the build's container and the log session are signalled with it, and the
676runner drains a build that is already dead. So =make deploy-runner=
677waits for a running build rather than orphaning it, and a build's result
678is retried for half a minute if gitbayd is restarting at that moment. A
679second SIGTERM ends the runner at once, abandoning the build to the
680reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
681the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
682=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
683the runner as a transient user unit under =systemd-run= and stops it
684under both kill modes. That last one needs a systemd user manager, so it
685skips in the container CI runs in; run it on a Linux host with
686=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
687
688*Validate podman mode on a scratch repository before pointing the runner
689at real ones.* Every deploy that switched the whole instance to
690containers and failed took CI down with it. Instead: create a throwaway
691repository the runner account can read (public, or granted read — a
692private one is "not found" to the runner and the build stays pending),
693give it one job that names the CI image, and deploy the runner with
694=-repos= naming only that repository. The production unit, with its real
695hardening, then claims nothing else; other repositories' builds queue
696until =-repos= is switched back, which is a pause, not an outage.
697
698#+begin_src sh
699gitbay repo create cmc/ci-smoke # then push a .gitbay/ci.yml naming the image
700sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
701systemctl daemon-reload && systemctl restart gitbay-runner
702gitbay build log cmc/ci-smoke 1 # green: switch -repos back, redeploy
703#+end_src
704
705*Do not deploy an isolating runner to a host that has not been
706prepared.* The runner is specified to refuse to start without a working
707podman rather than fall back to running builds unsandboxed — a fallback
708that silently drops isolation is worse than a stopped runner, because
709nothing surfaces it. On an unprepared host that refusal stops every
710build on the instance.
711
712The service drop-in carries =Delegate=yes= for rootless cgroup
713management and =ReadWritePaths= for podman's store under
714=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
715make read-only. Those paths are prefixed =-= so they are ignored when
716absent: the drop-in installs on unprepared hosts too, and a unit that
717refused to start would stop every build.
718The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
719names that repository too; a scoped runner claims nothing else.
720=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
721user: rootless storage belongs to that user, and root's prune would not
722see it. An unpruned image store on a 40GB host is a slow outage.
723
724* LFS storage
725
726Objects live content-addressed under =[lfs] root= (default
727=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
728(512MB default). Storage sits behind a small interface — an
729S3-compatible backend is a drop-in with the server proxying, and
730presigned URLs a later optimization. LFS objects do not travel with
731push mirrors (mirrors move git refs only), and gc does not yet collect
732orphaned objects.
733
734* Pages
735
736=[pages] domain = "example.site"= serves public repos' =pages= branches
737on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
738server; ACME issues per-subdomain certificates on demand (only for
739owners that exist). The domain must not be the site host or a parent of
740it — pages content runs its own scripts and must stay off the forge's
741origin.
742
743Users with repo admin claim custom domains with =repo domain add=.
744Claims activate only after a DNS TXT challenge proves control of the
745domain (=repo domain verify=, audit-logged); pending claims serve
746nothing, get no certificates, and expire after 7 days. ACME issues
747certificates only for verified hosts, so stray DNS pointed at the
748server gets nothing.
749
750* Security
751
752The [[Threat-Model]] file is the reference for what the forge
753trusts and refuses to do. Operational checklist:
754
755- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
756 (the module list is deliberately short — review it on each release),
757 and a short fuzz pass over every attacker-facing parser (pkt-line,
758 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
759 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
760 against main rather than per push, because =@latest= scans today's
761 advisory database and an advisory lands without anyone pushing;
762 =build trigger krz/gitbay vuln= runs it on demand.
763- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
764 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
765 configuration needed.
766- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
767 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
768 =LockPersonality=, =MemoryDenyWriteExecute=,
769 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
770 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
771- *OS patches* apply via =unattended-upgrades= (security origins,
772 auto-reboot 04:30 if required).
773- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
774 watched by =fail2ban=; gitbayd's own port 22 is throttled by
775 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
776 account's writes by =limits.write_rate=.
777- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
778journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
779webhook: disk, service, the daemon's own =/healthz= answer, certificate
780expiry, and the age of the newest full backup and database snapshot.
781It exits non-zero on an alert so the unit shows in =systemctl
782--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
783a certificate under 21 days, a full backup over 25 hours old, or a
784database snapshot over 2 hours old.
785
786=GET /healthz= is unauthenticated and cache-free: whether the database
787answers and which commit serves, 503 when it does not.
788- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
789 0750, owned by =gitbay=). The nightly archive plus provider snapshots
790 are the recovery path; for tighter RPO, add continuous replication
791 (litestream) against the same file — it coexists with the WAL.
792
793* Odds and ends
794
795- deleting a fork marks MRs sourced from it =source_gone=; their diffs
796 remain viewable and mergeable because the target repo owns the
797 objects.
798- =refs/merge-requests/*= is server-owned and unpushable by clients;
799 only =admin mr prune= removes one (see Maintenance).
800- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
801 the =events= table, which also feeds webhooks.
802- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.