.gitbay/wiki/Admin.org
739 lines · 37081 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21#+end_src
22
23=deploy/= in the source tree has a cloud-init file, a hardened systemd
24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
2622/80/443 without root.
27
28** The SSH port decision
29
30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
31 22 — move the host's admin sshd to another port. Remotes read
32 =git@host:owner/repo= with no port gymnastics.
33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
34 =AuthorizedKeysCommand=:
35 #+begin_example
36 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
37 AuthorizedKeysCommandUser gitbay
38 #+end_example
39 sshd requires that binary to be root-owned and not group/world
40 writable. Unknown keys fail authentication inside sshd, so system mode
41 requires =registration.mode = "closed"= (check-config enforces this).
42
43* Configuration reference
44
45=/etc/gitbay/config.toml=. =check-config= validates and names every
46contradiction; =--no-host-checks= skips port/path probes.
47=gitbayd admin config show= prints the configuration in effect as TOML,
48every default filled in and =smtp_pass= redacted. A file that fails
49validation still prints, followed by the contradiction.
50
51** [server]
52- =root= (default =/var/lib/gitbay=) — repositories, database, host
53 keys, ACME cache all live here.
54- =site_url= (required) — canonical =https://host=; drives ACME, clone
55 URLs, mail links.
56- =source_repo= (optional, =owner/name=) — the repository this instance
57 develops itself in. Startup warns when the running build's commit is
58 not on that repository's default branch, which is how a binary built
59 from an unmerged branch stops being invisible. Leave it unset unless
60 the instance hosts its own source.
61
62** [ssh]
63- =mode= — =embedded= | =system= (above).
64- =port= (22) — embedded listener port.
65- =host_keys= — list of private key paths; empty generates an ed25519
66 key at =<root>/ssh/host_ed25519=.
67
68** [http]
69- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
70- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
71 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
72 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
73 failing to bind it is a warning, not fatal. Requires an =https://=
74 site_url with a public DNS name.
75- =files=: =cert_file= + =key_file=.
76- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
77
78=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
79front of the daemon. A request from one of them is attributed, for API
80rate limiting, to the last =X-Forwarded-For= hop that is not itself a
81trusted proxy; from anyone else the header is ignored. Empty, the
82default, is right when gitbayd terminates TLS itself.
83
84** [web]
85- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
86 web routes are never registered; in accounts, browser sessions are
87 minted over SSH (=web login=), and users with write access can create
88 repos, comment, and make simple file edits (which commit unsigned,
89 honestly). =password_auth= is reserved and currently rejected.
90- =title= — the instance's display name in the rail and page titles;
91 empty falls back to the site host. Lower case is the convention for
92 gitbay itself.
93- =privacy_notice= — operator text shown on =/privacy= under the fixed
94 statement.
95
96** [registration]
97- =mode= — =closed= (default) | =invite= | =open=. invite/open require
98 [mail]. See the user guide for the flows.
99
100- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
101 self-registered account still unverified after that long is removed,
102 hourly and at start, audited as =pending.expired=.
103
104** [mail]
105- =smtp_host= (host:port, 587 assumed), =from=, optional =smtp_user= /
106 =smtp_pass=. STARTTLS when offered. Required for invite/open
107 registration and self-service =email add=; in closed mode you may omit
108 it entirely and assert addresses by hand (below).
109
110** [api]
111- =enabled= (false) — the JSON API surface; see [[API]]. Off
112 means no credential-bearing HTTP endpoint exists at all.
113
114** [webhooks]
115- =allow_local= (false) — permit webhook targets on loopback/private
116 addresses. Leave off unless you know why you need it (SSRF).
117
118** [limits]
119- =clone_timeout= (3600s) — cap on =repo import= fetches.
120- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
121- =max_asset_bytes= (512MB) — cap per uploaded release asset.
122- =max_snippet_bytes= (1MB) — cap per snippet file.
123- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
124- =max_repos_per_user= (0, unlimited) — repositories an account may own
125 directly; =repo create=, =fork= and =import= refuse past it.
126 Organizations are not capped.
127- =max_bytes_per_user= (0, unlimited) — disk the account's own
128 repositories may take; a push may be no larger than what is left.
129- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
130
131** [git_daemon]
132- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
133 Serves only public repositories that additionally ran
134 =repo settings git-daemon <repo> on=.
135
136** [mirrors]
137- =pull_interval_minutes= (15) — how often pull mirrors fetch their
138 upstream. Push mirrors sync shortly after each local ref update.
139 Mirror URLs pass the same SSRF rules as webhook targets.
140
141** [go_import]
142Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
143Requests with =?go-get=1= at or under the module path answer with the
144go-import meta tag pointing at the repository's HTTPS clone URL, so
145=go install host/module/cmd/...@latest= resolves. The repository should
146be public (the module path itself confirms it exists).
147
148* Users, email, invites
149
150Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
151backfills is a wrapper that dispatches the registry command of the same
152name as the host: an admin context with no account behind it, so its
153audit rows carry no actor and =source: host=. The same commands run in an
154instance admin's SSH session (=ssh git@<host> admin ...=) and write the
155same rows with the key fingerprint as source. One implementation, two
156credentials.
157
158#+begin_src sh
159gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
160gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
161gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
162#+end_src
163
164"Verified" means SMTP-confirmed or host-admin-asserted; the database
165records which. Verified emails are what make commit signatures
166meaningful — an unverified address never produces a =verified= badge.
167
168* Audit and account control
169
170The audit log is the security feed (events are the product feed): every
171successful mutating command with its argv and source credential (SSH key
172fingerprint or API), registrations, admin actions, force-pushes, and
173auth failures/throttling. Secrets never appear — they travel on stdin,
174never in argv.
175
176#+begin_src sh
177gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
178ssh git@<host> audit ... # the same, from an admin session (SSH only)
179ssh git@<host> admin user list [--state active|pending|disabled|admin]
180ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
181ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
182ssh git@<host> admin user promote <name> # grant instance admin
183ssh git@<host> admin user demote <name> # remove it; the last admin is refused
184gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
185gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
186gitbayd admin user enable <name> # sessions dropped, nothing deleted
187gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
188 # refused (with each blocker named) while
189 # the account owns repos, authored
190 # issues/MRs/comments/reviews, or is an
191 # org's only admin
192#+end_src
193
194=--actor= takes a username, or =-= for rows with no actor: host commands
195and auth failures. =--action= is a prefix, so =cmd repo= catches every
196repository command and =admin= every host or admin-session action.
197=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
198
199=admin user list= pages by username (=--limit=, =--cursor=) and carries
200each account's state and =last_seen=, the newest use of any of its SSH
201keys or API tokens. =admin user show= adds the keys with their last use,
202each address with how it was verified, PGP keys, org roles, the owned
203repository count, API token names, and live browser sessions. Both are
204SSH-only and refused to non-admins, like =audit=.
205
206Promotion needs an active account: a pending or disabled one is refused.
207Demotion is refused when it would leave no admin, over SSH and on the
208host alike, so the host-local =promote= is the way back in when the only
209admin key is lost.
210
211Instance admin carries no right on anyone's repository: policy does not
212consult it, and a private repository still answers not-found to an
213admin. Moderation goes through explicit overrides that skip the access
214check and write their own audit row:
215
216#+begin_src sh
217ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
218ssh git@<host> admin repo archive|unarchive <owner/name>
219ssh git@<host> admin repo visibility <owner/name> public|private
220ssh git@<host> admin repo delete <owner/name> --yes
221#+end_src
222
223Each lands in the audit log as =admin repo.<action>= naming the
224repository, on top of the =cmd= row every mutating command gets.
225
226=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
227per minute — successful auths never count and clear the slate.
228=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
229push.
230
231=limits.write_rate= (60) bounds *mutating commands per account per
232minute*. It is counted in the dispatcher, so SSH, the JSON API and the
233web spend one budget and a caller cannot refresh it by changing surface;
234=limits.api_rate= stays in front of it, bounding a network source rather
235than an account. A command is one token whatever it writes, so a bundle
236import costs one and only a loop of separate commands spends the budget.
237Read-only commands, the runner protocol (a build streams its log in many
238small writes) and the host CLI are exempt. Refusals exit 4 and say when
239to retry. A negative value turns the limit off; it matters most with
240=registration = "open"=, where every write also queues notification mail
241and webhook deliveries.
242
243* Queues
244
245Every background worker keeps a backlog and a failure state. An instance
246admin reads them all in one place:
247
248#+begin_src sh
249gitbay dashboard --json | jq .queues # webhooks, mail, mirrors, builds, deps
250#+end_src
251
252Per worker: pending, retrying (pending with a failed attempt) and
253dead-lettered counts with the oldest pending age, and the retrying or
254failed rows themselves, capped at twenty each. Builds list what is
255running and then what is pending, each since when, so a build no runner
256is scoped to claim is visible here rather than only in its repository;
257mirrors list the ones whose last sync failed;
258dependency checks list the ones whose last check errored. Non-admins get
259no =queues= key at all.
260
261In accounts mode the same read renders at =/admin=, linked from the rail
262for admins. Anyone else gets a 404 there.
263
264A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
265with the address redacted out of the relay's error. The id is the queue
266row: find it in the Mail table on =/admin=, or in =dashboard --json=,
267where the recipient and the unredacted error are. That is deliberate —
268see the Threat-Model page.
269
270* Maintenance
271
272#+begin_src sh
273gitbayd admin stats [--json] # counts, database size, per-repo disk
274ssh git@<host> admin stats [--json] # the same, from an admin session
275gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
276gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
277gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
278#+end_src
279
280A history rewrite leaves the commits it removed reachable through
281=refs/merge-requests/N/head= of the merge requests that landed them, so
282they stay fetchable by anyone who can read the repository. Nothing drops
283a head ref on its own — an open or source-gone MR is merged through it,
284and a merged or closed one keeps its diff readable through it — so the
285cleanup is a command an instance admin runs, naming the MRs:
286
287#+begin_src sh
288ssh git@<host> admin mr prune owner/name 1 2 3 --yes
289#+end_src
290
291It refuses an open or source-gone MR, deletes the named refs, runs
292=git gc --prune=now= on that one repository so the objects go at once
293rather than after git's two-week grace, leaves a system comment on each
294MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
295reviews and head sha; =mr diff= and the MR page say the head is gone.
296Run it when nothing is pushing to that repository: without the grace, a
297push caught between leaving quarantine and writing its ref loses its
298objects. Objects also survive in offsite backups until those are
299rewritten; see "Removing a repository's history from every snapshot".
300
301=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
302weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
303layout the source sent, so a first manual =admin gc= after a bulk
304import is worthwhile.
305
306* Backup and restore
307
308#+begin_src sh
309gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
310gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
311#+end_src
312
313One archive: a consistent SQLite snapshot (taken *before* the
314repositories are read, so the database never references objects the
315archive missed), every repository, and the SSH host keys. Excluded:
316hook socket, regenerated hook scripts, WAL files. Safe to run against a
317live daemon.
318
319=--verify= reads an archive back: the snapshot must pass SQLite's
320integrity check, and every repository the snapshot names must be in the
321archive. A database-only archive is checked for integrity and says so.
322Exit is non-zero on damage or a missing repository.
323
324Restore: extract into an empty directory, point =server.root= at it,
325start gitbayd. Host keys are preserved, so clients keep their
326known_hosts entries; hooks regenerate at startup.
327
328** Schedule and recovery point
329
330Two timers, because the two halves of the data have different exposure.
331
332- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
333- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
334 writes the SQLite snapshot alone, last 48 kept. A few MB against the
335 full archive's hundreds, which is what makes the frequency affordable.
336
337The split follows what a loss would actually cost. Repositories are git,
338so a mirror or any clone is a second copy; the database is the only copy
339of issues, merge requests, comments and review state. So the recovery
340point is about an hour for the data that exists nowhere else, and a day
341for the data that does.
342
343Continuous replication (litestream and similar) was considered and not
344adopted. It would take the database's recovery point to seconds, but the
345repositories would still be on the nightly archive, so a restore could
346produce a database referencing commits the repository backup does not
347have. Consistency between the two halves is worth more here than latency
348on one of them. Revisit if repository replication becomes continuous
349too.
350
351** Offsite copies
352
353bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
354=/var/lib/gitbay-stage= (the staged database copy) to an S3 bucket at
355Scaleway, with a key that can only add snapshots. The key that can
356remove them lives on the operator's machine, in
357=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
358cannot destroy its own history. Forgetting, pruning and rewriting all
359run from there.
360
361*** Removing a repository's history from every snapshot
362
363A history rewrite plus =admin mr prune= takes commits off the server,
364but every snapshot taken before it still holds them, and the retention
365window is the only thing that ages them out. To remove them now,
366rewrite the snapshots without that repository rather than forgetting
367the snapshots: everything else in them stays restorable. The next
368nightly run adds the repository back in its current state.
369
370Repositories are stored under the name they had on disk when each
371snapshot was taken, so a renamed repository needs every name it has
372carried. Check what an older snapshot holds before choosing the paths:
373
374#+begin_src sh
375set -a; . ~/.config/gitbay/offsite.env; set +a
376restic $RESTIC_OPTS snapshots
377restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
378#+end_src
379
380Then dry-run, apply, prune, and confirm nothing matches:
381
382#+begin_src sh
383EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
384restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
385restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
386restic $RESTIC_OPTS prune # drops the data nothing references
387restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
388#+end_src
389
390=--forget= is what makes the originals go; without it the rewritten
391snapshots sit beside them and the data stays referenced. Snapshot IDs
392change; their times do not. Done for krz/keycask (formerly rust-pass)
393on 2026-09-18, across 22 snapshots.
394
395* Upgrades
396
397Replace the binary, restart the unit. Migrations apply automatically and
398are transactional; hook scripts under =<root>/hooks= are rewritten at
399startup to point at the current binary path.
400
401* CI runner
402
403=gitbay-runner= executes builds queued by pushes and merge requests. It
404polls over SSH with a key of scope =runner=, which reaches only the
405runner protocol and read-only git (a runner executes arbitrary
406repository code, so the key it holds must not do more). A runner key
407claims builds only for the repositories it is attached to, by =repo
408runner add= from a repository admin or an instance admin; an admin key
409claims any. Users attach their own runners: see the Users page. For an
410instance runner, run it as a dedicated unprivileged user on a non-admin
411account. =admin user create --key= registers a full-scope key, so the
412runner key is added afterwards through a bootstrap key that is then
413removed, and attached to each repository it should build:
414
415#+begin_src sh
416useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
417sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
418ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
419gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
420ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
421ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
422rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
423gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
424#+end_src
425
426#+begin_src sh
427gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
428#+end_src
429
430=-jobs N= runs N builds at once. Claiming is one transaction that
431selects and updates, and each build works in its own =build-<id>=
432directory, so workers do not collide; idle polls are staggered across
433the interval so N of them do not wake together. The drop-in's weights
434below are per service, not per build, so raising =-jobs= divides them
435rather than multiplying the host's load.
436
437=admin runners= shows which account each runner polls as, and what each
438may claim. A runner key claims builds only for the repositories it is
439attached to: with none attached it claims nothing, and =-repos= may only
440narrow within them. An admin's full-scope key claims any repository —
441that is what =-repos= was for — and still works for the protocol during
442a rotation. A merge request head from a fork is built in the target
443repository as untrusted: the claim carries no secrets, and only a runner
444started with =-untrusted= takes it. Same-repository heads were built by
445their branch push and are not built again.
446
447=make deploy-runner= also installs
448=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
449=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
450the daemon or the backup timers, and =NoNewPrivileges=,
451=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
452and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
453and the runner's home. The e2e suite alone starts sixty
454daemon instances; without the drop-in a deploy's copy over the admin
455sshd stalled. Both deploy targets copy with =rsync --partial=, which
456resumes a stalled transfer.
457
458Under =-isolation podman=, the default and what bay1 runs, each build
459is confined to a container (see Container isolation below). Under
460=-isolation none= steps run directly on the host as the runner's user,
461so treat that machine as executing whatever your users push, and
462install the toolchains your builds need on it.
463
464A runner claims the oldest pending build among the repositories its key
465is attached to — for an admin key, the oldest in the instance. =-repos=
466narrows within that set, which is what makes a runner outside the server
467practical: one on a machine that should build a single project, or that
468holds credentials for one deployment, stays on it.
469
470Oldest-first is across everything the key may claim, so a repository
471with a deep queue holds every other repository the same runner serves;
472bay1 measured a 15-minute average wait on a day of merge request
473stacks from one repository. A runner attached to one repository cannot
474be starved. That is the rule, decided in krz/gitbay#207: a runner
475serving several repositories takes them oldest-first, and an operator
476who wants one repository never to wait on another runs a second
477runner attached to it alone. Nothing caps what an account queues,
478and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
479while the job's last build is pending or running, so a repository
480with no runner holds one row per scheduled job rather than one per
481tick, and a build runs only on a runner its owner attaches, so a busy
482schedule spends the owner's compute. Pushes are bounded by what an
483account can push.
484
485#+begin_src sh
486gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
487 -workdir /var/lib/gitbay-runner/work
488#+end_src
489
490Add =-untrusted= only with =-isolation podman=.
491
492gitbay.org's runner is attached to the forge's own repositories and
493the isolation canary, nothing else, because it shares the host with
494the forge; its unit names no =-repos=, the attachments are the
495boundary. Any other repository builds on a runner its owner attaches.
496
497=-repos= narrows an admin runner; for a runner key the attachments are
498the boundary, held by the server, and =-repos= may only name
499repositories among them. =-untrusted= makes a runner claim merge
500request heads from forks; the bay1 unit sets it because it isolates in
501podman. A runner without it builds trusted commits only.
502
503=gitbay dashboard= and =ssh git@<host> admin runners= list every key
504that has polled as a runner: the account, the key's fingerprint, when it
505last polled, the repositories it may claim — its attachments for a runner
506key, the =-repos= it asked for or =any= for an admin key — and the build
507it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
508that polled by mistake, the key itself untouched. =admin runners= also
509heads the list with the queue: builds
510pending now, and over the last day how many were claimed, how long they
511waited to be claimed (average and worst), and
512how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
513reported is failed by the scheduler's minute tick, whether or not any
514runner is still alive: within about two minutes of its log stream ending
515with no outcome reported — the runner reports right after closing the
516stream, retrying for half a minute if gitbayd is unreachable — or, if no
517stream was ever seen, at the deadline.
518
519Instance admin on the runner account only authorizes the claim/report
520protocol; it grants no repo access. A build that pushes back — a pages
521deploy, an archive publish, an automated MR branch — needs an explicit
522grant on that repo: =repo access grant <owner/name> ci write=. Private
523repos likewise need at least read for the clone.
524
525** Container isolation
526
527Builds run in a rootless podman container, one per job, with the
528workspace bind mounted and nothing else: the clone happens outside with
529the runner's key, so a step cannot read it. =-isolation none= keeps the
530old behaviour — steps on the host as the runner's user — for an instance
531where every repository is trusted. There is no automatic fallback: a
532runner started with =-isolation podman= that cannot find a working
533podman exits rather than running a build unsandboxed.
534
535gitbay's own jobs name =localhost/gitbay-ci:2=, built from
536=deploy/Containerfile.ci= on the runner host. A job's image must carry
537what its steps need: the suite drives real git, git-lfs, gpg and sshd and
538asserts they exist before running, so the stock runner default would fail
539it immediately. Build or rebuild it with:
540
541#+begin_src sh
542ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
543ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
544 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
545#+end_src
546
547The tag is deliberate rather than =:latest=: changing the file means
548bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
549change under it.
550
551=-image= names the image a job runs in when it declares none, and is
552required under =-isolation podman=: there is no built-in default,
553because an image this host does not have would fail every build. A job
554overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
555so a config file cannot turn it into podman arguments.
556
557=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
558-memory 4g=); unset means uncapped. The runner applies them itself: it
559creates a cgroup per build under its own delegated service cgroup,
560writes the limits there, and starts every podman process for the build
561inside it, with podman's cgroup handling off. Podman's own =--memory=
562and =--cpus= never applied under rootless cgroupfs, which is what a
563system service gets (krz/gitbay#188). The unit therefore needs
564=Delegate=yes=, which the drop-in sets; without it the runner refuses
565to start when a limit is set, and logs that builds run unconfined when
566none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
567and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
568swap: the memory cap is what keeps the forge alive when a build
569allocates without bound, and it sits above the e2e suite's 5GB peak
570rather than at a fair share. =OOMPolicy=continue= keeps systemd from
571stopping the runner when a build is OOM-killed.
572
573Each repository gets its own build home under the runner's workdir,
574mounted into its containers as =HOME=. Caches persist between builds of
575one repository and are never read by another's.
576
577*Images are provisioned, never pulled by a build.* The runner passes
578=--pull=never=. Two reasons, and the second is the better one: the
579service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
580holding a setuid file, which is nearly every distribution image; and on
581an instance where anyone can push a =ci.yml=, =image:= would otherwise
582mean "fetch and run anything from the internet". An operator pulls or
583builds what is allowed and a build picks among those. A job naming an
584image the host does not have fails with a message saying so.
585
586#+begin_src sh
587su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
588su - ci-runner -s /bin/sh -c "podman images"
589#+end_src
590
591Prepare a host before pointing an isolating runner at it:
592
593#+begin_src sh
594ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
595make deploy-runner
596#+end_src
597
598The script installs podman, delegates a subuid/subgid range to
599=ci-runner=, checks that user namespaces are enabled rather than
600assuming, enables lingering, and verifies rootless podman actually runs
601as that user. It is idempotent.
602
603The drop-in sets =NoNewPrivileges=no=, without which rootless podman
604cannot call =newuidmap= and the runner refuses to start. That is a
605considered trade, explained in the file and in the Threat-Model; if you
606run with =-isolation none=, set it back to =yes=.
607
608*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
609the build in flight, reports it, and exits; the drop-in's
610=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
611is what makes the signal reach the runner alone — under systemd's default
612the build's container and the log session are signalled with it, and the
613runner drains a build that is already dead. So =make deploy-runner=
614waits for a running build rather than orphaning it, and a build's result
615is retried for half a minute if gitbayd is restarting at that moment. A
616second SIGTERM ends the runner at once, abandoning the build to the
617reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
618the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
619=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
620the runner as a transient user unit under =systemd-run= and stops it
621under both kill modes. That last one needs a systemd user manager, so it
622skips in the container CI runs in; run it on a Linux host with
623=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
624
625*Validate podman mode on a scratch repository before pointing the runner
626at real ones.* Every deploy that switched the whole instance to
627containers and failed took CI down with it. Instead: create a throwaway
628repository the runner account can read (public, or granted read — a
629private one is "not found" to the runner and the build stays pending),
630give it one job that names the CI image, and deploy the runner with
631=-repos= naming only that repository. The production unit, with its real
632hardening, then claims nothing else; other repositories' builds queue
633until =-repos= is switched back, which is a pause, not an outage.
634
635#+begin_src sh
636gitbay repo create cmc/ci-smoke # then push a .gitbay/ci.yml naming the image
637sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
638systemctl daemon-reload && systemctl restart gitbay-runner
639gitbay build log cmc/ci-smoke 1 # green: switch -repos back, redeploy
640#+end_src
641
642*Do not deploy an isolating runner to a host that has not been
643prepared.* The runner is specified to refuse to start without a working
644podman rather than fall back to running builds unsandboxed — a fallback
645that silently drops isolation is worse than a stopped runner, because
646nothing surfaces it. On an unprepared host that refusal stops every
647build on the instance.
648
649The service drop-in carries =Delegate=yes= for rootless cgroup
650management and =ReadWritePaths= for podman's store under
651=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
652make read-only. Those paths are prefixed =-= so they are ignored when
653absent: the drop-in installs on unprepared hosts too, and a unit that
654refused to start would stop every build.
655The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
656names that repository too; a scoped runner claims nothing else.
657=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
658user: rootless storage belongs to that user, and root's prune would not
659see it. An unpruned image store on a 40GB host is a slow outage.
660
661* LFS storage
662
663Objects live content-addressed under =[lfs] root= (default
664=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
665(512MB default). Storage sits behind a small interface — an
666S3-compatible backend is a drop-in with the server proxying, and
667presigned URLs a later optimization. LFS objects do not travel with
668push mirrors (mirrors move git refs only), and gc does not yet collect
669orphaned objects.
670
671* Pages
672
673=[pages] domain = "example.site"= serves public repos' =pages= branches
674on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
675server; ACME issues per-subdomain certificates on demand (only for
676owners that exist). The domain must not be the site host or a parent of
677it — pages content runs its own scripts and must stay off the forge's
678origin.
679
680Users with repo admin claim custom domains with =repo domain add=.
681Claims activate only after a DNS TXT challenge proves control of the
682domain (=repo domain verify=, audit-logged); pending claims serve
683nothing, get no certificates, and expire after 7 days. ACME issues
684certificates only for verified hosts, so stray DNS pointed at the
685server gets nothing.
686
687* Security
688
689The [[Threat-Model]] file is the reference for what the forge
690trusts and refuses to do. Operational checklist:
691
692- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
693 (the module list is deliberately short — review it on each release),
694 and a short fuzz pass over every attacker-facing parser (pkt-line,
695 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
696 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
697 against main rather than per push, because =@latest= scans today's
698 advisory database and an advisory lands without anyone pushing;
699 =build trigger krz/gitbay vuln= runs it on demand.
700- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
701 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
702 configuration needed.
703- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
704 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
705 =LockPersonality=, =MemoryDenyWriteExecute=,
706 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
707 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
708- *OS patches* apply via =unattended-upgrades= (security origins,
709 auto-reboot 04:30 if required).
710- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
711 watched by =fail2ban=; gitbayd's own port 22 is throttled by
712 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
713 account's writes by =limits.write_rate=.
714- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
715journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
716webhook: disk, service, the daemon's own =/healthz= answer, certificate
717expiry, and the age of the newest full backup and database snapshot.
718It exits non-zero on an alert so the unit shows in =systemctl
719--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
720a certificate under 21 days, a full backup over 25 hours old, or a
721database snapshot over 2 hours old.
722
723=GET /healthz= is unauthenticated and cache-free: whether the database
724answers and which commit serves, 503 when it does not.
725- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
726 0750, owned by =gitbay=). The nightly archive plus provider snapshots
727 are the recovery path; for tighter RPO, add continuous replication
728 (litestream) against the same file — it coexists with the WAL.
729
730* Odds and ends
731
732- deleting a fork marks MRs sourced from it =source_gone=; their diffs
733 remain viewable and mergeable because the target repo owns the
734 objects.
735- =refs/merge-requests/*= is server-owned and unpushable by clients;
736 only =admin mr prune= removes one (see Maintenance).
737- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
738 the =events= table, which also feeds webhooks.
739- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.