.gitbay/wiki/Admin.org

  1#+title: gitbay admin guide
  2
  3One static binary (=gitbayd=), one SQLite file, bare repositories on
  4disk, and the system =git=. Schema migrations run automatically on
  5startup and on every admin command.
  6
  7* Install
  8
  9Build from source (=go build ./cmd/gitbayd=), install via the vanity
 10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
 11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
 12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
 13manifest (CGO off, trimpath, stripped — byte-identical per commit and
 14toolchain).
 15
 16#+begin_src sh
 17install -m 755 gitbayd /usr/local/bin/
 18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
 19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
 20gitbayd --config /etc/gitbay/config.toml check-config
 21#+end_src
 22
 23=deploy/= in the source tree has a cloud-init file, a hardened systemd
 24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
 25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
 2622/80/443 without root.
 27
 28** The SSH port decision
 29
 30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
 31  22 — move the host's admin sshd to another port. Remotes read
 32  =git@host:owner/repo= with no port gymnastics.
 33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
 34  =AuthorizedKeysCommand=:
 35  #+begin_example
 36  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
 37  AuthorizedKeysCommandUser gitbay
 38  #+end_example
 39  sshd requires that binary to be root-owned and not group/world
 40  writable. Unknown keys fail authentication inside sshd, so system mode
 41  requires =registration.mode = "closed"= (check-config enforces this).
 42
 43* Configuration reference
 44
 45=/etc/gitbay/config.toml=. =check-config= validates and names every
 46contradiction; =--no-host-checks= skips port/path probes.
 47=gitbayd admin config show= prints the configuration in effect as TOML,
 48every default filled in and =smtp_pass= redacted. A file that fails
 49validation still prints, followed by the contradiction.
 50
 51** [server]
 52- =root= (default =/var/lib/gitbay=) — repositories, database, host
 53  keys, ACME cache all live here.
 54- =site_url= (required) — canonical =https://host=; drives ACME, clone
 55  URLs, mail links.
 56- =source_repo= (optional, =owner/name=) — the repository this instance
 57  develops itself in. Startup warns when the running build's commit is
 58  not on that repository's default branch, which is how a binary built
 59  from an unmerged branch stops being invisible. Leave it unset unless
 60  the instance hosts its own source.
 61
 62** [ssh]
 63- =mode= — =embedded= | =system= (above).
 64- =port= (22) — embedded listener port.
 65- =host_keys= — list of private key paths; empty generates an ed25519
 66  key at =<root>/ssh/host_ed25519=.
 67
 68** [http]
 69- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
 70- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
 71  =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
 72  (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
 73  failing to bind it is a warning, not fatal. Requires an =https://=
 74  site_url with a public DNS name.
 75- =files=: =cert_file= + =key_file=.
 76- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
 77
 78=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
 79front of the daemon. A request from one of them is attributed, for API
 80rate limiting, to the last =X-Forwarded-For= hop that is not itself a
 81trusted proxy; from anyone else the header is ignored. Empty, the
 82default, is right when gitbayd terminates TLS itself.
 83
 84** [web]
 85- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
 86  web routes are never registered; in accounts, browser sessions are
 87  minted over SSH (=web login=), and users with write access can create
 88  repos, comment, and make simple file edits (which commit unsigned,
 89  honestly). =password_auth= is reserved and currently rejected.
 90- =title= — the instance's display name in the rail and page titles;
 91  empty falls back to the site host. Lower case is the convention for
 92  gitbay itself.
 93- =privacy_notice= — operator text shown on =/privacy= under the fixed
 94  statement.
 95
 96** [registration]
 97- =mode= — =closed= (default) | =invite= | =open=. invite/open require
 98  [mail]. See the user guide for the flows.
 99
100- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
101  self-registered account still unverified after that long is removed,
102  hourly and at start, audited as =pending.expired=.
103
104- =notify_admin= (false) — mail every instance admin when an account
105  becomes active: an invite redeemed, or an open-mode signup that
106  verified its address. The unverified row an open signup creates is
107  not reported, because anyone can post the form and mailing on that
108  would point a flood at the admins. Recipients are the verified
109  primary addresses of active admins who have activity mail on, the
110  same rule any other notice follows, so an admin with no verified
111  address hears nothing. The notice is queued, so a dead SMTP host
112  shows up in the admin page's Mail table instead of failing the
113  registration. Requires [mail].
114
115** [mail]
116- =smtp_host= (host:port, 587 assumed), =from=, optional =smtp_user= /
117  =smtp_pass=. STARTTLS when offered. Required for invite/open
118  registration and self-service =email add=; in closed mode you may omit
119  it entirely and assert addresses by hand (below).
120
121** [push]
122Push notifications to Apple devices, delivered by gitbayd talking to
123APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
124operator-supplied provider key. Off unless configured.
125
126- =enabled= (false).
127- =key_file= — path to the =.p8= provider key from Apple's developer
128  portal (Certificates, Identifiers & Profiles → Keys). It belongs at
129  =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
130  as. Read and validated at startup: it must parse as a PEM-wrapped
131  PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
132  than fill a queue nobody is watching.
133- =key_id=, =team_id= — the key's id and your Apple developer team id,
134  both from the same portal page.
135- =topic= — the app's bundle identifier. *An APNs key belongs to a
136  bundle ID.* gitbay.org pushes to the App Store build under its own
137  bundle id; a self-hoster who wants push ships their own iOS build
138  under their own bundle id, with its own =.p8= key from their own
139  developer account, and points =topic= at that id. There is no way to
140  push to someone else's build, by design — this is Apple's model, not
141  gitbay's.
142- =environment= — =production= or =sandbox=, naming the APNs host
143  rather than taking a URL, so a typo cannot aim the key at a host that
144  is not Apple's.
145
146All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
147are required when =enabled= is true; validation runs at config load,
148so a misconfigured =[push]= is caught before the daemon serves
149anything. The delivery queue (a device's undelivered and attempted
150pushes) is capped the same way the mail queue is, by =[retention]
151push=.
152
153** [api]
154- =enabled= (false) — the JSON API surface; see [[API]]. Off
155  means no credential-bearing HTTP endpoint exists at all.
156
157** [webhooks]
158- =allow_local= (false) — permit webhook targets on loopback/private
159  addresses. Leave off unless you know why you need it (SSRF).
160
161** [limits]
162- =clone_timeout= (3600s) — cap on =repo import= fetches.
163- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
164- =max_asset_bytes= (512MB) — cap per uploaded release asset.
165- =max_snippet_bytes= (1MB) — cap per snippet file.
166- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
167- =max_repos_per_user= (0, unlimited) — repositories an account may own
168  directly; =repo create=, =fork= and =import= refuse past it.
169  Organizations are not capped.
170- =max_bytes_per_user= (0, unlimited) — disk the account's own
171  repositories may take; a push may be no larger than what is left.
172- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
173
174** [git_daemon]
175- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
176  Serves only public repositories that additionally ran
177  =repo settings git-daemon <repo> on=.
178
179** [mirrors]
180- =pull_interval_minutes= (15) — how often pull mirrors fetch their
181  upstream. Push mirrors sync shortly after each local ref update.
182  Mirror URLs pass the same SSRF rules as webhook targets.
183
184** [go_import]
185Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
186Requests with =?go-get=1= at or under the module path answer with the
187go-import meta tag pointing at the repository's HTTPS clone URL, so
188=go install host/module/cmd/...@latest= resolves. The repository should
189be public (the module path itself confirms it exists).
190
191* Users, email, invites
192
193Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
194backfills is a wrapper that dispatches the registry command of the same
195name as the host: an admin context with no account behind it, so its
196audit rows carry no actor and =source: host=. The same commands run in an
197instance admin's SSH session (=ssh git@<host> admin ...=) and write the
198same rows with the key fingerprint as source. One implementation, two
199credentials.
200
201#+begin_src sh
202gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
203gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
204gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP
205#+end_src
206
207"Verified" means SMTP-confirmed or host-admin-asserted; the database
208records which. Verified emails are what make commit signatures
209meaningful — an unverified address never produces a =verified= badge.
210
211* Audit and account control
212
213The audit log is the security feed (events are the product feed): every
214successful mutating command with its argv and source credential (SSH key
215fingerprint or API), registrations, admin actions, force-pushes, and
216auth failures/throttling. Secrets never appear — they travel on stdin,
217never in argv.
218
219#+begin_src sh
220gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
221ssh git@<host> audit ...             # the same, from an admin session
222ssh git@<host> admin user list [--state active|pending|disabled|admin]
223ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
224ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default]   # per-account caps
225ssh git@<host> admin user promote <name>   # grant instance admin
226ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
227gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
228gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
229gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
230gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
231                                     #   refused (with each blocker named) while
232                                     #   the account owns repos, authored
233                                     #   issues/MRs/comments/reviews, or is an
234                                     #   org's only admin
235#+end_src
236
237=--actor= takes a username, or =-= for rows with no actor: host commands
238and auth failures. =--action= is a prefix, so =cmd repo= catches every
239repository command and =admin= every host or admin-session action.
240=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
241
242=admin user list= pages by username (=--limit=, =--cursor=) and carries
243each account's state and =last_seen=, the newest use of any of its SSH
244keys or API tokens. =admin user show= adds the keys with their last use,
245each address with how it was verified, PGP keys, org roles, the owned
246repository count, API token names, and live browser sessions. Both are
247Both are refused to non-admins, like =audit=, on every surface.
248
249=/admin/users= is the same list in a browser, linked from the admin
250page: the state filter the command takes, keyset paging on its cursor,
251and a row per account with promote, demote, disable and enable, each
252dispatching the command. Demote and disable ask for the username to be
253typed, since both take someone's access away. Creating and deleting an
254account, issuing an invite and asserting an address stay on the command
255line: each takes a key, mints a credential, or cannot be undone. A
256non-admin gets the 404 a missing page would, so the URL confirms
257nothing.
258
259Promotion needs an active account: a pending or disabled one is refused.
260Demotion is refused when it would leave no admin, over SSH, in the
261browser and on the host alike, so the host-local =promote= is the way
262back in when the only admin key is lost.
263
264Instance admin carries no right on anyone's repository: policy does not
265consult it, and a private repository still answers not-found to an
266admin. Moderation goes through explicit overrides that skip the access
267check and write their own audit row:
268
269#+begin_src sh
270ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
271ssh git@<host> admin repo archive|unarchive <owner/name>
272ssh git@<host> admin repo visibility <owner/name> public|private
273ssh git@<host> admin repo delete <owner/name> --yes
274#+end_src
275
276Each lands in the audit log as =admin repo.<action>= naming the
277repository, on top of the =cmd= row every mutating command gets.
278
279=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
280per minute — successful auths never count and clear the slate.
281=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
282push.
283
284=limits.write_rate= (60) bounds *mutating commands per account per
285minute*. It is counted in the dispatcher, so SSH, the JSON API and the
286web spend one budget and a caller cannot refresh it by changing surface;
287=limits.api_rate= stays in front of it, bounding a network source rather
288than an account. A command is one token whatever it writes, so a bundle
289import costs one and only a loop of separate commands spends the budget.
290Read-only commands, the runner protocol (a build streams its log in many
291small writes) and the host CLI are exempt. Refusals exit 4 and say when
292to retry. A negative value turns the limit off; it matters most with
293=registration = "open"=, where every write also queues notification mail
294and webhook deliveries.
295
296* Queues
297
298Every background worker keeps a backlog and a failure state. An instance
299admin reads them all in one place:
300
301#+begin_src sh
302gitbay dashboard --json | jq .queues   # webhooks, mail, push, mirrors, builds, deps
303#+end_src
304
305Per worker: pending, retrying (pending with a failed attempt) and
306dead-lettered counts with the oldest pending age, and the retrying or
307failed rows themselves, capped at twenty each. Builds list what is
308running and then what is pending, each since when, so a build no runner
309is scoped to claim is visible here rather than only in its repository;
310mirrors list the ones whose last sync failed;
311dependency checks list the ones whose last check errored. Non-admins get
312no =queues= key at all.
313
314Push rows name the device id, never the token. Watch this one after
315configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
316config validation, which can only check that the =.p8= parses, and then
317every send comes back =403 InvalidProviderToken= and dead-letters on its
318first attempt.
319
320In accounts mode the same read renders at =/admin=, linked from the rail
321for admins. Anyone else gets a 404 there.
322
323A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
324with the address redacted out of the relay's error. The id is the queue
325row: find it in the Mail table on =/admin=, or in =dashboard --json=,
326where the recipient and the unredacted error are. That is deliberate —
327see the Threat-Model page.
328
329* Maintenance
330
331#+begin_src sh
332gitbayd admin stats [--json]         # counts, database size, per-repo disk
333ssh git@<host> admin stats [--json]  # the same, from an admin session
334gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
335gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
336gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)
337#+end_src
338
339A history rewrite leaves the commits it removed reachable through
340=refs/merge-requests/N/head= of the merge requests that landed them, so
341they stay fetchable by anyone who can read the repository. Nothing drops
342a head ref on its own — an open or source-gone MR is merged through it,
343and a merged or closed one keeps its diff readable through it — so the
344cleanup is a command an instance admin runs, naming the MRs:
345
346#+begin_src sh
347ssh git@<host> admin mr prune owner/name 1 2 3 --yes
348#+end_src
349
350It refuses an open or source-gone MR, deletes the named refs, runs
351=git gc --prune=now= on that one repository so the objects go at once
352rather than after git's two-week grace, leaves a system comment on each
353MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
354reviews and head sha; =mr diff= and the MR page say the head is gone.
355Run it when nothing is pushing to that repository: without the grace, a
356push caught between leaving quarantine and writing its ref loses its
357objects. Objects also survive in offsite backups until those are
358rewritten; see "Removing a repository's history from every snapshot".
359
360=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
361weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
362layout the source sent, so a first manual =admin gc= after a bulk
363import is worthwhile.
364
365* Backup and restore
366
367#+begin_src sh
368gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
369gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back
370#+end_src
371
372One archive: a consistent SQLite snapshot (taken *before* the
373repositories are read, so the database never references objects the
374archive missed), every repository, and the SSH host keys. Excluded:
375hook socket, regenerated hook scripts, WAL files. Safe to run against a
376live daemon.
377
378=--verify= reads an archive back: the snapshot must pass SQLite's
379integrity check, and every repository the snapshot names must be in the
380archive. A database-only archive is checked for integrity and says so.
381Exit is non-zero on damage or a missing repository.
382
383Restore: extract into an empty directory, point =server.root= at it,
384start gitbayd. Host keys are preserved, so clients keep their
385known_hosts entries; hooks regenerate at startup.
386
387** Schedule and recovery point
388
389Two timers, because the two halves of the data have different exposure.
390
391- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
392- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
393  writes the SQLite snapshot alone, last 48 kept. A few MB against the
394  full archive's hundreds, which is what makes the frequency affordable.
395
396The split follows what a loss would actually cost. Repositories are git,
397so a mirror or any clone is a second copy; the database is the only copy
398of issues, merge requests, comments and review state. So the recovery
399point is about an hour for the data that exists nowhere else, and a day
400for the data that does.
401
402Continuous replication (litestream and similar) was considered and not
403adopted. It would take the database's recovery point to seconds, but the
404repositories would still be on the nightly archive, so a restore could
405produce a database referencing commits the repository backup does not
406have. Consistency between the two halves is worth more here than latency
407on one of them. Revisit if repository replication becomes continuous
408too.
409
410** Offsite copies
411
412bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
413=/var/lib/gitbay-stage= (the staged database copy) to an S3 bucket at
414Scaleway, with a key that can only add snapshots. The key that can
415remove them lives on the operator's machine, in
416=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
417cannot destroy its own history. Forgetting, pruning and rewriting all
418run from there.
419
420*** Removing a repository's history from every snapshot
421
422A history rewrite plus =admin mr prune= takes commits off the server,
423but every snapshot taken before it still holds them, and the retention
424window is the only thing that ages them out. To remove them now,
425rewrite the snapshots without that repository rather than forgetting
426the snapshots: everything else in them stays restorable. The next
427nightly run adds the repository back in its current state.
428
429Repositories are stored under the name they had on disk when each
430snapshot was taken, so a renamed repository needs every name it has
431carried. Check what an older snapshot holds before choosing the paths:
432
433#+begin_src sh
434set -a; . ~/.config/gitbay/offsite.env; set +a
435restic $RESTIC_OPTS snapshots
436restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
437#+end_src
438
439Then dry-run, apply, prune, and confirm nothing matches:
440
441#+begin_src sh
442EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
443restic $RESTIC_OPTS rewrite --dry-run $EXCL     # "would modify N snapshots"
444restic $RESTIC_OPTS rewrite --forget $EXCL      # new snapshots replace the originals
445restic $RESTIC_OPTS prune                       # drops the data nothing references
446restic $RESTIC_OPTS find <name>.git <old-name>.git   # expect no output
447#+end_src
448
449=--forget= is what makes the originals go; without it the rewritten
450snapshots sit beside them and the data stays referenced. Snapshot IDs
451change; their times do not. Done for krz/keycask (formerly rust-pass)
452on 2026-09-18, across 22 snapshots.
453
454* Upgrades
455
456Replace the binary, restart the unit. Migrations apply automatically and
457are transactional; hook scripts under =<root>/hooks= are rewritten at
458startup to point at the current binary path.
459
460* CI runner
461
462=gitbay-runner= executes builds queued by pushes and merge requests. It
463polls over SSH with a key of scope =runner=, which reaches only the
464runner protocol and read-only git (a runner executes arbitrary
465repository code, so the key it holds must not do more). A runner key
466claims builds only for the repositories it is attached to, by =repo
467runner add= from a repository admin or an instance admin; an admin key
468claims any. Users attach their own runners: see the Users page. For an
469instance runner, run it as a dedicated unprivileged user on a non-admin
470account. =admin user create --key= registers a full-scope key, so the
471runner key is added afterwards through a bootstrap key that is then
472removed, and attached to each repository it should build:
473
474#+begin_src sh
475useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
476sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
477ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
478gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
479ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
480ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
481rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
482gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
483#+end_src
484
485#+begin_src sh
486gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
487#+end_src
488
489=-jobs N= runs N builds at once. Claiming is one transaction that
490selects and updates, and each build works in its own =build-<id>=
491directory, so workers do not collide; idle polls are staggered across
492the interval so N of them do not wake together. The drop-in's weights
493below are per service, not per build, so raising =-jobs= divides them
494rather than multiplying the host's load.
495
496=admin runners= shows which account each runner polls as, and what each
497may claim. A runner key claims builds only for the repositories it is
498attached to: with none attached it claims nothing, and =-repos= may only
499narrow within them. An admin's full-scope key claims any repository —
500that is what =-repos= was for — and still works for the protocol during
501a rotation. A merge request head from a fork is built in the target
502repository as untrusted: the claim carries no secrets, and only a runner
503started with =-untrusted= takes it. Same-repository heads were built by
504their branch push and are not built again.
505
506=make deploy-runner= also installs
507=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
508=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
509the daemon or the backup timers, and =NoNewPrivileges=,
510=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
511and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
512and the runner's home. The e2e suite alone starts sixty
513daemon instances; without the drop-in a deploy's copy over the admin
514sshd stalled. Both deploy targets copy with =rsync --partial=, which
515resumes a stalled transfer.
516
517Under =-isolation podman=, the default and what bay1 runs, each build
518is confined to a container (see Container isolation below). Under
519=-isolation none= steps run directly on the host as the runner's user,
520so treat that machine as executing whatever your users push, and
521install the toolchains your builds need on it.
522
523A runner claims the oldest pending build among the repositories its key
524is attached to — for an admin key, the oldest in the instance. =-repos=
525narrows within that set, which is what makes a runner outside the server
526practical: one on a machine that should build a single project, or that
527holds credentials for one deployment, stays on it.
528
529Oldest-first is across everything the key may claim, so a repository
530with a deep queue holds every other repository the same runner serves;
531bay1 measured a 15-minute average wait on a day of merge request
532stacks from one repository. A runner attached to one repository cannot
533be starved. That is the rule, decided in krz/gitbay#207: a runner
534serving several repositories takes them oldest-first, and an operator
535who wants one repository never to wait on another runs a second
536runner attached to it alone. Nothing caps what an account queues,
537and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
538while the job's last build is pending or running, so a repository
539with no runner holds one row per scheduled job rather than one per
540tick, and a build runs only on a runner its owner attaches, so a busy
541schedule spends the owner's compute. Pushes are bounded by what an
542account can push.
543
544#+begin_src sh
545gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
546  -workdir /var/lib/gitbay-runner/work
547#+end_src
548
549Add =-untrusted= only with =-isolation podman=.
550
551gitbay.org's runner is attached to the forge's own repositories and
552the isolation canary, nothing else, because it shares the host with
553the forge; its unit names no =-repos=, the attachments are the
554boundary. Any other repository builds on a runner its owner attaches.
555
556=-repos= narrows an admin runner; for a runner key the attachments are
557the boundary, held by the server, and =-repos= may only name
558repositories among them. =-untrusted= makes a runner claim merge
559request heads from forks; the bay1 unit sets it because it isolates in
560podman. A runner without it builds trusted commits only.
561
562=gitbay dashboard= and =ssh git@<host> admin runners= list every key
563that has polled as a runner: the account, the key's fingerprint, when it
564last polled, the repositories it may claim — its attachments for a runner
565key, the =-repos= it asked for or =any= for an admin key — and the build
566it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
567that polled by mistake, the key itself untouched. =admin runners= also
568heads the list with the queue: builds
569pending now, and over the last day how many were claimed, how long they
570waited to be claimed (average and worst), and
571how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
572reported is failed by the scheduler's minute tick, whether or not any
573runner is still alive: within about two minutes of its log stream ending
574with no outcome reported — the runner reports right after closing the
575stream, retrying for half a minute if gitbayd is unreachable — or, if no
576stream was ever seen, at the deadline.
577
578Instance admin on the runner account only authorizes the claim/report
579protocol; it grants no repo access. A build that pushes back — a pages
580deploy, an archive publish, an automated MR branch — needs an explicit
581grant on that repo: =repo access grant <owner/name> ci write=. Private
582repos likewise need at least read for the clone.
583
584** Container isolation
585
586Builds run in a rootless podman container, one per job, with the
587workspace bind mounted and nothing else: the clone happens outside with
588the runner's key, so a step cannot read it. =-isolation none= keeps the
589old behaviour — steps on the host as the runner's user — for an instance
590where every repository is trusted. There is no automatic fallback: a
591runner started with =-isolation podman= that cannot find a working
592podman exits rather than running a build unsandboxed.
593
594gitbay's own jobs name =localhost/gitbay-ci:2=, built from
595=deploy/Containerfile.ci= on the runner host. A job's image must carry
596what its steps need: the suite drives real git, git-lfs, gpg and sshd and
597asserts they exist before running, so the stock runner default would fail
598it immediately. Build or rebuild it with:
599
600#+begin_src sh
601ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
602ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
603  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
604#+end_src
605
606The tag is deliberate rather than =:latest=: changing the file means
607bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
608change under it.
609
610=-image= names the image a job runs in when it declares none, and is
611required under =-isolation podman=: there is no built-in default,
612because an image this host does not have would fail every build. A job
613overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
614so a config file cannot turn it into podman arguments.
615
616=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
617-memory 4g=); unset means uncapped. The runner applies them itself: it
618creates a cgroup per build under its own delegated service cgroup,
619writes the limits there, and starts every podman process for the build
620inside it, with podman's cgroup handling off. Podman's own =--memory=
621and =--cpus= never applied under rootless cgroupfs, which is what a
622system service gets (krz/gitbay#188). The unit therefore needs
623=Delegate=yes=, which the drop-in sets; without it the runner refuses
624to start when a limit is set, and logs that builds run unconfined when
625none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
626and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
627swap: the memory cap is what keeps the forge alive when a build
628allocates without bound, and it sits above the e2e suite's 5GB peak
629rather than at a fair share. =OOMPolicy=continue= keeps systemd from
630stopping the runner when a build is OOM-killed.
631
632Each repository gets its own build home under the runner's workdir,
633mounted into its containers as =HOME=. Caches persist between builds of
634one repository and are never read by another's.
635
636*Images are provisioned, never pulled by a build.* The runner passes
637=--pull=never=. Two reasons, and the second is the better one: the
638service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
639holding a setuid file, which is nearly every distribution image; and on
640an instance where anyone can push a =ci.yml=, =image:= would otherwise
641mean "fetch and run anything from the internet". An operator pulls or
642builds what is allowed and a build picks among those. A job naming an
643image the host does not have fails with a message saying so.
644
645#+begin_src sh
646su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
647su - ci-runner -s /bin/sh -c "podman images"
648#+end_src
649
650Prepare a host before pointing an isolating runner at it:
651
652#+begin_src sh
653ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
654make deploy-runner
655#+end_src
656
657The script installs podman, delegates a subuid/subgid range to
658=ci-runner=, checks that user namespaces are enabled rather than
659assuming, enables lingering, and verifies rootless podman actually runs
660as that user. It is idempotent.
661
662The drop-in sets =NoNewPrivileges=no=, without which rootless podman
663cannot call =newuidmap= and the runner refuses to start. That is a
664considered trade, explained in the file and in the Threat-Model; if you
665run with =-isolation none=, set it back to =yes=.
666
667*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
668the build in flight, reports it, and exits; the drop-in's
669=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
670is what makes the signal reach the runner alone — under systemd's default
671the build's container and the log session are signalled with it, and the
672runner drains a build that is already dead. So =make deploy-runner=
673waits for a running build rather than orphaning it, and a build's result
674is retried for half a minute if gitbayd is restarting at that moment. A
675second SIGTERM ends the runner at once, abandoning the build to the
676reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
677the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
678=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
679the runner as a transient user unit under =systemd-run= and stops it
680under both kill modes. That last one needs a systemd user manager, so it
681skips in the container CI runs in; run it on a Linux host with
682=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
683
684*Validate podman mode on a scratch repository before pointing the runner
685at real ones.* Every deploy that switched the whole instance to
686containers and failed took CI down with it. Instead: create a throwaway
687repository the runner account can read (public, or granted read — a
688private one is "not found" to the runner and the build stays pending),
689give it one job that names the CI image, and deploy the runner with
690=-repos= naming only that repository. The production unit, with its real
691hardening, then claims nothing else; other repositories' builds queue
692until =-repos= is switched back, which is a pause, not an outage.
693
694#+begin_src sh
695gitbay repo create cmc/ci-smoke          # then push a .gitbay/ci.yml naming the image
696sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
697systemctl daemon-reload && systemctl restart gitbay-runner
698gitbay build log cmc/ci-smoke 1         # green: switch -repos back, redeploy
699#+end_src
700
701*Do not deploy an isolating runner to a host that has not been
702prepared.* The runner is specified to refuse to start without a working
703podman rather than fall back to running builds unsandboxed — a fallback
704that silently drops isolation is worse than a stopped runner, because
705nothing surfaces it. On an unprepared host that refusal stops every
706build on the instance.
707
708The service drop-in carries =Delegate=yes= for rootless cgroup
709management and =ReadWritePaths= for podman's store under
710=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
711make read-only. Those paths are prefixed =-= so they are ignored when
712absent: the drop-in installs on unprepared hosts too, and a unit that
713refused to start would stop every build.
714The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
715names that repository too; a scoped runner claims nothing else.
716=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
717user: rootless storage belongs to that user, and root's prune would not
718see it. An unpruned image store on a 40GB host is a slow outage.
719
720* LFS storage
721
722Objects live content-addressed under =[lfs] root= (default
723=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
724(512MB default). Storage sits behind a small interface — an
725S3-compatible backend is a drop-in with the server proxying, and
726presigned URLs a later optimization. LFS objects do not travel with
727push mirrors (mirrors move git refs only), and gc does not yet collect
728orphaned objects.
729
730* Pages
731
732=[pages] domain = "example.site"= serves public repos' =pages= branches
733on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
734server; ACME issues per-subdomain certificates on demand (only for
735owners that exist). The domain must not be the site host or a parent of
736it — pages content runs its own scripts and must stay off the forge's
737origin.
738
739Users with repo admin claim custom domains with =repo domain add=.
740Claims activate only after a DNS TXT challenge proves control of the
741domain (=repo domain verify=, audit-logged); pending claims serve
742nothing, get no certificates, and expire after 7 days. ACME issues
743certificates only for verified hosts, so stray DNS pointed at the
744server gets nothing.
745
746* Security
747
748The [[Threat-Model]] file is the reference for what the forge
749trusts and refuses to do. Operational checklist:
750
751- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
752  (the module list is deliberately short — review it on each release),
753  and a short fuzz pass over every attacker-facing parser (pkt-line,
754  commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
755  tagging a release. CI's own =vuln= job runs =govulncheck= nightly
756  against main rather than per push, because =@latest= scans today's
757  advisory database and an advisory lands without anyone pushing;
758  =build trigger krz/gitbay vuln= runs it on demand.
759- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
760  DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
761  configuration needed.
762- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
763  gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
764  =LockPersonality=, =MemoryDenyWriteExecute=,
765  =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
766  INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
767- *OS patches* apply via =unattended-upgrades= (security origins,
768  auto-reboot 04:30 if required).
769- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
770  watched by =fail2ban=; gitbayd's own port 22 is throttled by
771  =limits.ssh_auth_rate= (auth failures per IP per minute), and every
772  account's writes by =limits.write_rate=.
773- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
774journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
775webhook: disk, service, the daemon's own =/healthz= answer, certificate
776expiry, and the age of the newest full backup and database snapshot.
777It exits non-zero on an alert so the unit shows in =systemctl
778--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
779a certificate under 21 days, a full backup over 25 hours old, or a
780database snapshot over 2 hours old.
781
782=GET /healthz= is unauthenticated and cache-free: whether the database
783answers and which commit serves, 503 when it does not.
784- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
785  0750, owned by =gitbay=). The nightly archive plus provider snapshots
786  are the recovery path; for tighter RPO, add continuous replication
787  (litestream) against the same file — it coexists with the WAL.
788
789* Odds and ends
790
791- deleting a fork marks MRs sourced from it =source_gone=; their diffs
792  remain viewable and mergeable because the target repo owns the
793  objects.
794- =refs/merge-requests/*= is server-owned and unpushable by clients;
795  only =admin mr prune= removes one (see Maintenance).
796- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
797  the =events= table, which also feeds webhooks.
798- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.