.gitbay/wiki/Admin.org

  1#+title: gitbay admin guide
  2
  3One static binary (=gitbayd=), one SQLite file, bare repositories on
  4disk, and the system =git=. Schema migrations run automatically on
  5startup and on every admin command.
  6
  7* Install
  8
  9Build from source (=go build ./cmd/gitbayd=), install via the vanity
 10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
 11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
 12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
 13manifest (CGO off, trimpath, stripped — byte-identical per commit and
 14toolchain).
 15
 16#+begin_src sh
 17install -m 755 gitbayd /usr/local/bin/
 18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
 19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
 20gitbayd --config /etc/gitbay/config.toml check-config
 21#+end_src
 22
 23=deploy/= in the source tree has a cloud-init file, a hardened systemd
 24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
 25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
 2622/80/443 without root.
 27
 28** The SSH port decision
 29
 30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
 31  22 — move the host's admin sshd to another port. Remotes read
 32  =git@host:owner/repo= with no port gymnastics.
 33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
 34  =AuthorizedKeysCommand=:
 35  #+begin_example
 36  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
 37  AuthorizedKeysCommandUser gitbay
 38  #+end_example
 39  sshd requires that binary to be root-owned and not group/world
 40  writable. Unknown keys fail authentication inside sshd, so system mode
 41  requires =registration.mode = "closed"= (check-config enforces this).
 42
 43* Configuration reference
 44
 45=/etc/gitbay/config.toml=. =check-config= validates and names every
 46contradiction; =--no-host-checks= skips port/path probes.
 47=gitbayd admin config show= prints the configuration in effect as TOML,
 48every default filled in and =smtp_pass= redacted. A file that fails
 49validation still prints, followed by the contradiction.
 50
 51** [server]
 52- =root= (default =/var/lib/gitbay=) — repositories, database, host
 53  keys, ACME cache all live here.
 54- =site_url= (required) — canonical =https://host=; drives ACME, clone
 55  URLs, mail links.
 56- =source_repo= (optional, =owner/name=) — the repository this instance
 57  develops itself in. Startup warns when the running build's commit is
 58  not on that repository's default branch, which is how a binary built
 59  from an unmerged branch stops being invisible. Leave it unset unless
 60  the instance hosts its own source.
 61
 62** [ssh]
 63- =mode= — =embedded= | =system= (above).
 64- =port= (22) — embedded listener port.
 65- =host_keys= — list of private key paths; empty generates an ed25519
 66  key at =<root>/ssh/host_ed25519=.
 67
 68** [http]
 69- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
 70- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
 71  =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
 72  (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
 73  failing to bind it is a warning, not fatal. Requires an =https://=
 74  site_url with a public DNS name.
 75- =files=: =cert_file= + =key_file=.
 76- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
 77
 78=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
 79front of the daemon. A request from one of them is attributed, for API
 80rate limiting, to the last =X-Forwarded-For= hop that is not itself a
 81trusted proxy; from anyone else the header is ignored. Empty, the
 82default, is right when gitbayd terminates TLS itself.
 83
 84** [web]
 85- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
 86  web routes are never registered; in accounts, browser sessions are
 87  minted over SSH (=web login=), and users with write access can create
 88  repos, comment, and make simple file edits (which commit unsigned,
 89  honestly). =password_auth= is reserved and currently rejected.
 90
 91** [registration]
 92- =mode= — =closed= (default) | =invite= | =open=. invite/open require
 93  [mail]. See the user guide for the flows.
 94
 95- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
 96  self-registered account still unverified after that long is removed,
 97  hourly and at start, audited as =pending.expired=.
 98
 99** [mail]
100- =smtp_host= (host:port, 587 assumed), =from=, optional =smtp_user= /
101  =smtp_pass=. STARTTLS when offered. Required for invite/open
102  registration and self-service =email add=; in closed mode you may omit
103  it entirely and assert addresses by hand (below).
104
105** [api]
106- =enabled= (false) — the JSON API surface; see [[API]]. Off
107  means no credential-bearing HTTP endpoint exists at all.
108
109** [webhooks]
110- =allow_local= (false) — permit webhook targets on loopback/private
111  addresses. Leave off unless you know why you need it (SSRF).
112
113** [limits]
114- =clone_timeout= (3600s) — cap on =repo import= fetches.
115- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
116- =max_asset_bytes= (512MB) — cap per uploaded release asset.
117- =max_snippet_bytes= (1MB) — cap per snippet file.
118- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
119- =max_repos_per_user= (0, unlimited) — repositories an account may own
120  directly; =repo create=, =fork= and =import= refuse past it.
121  Organizations are not capped.
122- =max_bytes_per_user= (0, unlimited) — disk the account's own
123  repositories may take; a push may be no larger than what is left.
124- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
125
126** [git_daemon]
127- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
128  Serves only public repositories that additionally ran
129  =repo settings git-daemon <repo> on=.
130
131** [mirrors]
132- =pull_interval_minutes= (15) — how often pull mirrors fetch their
133  upstream. Push mirrors sync shortly after each local ref update.
134  Mirror URLs pass the same SSRF rules as webhook targets.
135
136** [go_import]
137Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
138Requests with =?go-get=1= at or under the module path answer with the
139go-import meta tag pointing at the repository's HTTPS clone URL, so
140=go install host/module/cmd/...@latest= resolves. The repository should
141be public (the module path itself confirms it exists).
142
143* Users, email, invites
144
145Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
146backfills is a wrapper that dispatches the registry command of the same
147name as the host: an admin context with no account behind it, so its
148audit rows carry no actor and =source: host=. The same commands run in an
149instance admin's SSH session (=ssh git@<host> admin ...=) and write the
150same rows with the key fingerprint as source. One implementation, two
151credentials.
152
153#+begin_src sh
154gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
155gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
156gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP
157#+end_src
158
159"Verified" means SMTP-confirmed or host-admin-asserted; the database
160records which. Verified emails are what make commit signatures
161meaningful — an unverified address never produces a =verified= badge.
162
163* Audit and account control
164
165The audit log is the security feed (events are the product feed): every
166successful mutating command with its argv and source credential (SSH key
167fingerprint or API), registrations, admin actions, force-pushes, and
168auth failures/throttling. Secrets never appear — they travel on stdin,
169never in argv.
170
171#+begin_src sh
172gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
173ssh git@<host> audit ...             # the same, from an admin session (SSH only)
174ssh git@<host> admin user list [--state active|pending|disabled|admin]
175ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
176ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default]   # per-account caps
177ssh git@<host> admin user promote <name>   # grant instance admin
178ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
179gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
180gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
181gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
182gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
183                                     #   refused (with each blocker named) while
184                                     #   the account owns repos, authored
185                                     #   issues/MRs/comments/reviews, or is an
186                                     #   org's only admin
187#+end_src
188
189=--actor= takes a username, or =-= for rows with no actor: host commands
190and auth failures. =--action= is a prefix, so =cmd repo= catches every
191repository command and =admin= every host or admin-session action.
192=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
193
194=admin user list= pages by username (=--limit=, =--cursor=) and carries
195each account's state and =last_seen=, the newest use of any of its SSH
196keys or API tokens. =admin user show= adds the keys with their last use,
197each address with how it was verified, PGP keys, org roles, the owned
198repository count, API token names, and live browser sessions. Both are
199SSH-only and refused to non-admins, like =audit=.
200
201Promotion needs an active account: a pending or disabled one is refused.
202Demotion is refused when it would leave no admin, over SSH and on the
203host alike, so the host-local =promote= is the way back in when the only
204admin key is lost.
205
206Instance admin carries no right on anyone's repository: policy does not
207consult it, and a private repository still answers not-found to an
208admin. Moderation goes through explicit overrides that skip the access
209check and write their own audit row:
210
211#+begin_src sh
212ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
213ssh git@<host> admin repo archive|unarchive <owner/name>
214ssh git@<host> admin repo visibility <owner/name> public|private
215ssh git@<host> admin repo delete <owner/name> --yes
216#+end_src
217
218Each lands in the audit log as =admin repo.<action>= naming the
219repository, on top of the =cmd= row every mutating command gets.
220
221=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
222per minute — successful auths never count and clear the slate.
223=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
224push.
225
226=limits.write_rate= (60) bounds *mutating commands per account per
227minute*. It is counted in the dispatcher, so SSH, the JSON API and the
228web spend one budget and a caller cannot refresh it by changing surface;
229=limits.api_rate= stays in front of it, bounding a network source rather
230than an account. A command is one token whatever it writes, so a bundle
231import costs one and only a loop of separate commands spends the budget.
232Read-only commands, the runner protocol (a build streams its log in many
233small writes) and the host CLI are exempt. Refusals exit 4 and say when
234to retry. A negative value turns the limit off; it matters most with
235=registration = "open"=, where every write also queues notification mail
236and webhook deliveries.
237
238* Queues
239
240Every background worker keeps a backlog and a failure state. An instance
241admin reads them all in one place:
242
243#+begin_src sh
244gitbay dashboard --json | jq .queues   # webhooks, mail, mirrors, builds, deps
245#+end_src
246
247Per worker: pending, retrying (pending with a failed attempt) and
248dead-lettered counts with the oldest pending age, and the retrying or
249failed rows themselves, capped at twenty each. Builds list what is
250running and then what is pending, each since when, so a build no runner
251is scoped to claim is visible here rather than only in its repository;
252mirrors list the ones whose last sync failed;
253dependency checks list the ones whose last check errored. Non-admins get
254no =queues= key at all.
255
256In accounts mode the same read renders at =/admin=, linked from the rail
257for admins. Anyone else gets a 404 there.
258
259A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
260with the address redacted out of the relay's error. The id is the queue
261row: find it in the Mail table on =/admin=, or in =dashboard --json=,
262where the recipient and the unredacted error are. That is deliberate —
263see the Threat-Model page.
264
265* Maintenance
266
267#+begin_src sh
268gitbayd admin stats [--json]         # counts, database size, per-repo disk
269ssh git@<host> admin stats [--json]  # the same, from an admin session
270gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
271gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
272gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)
273#+end_src
274
275=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
276weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
277layout the source sent, so a first manual =admin gc= after a bulk
278import is worthwhile.
279
280* Backup and restore
281
282#+begin_src sh
283gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
284gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back
285#+end_src
286
287One archive: a consistent SQLite snapshot (taken *before* the
288repositories are read, so the database never references objects the
289archive missed), every repository, and the SSH host keys. Excluded:
290hook socket, regenerated hook scripts, WAL files. Safe to run against a
291live daemon.
292
293=--verify= reads an archive back: the snapshot must pass SQLite's
294integrity check, and every repository the snapshot names must be in the
295archive. A database-only archive is checked for integrity and says so.
296Exit is non-zero on damage or a missing repository.
297
298Restore: extract into an empty directory, point =server.root= at it,
299start gitbayd. Host keys are preserved, so clients keep their
300known_hosts entries; hooks regenerate at startup.
301
302** Schedule and recovery point
303
304Two timers, because the two halves of the data have different exposure.
305
306- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
307- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
308  writes the SQLite snapshot alone, last 48 kept. A few MB against the
309  full archive's hundreds, which is what makes the frequency affordable.
310
311The split follows what a loss would actually cost. Repositories are git,
312so a mirror or any clone is a second copy; the database is the only copy
313of issues, merge requests, comments and review state. So the recovery
314point is about an hour for the data that exists nowhere else, and a day
315for the data that does.
316
317Continuous replication (litestream and similar) was considered and not
318adopted. It would take the database's recovery point to seconds, but the
319repositories would still be on the nightly archive, so a restore could
320produce a database referencing commits the repository backup does not
321have. Consistency between the two halves is worth more here than latency
322on one of them. Revisit if repository replication becomes continuous
323too.
324
325* Upgrades
326
327Replace the binary, restart the unit. Migrations apply automatically and
328are transactional; hook scripts under =<root>/hooks= are rewritten at
329startup to point at the current binary path.
330
331* CI runner
332
333=gitbay-runner= executes builds queued by pushes and merge requests. It
334polls over SSH with a key of scope =runner=, which reaches only the
335runner protocol and read-only git (a runner executes arbitrary
336repository code, so the key it holds must not do more). A runner key
337claims builds only for the repositories it is attached to, by =repo
338runner add= from a repository admin or an instance admin; an admin key
339claims any. Users attach their own runners: see the Users page. For an
340instance runner, run it as a dedicated unprivileged user on a non-admin
341account. =admin user create --key= registers a full-scope key, so the
342runner key is added afterwards through a bootstrap key that is then
343removed, and attached to each repository it should build:
344
345#+begin_src sh
346useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
347sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
348ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
349gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
350ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
351ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
352rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
353gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
354#+end_src
355
356#+begin_src sh
357gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
358#+end_src
359
360=-jobs N= runs N builds at once. Claiming is one transaction that
361selects and updates, and each build works in its own =build-<id>=
362directory, so workers do not collide; idle polls are staggered across
363the interval so N of them do not wake together. The drop-in's weights
364below are per service, not per build, so raising =-jobs= divides them
365rather than multiplying the host's load.
366
367=admin runners= shows which account each runner polls as, and what each
368may claim. A runner key claims builds only for the repositories it is
369attached to: with none attached it claims nothing, and =-repos= may only
370narrow within them. An admin's full-scope key claims any repository —
371that is what =-repos= was for — and still works for the protocol during
372a rotation. A merge request head from a fork is built in the target
373repository as untrusted: the claim carries no secrets, and only a runner
374started with =-untrusted= takes it. Same-repository heads were built by
375their branch push and are not built again.
376
377=make deploy-runner= also installs
378=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
379=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
380the daemon or the backup timers, and =NoNewPrivileges=,
381=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
382and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
383and the runner's home. The e2e suite alone starts sixty
384daemon instances; without the drop-in a deploy's copy over the admin
385sshd stalled. Both deploy targets copy with =rsync --partial=, which
386resumes a stalled transfer.
387
388Under =-isolation podman=, the default and what bay1 runs, each build
389is confined to a container (see Container isolation below). Under
390=-isolation none= steps run directly on the host as the runner's user,
391so treat that machine as executing whatever your users push, and
392install the toolchains your builds need on it.
393
394A runner claims the oldest pending build among the repositories its key
395is attached to — for an admin key, the oldest in the instance. =-repos=
396narrows within that set, which is what makes a runner outside the server
397practical: one on a machine that should build a single project, or that
398holds credentials for one deployment, stays on it.
399
400Oldest-first is across everything the key may claim, so a repository
401with a deep queue holds every other repository the same runner serves;
402bay1 measured a 15-minute average wait on a day of merge request
403stacks from one repository. A runner attached to one repository cannot
404be starved. That is the rule, decided in krz/gitbay#207: a runner
405serving several repositories takes them oldest-first, and an operator
406who wants one repository never to wait on another runs a second
407runner attached to it alone. Nothing caps what an account queues,
408and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
409while the job's last build is pending or running, so a repository
410with no runner holds one row per scheduled job rather than one per
411tick, and a build runs only on a runner its owner attaches, so a busy
412schedule spends the owner's compute. Pushes are bounded by what an
413account can push.
414
415#+begin_src sh
416gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
417  -workdir /var/lib/gitbay-runner/work
418#+end_src
419
420Add =-untrusted= only with =-isolation podman=.
421
422gitbay.org's runner is attached to the forge's own repositories and
423the isolation canary, nothing else, because it shares the host with
424the forge; its unit names no =-repos=, the attachments are the
425boundary. Any other repository builds on a runner its owner attaches.
426
427=-repos= narrows an admin runner; for a runner key the attachments are
428the boundary, held by the server, and =-repos= may only name
429repositories among them. =-untrusted= makes a runner claim merge
430request heads from forks; the bay1 unit sets it because it isolates in
431podman. A runner without it builds trusted commits only.
432
433=gitbay dashboard= and =ssh git@<host> admin runners= list every key
434that has polled as a runner: the account, the key's fingerprint, when it
435last polled, the repositories it may claim — its attachments for a runner
436key, the =-repos= it asked for or =any= for an admin key — and the build
437it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
438that polled by mistake, the key itself untouched. =admin runners= also
439heads the list with the queue: builds
440pending now, and over the last day how many were claimed, how long they
441waited to be claimed (average and worst), and
442how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
443reported is failed by the scheduler's minute tick, whether or not any
444runner is still alive: within about two minutes of its log stream ending
445with no outcome reported — the runner reports right after closing the
446stream, retrying for half a minute if gitbayd is unreachable — or, if no
447stream was ever seen, at the deadline.
448
449Instance admin on the runner account only authorizes the claim/report
450protocol; it grants no repo access. A build that pushes back — a pages
451deploy, an archive publish, an automated MR branch — needs an explicit
452grant on that repo: =repo access grant <owner/name> ci write=. Private
453repos likewise need at least read for the clone.
454
455** Container isolation
456
457Builds run in a rootless podman container, one per job, with the
458workspace bind mounted and nothing else: the clone happens outside with
459the runner's key, so a step cannot read it. =-isolation none= keeps the
460old behaviour — steps on the host as the runner's user — for an instance
461where every repository is trusted. There is no automatic fallback: a
462runner started with =-isolation podman= that cannot find a working
463podman exits rather than running a build unsandboxed.
464
465gitbay's own jobs name =localhost/gitbay-ci:2=, built from
466=deploy/Containerfile.ci= on the runner host. A job's image must carry
467what its steps need: the suite drives real git, git-lfs, gpg and sshd and
468asserts they exist before running, so the stock runner default would fail
469it immediately. Build or rebuild it with:
470
471#+begin_src sh
472ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
473ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
474  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
475#+end_src
476
477The tag is deliberate rather than =:latest=: changing the file means
478bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
479change under it.
480
481=-image= names the image a job runs in when it declares none, and is
482required under =-isolation podman=: there is no built-in default,
483because an image this host does not have would fail every build. A job
484overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
485so a config file cannot turn it into podman arguments.
486
487=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
488-memory 4g=); unset means uncapped. The runner applies them itself: it
489creates a cgroup per build under its own delegated service cgroup,
490writes the limits there, and starts every podman process for the build
491inside it, with podman's cgroup handling off. Podman's own =--memory=
492and =--cpus= never applied under rootless cgroupfs, which is what a
493system service gets (krz/gitbay#188). The unit therefore needs
494=Delegate=yes=, which the drop-in sets; without it the runner refuses
495to start when a limit is set, and logs that builds run unconfined when
496none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
497and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
498swap: the memory cap is what keeps the forge alive when a build
499allocates without bound, and it sits above the e2e suite's 5GB peak
500rather than at a fair share. =OOMPolicy=continue= keeps systemd from
501stopping the runner when a build is OOM-killed.
502
503Each repository gets its own build home under the runner's workdir,
504mounted into its containers as =HOME=. Caches persist between builds of
505one repository and are never read by another's.
506
507*Images are provisioned, never pulled by a build.* The runner passes
508=--pull=never=. Two reasons, and the second is the better one: the
509service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
510holding a setuid file, which is nearly every distribution image; and on
511an instance where anyone can push a =ci.yml=, =image:= would otherwise
512mean "fetch and run anything from the internet". An operator pulls or
513builds what is allowed and a build picks among those. A job naming an
514image the host does not have fails with a message saying so.
515
516#+begin_src sh
517su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
518su - ci-runner -s /bin/sh -c "podman images"
519#+end_src
520
521Prepare a host before pointing an isolating runner at it:
522
523#+begin_src sh
524ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
525make deploy-runner
526#+end_src
527
528The script installs podman, delegates a subuid/subgid range to
529=ci-runner=, checks that user namespaces are enabled rather than
530assuming, enables lingering, and verifies rootless podman actually runs
531as that user. It is idempotent.
532
533The drop-in sets =NoNewPrivileges=no=, without which rootless podman
534cannot call =newuidmap= and the runner refuses to start. That is a
535considered trade, explained in the file and in the Threat-Model; if you
536run with =-isolation none=, set it back to =yes=.
537
538*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
539the build in flight, reports it, and exits; the drop-in's
540=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
541is what makes the signal reach the runner alone — under systemd's default
542the build's container and the log session are signalled with it, and the
543runner drains a build that is already dead. So =make deploy-runner=
544waits for a running build rather than orphaning it, and a build's result
545is retried for half a minute if gitbayd is restarting at that moment. A
546second SIGTERM ends the runner at once, abandoning the build to the
547reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
548the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
549=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
550the runner as a transient user unit under =systemd-run= and stops it
551under both kill modes. That last one needs a systemd user manager, so it
552skips in the container CI runs in; run it on a Linux host with
553=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
554
555*Validate podman mode on a scratch repository before pointing the runner
556at real ones.* Every deploy that switched the whole instance to
557containers and failed took CI down with it. Instead: create a throwaway
558repository the runner account can read (public, or granted read — a
559private one is "not found" to the runner and the build stays pending),
560give it one job that names the CI image, and deploy the runner with
561=-repos= naming only that repository. The production unit, with its real
562hardening, then claims nothing else; other repositories' builds queue
563until =-repos= is switched back, which is a pause, not an outage.
564
565#+begin_src sh
566gitbay repo create cmc/ci-smoke          # then push a .gitbay/ci.yml naming the image
567sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
568systemctl daemon-reload && systemctl restart gitbay-runner
569gitbay build log cmc/ci-smoke 1         # green: switch -repos back, redeploy
570#+end_src
571
572*Do not deploy an isolating runner to a host that has not been
573prepared.* The runner is specified to refuse to start without a working
574podman rather than fall back to running builds unsandboxed — a fallback
575that silently drops isolation is worse than a stopped runner, because
576nothing surfaces it. On an unprepared host that refusal stops every
577build on the instance.
578
579The service drop-in carries =Delegate=yes= for rootless cgroup
580management and =ReadWritePaths= for podman's store under
581=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
582make read-only. Those paths are prefixed =-= so they are ignored when
583absent: the drop-in installs on unprepared hosts too, and a unit that
584refused to start would stop every build.
585The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
586names that repository too; a scoped runner claims nothing else.
587=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
588user: rootless storage belongs to that user, and root's prune would not
589see it. An unpruned image store on a 40GB host is a slow outage.
590
591* LFS storage
592
593Objects live content-addressed under =[lfs] root= (default
594=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
595(512MB default). Storage sits behind a small interface — an
596S3-compatible backend is a drop-in with the server proxying, and
597presigned URLs a later optimization. LFS objects do not travel with
598push mirrors (mirrors move git refs only), and gc does not yet collect
599orphaned objects.
600
601* Pages
602
603=[pages] domain = "example.site"= serves public repos' =pages= branches
604on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
605server; ACME issues per-subdomain certificates on demand (only for
606owners that exist). The domain must not be the site host or a parent of
607it — pages content runs its own scripts and must stay off the forge's
608origin.
609
610Users with repo admin claim custom domains with =repo domain add=.
611Claims activate only after a DNS TXT challenge proves control of the
612domain (=repo domain verify=, audit-logged); pending claims serve
613nothing, get no certificates, and expire after 7 days. ACME issues
614certificates only for verified hosts, so stray DNS pointed at the
615server gets nothing.
616
617* Security
618
619The [[Threat-Model]] file is the reference for what the forge
620trusts and refuses to do. Operational checklist:
621
622- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
623  (the module list is deliberately short — review it on each release),
624  and a short fuzz pass over every attacker-facing parser (pkt-line,
625  commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
626  tagging a release. CI's own =vuln= job runs =govulncheck= nightly
627  against main rather than per push, because =@latest= scans today's
628  advisory database and an advisory lands without anyone pushing;
629  =build trigger krz/gitbay vuln= runs it on demand.
630- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
631  DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
632  configuration needed.
633- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
634  gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
635  =LockPersonality=, =MemoryDenyWriteExecute=,
636  =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
637  INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
638- *OS patches* apply via =unattended-upgrades= (security origins,
639  auto-reboot 04:30 if required).
640- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
641  watched by =fail2ban=; gitbayd's own port 22 is throttled by
642  =limits.ssh_auth_rate= (auth failures per IP per minute), and every
643  account's writes by =limits.write_rate=.
644- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
645journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
646webhook: disk, service, the daemon's own =/healthz= answer, certificate
647expiry, and the age of the newest full backup and database snapshot.
648It exits non-zero on an alert so the unit shows in =systemctl
649--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
650a certificate under 21 days, a full backup over 25 hours old, or a
651database snapshot over 2 hours old.
652
653=GET /healthz= is unauthenticated and cache-free: whether the database
654answers and which commit serves, 503 when it does not.
655- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
656  0750, owned by =gitbay=). The nightly archive plus provider snapshots
657  are the recovery path; for tighter RPO, add continuous replication
658  (litestream) against the same file — it coexists with the WAL.
659
660* Odds and ends
661
662- deleting a fork marks MRs sourced from it =source_gone=; their diffs
663  remain viewable and mergeable because the target repo owns the
664  objects.
665- =refs/merge-requests/*= is server-owned and unpushable by clients.
666- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
667  the =events= table, which also feeds webhooks.
668- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.