.gitbay/wiki/Admin.org
666 lines · 33471 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21#+end_src
22
23=deploy/= in the source tree has a cloud-init file, a hardened systemd
24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
2622/80/443 without root.
27
28** The SSH port decision
29
30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
31 22 — move the host's admin sshd to another port. Remotes read
32 =git@host:owner/repo= with no port gymnastics.
33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
34 =AuthorizedKeysCommand=:
35 #+begin_example
36 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
37 AuthorizedKeysCommandUser gitbay
38 #+end_example
39 sshd requires that binary to be root-owned and not group/world
40 writable. Unknown keys fail authentication inside sshd, so system mode
41 requires =registration.mode = "closed"= (check-config enforces this).
42
43* Configuration reference
44
45=/etc/gitbay/config.toml=. =check-config= validates and names every
46contradiction; =--no-host-checks= skips port/path probes.
47=gitbayd admin config show= prints the configuration in effect as TOML,
48every default filled in and =smtp_pass= redacted. A file that fails
49validation still prints, followed by the contradiction.
50
51** [server]
52- =root= (default =/var/lib/gitbay=) — repositories, database, host
53 keys, ACME cache all live here.
54- =site_url= (required) — canonical =https://host=; drives ACME, clone
55 URLs, mail links.
56- =source_repo= (optional, =owner/name=) — the repository this instance
57 develops itself in. Startup warns when the running build's commit is
58 not on that repository's default branch, which is how a binary built
59 from an unmerged branch stops being invisible. Leave it unset unless
60 the instance hosts its own source.
61
62** [ssh]
63- =mode= — =embedded= | =system= (above).
64- =port= (22) — embedded listener port.
65- =host_keys= — list of private key paths; empty generates an ed25519
66 key at =<root>/ssh/host_ed25519=.
67
68** [http]
69- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
70- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
71 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
72 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
73 failing to bind it is a warning, not fatal. Requires an =https://=
74 site_url with a public DNS name.
75- =files=: =cert_file= + =key_file=.
76- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
77
78=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
79front of the daemon. A request from one of them is attributed, for API
80rate limiting, to the last =X-Forwarded-For= hop that is not itself a
81trusted proxy; from anyone else the header is ignored. Empty, the
82default, is right when gitbayd terminates TLS itself.
83
84** [web]
85- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
86 web routes are never registered; in accounts, browser sessions are
87 minted over SSH (=web login=), and users with write access can create
88 repos, comment, and make simple file edits (which commit unsigned,
89 honestly). =password_auth= is reserved and currently rejected.
90
91** [registration]
92- =mode= — =closed= (default) | =invite= | =open=. invite/open require
93 [mail]. See the user guide for the flows.
94
95- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
96 self-registered account still unverified after that long is removed,
97 hourly and at start, audited as =pending.expired=.
98
99** [mail]
100- =smtp_host= (host:port, 587 assumed), =from=, optional =smtp_user= /
101 =smtp_pass=. STARTTLS when offered. Required for invite/open
102 registration and self-service =email add=; in closed mode you may omit
103 it entirely and assert addresses by hand (below).
104
105** [api]
106- =enabled= (false) — the JSON API surface; see [[API]]. Off
107 means no credential-bearing HTTP endpoint exists at all.
108
109** [webhooks]
110- =allow_local= (false) — permit webhook targets on loopback/private
111 addresses. Leave off unless you know why you need it (SSRF).
112
113** [limits]
114- =clone_timeout= (3600s) — cap on =repo import= fetches.
115- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
116- =max_asset_bytes= (512MB) — cap per uploaded release asset.
117- =max_repos_per_user= (0, unlimited) — repositories an account may own
118 directly; =repo create=, =fork= and =import= refuse past it.
119 Organizations are not capped.
120- =max_bytes_per_user= (0, unlimited) — disk the account's own
121 repositories may take; a push may be no larger than what is left.
122- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
123
124** [git_daemon]
125- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
126 Serves only public repositories that additionally ran
127 =repo settings git-daemon <repo> on=.
128
129** [mirrors]
130- =pull_interval_minutes= (15) — how often pull mirrors fetch their
131 upstream. Push mirrors sync shortly after each local ref update.
132 Mirror URLs pass the same SSRF rules as webhook targets.
133
134** [go_import]
135Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
136Requests with =?go-get=1= at or under the module path answer with the
137go-import meta tag pointing at the repository's HTTPS clone URL, so
138=go install host/module/cmd/...@latest= resolves. The repository should
139be public (the module path itself confirms it exists).
140
141* Users, email, invites
142
143Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
144backfills is a wrapper that dispatches the registry command of the same
145name as the host: an admin context with no account behind it, so its
146audit rows carry no actor and =source: host=. The same commands run in an
147instance admin's SSH session (=ssh git@<host> admin ...=) and write the
148same rows with the key fingerprint as source. One implementation, two
149credentials.
150
151#+begin_src sh
152gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
153gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
154gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
155#+end_src
156
157"Verified" means SMTP-confirmed or host-admin-asserted; the database
158records which. Verified emails are what make commit signatures
159meaningful — an unverified address never produces a =verified= badge.
160
161* Audit and account control
162
163The audit log is the security feed (events are the product feed): every
164successful mutating command with its argv and source credential (SSH key
165fingerprint or API), registrations, admin actions, force-pushes, and
166auth failures/throttling. Secrets never appear — they travel on stdin,
167never in argv.
168
169#+begin_src sh
170gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
171ssh git@<host> audit ... # the same, from an admin session (SSH only)
172ssh git@<host> admin user list [--state active|pending|disabled|admin]
173ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
174ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
175ssh git@<host> admin user promote <name> # grant instance admin
176ssh git@<host> admin user demote <name> # remove it; the last admin is refused
177gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
178gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
179gitbayd admin user enable <name> # sessions dropped, nothing deleted
180gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
181 # refused (with each blocker named) while
182 # the account owns repos, authored
183 # issues/MRs/comments/reviews, or is an
184 # org's only admin
185#+end_src
186
187=--actor= takes a username, or =-= for rows with no actor: host commands
188and auth failures. =--action= is a prefix, so =cmd repo= catches every
189repository command and =admin= every host or admin-session action.
190=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
191
192=admin user list= pages by username (=--limit=, =--cursor=) and carries
193each account's state and =last_seen=, the newest use of any of its SSH
194keys or API tokens. =admin user show= adds the keys with their last use,
195each address with how it was verified, PGP keys, org roles, the owned
196repository count, API token names, and live browser sessions. Both are
197SSH-only and refused to non-admins, like =audit=.
198
199Promotion needs an active account: a pending or disabled one is refused.
200Demotion is refused when it would leave no admin, over SSH and on the
201host alike, so the host-local =promote= is the way back in when the only
202admin key is lost.
203
204Instance admin carries no right on anyone's repository: policy does not
205consult it, and a private repository still answers not-found to an
206admin. Moderation goes through explicit overrides that skip the access
207check and write their own audit row:
208
209#+begin_src sh
210ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
211ssh git@<host> admin repo archive|unarchive <owner/name>
212ssh git@<host> admin repo visibility <owner/name> public|private
213ssh git@<host> admin repo delete <owner/name> --yes
214#+end_src
215
216Each lands in the audit log as =admin repo.<action>= naming the
217repository, on top of the =cmd= row every mutating command gets.
218
219=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
220per minute — successful auths never count and clear the slate.
221=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
222push.
223
224=limits.write_rate= (60) bounds *mutating commands per account per
225minute*. It is counted in the dispatcher, so SSH, the JSON API and the
226web spend one budget and a caller cannot refresh it by changing surface;
227=limits.api_rate= stays in front of it, bounding a network source rather
228than an account. A command is one token whatever it writes, so a bundle
229import costs one and only a loop of separate commands spends the budget.
230Read-only commands, the runner protocol (a build streams its log in many
231small writes) and the host CLI are exempt. Refusals exit 4 and say when
232to retry. A negative value turns the limit off; it matters most with
233=registration = "open"=, where every write also queues notification mail
234and webhook deliveries.
235
236* Queues
237
238Every background worker keeps a backlog and a failure state. An instance
239admin reads them all in one place:
240
241#+begin_src sh
242gitbay dashboard --json | jq .queues # webhooks, mail, mirrors, builds, deps
243#+end_src
244
245Per worker: pending, retrying (pending with a failed attempt) and
246dead-lettered counts with the oldest pending age, and the retrying or
247failed rows themselves, capped at twenty each. Builds list what is
248running and then what is pending, each since when, so a build no runner
249is scoped to claim is visible here rather than only in its repository;
250mirrors list the ones whose last sync failed;
251dependency checks list the ones whose last check errored. Non-admins get
252no =queues= key at all.
253
254In accounts mode the same read renders at =/admin=, linked from the rail
255for admins. Anyone else gets a 404 there.
256
257A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
258with the address redacted out of the relay's error. The id is the queue
259row: find it in the Mail table on =/admin=, or in =dashboard --json=,
260where the recipient and the unredacted error are. That is deliberate —
261see the Threat-Model page.
262
263* Maintenance
264
265#+begin_src sh
266gitbayd admin stats [--json] # counts, database size, per-repo disk
267ssh git@<host> admin stats [--json] # the same, from an admin session
268gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
269gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
270gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
271#+end_src
272
273=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
274weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
275layout the source sent, so a first manual =admin gc= after a bulk
276import is worthwhile.
277
278* Backup and restore
279
280#+begin_src sh
281gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
282gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
283#+end_src
284
285One archive: a consistent SQLite snapshot (taken *before* the
286repositories are read, so the database never references objects the
287archive missed), every repository, and the SSH host keys. Excluded:
288hook socket, regenerated hook scripts, WAL files. Safe to run against a
289live daemon.
290
291=--verify= reads an archive back: the snapshot must pass SQLite's
292integrity check, and every repository the snapshot names must be in the
293archive. A database-only archive is checked for integrity and says so.
294Exit is non-zero on damage or a missing repository.
295
296Restore: extract into an empty directory, point =server.root= at it,
297start gitbayd. Host keys are preserved, so clients keep their
298known_hosts entries; hooks regenerate at startup.
299
300** Schedule and recovery point
301
302Two timers, because the two halves of the data have different exposure.
303
304- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
305- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
306 writes the SQLite snapshot alone, last 48 kept. A few MB against the
307 full archive's hundreds, which is what makes the frequency affordable.
308
309The split follows what a loss would actually cost. Repositories are git,
310so a mirror or any clone is a second copy; the database is the only copy
311of issues, merge requests, comments and review state. So the recovery
312point is about an hour for the data that exists nowhere else, and a day
313for the data that does.
314
315Continuous replication (litestream and similar) was considered and not
316adopted. It would take the database's recovery point to seconds, but the
317repositories would still be on the nightly archive, so a restore could
318produce a database referencing commits the repository backup does not
319have. Consistency between the two halves is worth more here than latency
320on one of them. Revisit if repository replication becomes continuous
321too.
322
323* Upgrades
324
325Replace the binary, restart the unit. Migrations apply automatically and
326are transactional; hook scripts under =<root>/hooks= are rewritten at
327startup to point at the current binary path.
328
329* CI runner
330
331=gitbay-runner= executes builds queued by pushes and merge requests. It
332polls over SSH with a key of scope =runner=, which reaches only the
333runner protocol and read-only git (a runner executes arbitrary
334repository code, so the key it holds must not do more). A runner key
335claims builds only for the repositories it is attached to, by =repo
336runner add= from a repository admin or an instance admin; an admin key
337claims any. Users attach their own runners: see the Users page. For an
338instance runner, run it as a dedicated unprivileged user on a non-admin
339account. =admin user create --key= registers a full-scope key, so the
340runner key is added afterwards through a bootstrap key that is then
341removed, and attached to each repository it should build:
342
343#+begin_src sh
344useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
345sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
346ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
347gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
348ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
349ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
350rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
351gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
352#+end_src
353
354#+begin_src sh
355gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
356#+end_src
357
358=-jobs N= runs N builds at once. Claiming is one transaction that
359selects and updates, and each build works in its own =build-<id>=
360directory, so workers do not collide; idle polls are staggered across
361the interval so N of them do not wake together. The drop-in's weights
362below are per service, not per build, so raising =-jobs= divides them
363rather than multiplying the host's load.
364
365=admin runners= shows which account each runner polls as, and what each
366may claim. A runner key claims builds only for the repositories it is
367attached to: with none attached it claims nothing, and =-repos= may only
368narrow within them. An admin's full-scope key claims any repository —
369that is what =-repos= was for — and still works for the protocol during
370a rotation. A merge request head from a fork is built in the target
371repository as untrusted: the claim carries no secrets, and only a runner
372started with =-untrusted= takes it. Same-repository heads were built by
373their branch push and are not built again.
374
375=make deploy-runner= also installs
376=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
377=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
378the daemon or the backup timers, and =NoNewPrivileges=,
379=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
380and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
381and the runner's home. The e2e suite alone starts sixty
382daemon instances; without the drop-in a deploy's copy over the admin
383sshd stalled. Both deploy targets copy with =rsync --partial=, which
384resumes a stalled transfer.
385
386Under =-isolation podman=, the default and what bay1 runs, each build
387is confined to a container (see Container isolation below). Under
388=-isolation none= steps run directly on the host as the runner's user,
389so treat that machine as executing whatever your users push, and
390install the toolchains your builds need on it.
391
392A runner claims the oldest pending build among the repositories its key
393is attached to — for an admin key, the oldest in the instance. =-repos=
394narrows within that set, which is what makes a runner outside the server
395practical: one on a machine that should build a single project, or that
396holds credentials for one deployment, stays on it.
397
398Oldest-first is across everything the key may claim, so a repository
399with a deep queue holds every other repository the same runner serves;
400bay1 measured a 15-minute average wait on a day of merge request
401stacks from one repository. A runner attached to one repository cannot
402be starved. That is the rule, decided in krz/gitbay#207: a runner
403serving several repositories takes them oldest-first, and an operator
404who wants one repository never to wait on another runs a second
405runner attached to it alone. Nothing caps what an account queues,
406and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
407while the job's last build is pending or running, so a repository
408with no runner holds one row per scheduled job rather than one per
409tick, and a build runs only on a runner its owner attaches, so a busy
410schedule spends the owner's compute. Pushes are bounded by what an
411account can push.
412
413#+begin_src sh
414gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
415 -workdir /var/lib/gitbay-runner/work
416#+end_src
417
418Add =-untrusted= only with =-isolation podman=.
419
420gitbay.org's runner is attached to the forge's own repositories and
421the isolation canary, nothing else, because it shares the host with
422the forge; its unit names no =-repos=, the attachments are the
423boundary. Any other repository builds on a runner its owner attaches.
424
425=-repos= narrows an admin runner; for a runner key the attachments are
426the boundary, held by the server, and =-repos= may only name
427repositories among them. =-untrusted= makes a runner claim merge
428request heads from forks; the bay1 unit sets it because it isolates in
429podman. A runner without it builds trusted commits only.
430
431=gitbay dashboard= and =ssh git@<host> admin runners= list every key
432that has polled as a runner: the account, the key's fingerprint, when it
433last polled, the repositories it may claim — its attachments for a runner
434key, the =-repos= it asked for or =any= for an admin key — and the build
435it holds; =admin runners forget <fingerprint>= drops the row for a key
436that polled by mistake, the key itself untouched. =admin runners= also
437heads the list with the queue: builds
438pending now, and over the last day how many were claimed, how long they
439waited to be claimed (average and worst), and
440how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
441reported is failed by the scheduler's minute tick, whether or not any
442runner is still alive: within about two minutes of its log stream ending
443with no outcome reported — the runner reports right after closing the
444stream, retrying for half a minute if gitbayd is unreachable — or, if no
445stream was ever seen, at the deadline.
446
447Instance admin on the runner account only authorizes the claim/report
448protocol; it grants no repo access. A build that pushes back — a pages
449deploy, an archive publish, an automated MR branch — needs an explicit
450grant on that repo: =repo access grant <owner/name> ci write=. Private
451repos likewise need at least read for the clone.
452
453** Container isolation
454
455Builds run in a rootless podman container, one per job, with the
456workspace bind mounted and nothing else: the clone happens outside with
457the runner's key, so a step cannot read it. =-isolation none= keeps the
458old behaviour — steps on the host as the runner's user — for an instance
459where every repository is trusted. There is no automatic fallback: a
460runner started with =-isolation podman= that cannot find a working
461podman exits rather than running a build unsandboxed.
462
463gitbay's own jobs name =localhost/gitbay-ci:2=, built from
464=deploy/Containerfile.ci= on the runner host. A job's image must carry
465what its steps need: the suite drives real git, git-lfs, gpg and sshd and
466asserts they exist before running, so the stock runner default would fail
467it immediately. Build or rebuild it with:
468
469#+begin_src sh
470ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
471ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
472 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
473#+end_src
474
475The tag is deliberate rather than =:latest=: changing the file means
476bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
477change under it.
478
479=-image= names the image a job runs in when it declares none, and is
480required under =-isolation podman=: there is no built-in default,
481because an image this host does not have would fail every build. A job
482overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
483so a config file cannot turn it into podman arguments.
484
485=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
486-memory 4g=); unset means uncapped. The runner applies them itself: it
487creates a cgroup per build under its own delegated service cgroup,
488writes the limits there, and starts every podman process for the build
489inside it, with podman's cgroup handling off. Podman's own =--memory=
490and =--cpus= never applied under rootless cgroupfs, which is what a
491system service gets (krz/gitbay#188). The unit therefore needs
492=Delegate=yes=, which the drop-in sets; without it the runner refuses
493to start when a limit is set, and logs that builds run unconfined when
494none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
495and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
496swap: the memory cap is what keeps the forge alive when a build
497allocates without bound, and it sits above the e2e suite's 5GB peak
498rather than at a fair share. =OOMPolicy=continue= keeps systemd from
499stopping the runner when a build is OOM-killed.
500
501Each repository gets its own build home under the runner's workdir,
502mounted into its containers as =HOME=. Caches persist between builds of
503one repository and are never read by another's.
504
505*Images are provisioned, never pulled by a build.* The runner passes
506=--pull=never=. Two reasons, and the second is the better one: the
507service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
508holding a setuid file, which is nearly every distribution image; and on
509an instance where anyone can push a =ci.yml=, =image:= would otherwise
510mean "fetch and run anything from the internet". An operator pulls or
511builds what is allowed and a build picks among those. A job naming an
512image the host does not have fails with a message saying so.
513
514#+begin_src sh
515su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
516su - ci-runner -s /bin/sh -c "podman images"
517#+end_src
518
519Prepare a host before pointing an isolating runner at it:
520
521#+begin_src sh
522ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
523make deploy-runner
524#+end_src
525
526The script installs podman, delegates a subuid/subgid range to
527=ci-runner=, checks that user namespaces are enabled rather than
528assuming, enables lingering, and verifies rootless podman actually runs
529as that user. It is idempotent.
530
531The drop-in sets =NoNewPrivileges=no=, without which rootless podman
532cannot call =newuidmap= and the runner refuses to start. That is a
533considered trade, explained in the file and in the Threat-Model; if you
534run with =-isolation none=, set it back to =yes=.
535
536*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
537the build in flight, reports it, and exits; the drop-in's
538=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
539is what makes the signal reach the runner alone — under systemd's default
540the build's container and the log session are signalled with it, and the
541runner drains a build that is already dead. So =make deploy-runner=
542waits for a running build rather than orphaning it, and a build's result
543is retried for half a minute if gitbayd is restarting at that moment. A
544second SIGTERM ends the runner at once, abandoning the build to the
545reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
546the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
547=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
548the runner as a transient user unit under =systemd-run= and stops it
549under both kill modes. That last one needs a systemd user manager, so it
550skips in the container CI runs in; run it on a Linux host with
551=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
552
553*Validate podman mode on a scratch repository before pointing the runner
554at real ones.* Every deploy that switched the whole instance to
555containers and failed took CI down with it. Instead: create a throwaway
556repository the runner account can read (public, or granted read — a
557private one is "not found" to the runner and the build stays pending),
558give it one job that names the CI image, and deploy the runner with
559=-repos= naming only that repository. The production unit, with its real
560hardening, then claims nothing else; other repositories' builds queue
561until =-repos= is switched back, which is a pause, not an outage.
562
563#+begin_src sh
564gitbay repo create cmc/ci-smoke # then push a .gitbay/ci.yml naming the image
565sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
566systemctl daemon-reload && systemctl restart gitbay-runner
567gitbay build log cmc/ci-smoke 1 # green: switch -repos back, redeploy
568#+end_src
569
570*Do not deploy an isolating runner to a host that has not been
571prepared.* The runner is specified to refuse to start without a working
572podman rather than fall back to running builds unsandboxed — a fallback
573that silently drops isolation is worse than a stopped runner, because
574nothing surfaces it. On an unprepared host that refusal stops every
575build on the instance.
576
577The service drop-in carries =Delegate=yes= for rootless cgroup
578management and =ReadWritePaths= for podman's store under
579=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
580make read-only. Those paths are prefixed =-= so they are ignored when
581absent: the drop-in installs on unprepared hosts too, and a unit that
582refused to start would stop every build.
583The nightly canary on =cmc/ci-smoke= only runs if the runner's =-repos=
584names that repository too; a scoped runner claims nothing else.
585=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
586user: rootless storage belongs to that user, and root's prune would not
587see it. An unpruned image store on a 40GB host is a slow outage.
588
589* LFS storage
590
591Objects live content-addressed under =[lfs] root= (default
592=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
593(512MB default). Storage sits behind a small interface — an
594S3-compatible backend is a drop-in with the server proxying, and
595presigned URLs a later optimization. LFS objects do not travel with
596push mirrors (mirrors move git refs only), and gc does not yet collect
597orphaned objects.
598
599* Pages
600
601=[pages] domain = "example.site"= serves public repos' =pages= branches
602on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
603server; ACME issues per-subdomain certificates on demand (only for
604owners that exist). The domain must not be the site host or a parent of
605it — pages content runs its own scripts and must stay off the forge's
606origin.
607
608Users with repo admin claim custom domains with =repo domain add=.
609Claims activate only after a DNS TXT challenge proves control of the
610domain (=repo domain verify=, audit-logged); pending claims serve
611nothing, get no certificates, and expire after 7 days. ACME issues
612certificates only for verified hosts, so stray DNS pointed at the
613server gets nothing.
614
615* Security
616
617The [[Threat-Model]] file is the reference for what the forge
618trusts and refuses to do. Operational checklist:
619
620- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
621 (the module list is deliberately short — review it on each release),
622 and a short fuzz pass over every attacker-facing parser (pkt-line,
623 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
624 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
625 against main rather than per push, because =@latest= scans today's
626 advisory database and an advisory lands without anyone pushing;
627 =build trigger krz/gitbay vuln= runs it on demand.
628- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
629 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
630 configuration needed.
631- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
632 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
633 =LockPersonality=, =MemoryDenyWriteExecute=,
634 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
635 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
636- *OS patches* apply via =unattended-upgrades= (security origins,
637 auto-reboot 04:30 if required).
638- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
639 watched by =fail2ban=; gitbayd's own port 22 is throttled by
640 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
641 account's writes by =limits.write_rate=.
642- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
643journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
644webhook: disk, service, the daemon's own =/healthz= answer, certificate
645expiry, and the age of the newest full backup and database snapshot.
646It exits non-zero on an alert so the unit shows in =systemctl
647--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
648a certificate under 21 days, a full backup over 25 hours old, or a
649database snapshot over 2 hours old.
650
651=GET /healthz= is unauthenticated and cache-free: whether the database
652answers and which commit serves, 503 when it does not.
653- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
654 0750, owned by =gitbay=). The nightly archive plus provider snapshots
655 are the recovery path; for tighter RPO, add continuous replication
656 (litestream) against the same file — it coexists with the WAL.
657
658* Odds and ends
659
660- deleting a fork marks MRs sourced from it =source_gone=; their diffs
661 remain viewable and mergeable because the target repo owns the
662 objects.
663- =refs/merge-requests/*= is server-owned and unpushable by clients.
664- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
665 the =events= table, which also feeds webhooks.
666- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.