.gitbay/wiki/Admin.org
1254 lines · 66567 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21gitbayd --config /etc/gitbay/config.toml admin secrets init
22chown gitbay:gitbay /etc/gitbay/secret.key
23#+end_src
24
25=admin secrets init= refuses when the key file already exists, so a
26reinstall on the same host should skip it — =deploy/install.sh= does
27this with a file check before running it.
28
29=deploy/= in the source tree has a cloud-init file, a hardened systemd
30unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
31the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
3222/80/443 without root.
33
34** The SSH port decision
35
36- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
37 22 — move the host's admin sshd to another port. Remotes read
38 =git@host:owner/repo= with no port gymnastics.
39- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
40 =AuthorizedKeysCommand=:
41 #+begin_example
42 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
43 AuthorizedKeysCommandUser gitbay
44 #+end_example
45 sshd requires that binary to be root-owned and not group/world
46 writable. Unknown keys fail authentication inside sshd, so system mode
47 requires =registration.mode = "closed"= (check-config enforces this).
48 Under the forced command the CLI's leading =--term=<cols>[,color]=
49 argument works as is; a client that sets =GITBAY_TERM= instead needs
50 =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
51 handled by the host's sshd, not gitbayd.
52
53* Configuration reference
54
55=/etc/gitbay/config.toml=. =check-config= validates and names every
56contradiction; =--no-host-checks= skips port/path probes.
57=gitbayd admin config show= prints the configuration in effect as TOML,
58every default filled in and =smtp_pass= redacted. A file that fails
59validation still prints, followed by the contradiction.
60
61** [server]
62- =root= (default =/var/lib/gitbay=) — repositories, database, host
63 keys, ACME cache all live here.
64- =site_url= (required) — canonical =https://host=; drives ACME, clone
65 URLs, mail links.
66- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
67 seal CI secrets, webhook secrets, mirror tokens and push device
68 tokens in the database. Must be outside =root=, mode 0600, readable
69 by the daemon's user. See "Secret key" below.
70- =source_repo= (optional, =owner/name=) — the repository this instance
71 develops itself in. Startup warns when the running build's commit is
72 not on that repository's default branch, which is how a binary built
73 from an unmerged branch stops being invisible. Leave it unset unless
74 the instance hosts its own source.
75
76** [ssh]
77- =mode= — =embedded= | =system= (above).
78- =port= (22) — embedded listener port.
79- =host_keys= — list of private key paths; empty generates an ed25519
80 key at =<root>/ssh/host_ed25519=.
81
82** [http]
83- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
84- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
85 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
86 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
87 failing to bind it is a warning, not fatal. Requires an =https://=
88 site_url with a public DNS name.
89- =files=: =cert_file= + =key_file=.
90- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
91- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
92 =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
93 release the binary was built with. =openssl s_client -connect
94 <host>:443 -tls1_1= fails the handshake.
95
96=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
97front of the daemon. A request from one of them is attributed, for API
98rate limiting, to the last =X-Forwarded-For= hop that is not itself a
99trusted proxy; from anyone else the header is ignored. Empty, the
100default, is right when gitbayd terminates TLS itself.
101
102A reverse proxy in front of gitbay must not buffer responses, or the
103build page's live log arrives only when the build ends;
104=X-Accel-Buffering: no= covers nginx.
105
106** [web]
107- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
108 web routes are never registered; in accounts, browser sessions are
109 minted over SSH (=web login=), and users with write access can create
110 repos, comment, and make simple file edits (which commit unsigned,
111 honestly). =password_auth= is reserved and currently rejected.
112- =title= — the instance's display name in the rail and page titles;
113 empty falls back to the site host. Lower case is the convention for
114 gitbay itself.
115- =privacy_notice= — operator text shown on =/privacy= under the fixed
116 statement.
117
118** [registration]
119- =mode= — =closed= (default) | =invite= | =open=. invite/open require
120 [mail]. See the user guide for the flows.
121
122- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
123 self-registered account still unverified after that long is removed,
124 hourly and at start, audited as =pending.expired=.
125
126- =notify_admin= (false) — mail every instance admin when an account
127 becomes active: an invite redeemed, or an open-mode signup that
128 verified its address. The unverified row an open signup creates is
129 not reported, because anyone can post the form and mailing on that
130 would point a flood at the admins. Recipients are the verified
131 primary addresses of active admins who have activity mail on, the
132 same rule any other notice follows, so an admin with no verified
133 address hears nothing. The notice is queued, so a dead SMTP host
134 shows up in the admin page's Mail table instead of failing the
135 registration. Requires [mail].
136
137** [mail]
138- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
139 =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
140 registration and self-service =email add=; in closed mode you may omit
141 it entirely and assert addresses by hand (below).
142- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
143 for relays on 465).
144- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
145 gets no mail: delivery fails and retries, and the admin page's Mail
146 table shows the error. Unset, it is on for every relay except
147 =localhost= and loopback addresses; set =false= to allow plaintext
148 to a remote relay.
149
150** [mail.inbound]
151Reply by mail: a reply to issue or merge request mail posts a comment
152(#295). gitbayd polls a mailbox over IMAP; no port is opened for it.
153Off unless =enabled=, and it requires =[mail] smtp_host=, since the
154replies answer mail gitbayd sends.
155
156#+begin_src toml
157[mail.inbound]
158enabled = true
159imap_host = "imap.example.org" # host:port; 993, or 143 with tls = "starttls"
160tls = "implicit" # or "starttls"; there is no plaintext setting
161user = "reply@gitbay.example"
162password_file = "/etc/gitbay/imap.pass" # one line, mode 0600, owned by gitbayd's user
163mailbox = "INBOX" # default
164poll_interval = "1m" # default; at least 10s
165reply_address = "reply@gitbay.example"
166trusted_authserv_id = "mx.example.org" # the mail host's Authentication-Results id
167#+end_src
168
169- The password is read from =password_file= at every connection and
170 never appears in the config, argv or the log. An unreadable file, or
171 one readable by group or others, stops the daemon at start.
172- =reply_address= is what each Reply-To is built from:
173 =reply@gitbay.example= becomes =reply+<token>@gitbay.example=. The
174 mailbox must receive that: give it a plus-addressing (subaddress)
175 mailbox, as most hosted mail does by default, or a catch-all for the
176 domain. Point the domain's MX at the mail host as for any other
177 mailbox; nothing about it involves gitbayd.
178- Use a mailbox that holds nothing else. gitbayd reads every unseen
179 message in it and marks each one =\Seen= when it is handled, posted
180 or refused. A message that fails for a reason that may pass (the
181 database busy, the server refusing that one fetch) stays unseen and is
182 tried again on the next poll, up to five times, then is marked seen
183 and audited. A dropped connection or a timeout ends the poll and
184 counts against no message. A
185 message over 10 MiB is refused by its size without being fetched. A
186 server that sends more than about 11 MiB, or more than a thousand
187 untagged responses, for one command has its connection closed; one
188 poll handles at most ten thousand messages.
189- =trusted_authserv_id= is required on any instance reachable from the
190 internet. It names the authserv-id the mail host writes at the start
191 of its =Authentication-Results= header (Gmail's is =mx.google.com=).
192 With it set, a reply is posted only when the topmost header with that
193 id shows =dmarc=pass= with =header.from= equal to the From domain, or
194 =dkim=pass= with a =header.d= in relaxed alignment with it (the same
195 organizational domain by the public suffix list; a public suffix such
196 as =github.io= aligns with nothing). The header is parsed per RFC
197 8601, so text inside a quoted string or a comment (a quoted MAIL FROM
198 local part, a reason) is never read as a result. Lower headers
199 claiming the same id are the sender's and are not read. This is only safe when the mail host
200 removes incoming =Authentication-Results= headers that claim its id,
201 as RFC 8601 asks; Gmail, Fastmail and Migadu do. Check yours before
202 relying on it. Unset, the daemon logs a warning at start and =admin
203 mail inbound check= repeats it: without it, =From= is whatever the
204 sender wrote.
205- =gitbay admin mail inbound check= logs in, opens the mailbox
206 read-only (EXAMINE) and reports the message and unseen counts, so a
207 check never marks a reply seen before the poller reads it. With
208 inbound off it says so and exits 0. Poll failures are logged as
209 =mail reply: poll failed= with the server and the IMAP error.
210
211A reply is posted when all of these hold, checked when it is read:
212
2131. It is not an automatic reply (=Auto-Submitted=, =Precedence: bulk=
214 and the like).
2152. A recipient header carries a reply address whose token verifies
216 and has not expired (thirty days from the mail it came on).
2173. The token's account exists, is active and not disabled, still has
218 reply by mail on, and was created before the token was: an id freed
219 by a delete and reused is not the account the token named. The same
220 holds for the repository.
2214. =From= is one of that account's verified addresses, and, with
222 =trusted_authserv_id= set, the mail host authenticated it.
2235. The message has a =text/plain= part (HTML-only mail is refused, not
224 converted), and what is left after quoted text and the signature
225 are removed is not empty and fits a comment (64 KiB).
2266. No earlier copy of the message (same =Message-ID=, account and
227 thread) posted.
2287. =issue comment= or =mr comment=, dispatched as the account, accepts
229 it: the account can still read the repository, the thread exists,
230 the repository is not archived, the write budget is not spent.
231
232The comment's audit row is =cmd issue comment= (or =mr comment=) with
233=source: mail=. A refusal writes a =refused mail reply= row with the
234reason and the =Message-ID=, never the message's content, at most sixty
235a minute, and sends nothing back. See Threat-Model for why the token
236and the sender address are both required.
237
238** [push]
239Push notifications to Apple devices, delivered by gitbayd talking to
240APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
241operator-supplied provider key. Off unless configured.
242
243- =enabled= (false).
244- =key_file= — path to the =.p8= provider key from Apple's developer
245 portal (Certificates, Identifiers & Profiles → Keys). It belongs at
246 =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
247 as. Read and validated at startup: it must parse as a PEM-wrapped
248 PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
249 than fill a queue nobody is watching.
250- =key_id=, =team_id= — the key's id and your Apple developer team id,
251 both from the same portal page.
252- =topic= — the app's bundle identifier. *An APNs key belongs to a
253 bundle ID.* gitbay.org pushes to the App Store build under its own
254 bundle id; a self-hoster who wants push ships their own iOS build
255 under their own bundle id, with its own =.p8= key from their own
256 developer account, and points =topic= at that id. There is no way to
257 push to someone else's build, by design — this is Apple's model, not
258 gitbay's.
259- =environment= — =production= or =sandbox=, naming the APNs host
260 rather than taking a URL, so a typo cannot aim the key at a host that
261 is not Apple's.
262
263All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
264are required when =enabled= is true; validation runs at config load,
265so a misconfigured =[push]= is caught before the daemon serves
266anything. The delivery queue (a device's undelivered and attempted
267pushes) is capped the same way the mail queue is, by =[retention]
268push=.
269
270** [backup]
271- =age_recipients= (optional) — age public keys (=age1...=). When set,
272 =admin backup= encrypts every archive to them and appends =.age= to
273 its name. Generate the pair off the host with =age-keygen=; only the
274 public key goes here, so the host writes archives it cannot read.
275 The restic copy is unaffected: the offsite job stages its own
276 =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
277 not the archives.
278
279** [api]
280- =enabled= (false) — the JSON API surface; see [[API]]. Off
281 means no credential-bearing HTTP endpoint exists at all.
282
283** [webhooks]
284- =allow_local= (false) — permit webhook, mirror and import targets on
285 loopback, private, shared (100.64.0.0/10), link-local or multicast
286 addresses. Leave off unless you know why you need it (SSRF).
287
288** [limits]
289- =clone_timeout= (3600s) — cap on =repo import= fetches. An import
290 takes http and https URLs only, passes the same address check as a
291 mirror sync and is pinned the same way, so it needs git 2.37 too.
292- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
293- =max_asset_bytes= (512MB) — cap per uploaded release asset.
294- =max_snippet_bytes= (1MB) — cap per snippet file.
295- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
296- =max_repos_per_user= (0, unlimited) — repositories an account may own
297 directly; =repo create=, =fork= and =import= refuse past it.
298 Organizations are not capped.
299- =max_bytes_per_user= (0, unlimited) — disk the account's own
300 repositories may take; a push may be no larger than what is left.
301- =pack_concurrency= (3), =pack_per_principal= (2), =pack_queue= (32),
302 =pack_queue_wait= (="60s"=) — git pack generation (clones, fetches,
303 =git archive --remote=, web archive downloads) over SSH, smart HTTP
304 and git:// shares one
305 budget: this many at once, this many per account (per client
306 address when anonymous: an IPv4 address, or an IPv6 /64), and this
307 many waiting for at most the wait. Anonymous clients together hold
308 at most =pack_concurrency= − 1 slots when it is above 1: an account
309 (SSH key, bearer token or web session) can take the last slot when it
310 is free, and anonymous clients cannot hold it.
311 Past that an SSH client gets "the server is busy…" and exit 1, HTTP
312 gets 503 with =Retry-After: 30=, git:// an =ERR= line; the daemon
313 logs a =pack limit= warning naming the transport and whether the
314 client was signed in, at most once a minute per transport. An SSH client
315 that disconnects while queued leaves the queue; an HTTP or git:// one
316 keeps its place until the wait runs out. A running clone is killed
317 when its client disconnects, or when no write to the client completes
318 for two minutes: a client reading below about 550 B/s, or an HTTP
319 request body that takes over two minutes with nothing written back,
320 is cut. Ref listings (info/refs, protocol v2
321 =ls-refs=), pushes and =repo download= are outside the budget. For the
322 three counts 0 means the default and a negative value turns that
323 bound off. The defaults suit a four-core host; see [[Performance]].
324 With =ssh.mode = "system"= each SSH session is its own process and
325 SSH clones are not counted.
326- =max_pack_bytes= (2 GiB) — the largest pack one push may send,
327 enforced as =receive.maxInputSize= and lowered to what an owner's
328 storage quota has left.
329- =ssh_auth_rate= (10) — SSH authentication failures per client address
330 per minute on the embedded listener; see below.
331
332** [git_daemon]
333- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
334 Serves only public repositories that additionally ran
335 =repo settings git-daemon <repo> on=.
336
337** [mirrors]
338- =pull_interval_minutes= (15) — how often pull mirrors fetch their
339 upstream. Push mirrors sync shortly after each local ref update.
340 Mirror URLs pass the same SSRF rules as webhook targets, when saved
341 and again before every sync; git then connects only to the addresses
342 that were checked (=http.curloptResolve=) and does not follow
343 redirects, so a mirror of a renamed repository fails until its URL
344 is updated. Needs git 2.37 or later on the server; with an older git
345 the worker logs an error at start and syncs no mirror, recording the
346 reason on each. Sync ignores the system and global gitconfig.
347
348** [go_import]
349Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
350Requests with =?go-get=1= at or under the module path answer with the
351go-import meta tag pointing at the repository's HTTPS clone URL, so
352=go install host/module/cmd/...@latest= resolves. The repository should
353be public (the module path itself confirms it exists).
354
355* Users, email, invites
356
357Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
358backfills is a wrapper that dispatches the registry command of the same
359name as the host: an admin context with no account behind it, so its
360audit rows carry no actor and =source: host=. The same commands run in an
361instance admin's SSH session (=ssh git@<host> admin ...=) and write the
362same rows with the key fingerprint as source. One implementation, two
363credentials.
364
365#+begin_src sh
366gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
367gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
368gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
369#+end_src
370
371"Verified" means SMTP-confirmed or host-admin-asserted; the database
372records which. Verified emails are what make commit signatures
373meaningful — an unverified address never produces a =verified= badge.
374
375* Audit and account control
376
377The audit log is the security feed (events are the product feed): every
378successful mutating command with its argv and source credential (SSH key
379fingerprint or API), every refused one (exit 3 or 4) as =refused
380<command>=, refused pushes as =refused git-receive-pack= (the access
381check) or =refused push= (a branch or tag rule, a release anchor or an
382unsigned commit, with the repository and ref names), hook socket
383requests failing the peer or push-token check as =refused hook= (never
384the token), registrations, admin actions, force-pushes, and auth
385failures/throttling. A refusal row
386keeps the flag names and the first positional, not the values.
387Refusals are recorded up to ten a minute per account and 600 a minute
388across the instance; past either, one =refused.throttled= row stands
389for the rest of that minute. The caps bound the embedded listener, the
390web and the API; under =ssh.mode = "system"= each =gitbayd shell=
391connection counts separately. Secrets never appear — they travel on
392stdin, never in argv.
393
394Each row carries the SHA-256 of the row before it. =gitbayd admin audit
395verify= opens the store as other admin commands do, applying pending
396migrations, so run it with the binary that matches the daemon. It
397recomputes the chain and exits 1 naming the first row that was
398edited or whose predecessor was removed. The chain is unkeyed: whoever
399can write the database can recompute every hash after an edit, and
400verify then finds nothing. It catches an edit only when the later
401hashes were not recomputed. Retention removing the oldest
402rows is not a break. Rows written before the chain existed are counted
403and skipped; when every row is such a row, verify warns and exits 1,
404since clearing the hash columns looks the same. After an upgrade that
405clears with the first new audit row.
406
407Removing the newest rows leaves no break, and neither do rows written
408afterwards under the freed ids. The database cannot show either. The
409daemon logs every row it writes to its journal, outside the database
410(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
411id and hash: comparing them with the newest journal line is the check
412for any change, recomputed hashes included. Rows written by
413host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
414"system"=, are not copied to the journal.
415
416#+begin_src sh
417gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
418gitbayd admin audit verify # check the hash chain; exit 1 names the first bad row
419ssh git@<host> audit ... # the same, from an admin session
420ssh git@<host> admin user list [--state active|pending|disabled|admin]
421ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
422ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
423ssh git@<host> admin user promote <name> # grant instance admin
424ssh git@<host> admin user demote <name> # remove it; the last admin is refused
425gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
426gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
427gitbayd admin user enable <name> # sessions dropped, nothing deleted
428gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
429 # refused (with each blocker named) while
430 # the account owns repos, authored
431 # issues/MRs/comments/reviews, or is an
432 # org's only admin
433#+end_src
434
435=--actor= takes a username, or =-= for rows with no actor: host commands
436and auth failures. =--action= is a prefix, so =cmd repo= catches every
437repository command and =admin= every host or admin-session action.
438=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
439
440=admin user list= pages by username (=--limit=, =--cursor=) and carries
441each account's state and =last_seen=, the newest use of any of its SSH
442keys or API tokens. =admin user show= adds the keys with their last use,
443each address with how it was verified, PGP keys, org roles, the owned
444repository count, API token names, and live browser sessions. Both are
445Both are refused to non-admins, like =audit=, on every surface.
446
447=/admin/users= is the same list in a browser, linked from the admin
448page: the state filter the command takes, keyset paging on its cursor,
449and a row per account with promote, demote, disable and enable, each
450dispatching the command. Demote and disable ask for the username to be
451typed, since both take someone's access away. Creating and deleting an
452account, issuing an invite and asserting an address stay on the command
453line: each takes a key, mints a credential, or cannot be undone. A
454non-admin gets the 404 a missing page would, so the URL confirms
455nothing.
456
457Promotion needs an active account: a pending or disabled one is refused.
458Demotion is refused when it would leave no admin, over SSH, in the
459browser and on the host alike, so the host-local =promote= is the way
460back in when the only admin key is lost.
461
462Instance admin carries no right on anyone's repository: policy does not
463consult it, and a private repository still answers not-found to an
464admin. Moderation goes through explicit overrides that skip the access
465check and write their own audit row:
466
467#+begin_src sh
468ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
469ssh git@<host> admin repo archive|unarchive <owner/name>
470ssh git@<host> admin repo visibility <owner/name> public|private
471ssh git@<host> admin repo delete <owner/name> --yes
472#+end_src
473
474Each lands in the audit log as =admin repo.<action>= naming the
475repository, on top of the =cmd= row every mutating command gets.
476
477=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
478per minute — successful auths never count and clear the slate.
479=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
480push.
481
482=limits.write_rate= (60) bounds *mutating commands per account per
483minute*. It is counted in the dispatcher, so SSH, the JSON API and the
484web spend one budget and a caller cannot refresh it by changing surface;
485=limits.api_rate= stays in front of it, bounding a network source rather
486than an account. A command is one token whatever it writes, so a bundle
487import costs one and only a loop of separate commands spends the budget.
488Read-only commands, the runner protocol (a build streams its log in many
489small writes) and the host CLI are exempt. Refusals exit 4 and say when
490to retry. A negative value turns the limit off; it matters most with
491=registration = "open"=, where every write also queues notification mail
492and webhook deliveries.
493
494* Queues
495
496Every background worker keeps a backlog and a failure state. An instance
497admin reads them all in one place:
498
499#+begin_src sh
500gitbay dashboard --json | jq .queues # webhooks, mail, push, mirrors, builds, deps
501#+end_src
502
503Per worker: pending, retrying (pending with a failed attempt) and
504dead-lettered counts with the oldest pending age, and the retrying or
505failed rows themselves, capped at twenty each. Builds list what is
506running and then what is pending, each since when, so a build no runner
507is scoped to claim is visible here rather than only in its repository;
508mirrors list the ones whose last sync failed;
509dependency checks list the ones whose last check errored. Non-admins get
510no =queues= key at all.
511
512Push rows name the device id, never the token. Watch this one after
513configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
514config validation, which can only check that the =.p8= parses, and then
515every send comes back =403 InvalidProviderToken= and dead-letters on its
516first attempt.
517
518In accounts mode the same read renders at =/admin=, linked from the rail
519for admins. Anyone else gets a 404 there.
520
521A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
522with the address redacted out of the relay's error. The id is the queue
523row: find it in the Mail table on =/admin=, or in =dashboard --json=,
524where the recipient and the unredacted error are. That is deliberate —
525see the Threat-Model page.
526
527* Maintenance
528
529#+begin_src sh
530gitbayd admin stats [--json] # counts, database size, per-repo disk
531ssh git@<host> admin stats [--json] # the same, from an admin session
532gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
533gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
534gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
535#+end_src
536
537A history rewrite leaves the commits it removed reachable through
538=refs/merge-requests/N/head= of the merge requests that landed them, so
539they stay fetchable by anyone who can read the repository. Nothing drops
540a head ref on its own — an open or source-gone MR is merged through it,
541and a merged or closed one keeps its diff readable through it — so the
542cleanup is a command an instance admin runs, naming the MRs:
543
544#+begin_src sh
545ssh git@<host> admin mr prune owner/name 1 2 3 --yes
546#+end_src
547
548It refuses an open or source-gone MR, deletes the named refs, runs
549=git gc --prune=now= on that one repository so the objects go at once
550rather than after git's two-week grace, leaves a system comment on each
551MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
552reviews and head sha; =mr diff= and the MR page say the head is gone.
553Run it when nothing is pushing to that repository: without the grace, a
554push caught between leaving quarantine and writing its ref loses its
555objects. Objects also survive in offsite backups until those are
556rewritten; see "Removing a repository's history from every snapshot".
557
558=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
559weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
560layout the source sent, so a first manual =admin gc= after a bulk
561import is worthwhile.
562
563* Backup and restore
564
565#+begin_src sh
566gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
567gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
568#+end_src
569
570One archive: a consistent SQLite snapshot (taken *before* the
571repositories are read, so the database never references objects the
572archive missed), every repository, and the SSH host keys. Excluded:
573hook socket, regenerated hook scripts, WAL files. Safe to run against a
574live daemon. =--out= must be outside =server.root=, or the next full
575backup would carry the archive. Each run removes the snapshot
576directories (=.gitbay-snap-*=) and temporary archives (=.*.tmp-*=) a
577killed run left beside its archive once they are a day old, and prints
578each one it removes.
579
580=--verify= reads an archive back: the snapshot must pass SQLite's
581integrity check, every repository the snapshot names must be in the
582archive, and each must pass =git fsck --connectivity-only=. Every
583release asset the snapshot names must be in its repository with its
584recorded size and sha256, and every LFS object in the archive must
585hash to its name. LFS objects are named by pointer files in git
586history rather than by the database, so a pointer whose object was
587never uploaded is not caught, and with =[lfs] root= outside
588=server.root= the archive holds no LFS objects at all. It extracts the
589repositories to a temporary directory for the checks, so it needs free
590space the size of the repositories. A database-only archive is checked
591for integrity and says so. Exit is non-zero on damage, a missing
592repository, a missing object, or a missing or altered asset or LFS
593object.
594
595A full backup holds =<root>/backup.lock= from its database snapshot to
596its last repository. While it runs, =repo delete=, =repo rename=,
597=repo transfer=, =admin repo delete=, =org rename=, =admin mr prune=
598and =gitbayd admin gc= refuse with "a backup is running"; retry when it
599finishes. Database-only backups take no lock. A pack that git's own
600automatic gc removes during the walk is skipped; each repository's refs
601are archived before its objects, so the refs still find their objects,
602and =--verify= reports it if one does not.
603
604Verifying an archive of unknown origin: run it as an unprivileged
605user. The connectivity check runs git with =--git-dir= on each
606extracted repository, so a directory that is not a repository fails
607rather than git checking an enclosing one. =objects/info/alternates=
608and a =commondir= directly in a =*.git= directory are not extracted,
609so an archive cannot use either to have git read another repository's
610objects or refs on the host. Git still reads each archived
611repository's own =config=.
612
613Archives carry a directory entry for every directory, including an
614empty one, so a bare repository whose refs are all packed restores as
615a repository. Archives written before this release do not: extracting
616one can leave a repository's =refs/= directory missing, which stops
617git from recognizing it as a repository at all. =gitbayd admin backup
618--verify <archive>= names the repositories this affects; the fix is
619=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
620which it opens normally.
621
622With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
623and =--verify= needs the private key:
624
625#+begin_src sh
626gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
627age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
628#+end_src
629
630The identity lives off the host (with the secret key file and the
631restic credentials), so verifying an encrypted archive happens there
632or on a restore host.
633
634Restore: extract into an empty directory, point =server.root= at it,
635restore =server.secret_key_file= from its own copy (mode 0600, owned
636by the account gitbayd runs as), then start gitbayd. No archive carries
637the key file, and without it gitbayd refuses to start. Host keys are
638preserved, so clients keep their known_hosts entries; hooks regenerate
639at startup.
640
641** Schedule and recovery point
642
643Two timers, because the two halves of the data have different exposure.
644
645- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
646- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
647 writes the SQLite snapshot alone, last 48 kept. A few MB against the
648 full archive's hundreds, which is what makes the frequency affordable.
649
650The split follows what a loss would actually cost. Repositories are git,
651so a mirror or any clone is a second copy; the database is the only copy
652of issues, merge requests, comments and review state. So the recovery
653point is about an hour for the data that exists nowhere else, and a day
654for the data that does.
655
656Continuous replication (litestream and similar) was considered and not
657adopted. It would take the database's recovery point to seconds, but the
658repositories would still be on the nightly archive, so a restore could
659produce a database referencing commits the repository backup does not
660have. Consistency between the two halves is worth more here than latency
661on one of them. Revisit if repository replication becomes continuous
662too.
663
664** Offsite copies
665
666bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
667=/var/lib/gitbay-stage= (a database copy and =config.toml=, staged
668there) to an S3 bucket at Scaleway, with a key that can only add
669snapshots. Nothing else under =/etc/gitbay= is in it: not the secret
670key file, not =apns.p8=. The key that can
671remove them lives on the operator's machine, in
672=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
673cannot destroy its own history. Forgetting, pruning and rewriting all
674run from there.
675
676*** Removing a repository's history from every snapshot
677
678A history rewrite plus =admin mr prune= takes commits off the server,
679but every snapshot taken before it still holds them, and the retention
680window is the only thing that ages them out. To remove them now,
681rewrite the snapshots without that repository rather than forgetting
682the snapshots: everything else in them stays restorable. The next
683nightly run adds the repository back in its current state.
684
685Repositories are stored under the name they had on disk when each
686snapshot was taken, so a renamed repository needs every name it has
687carried. Check what an older snapshot holds before choosing the paths:
688
689#+begin_src sh
690set -a; . ~/.config/gitbay/offsite.env; set +a
691restic $RESTIC_OPTS snapshots
692restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
693#+end_src
694
695Then dry-run, apply, prune, and confirm nothing matches:
696
697#+begin_src sh
698EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
699restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
700restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
701restic $RESTIC_OPTS prune # drops the data nothing references
702restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
703#+end_src
704
705=--forget= is what makes the originals go; without it the rewritten
706snapshots sit beside them and the data stays referenced. Snapshot IDs
707change; their times do not. Done for krz/keycask (formerly rust-pass)
708on 2026-09-18, across 22 snapshots.
709
710** Secret key
711
712CI secrets, webhook secrets, mirror tokens and APNs device tokens are
713stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
714each value prefixed with the id of the key that sealed it
715(=gbs1:<id>:=). The key file is not in the database, not under
716=server.root=, and therefore in neither the local archives nor the
717main restic repository. It must be copied off the host separately;
718without it a restored database's secrets cannot be opened, and
719gitbayd refuses to start against them. A separate restic repository
720for it and =apns.p8= is planned (runbook D of the data-at-rest plan)
721and not yet in place, so today the only off-host copy is one the
722operator makes by hand after =init= and after every =rotate=.
723
724#+begin_src sh
725gitbayd admin secrets init # once; deploy/install.sh does it on first install
726gitbayd admin secrets check # open every value, count by key
727gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
728#+end_src
729
730- Missing file: every gitbayd process that opens the database refuses
731 to run and names the path, including =serve= and, in system mode,
732 =authorized-keys=. =migrate= does not need it.
733- Backups: =admin backup= and =admin backup --verify= do not need the
734 key file; a restore does.
735- Wrong key: =serve= stops at startup naming the first row that does
736 not open; =secrets check= does the same without starting anything.
737- Upgrade: the first start after the upgrade seals every value still
738 in clear and logs =sealed secret values=.
739- Rotation: =rotate= adds a key, reseals every value under it in one
740 transaction, then removes the old keys. Run it as root, since it
741 replaces the key file in =/etc/gitbay=; the file keeps its owner.
742 The daemon re-reads the file when it changes, so it needs no
743 restart. Copy the new file off the host afterwards. =init= and
744 =rotate= hold an flock on =<key file>.lock= while they run, so a
745 second run waits for the first.
746- Push devices are looked up by the SHA-256 of their token
747 (=push_devices.token_hash=), since two seals of one token differ.
748
749** Restore drill
750
751A restore onto a clean host, run quarterly (January, April, July,
752October) and after any change to the backup code
753(=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite
754job), and recorded below. The disaster it rehearses is losing bay1, so
755the local archives are gone with it and the sources are the main
756offsite restic repository (repositories, LFS, the staged database,
757=config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys
758repository once runbook D creates it; until then the operator's
759hand-made copy), and the operator's password manager (=offsite.env=,
760the keys repository's password and token once it exists,
761=backup-identity.txt=). The full steps are runbook C of the
762data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
763
764=gitbayd admin restore-drill= does the archive half. It extracts a
765full archive into an empty or absent directory, runs every =--verify=
766check on the extracted copy, and prints what was restored, the newest
767issue, issue comment, merge request comment and push in the restored
768database, and the elapsed time. Exit is non-zero if any check fails.
769
770#+begin_src sh
771gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
772gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill
773#+end_src
774
775What to restore, and from where:
776
777| Item | Source | Path on the drill host |
778|-------------------------------+-------------------------------------------------------------+------------------------------------------------|
779| Database | staged copy in =/var/lib/gitbay-stage= (restic), or an archive | =<root>/gitbay.db= |
780| Repositories | =/var/lib/gitbay/repos= (restic), or an archive | =<root>/repos= |
781| LFS objects | =/var/lib/gitbay/lfs= (restic), or an archive | =<root>/lfs= (or =[lfs] root=) |
782| Release assets | inside each repository (=gitbay-releases/=) | with the repositories |
783| Host keys | =/var/lib/gitbay/ssh= (restic), or an archive | =<root>/ssh= (or =[ssh] host_keys=) |
784| =config.toml= | =/var/lib/gitbay-stage/config.toml= (restic) | =/etc/gitbay/config.toml= |
785| =secret.key=, =apns.p8= | the off-host copy; no archive or main snapshot carries them | =/etc/gitbay/=, mode 0600, owned by =gitbay= |
786
787From the offsite path (restic is not run by any gitbay command):
788
789#+begin_src sh
790restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
791cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
792gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
793gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root
794#+end_src
795
796The second archive is how the restic tree gets the same checks and
797timestamps; =/tmp/drill-root= is discarded afterwards.
798
799What to check, each a column below:
800
801- DB integrity, connectivity, release assets, LFS: =restore-drill=
802 prints =integrity ok=, =connectivity ok on N repositories=, =release
803 assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin
804 stats --json= on the source at the snapshot time.
805- Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets
806 check= opens every value.
807- Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's
808 fingerprint.
809- Config: =gitbayd --config /etc/gitbay/config.toml check-config=.
810- Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over
811 SSH succeed.
812
813Time to service runs from the clean host's first root login to the
814first successful =git clone= over SSH from it. The recovery point is
815the time of the newest restic snapshot restored; record beside it the
816newest issue, comment and push =restore-drill= printed, which show how
817much activity the restore carries.
818
819The first drill ran on the operator's laptop rather than a provisioned
820host, so its time to service has no provisioning in it, and it could
821not check secrets: no copy of =secret.key= was on the machine, and
822gitbayd refuses to start while any sealed value does not open. The
823sealed values were cleared in the drill copy to reach service; #305
824tracks proving the off-host key.
825
826| Date | Host | Snapshot restored (UTC) | Newest issue / comment / push | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
827|------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
828| 2026-09-29 | laptop (macOS), restic from offsite | 2026-09-29 00:19:15 (ccd646cf) | 00:05:17 / 00:09:44 / 00:15:31 | 8m43s (restore 1m49s, checks 33s) | ok | ok, 69/69 | ok, 2 | ok, 529 | matches | not checked (#305) | 4.99 GiB restored; 71 sealed values cleared in the drill copy to start |
829
830* Upgrades
831
832Replace the binary, restart the unit. Migrations apply automatically and
833are transactional; hook scripts under =<root>/hooks= are rewritten at
834startup to point at the current binary path.
835
836* CI runner
837
838=gitbay-runner= executes builds queued by pushes and merge requests. It
839polls over SSH with a key of scope =runner=, which reaches only the
840runner protocol and read-only git (a runner executes arbitrary
841repository code, so the key it holds must not do more). A runner key
842claims builds only for the repositories it is attached to, by =repo
843runner add= from a repository admin or an instance admin; an admin key
844claims any. Users attach their own runners: see the Users page. For an
845instance runner, run it as a dedicated unprivileged user on a non-admin
846account. =admin user create --key= registers a full-scope key, so the
847runner key is added afterwards through a bootstrap key that is then
848removed, and attached to each repository it should build:
849
850#+begin_src sh
851useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
852sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
853ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
854gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
855ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
856ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
857rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
858gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
859#+end_src
860
861#+begin_src sh
862gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
863#+end_src
864
865=-jobs N= runs N builds at once. Claiming is one transaction that
866selects and updates, and each build works in its own =build-<id>=
867directory, so workers do not collide; idle polls are staggered across
868the interval so N of them do not wake together. The drop-in's weights
869below are per service, not per build, so raising =-jobs= divides them
870rather than multiplying the host's load.
871
872=admin runners= shows which account each runner polls as, and what each
873may claim. A runner key claims builds only for the repositories it is
874attached to: with none attached it claims nothing, and =-repos= may only
875narrow within them. An admin's full-scope key claims any repository —
876that is what =-repos= was for — and still works for the protocol during
877a rotation. A merge request head from a fork is built in the target
878repository as untrusted: the claim carries no secrets, and only a runner
879started with =-untrusted= takes it. Same-repository heads were built by
880their branch push and are not built again.
881
882=make deploy-runner= also installs
883=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
884=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
885the daemon or the backup timers, and =NoNewPrivileges=,
886=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
887and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
888and the runner's home. The e2e suite alone starts sixty
889daemon instances; without the drop-in a deploy's copy over the admin
890sshd stalled. Both deploy targets copy with =rsync --partial=, which
891resumes a stalled transfer.
892
893Under =-isolation podman=, the default and what bay1 runs, each build
894is confined to a container (see Container isolation below). Under
895=-isolation none= steps run directly on the host as the runner's user,
896so treat that machine as executing whatever your users push, and
897install the toolchains your builds need on it.
898
899A runner claims the oldest pending build among the repositories its key
900is attached to — for an admin key, the oldest in the instance. =-repos=
901narrows within that set, which is what makes a runner outside the server
902practical: one on a machine that should build a single project, or that
903holds credentials for one deployment, stays on it.
904
905Oldest-first is across everything the key may claim, so a repository
906with a deep queue holds every other repository the same runner serves;
907bay1 measured a 15-minute average wait on a day of merge request
908stacks from one repository. A runner attached to one repository cannot
909be starved. That is the rule, decided in krz/gitbay#207: a runner
910serving several repositories takes them oldest-first, and an operator
911who wants one repository never to wait on another runs a second
912runner attached to it alone. Nothing caps what an account queues,
913and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
914while the job's last build is pending or running, so a repository
915with no runner holds one row per scheduled job rather than one per
916tick, and a build runs only on a runner its owner attaches, so a busy
917schedule spends the owner's compute. Pushes are bounded by what an
918account can push.
919
920#+begin_src sh
921gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
922 -workdir /var/lib/gitbay-runner/work
923#+end_src
924
925Add =-untrusted= only with =-isolation podman=.
926
927gitbay.org's runner is attached to the forge's own repositories and
928the isolation canary, nothing else, because it shares the host with
929the forge; its unit names no =-repos=, the attachments are the
930boundary. Any other repository builds on a runner its owner attaches.
931
932=-repos= narrows an admin runner; for a runner key the attachments are
933the boundary, held by the server, and =-repos= may only name
934repositories among them. =-untrusted= makes a runner claim merge
935request heads from forks; the bay1 unit sets it because it isolates in
936podman. A runner without it builds trusted commits only.
937
938=gitbay dashboard= and =ssh git@<host> admin runners= list every key
939that has polled as a runner: the account, the key's fingerprint, when it
940last polled, the repositories it may claim — its attachments for a runner
941key, the =-repos= it asked for or =any= for an admin key — and the build
942it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
943that polled by mistake, the key itself untouched. =admin runners= also
944heads the list with the queue: builds
945pending now, and over the last day how many were claimed, how long they
946waited to be claimed (average and worst), and
947how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
948reported is failed by the scheduler's minute tick, whether or not any
949runner is still alive: within about two minutes of its log stream ending
950with no outcome reported — the runner reports right after closing the
951stream, retrying for half a minute if gitbayd is unreachable — or, if no
952stream was ever seen, at the deadline.
953
954Instance admin on the runner account only authorizes the claim/report
955protocol; it grants no repo access. A build that pushes back — a pages
956deploy, an archive publish, an automated MR branch — needs an explicit
957grant on that repo: =repo access grant <owner/name> ci write=. Private
958repos likewise need at least read for the clone.
959
960** Container isolation
961
962Builds run in a rootless podman container, one per job, with the
963workspace bind mounted and nothing else: the clone happens outside with
964the runner's key, so a step cannot read it. =-isolation none= keeps the
965old behaviour — steps on the host as the runner's user — for an instance
966where every repository is trusted. There is no automatic fallback: a
967runner started with =-isolation podman= that cannot find a working
968podman exits rather than running a build unsandboxed.
969
970gitbay's own jobs name =localhost/gitbay-ci:2=, built from
971=deploy/Containerfile.ci= on the runner host. A job's image must carry
972what its steps need: the suite drives real git, git-lfs, gpg and sshd and
973asserts they exist before running, so the stock runner default would fail
974it immediately. Build or rebuild it with:
975
976#+begin_src sh
977ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
978ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
979 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
980#+end_src
981
982The tag is deliberate rather than =:latest=: changing the file means
983bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
984change under it.
985
986=-image= names the image a job runs in when it declares none, and is
987required under =-isolation podman=: there is no built-in default,
988because an image this host does not have would fail every build. A job
989overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
990so a config file cannot turn it into podman arguments.
991
992=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
993-memory 4g=); unset means uncapped. The runner applies them itself: it
994creates a cgroup per build under its own delegated service cgroup,
995writes the limits there, and starts every podman process for the build
996inside it, with podman's cgroup handling off. Podman's own =--memory=
997and =--cpus= never applied under rootless cgroupfs, which is what a
998system service gets (krz/gitbay#188). The unit therefore needs
999=Delegate=yes=, which the drop-in sets; without it the runner refuses
1000to start when a limit is set, and logs that builds run unconfined when
1001none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
1002and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
1003swap: the memory cap is what keeps the forge alive when a build
1004allocates without bound, and it sits above the e2e suite's 5GB peak
1005rather than at a fair share. =OOMPolicy=continue= keeps systemd from
1006stopping the runner when a build is OOM-killed.
1007
1008A trusted build's home is its repository's, under
1009=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
1010=HOME=: caches persist between trusted builds of one repository and are
1011never read by another's. An untrusted build — a merge request head from
1012a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
1013the build ends. Homes under =<workdir>/home= are from runners before
1014krz/gitbay#255, which shared them with untrusted builds; nothing reads
1015them any more, and they can be deleted.
1016
1017*Images are provisioned, never pulled by a build.* The runner passes
1018=--pull=never=. Two reasons, and the second is the better one: the
1019service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
1020holding a setuid file, which is nearly every distribution image; and on
1021an instance where anyone can push a =ci.yml=, =image:= would otherwise
1022mean "fetch and run anything from the internet". An operator pulls or
1023builds what is allowed and a build picks among those. A job naming an
1024image the host does not have fails with a message saying so.
1025
1026#+begin_src sh
1027su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
1028su - ci-runner -s /bin/sh -c "podman images"
1029#+end_src
1030
1031Prepare a host before pointing an isolating runner at it:
1032
1033#+begin_src sh
1034ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
1035make deploy-runner
1036#+end_src
1037
1038The script installs podman, delegates a subuid/subgid range to
1039=ci-runner=, checks that user namespaces are enabled rather than
1040assuming, enables lingering, and verifies rootless podman actually runs
1041as that user. It is idempotent.
1042
1043It also installs nftables. =make deploy-runner= ships
1044=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
1045with =gitbay-runner-egress.service=, which loads it and which the
1046runner's unit requires; it checks the file with =nft -c=, reloads the
1047unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
1048restarting the runner: =127.0.0.1:22= and the public 22 must answer,
10492222 must not. The table limits the runner's user to =127.0.0.1:22=,
1050DNS on loopback, and 22, 80 and 443 on the host's public address; the
1051Threat-Model page says why.
1052
1053That table cannot tell a build from the runner, since both run as
1054=ci-runner=. =deploy/gitbay-runner-builds.nft=, shipped as
1055=/etc/gitbay-runner/builds.nft=, matches by cgroup instead: the runner
1056starts each build under =builds/trusted= or =builds/untrusted= in its
1057service cgroup, and the table closes the host's loopback to every
1058build, limits an untrusted build to TCP 80 and 443 and DNS, and closes
1059private ranges to both (the CI page has the table). nftables resolves a
1060cgroup path to its id when the table loads, and the service cgroup is
1061new on every start, so the drop-in's =ExecStartPre= creates the two
1062cgroups, hands them to =ci-runner= and loads the table before the
1063runner starts; a table that does not load stops the start. =make
1064deploy-runner= checks the file's syntax before the restart and that the
1065table is loaded after it. If =/etc/resolv.conf= names a nameserver in a
1066private range, allow it in the file first or builds resolve nothing.
1067
1068A restart of =nftables.service= flushes both tables;
1069=systemctl reload gitbay-runner-egress= restores both.
1070
1071The egress unit's stop and the rollback below use =nft destroy=,
1072which needs nftables 1.0.8 or later; check =nft --version= on a new
1073host.
1074
1075Checking the tables on a running host, during a build (the flood test
1076below waits a minute before its first probe, for this):
1077
1078#+begin_src sh
1079nft list table inet gitbay_builds # counters on the reject rules
1080for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
1081# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>
1082#+end_src
1083
1084A pasta process anywhere else — the runner's own =runner= cgroup, a
1085user slice — means the builds table does not see that build's traffic
1086and only the first table applies. Run
1087=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
1088once as a push and once with =--untrusted=. It fails if the runner was
1089locked out or if the build log lacks the lines only a working table
1090produces. The authoritative proof that pasta's sockets are in the
1091build's cgroup is the untrusted run: =169.254.1.2:22= and
1092=github.com:22= refused, all twelve logins refused, and the counters on
1093the =untrusted= chain's rejects rising. The first table lets
1094=ci-runner= reach both of those addresses.
1095
1096To take the builds table out, on the host:
1097
1098#+begin_src sh
1099sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
1100rm /etc/gitbay-runner/builds.nft
1101systemctl daemon-reload
1102nft destroy table inet gitbay_builds
1103#+end_src
1104
1105The runner needs no restart. It still places builds under
1106=builds/trusted= and =builds/untrusted=, but with the file gone neither
1107a runner start nor =systemctl reload gitbay-runner-egress= loads the
1108table again, and builds keep the first table and =--no-map-gw=. A
1109runner that takes =-untrusted= or polls over loopback refuses to start
1110without build cgroups at all, since the table would match nothing. The
1111next =make deploy-runner= installs the file and the lines again.
1112
1113The drop-in sets =NoNewPrivileges=no=, without which rootless podman
1114cannot call =newuidmap= and the runner refuses to start. That is a
1115considered trade, explained in the file and in the Threat-Model; if you
1116run with =-isolation none=, set it back to =yes=.
1117
1118*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
1119the build in flight, reports it, and exits; the drop-in's
1120=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
1121is what makes the signal reach the runner alone — under systemd's default
1122the build's container and the log session are signalled with it, and the
1123runner drains a build that is already dead. So =make deploy-runner=
1124waits for a running build rather than orphaning it, and a build's result
1125is retried for half a minute if gitbayd is restarting at that moment. A
1126second SIGTERM ends the runner at once, abandoning the build to the
1127reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
1128the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
1129=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
1130the runner as a transient user unit under =systemd-run= and stops it
1131under both kill modes. That last one needs a systemd user manager, so it
1132skips in the container CI runs in; run it on a Linux host with
1133=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
1134
1135*Validate podman mode on a scratch repository before pointing the runner
1136at real ones.* Every deploy that switched the whole instance to
1137containers and failed took CI down with it. Instead: create a throwaway
1138repository the runner account can read (public, or granted read — a
1139private one is "not found" to the runner and the build stays pending),
1140give it one job that names the CI image, attach the runner's key to it,
1141and deploy the runner with =-repos= naming only that repository. The
1142production unit, with its real hardening, then claims nothing else;
1143other repositories' builds queue until =-repos= is removed again, which
1144is a pause, not an outage.
1145
1146#+begin_src sh
1147# on a machine with an admin gitbay identity (root on the host has none),
1148# with the runner's public key copied from the host:
1149gitbay repo create cmc/runner-scratch # then push a .gitbay/ci.yml naming the image
1150scp -P 2222 root@<host>:/var/lib/gitbay-runner/.ssh/id_ed25519.pub runner.pub
1151gitbay repo runner add cmc/runner-scratch < runner.pub
1152# on the host:
1153sed -i 's#^ExecStart=/usr/local/bin/gitbay-runner #&-repos cmc/runner-scratch #' /etc/systemd/system/gitbay-runner.service.d/override.conf
1154systemctl daemon-reload && systemctl restart gitbay-runner
1155# back on the admin machine:
1156gitbay build log cmc/runner-scratch 1 # green: remove -repos, redeploy, delete the scratch repository
1157#+end_src
1158
1159*Do not deploy an isolating runner to a host that has not been
1160prepared.* The runner is specified to refuse to start without a working
1161podman rather than fall back to running builds unsandboxed — a fallback
1162that silently drops isolation is worse than a stopped runner, because
1163nothing surfaces it. On an unprepared host that refusal stops every
1164build on the instance.
1165
1166The service drop-in carries =Delegate=yes= for rootless cgroup
1167management and =ReadWritePaths= for podman's store under
1168=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
1169make read-only. Those paths are prefixed =-= so they are ignored when
1170absent: the drop-in installs on unprepared hosts too, and a unit that
1171refused to start would stop every build.
1172=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
1173user: rootless storage belongs to that user, and root's prune would not
1174see it. An unpruned image store on a 40GB host is a slow outage.
1175
1176* LFS storage
1177
1178Objects live content-addressed under =[lfs] root= (default
1179=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
1180(512MB default). Storage sits behind a small interface — an
1181S3-compatible backend is a drop-in with the server proxying, and
1182presigned URLs a later optimization. LFS objects do not travel with
1183push mirrors (mirrors move git refs only), and gc does not yet collect
1184orphaned objects.
1185
1186* Pages
1187
1188=[pages] domain = "example.site"= serves public repos' =pages= branches
1189on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
1190server; ACME issues per-subdomain certificates on demand (only for
1191owners that exist). The domain must not be the site host or a parent of
1192it — pages content runs its own scripts and must stay off the forge's
1193origin.
1194
1195Users with repo admin claim custom domains with =repo domain add=.
1196Claims activate only after a DNS TXT challenge proves control of the
1197domain (=repo domain verify=, audit-logged); pending claims serve
1198nothing, get no certificates, and expire after 7 days. ACME issues
1199certificates only for verified hosts, so stray DNS pointed at the
1200server gets nothing.
1201
1202* Security
1203
1204The [[Threat-Model]] file is the reference for what the forge
1205trusts and refuses to do. Operational checklist:
1206
1207- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
1208 (the module list is deliberately short — review it on each release),
1209 and a short fuzz pass over every attacker-facing parser (pkt-line,
1210 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
1211 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
1212 against main rather than per push, because =@latest= scans today's
1213 advisory database and an advisory lands without anyone pushing;
1214 =build trigger krz/gitbay vuln= runs it on demand.
1215- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
1216 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
1217 configuration needed.
1218- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
1219 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
1220 =LockPersonality=, =MemoryDenyWriteExecute=,
1221 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
1222 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
1223- *OS patches* apply via =unattended-upgrades= (security origins,
1224 auto-reboot 04:30 if required).
1225- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
1226 watched by =fail2ban=; gitbayd's own port 22 is throttled by
1227 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
1228 account's writes by =limits.write_rate=.
1229- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
1230journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
1231webhook: disk, service, the daemon's own =/healthz= answer, certificate
1232expiry, and the age of the newest full backup and database snapshot.
1233It exits non-zero on an alert so the unit shows in =systemctl
1234--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
1235a certificate under 21 days, a full backup over 25 hours old, or a
1236database snapshot over 2 hours old.
1237
1238=GET /healthz= is unauthenticated and cache-free: whether the database
1239answers and which commit serves, 503 when it does not.
1240- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
1241 0750, owned by =gitbay=). The nightly archive plus provider snapshots
1242 are the recovery path; for tighter RPO, add continuous replication
1243 (litestream) against the same file — it coexists with the WAL.
1244
1245* Odds and ends
1246
1247- deleting a fork marks MRs sourced from it =source_gone=; their diffs
1248 remain viewable and mergeable because the target repo owns the
1249 objects.
1250- =refs/merge-requests/*= is server-owned and unpushable by clients;
1251 only =admin mr prune= removes one (see Maintenance).
1252- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
1253 the =events= table, which also feeds webhooks.
1254- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.