.gitbay/wiki/Admin.org
1361 lines · 73366 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21gitbayd --config /etc/gitbay/config.toml admin secrets init
22chown gitbay:gitbay /etc/gitbay/secret.key
23#+end_src
24
25=admin secrets init= refuses when the key file already exists, so a
26reinstall on the same host should skip it — =deploy/install.sh= does
27this with a file check before running it.
28
29=deploy/= in the source tree has a cloud-init file, a hardened systemd
30unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
31the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
3222/80/443 without root.
33
34** The SSH port decision
35
36- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
37 22 — move the host's admin sshd to another port. Remotes read
38 =git@host:owner/repo= with no port gymnastics.
39- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
40 =AuthorizedKeysCommand=:
41 #+begin_example
42 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
43 AuthorizedKeysCommandUser gitbay
44 #+end_example
45 sshd requires that binary to be root-owned and not group/world
46 writable. Unknown keys fail authentication inside sshd, so system mode
47 requires =registration.mode = "closed"= (check-config enforces this).
48 Under the forced command the CLI's leading =--term=<cols>[,color]=
49 argument works as is; a client that sets =GITBAY_TERM= instead needs
50 =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
51 handled by the host's sshd, not gitbayd.
52
53* Configuration reference
54
55=/etc/gitbay/config.toml=. =check-config= validates and names every
56contradiction; =--no-host-checks= skips port/path probes.
57=gitbayd admin config show= prints the configuration in effect as TOML,
58every default filled in and =smtp_pass= redacted. A file that fails
59validation still prints, followed by the contradiction.
60
61** [server]
62- =root= (default =/var/lib/gitbay=) — repositories, database, host
63 keys, ACME cache all live here.
64- =site_url= (required) — canonical =https://host=; drives ACME, clone
65 URLs, mail links.
66- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
67 seal CI secrets, webhook secrets, mirror tokens and push device
68 tokens in the database. Must be outside =root=, mode 0600, readable
69 by the daemon's user. See "Secret key" below.
70- =source_repo= (optional, =owner/name=) — the repository this instance
71 develops itself in. Startup warns when the running build's commit is
72 not on that repository's default branch, which is how a binary built
73 from an unmerged branch stops being invisible. Leave it unset unless
74 the instance hosts its own source.
75
76** [ssh]
77- =mode= — =embedded= | =system= (above).
78- =port= (22) — embedded listener port.
79- =host_keys= — list of private key paths; empty generates an ed25519
80 key at =<root>/ssh/host_ed25519=.
81
82** [http]
83- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
84- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
85 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
86 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
87 failing to bind it is a warning, not fatal. Requires an =https://=
88 site_url with a public DNS name.
89- =files=: =cert_file= + =key_file=.
90- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
91- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
92 =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
93 release the binary was built with. =openssl s_client -connect
94 <host>:443 -tls1_1= fails the handshake.
95
96=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
97front of the daemon. A request from one of them is attributed, for API
98rate limiting, to the last =X-Forwarded-For= hop that is not itself a
99trusted proxy; from anyone else the header is ignored. Empty, the
100default, is right when gitbayd terminates TLS itself.
101
102A reverse proxy in front of gitbay must not buffer responses, or the
103build page's live log arrives only when the build ends;
104=X-Accel-Buffering: no= covers nginx.
105
106** [web]
107- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
108 web routes are never registered; in accounts, browser sessions are
109 minted over SSH (=web login=), and users with write access can create
110 repos, comment, and make simple file edits (which commit unsigned,
111 honestly). =password_auth= is reserved and currently rejected.
112- =title= — the instance's display name in the rail and page titles;
113 empty falls back to the site host. Lower case is the convention for
114 gitbay itself.
115- =privacy_notice= — operator text shown on =/privacy= under the fixed
116 statement.
117
118** [registration]
119- =mode= — =closed= (default) | =invite= | =open=. invite/open require
120 [mail]. See the user guide for the flows.
121
122- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
123 self-registered account still unverified after that long is removed,
124 hourly and at start, audited as =pending.expired=.
125
126- =notify_admin= (false) — mail every instance admin when an account
127 becomes active: an invite redeemed, or an open-mode signup that
128 verified its address. The unverified row an open signup creates is
129 not reported, because anyone can post the form and mailing on that
130 would point a flood at the admins. Recipients are the verified
131 primary addresses of active admins who have activity mail on, the
132 same rule any other notice follows, so an admin with no verified
133 address hears nothing. The notice is queued, so a dead SMTP host
134 shows up in the admin page's Mail table instead of failing the
135 registration. Requires [mail].
136
137** [mail]
138- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
139 =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
140 registration and self-service =email add=; in closed mode you may omit
141 it entirely and assert addresses by hand (below).
142- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
143 for relays on 465).
144- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
145 gets no mail: delivery fails and retries, and the admin page's Mail
146 table shows the error. Unset, it is on for every relay except
147 =localhost= and loopback addresses; set =false= to allow plaintext
148 to a remote relay.
149
150** [mail.inbound]
151Reply by mail: a reply to issue or merge request mail posts a comment
152(#295). gitbayd polls a mailbox over IMAP; no port is opened for it.
153Off unless =enabled=, and it requires =[mail] smtp_host=, since the
154replies answer mail gitbayd sends.
155
156#+begin_src toml
157[mail.inbound]
158enabled = true
159imap_host = "imap.example.org" # host:port; 993, or 143 with tls = "starttls"
160tls = "implicit" # or "starttls"; there is no plaintext setting
161user = "reply@gitbay.example"
162password_file = "/etc/gitbay/imap.pass" # one line, mode 0600, owned by gitbayd's user
163mailbox = "INBOX" # default
164poll_interval = "1m" # default; at least 10s
165reply_address = "reply@gitbay.example"
166require_dkim = true # recommended; see below
167trusted_authserv_id = "mx.example.org" # the mail host's Authentication-Results id
168#+end_src
169
170- The password is read from =password_file= at every connection and
171 never appears in the config, argv or the log. An unreadable file, or
172 one readable by group or others, stops the daemon at start.
173- =reply_address= is what each Reply-To is built from:
174 =reply@gitbay.example= becomes =reply+<token>@gitbay.example=. The
175 mailbox must receive that. Hosts that deliver subaddresses to the
176 mailbox need nothing more; others need a wildcard rule. Migadu, for
177 one, sent =threads+x@gitbay.org= nowhere until a rewrite from
178 =threads+*= to =threads@gitbay.org= was added. A domain catch-all only
179 works if it points at this mailbox, which then also holds everything
180 else sent to the domain. Send a test to =<reply_address>+test= and
181 expect a =refused mail reply= audit row with "malformed reply token"
182 within a poll interval (=gitbay audit --action 'refused mail'=). Point
183 the domain's MX at the mail host as for any other mailbox; nothing
184 about it involves gitbayd.
185- Use a mailbox that holds nothing else. gitbayd reads every unseen
186 message in it and marks each one =\Seen= when it is handled, posted
187 or refused. A message that fails for a reason that may pass (the
188 database busy, the server refusing that one fetch) stays unseen and is
189 tried again on the next poll, up to five times, then is marked seen
190 and audited. A dropped connection or a timeout ends the poll and
191 counts against no message. A
192 message over 10 MiB is refused by its size without being fetched. A
193 server that sends more than about 11 MiB, or more than a thousand
194 untagged responses, for one command has its connection closed; one
195 poll handles at most ten thousand messages.
196- Whatever the settings, a message is refused when a header field name
197 is not RFC 5322 =ftext= (=From : x=, a space or a non-ASCII byte in a
198 name), when it does not have exactly one =From=, or when it has more
199 than one =To=, =Cc=, =Message-ID=, =Content-Type= or
200 =Content-Transfer-Encoding=.
201- =require_dkim= (default false) should be on for any instance
202 reachable from the internet. With it, a reply is posted only when
203 gitbayd itself verifies one of its DKIM signatures (RFC 6376) with a
204 =d== in relaxed alignment with the From domain (the same
205 organizational domain by the public suffix list; =d=github.io= aligns
206 with nothing) and an =h== that covers =From=, the =To= or =Cc= holding
207 the reply address, =Content-Type=, and =Message-ID= when the message
208 has one. A =Content-Transfer-Encoding= outside =h== is accepted only
209 when it is =7bit=, =8bit= or =binary=, which leave the decoded body as
210 it is; Thunderbird, for one, does not sign it. An unsigned
211 =quoted-printable= or =base64= is refused. The reply
212 address is read only from =To= or =Cc=: a reply that reached the
213 mailbox by Bcc, with the address only in =Delivered-To=, is refused
214 ("reply address not in To or Cc"). It needs nothing from the mail
215 host, so it works where the host adds no =Authentication-Results=.
216 rsa-sha256 (keys of 1024 bits or more) and ed25519-sha256 are
217 accepted, with simple or relaxed canonicalization; rsa-sha1, a body
218 length tag (=l==), an expired =x== and a =t== more than fifteen
219 minutes ahead are refused. Only the first five signatures are
220 checked. The key is looked up at =<s>._domainkey.<d>= with a
221 five-second timeout and cached for fifteen minutes (the resolver does
222 not report the record's TTL); a lookup that fails for a reason that
223 may pass (a timeout, SERVFAIL) leaves the message for the next poll,
224 up to the five tries above, while a missing key refuses it.
225 Signatures are checked on the message as fetched. Each passing
226 signature is recorded with the =Message-ID=, so a copy of the same
227 signed message posts once even with unsigned fields changed.
228- Either =require_dkim= or =trusted_authserv_id= passing is enough to
229 authenticate =From=; when both are set, a reply needs only one of
230 them, and a refusal names both reasons. With only
231 =trusted_authserv_id=, the reply address may come from any recipient
232 field (=Delivered-To=, =X-Original-To=, =Envelope-To=, =To=, =Cc=),
233 since the mail host vouches for the sender and not for the fields.
234 Set both when the mail host adds =Authentication-Results= for most
235 senders but not all.
236- =trusted_authserv_id= names the authserv-id the mail host writes at
237 the start of its =Authentication-Results= header (Gmail's is =mx.google.com=).
238 With it set, a reply is posted only when the topmost header with that
239 id shows =dmarc=pass= with =header.from= equal to the From domain, or
240 =dkim=pass= with a =header.d= in relaxed alignment with it (the same
241 organizational domain by the public suffix list; a public suffix such
242 as =github.io= aligns with nothing). The header is parsed per RFC
243 8601, so text inside a quoted string or a comment (a quoted MAIL FROM
244 local part, a reason) is never read as a result. Lower headers
245 claiming the same id are the sender's and are not read. This is only safe when the mail host
246 removes incoming =Authentication-Results= headers that claim its id,
247 as RFC 8601 asks; Gmail, Fastmail and Migadu do. Check yours before
248 relying on it. With neither this nor =require_dkim= set, the daemon
249 logs a warning at start and =admin mail inbound check= repeats it:
250 =From= is then whatever the sender wrote. Mail between two addresses
251 at the same host may carry no =Authentication-Results= at all: at
252 Migadu, mail from another
253 Migadu-hosted domain is delivered through its outbound path and gets
254 none, so setting the id alone refuses every reply from such users.
255 That mail does carry a DKIM signature aligned with =From= (for
256 example =d=cleberg.net; s=key1; a=rsa-sha256; c=simple/simple;
257 h=from:to:subject:date:message-id:mime-version:content-type=), which
258 is why gitbay.org sets =require_dkim= instead.
259- =gitbay admin mail inbound check= logs in, opens the mailbox
260 read-only (EXAMINE) and reports the message and unseen counts and how
261 =From= is authenticated (=require_dkim=, =trusted_authserv_id=), so a
262 check never marks a reply seen before the poller reads it. It warns
263 when neither is set. With inbound off it says so and exits 0. Poll
264 failures are logged as =mail reply: poll failed= with the server and
265 the IMAP error.
266
267A reply is posted when all of these hold, checked when it is read:
268
2691. It is not an automatic reply (=Auto-Submitted=, =Precedence: bulk=
270 and the like).
2712. A recipient header carries a reply address whose token verifies
272 and has not expired (thirty days from the mail it came on).
2733. The token's account exists, is active and not disabled, still has
274 reply by mail on, and was created before the token was: an id freed
275 by a delete and reused is not the account the token named. The same
276 holds for the repository.
2774. =From= is one of that account's verified addresses, and, with
278 =require_dkim= or =trusted_authserv_id= set, a DKIM signature
279 gitbayd verified or the mail host's result authenticated it.
2805. The message has a =text/plain= part (HTML-only mail is refused, not
281 converted), and what is left after quoted text and the signature
282 are removed is not empty and fits a comment (64 KiB).
2836. No earlier copy of the message (same =Message-ID=, account and
284 thread) posted.
2857. =issue comment= or =mr comment=, dispatched as the account, accepts
286 it: the account can still read the repository, the thread exists,
287 the repository is not archived, the write budget is not spent.
288
289The comment's audit row is =cmd issue comment= (or =mr comment=) with
290=source: mail=. A refusal writes a =refused mail reply= row with the
291reason and the =Message-ID=, never the message's content, at most sixty
292a minute, and sends nothing back. See Threat-Model for why the token
293and the sender address are both required.
294
295** [push]
296Push notifications to Apple devices, delivered by gitbayd talking to
297APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
298operator-supplied provider key. Off unless configured.
299
300- =enabled= (false).
301- =key_file= — path to the =.p8= provider key from Apple's developer
302 portal (Certificates, Identifiers & Profiles → Keys). It belongs at
303 =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
304 as. Read and validated at startup: it must parse as a PEM-wrapped
305 PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
306 than fill a queue nobody is watching.
307- =key_id=, =team_id= — the key's id and your Apple developer team id,
308 both from the same portal page.
309- =topic= — the app's bundle identifier. *An APNs key belongs to a
310 bundle ID.* gitbay.org pushes to the App Store build under its own
311 bundle id; a self-hoster who wants push ships their own iOS build
312 under their own bundle id, with its own =.p8= key from their own
313 developer account, and points =topic= at that id. There is no way to
314 push to someone else's build, by design — this is Apple's model, not
315 gitbay's.
316- =environment= — =production= or =sandbox=, naming the APNs host
317 rather than taking a URL, so a typo cannot aim the key at a host that
318 is not Apple's.
319
320All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
321are required when =enabled= is true; validation runs at config load,
322so a misconfigured =[push]= is caught before the daemon serves
323anything. The delivery queue (a device's undelivered and attempted
324pushes) is capped the same way the mail queue is, by =[retention]
325push=.
326
327** [backup]
328- =age_recipients= (optional) — age public keys (=age1...=). When set,
329 =admin backup= encrypts every archive to them and appends =.age= to
330 its name. Generate the pair off the host with =age-keygen=; only the
331 public key goes here, so the host writes archives it cannot read.
332 The restic copy is unaffected: the offsite job stages its own
333 =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
334 not the archives.
335
336** [api]
337- =enabled= (false) — the JSON API surface; see [[API]]. Off
338 means no credential-bearing HTTP endpoint exists at all.
339
340** [webhooks]
341- =allow_local= (false) — permit webhook, mirror and import targets on
342 loopback, private, shared (100.64.0.0/10), link-local or multicast
343 addresses. Leave off unless you know why you need it (SSRF).
344
345** [limits]
346- =clone_timeout= (3600s) — cap on =repo import= fetches. An import
347 takes http and https URLs only, passes the same address check as a
348 mirror sync and is pinned the same way, so it needs git 2.37 too.
349- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
350- =max_asset_bytes= (512MB) — cap per uploaded release asset.
351- =max_snippet_bytes= (1MB) — cap per snippet file.
352- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
353- =max_repos_per_user= (0, unlimited) — repositories an account may own
354 directly; =repo create=, =fork= and =import= refuse past it.
355 Organizations are not capped.
356- =max_bytes_per_user= (0, unlimited) — disk the account's own
357 repositories may take; a push may be no larger than what is left.
358- =pack_concurrency= (3), =pack_per_principal= (2), =pack_queue= (32),
359 =pack_queue_wait= (="60s"=) — git pack generation (clones, fetches,
360 =git archive --remote=, web archive downloads, =repo download= over
361 SSH and the API) over SSH, smart HTTP and git:// shares one
362 budget: this many at once, this many per account (per client
363 address when anonymous: an IPv4 address, or an IPv6 /64), and this
364 many waiting for at most the wait. Anonymous clients together hold
365 at most =pack_concurrency= − 1 slots when it is above 1: an account
366 (SSH key, bearer token or web session) can take the last slot when it
367 is free, and anonymous clients cannot hold it.
368 Past that an SSH client gets "the server is busy…" and exit 1, HTTP
369 gets 503 with =Retry-After: 30=, git:// an =ERR= line; the daemon
370 logs a =pack limit= warning naming the transport and whether the
371 client was signed in, at most once a minute per transport. An SSH client
372 that disconnects while queued leaves the queue; an HTTP or git:// one
373 keeps its place until the wait runs out. A running clone is killed
374 when its client disconnects, or when no write to the client completes
375 for two minutes: a client reading below about 550 B/s, or an HTTP
376 request body that takes over two minutes with nothing written back,
377 is cut. Ref listings (info/refs, protocol v2
378 =ls-refs=) and pushes are outside the budget. A busy =repo download=
379 exits 1 with the same message; over the API it is a 503 with
380 =Retry-After: 60=. For the
381 three counts 0 means the default and a negative value turns that
382 bound off. The defaults suit a four-core host; see [[Performance]].
383 With =ssh.mode = "system"= each SSH session is its own process and
384 SSH clones are not counted.
385- =push_concurrency= (2), =push_per_principal= (1), =push_queue= (16),
386 =push_queue_wait= (="60s"=) — =git receive-pack= over SSH, on a
387 budget of its own so clones cannot starve pushes or the reverse:
388 this many at once, this many per account or deploy key, and this
389 many waiting for at most the wait. One principal may have up to half
390 of =push_queue= (at least one) waiting, so parallel pushes from one
391 account queue rather than being refused, but cannot fill the queue.
392 A deploy key is its own principal, apart from the account that
393 registered it, so an account with deploy keys can hold
394 =push_per_principal= slots per key;
395 the global cap still holds. A bot account such as a runner's counts
396 like any other account. The slot is taken after the access checks and held
397 until receive-pack exits, which is after =post-receive= (merge
398 detection, CI queueing, webhooks) has run. Past that the client gets
399 "the server is busy: it is at its limit of concurrent pushes…" and
400 exit 1, and the daemon logs a =push limit= warning at most once a
401 minute. A client that disconnects while queued leaves the queue; one
402 that disconnects mid-push ends receive-pack and frees its slot, and
403 a revoked key kills it. 0 and negative values read as for =pack_*=.
404 HTTP and git:// carry no pushes. With =ssh.mode = "system"= pushes
405 are not counted and the two timeouts below do not apply.
406- =push_idle= (="60s"=), =push_receive_timeout= (="15m"=) — a push
407 holding a slot is killed, its process group with it, when its
408 pre-receive has not started =push_receive_timeout= after it took the
409 slot, or when no byte has been read from the client and none written
410 to it for =push_idle=. The idle rule starts when the pack signature
411 arrives from the client or pre-receive starts, whichever is first
412 (a delete-only push sends no pack): before that the client may be
413 silent while its =pack-objects= counts and compresses, and only
414 =push_receive_timeout= applies. receive-pack runs with
415 =receive.keepAlive= set to a quarter of =push_idle= (1 to 5 seconds),
416 so the keepalives it sends to side-band clients while it indexes and
417 runs hooks keep a working push alive. Once pre-receive has started
418 only =push_idle= applies: post-receive is never cut short by the
419 clock. A push killed before pre-receive answers updates no refs.
420- =max_pack_bytes= (2 GiB) — the largest pack one push may send,
421 enforced as =receive.maxInputSize= and lowered to what an owner's
422 storage quota has left.
423- =ssh_auth_rate= (10) — SSH authentication failures per client address
424 per minute on the embedded listener; see below.
425
426** [git_daemon]
427- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
428 Serves only public repositories that additionally ran
429 =repo settings git-daemon <repo> on=.
430
431** [mirrors]
432- =pull_interval_minutes= (15) — how often pull mirrors fetch their
433 upstream. Push mirrors sync shortly after each local ref update.
434 Mirror URLs pass the same SSRF rules as webhook targets, when saved
435 and again before every sync; git then connects only to the addresses
436 that were checked (=http.curloptResolve=) and does not follow
437 redirects, so a mirror of a renamed repository fails until its URL
438 is updated. Needs git 2.37 or later on the server; with an older git
439 the worker logs an error at start and syncs no mirror, recording the
440 reason on each. Sync ignores the system and global gitconfig.
441
442** [go_import]
443Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
444Requests with =?go-get=1= at or under the module path answer with the
445go-import meta tag pointing at the repository's HTTPS clone URL, so
446=go install host/module/cmd/...@latest= resolves. The repository should
447be public (the module path itself confirms it exists).
448
449* Users, email, invites
450
451Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
452backfills is a wrapper that dispatches the registry command of the same
453name as the host: an admin context with no account behind it, so its
454audit rows carry no actor and =source: host=. The same commands run in an
455instance admin's SSH session (=ssh git@<host> admin ...=) and write the
456same rows with the key fingerprint as source. One implementation, two
457credentials.
458
459#+begin_src sh
460gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
461gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
462gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
463#+end_src
464
465"Verified" means SMTP-confirmed or host-admin-asserted; the database
466records which. Verified emails are what make commit signatures
467meaningful — an unverified address never produces a =verified= badge.
468
469* Audit and account control
470
471The audit log is the security feed (events are the product feed): every
472successful mutating command with its argv and source credential (SSH key
473fingerprint or API), every refused one (exit 3 or 4) as =refused
474<command>=, refused pushes as =refused git-receive-pack= (the access
475check) or =refused push= (a branch or tag rule, a release anchor or an
476unsigned commit, with the repository and ref names), hook socket
477requests failing the peer or push-token check as =refused hook= (never
478the token), registrations, admin actions, force-pushes, and auth
479failures/throttling. A refusal row
480keeps the flag names and the first positional, not the values.
481Refusals are recorded up to ten a minute per account and 600 a minute
482across the instance; past either, one =refused.throttled= row stands
483for the rest of that minute. The caps bound the embedded listener, the
484web and the API; under =ssh.mode = "system"= each =gitbayd shell=
485connection counts separately. Secrets never appear — they travel on
486stdin, never in argv.
487
488Each row carries the SHA-256 of the row before it. =gitbayd admin audit
489verify= opens the store as other admin commands do, applying pending
490migrations, so run it with the binary that matches the daemon. It
491recomputes the chain and exits 1 naming the first row that was
492edited or whose predecessor was removed. The chain is unkeyed: whoever
493can write the database can recompute every hash after an edit, and
494verify then finds nothing. It catches an edit only when the later
495hashes were not recomputed. Retention removing the oldest
496rows is not a break. Rows written before the chain existed are counted
497and skipped; when every row is such a row, verify warns and exits 1,
498since clearing the hash columns looks the same. After an upgrade that
499clears with the first new audit row.
500
501Removing the newest rows leaves no break, and neither do rows written
502afterwards under the freed ids. The database cannot show either. The
503daemon logs every row it writes to its journal, outside the database
504(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
505id and hash: comparing them with the newest journal line is the check
506for any change, recomputed hashes included. Rows written by
507host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
508"system"=, are not copied to the journal.
509
510#+begin_src sh
511gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
512gitbayd admin audit verify # check the hash chain; exit 1 names the first bad row
513ssh git@<host> audit ... # the same, from an admin session
514ssh git@<host> admin user list [--state active|pending|disabled|admin]
515ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
516ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
517ssh git@<host> admin user promote <name> # grant instance admin
518ssh git@<host> admin user demote <name> # remove it; the last admin is refused
519gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
520gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
521gitbayd admin user enable <name> # sessions dropped, nothing deleted
522gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
523 # refused (with each blocker named) while
524 # the account owns repos, authored
525 # issues/MRs/comments/reviews, or is an
526 # org's only admin
527#+end_src
528
529=--actor= takes a username, or =-= for rows with no actor: host commands
530and auth failures. =--action= is a prefix, so =cmd repo= catches every
531repository command and =admin= every host or admin-session action.
532=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
533
534=admin user list= pages by username (=--limit=, =--cursor=) and carries
535each account's state and =last_seen=, the newest use of any of its SSH
536keys or API tokens. =admin user show= adds the keys with their last use,
537each address with how it was verified, PGP keys, org roles, the owned
538repository count, API token names, and live browser sessions. Both are
539Both are refused to non-admins, like =audit=, on every surface.
540
541=/admin/users= is the same list in a browser, linked from the admin
542page: the state filter the command takes, keyset paging on its cursor,
543and a row per account with promote, demote, disable and enable, each
544dispatching the command. Demote and disable ask for the username to be
545typed, since both take someone's access away. Creating and deleting an
546account, issuing an invite and asserting an address stay on the command
547line: each takes a key, mints a credential, or cannot be undone. A
548non-admin gets the 404 a missing page would, so the URL confirms
549nothing.
550
551Promotion needs an active account: a pending or disabled one is refused.
552Demotion is refused when it would leave no admin, over SSH, in the
553browser and on the host alike, so the host-local =promote= is the way
554back in when the only admin key is lost.
555
556Instance admin carries no right on anyone's repository: policy does not
557consult it, and a private repository still answers not-found to an
558admin. Moderation goes through explicit overrides that skip the access
559check and write their own audit row:
560
561#+begin_src sh
562ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
563ssh git@<host> admin repo archive|unarchive <owner/name>
564ssh git@<host> admin repo visibility <owner/name> public|private
565ssh git@<host> admin repo delete <owner/name> --yes
566#+end_src
567
568Each lands in the audit log as =admin repo.<action>= naming the
569repository, on top of the =cmd= row every mutating command gets.
570
571=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
572per minute — successful auths never count and clear the slate.
573=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
574push.
575
576=limits.write_rate= (60) bounds *mutating commands per account per
577minute*. It is counted in the dispatcher, so SSH, the JSON API and the
578web spend one budget and a caller cannot refresh it by changing surface;
579=limits.api_rate= stays in front of it, bounding a network source rather
580than an account. A command is one token whatever it writes, so a bundle
581import costs one and only a loop of separate commands spends the budget.
582Read-only commands, the runner protocol (a build streams its log in many
583small writes) and the host CLI are exempt. Refusals exit 4 and say when
584to retry. A negative value turns the limit off; it matters most with
585=registration = "open"=, where every write also queues notification mail
586and webhook deliveries.
587
588* Queues
589
590Every background worker keeps a backlog and a failure state. An instance
591admin reads them all in one place:
592
593#+begin_src sh
594gitbay dashboard --json | jq .queues # webhooks, mail, push, mirrors, builds, deps
595#+end_src
596
597Per worker: pending, retrying (pending with a failed attempt) and
598dead-lettered counts with the oldest pending age, and the retrying or
599failed rows themselves, capped at twenty each. Builds list what is
600running and then what is pending, each since when, so a build no runner
601is scoped to claim is visible here rather than only in its repository;
602mirrors list the ones whose last sync failed;
603dependency checks list the ones whose last check errored. Non-admins get
604no =queues= key at all.
605
606Push rows name the device id, never the token. Watch this one after
607configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
608config validation, which can only check that the =.p8= parses, and then
609every send comes back =403 InvalidProviderToken= and dead-letters on its
610first attempt.
611
612In accounts mode the same read renders at =/admin=, linked from the rail
613for admins. Anyone else gets a 404 there.
614
615A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
616with the address redacted out of the relay's error. The id is the queue
617row: find it in the Mail table on =/admin=, or in =dashboard --json=,
618where the recipient and the unredacted error are. That is deliberate —
619see the Threat-Model page.
620
621* Maintenance
622
623#+begin_src sh
624gitbayd admin stats [--json] # counts, database size, per-repo disk
625ssh git@<host> admin stats [--json] # the same, from an admin session
626gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
627gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
628gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
629#+end_src
630
631A history rewrite leaves the commits it removed reachable through
632=refs/merge-requests/N/head= of the merge requests that landed them, so
633they stay fetchable by anyone who can read the repository. Nothing drops
634a head ref on its own — an open or source-gone MR is merged through it,
635and a merged or closed one keeps its diff readable through it — so the
636cleanup is a command an instance admin runs, naming the MRs:
637
638#+begin_src sh
639ssh git@<host> admin mr prune owner/name 1 2 3 --yes
640#+end_src
641
642It refuses an open or source-gone MR, deletes the named refs, runs
643=git gc --prune=now= on that one repository so the objects go at once
644rather than after git's two-week grace, leaves a system comment on each
645MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
646reviews and head sha; =mr diff= and the MR page say the head is gone.
647Run it when nothing is pushing to that repository: without the grace, a
648push caught between leaving quarantine and writing its ref loses its
649objects. Objects also survive in offsite backups until those are
650rewritten; see "Removing a repository's history from every snapshot".
651
652=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
653weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
654layout the source sent, so a first manual =admin gc= after a bulk
655import is worthwhile.
656
657* Backup and restore
658
659#+begin_src sh
660gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
661gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
662#+end_src
663
664One archive: a consistent SQLite snapshot (taken *before* the
665repositories are read, so the database never references objects the
666archive missed), every repository, and the SSH host keys. Excluded:
667hook socket, regenerated hook scripts, WAL files. Safe to run against a
668live daemon. =--out= must be outside =server.root=, or the next full
669backup would carry the archive. Each run removes the snapshot
670directories (=.gitbay-snap-*=) and temporary archives (=.*.tmp-*=) a
671killed run left beside its archive once they are a day old, and prints
672each one it removes.
673
674=--verify= reads an archive back: the snapshot must pass SQLite's
675integrity check, every repository the snapshot names must be in the
676archive, and each must pass =git fsck --connectivity-only=. Every
677release asset the snapshot names must be in its repository with its
678recorded size and sha256, and every LFS object in the archive must
679hash to its name. LFS objects are named by pointer files in git
680history rather than by the database, so a pointer whose object was
681never uploaded is not caught, and with =[lfs] root= outside
682=server.root= the archive holds no LFS objects at all. It extracts the
683repositories to a temporary directory for the checks, so it needs free
684space the size of the repositories. A database-only archive is checked
685for integrity and says so. Exit is non-zero on damage, a missing
686repository, a missing object, or a missing or altered asset or LFS
687object.
688
689A full backup holds =<root>/backup.lock= from its database snapshot to
690its last repository. While it runs, =repo delete=, =repo rename=,
691=repo transfer=, =admin repo delete=, =org rename=, =admin mr prune=
692and =gitbayd admin gc= refuse with "a backup is running"; retry when it
693finishes. Database-only backups take no lock. A pack that git's own
694automatic gc removes during the walk is skipped; each repository's refs
695are archived before its objects, so the refs still find their objects,
696and =--verify= reports it if one does not.
697
698Verifying an archive of unknown origin: run it as an unprivileged
699user. The connectivity check runs git with =--git-dir= on each
700extracted repository, so a directory that is not a repository fails
701rather than git checking an enclosing one. =objects/info/alternates=
702and a =commondir= directly in a =*.git= directory are not extracted,
703so an archive cannot use either to have git read another repository's
704objects or refs on the host. Git still reads each archived
705repository's own =config=.
706
707Archives carry a directory entry for every directory, including an
708empty one, so a bare repository whose refs are all packed restores as
709a repository. Archives written before this release do not: extracting
710one can leave a repository's =refs/= directory missing, which stops
711git from recognizing it as a repository at all. =gitbayd admin backup
712--verify <archive>= names the repositories this affects; the fix is
713=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
714which it opens normally.
715
716With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
717and =--verify= needs the private key:
718
719#+begin_src sh
720gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
721age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
722#+end_src
723
724The identity lives off the host (with the secret key file and the
725restic credentials), so verifying an encrypted archive happens there
726or on a restore host.
727
728Restore: extract into an empty directory, point =server.root= at it,
729restore =server.secret_key_file= from its own copy (mode 0600, owned
730by the account gitbayd runs as), then start gitbayd. No archive carries
731the key file, and without it gitbayd refuses to start. Host keys are
732preserved, so clients keep their known_hosts entries; hooks regenerate
733at startup.
734
735** Schedule and recovery point
736
737Two timers, because the two halves of the data have different exposure.
738
739- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
740- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
741 writes the SQLite snapshot alone, last 48 kept. A few MB against the
742 full archive's hundreds, which is what makes the frequency affordable.
743
744The split follows what a loss would actually cost. Repositories are git,
745so a mirror or any clone is a second copy; the database is the only copy
746of issues, merge requests, comments and review state. So the recovery
747point is about an hour for the data that exists nowhere else, and a day
748for the data that does.
749
750Continuous replication (litestream and similar) was considered and not
751adopted. It would take the database's recovery point to seconds, but the
752repositories would still be on the nightly archive, so a restore could
753produce a database referencing commits the repository backup does not
754have. Consistency between the two halves is worth more here than latency
755on one of them. Revisit if repository replication becomes continuous
756too.
757
758** Offsite copies
759
760bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
761=/var/lib/gitbay-stage= (a database copy and =config.toml=, staged
762there) to an S3 bucket at Scaleway, with a key that can only add
763snapshots. Nothing else under =/etc/gitbay= is in it: not the secret
764key file, not =apns.p8=. The key that can
765remove them lives on the operator's machine, in
766=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
767cannot destroy its own history. Forgetting, pruning and rewriting all
768run from there.
769
770*** Removing a repository's history from every snapshot
771
772A history rewrite plus =admin mr prune= takes commits off the server,
773but every snapshot taken before it still holds them, and the retention
774window is the only thing that ages them out. To remove them now,
775rewrite the snapshots without that repository rather than forgetting
776the snapshots: everything else in them stays restorable. The next
777nightly run adds the repository back in its current state.
778
779Repositories are stored under the name they had on disk when each
780snapshot was taken, so a renamed repository needs every name it has
781carried. Check what an older snapshot holds before choosing the paths:
782
783#+begin_src sh
784set -a; . ~/.config/gitbay/offsite.env; set +a
785restic $RESTIC_OPTS snapshots
786restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
787#+end_src
788
789Then dry-run, apply, prune, and confirm nothing matches:
790
791#+begin_src sh
792EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
793restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
794restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
795restic $RESTIC_OPTS prune # drops the data nothing references
796restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
797#+end_src
798
799=--forget= is what makes the originals go; without it the rewritten
800snapshots sit beside them and the data stays referenced. Snapshot IDs
801change; their times do not. Done for krz/keycask (formerly rust-pass)
802on 2026-09-18, across 22 snapshots.
803
804** Secret key
805
806CI secrets, webhook secrets, mirror tokens and APNs device tokens are
807stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
808each value prefixed with the id of the key that sealed it
809(=gbs1:<id>:=). The key file is not in the database, not under
810=server.root=, and therefore in neither the local archives nor the
811main restic repository. It must be copied off the host separately;
812without it a restored database's secrets cannot be opened, and
813gitbayd refuses to start against them. A separate restic repository
814for it and =apns.p8= is planned (runbook D of the data-at-rest plan)
815and not yet in place, so today the only off-host copy is one the
816operator makes by hand after =init= and after every =rotate=.
817
818#+begin_src sh
819gitbayd admin secrets init # once; deploy/install.sh does it on first install
820gitbayd admin secrets check # open every value, count by key
821gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
822#+end_src
823
824- Missing file: every gitbayd process that opens the database refuses
825 to run and names the path, including =serve= and, in system mode,
826 =authorized-keys=. =migrate= does not need it.
827- Backups: =admin backup= and =admin backup --verify= do not need the
828 key file; a restore does.
829- Wrong key: =serve= stops at startup naming the first row that does
830 not open; =secrets check= does the same without starting anything.
831- Upgrade: the first start after the upgrade seals every value still
832 in clear and logs =sealed secret values=.
833- Rotation: =rotate= adds a key, reseals every value under it in one
834 transaction, then removes the old keys. Run it as root, since it
835 replaces the key file in =/etc/gitbay=; the file keeps its owner.
836 The daemon re-reads the file when it changes, so it needs no
837 restart. Copy the new file off the host afterwards. =init= and
838 =rotate= hold an flock on =<key file>.lock= while they run, so a
839 second run waits for the first.
840- Push devices are looked up by the SHA-256 of their token
841 (=push_devices.token_hash=), since two seals of one token differ.
842
843** Restore drill
844
845A restore onto a clean host, run quarterly (January, April, July,
846October) and after any change to the backup code
847(=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite
848job), and recorded below. The disaster it rehearses is losing bay1, so
849the local archives are gone with it and the sources are the main
850offsite restic repository (repositories, LFS, the staged database,
851=config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys
852repository once runbook D creates it; until then the operator's
853hand-made copy), and the operator's password manager (=offsite.env=,
854the keys repository's password and token once it exists,
855=backup-identity.txt=). The full steps are runbook C of the
856data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
857
858=gitbayd admin restore-drill= does the archive half. It extracts a
859full archive into an empty or absent directory, runs every =--verify=
860check on the extracted copy, and prints what was restored, the newest
861issue, issue comment, merge request comment and push in the restored
862database, and the elapsed time. Exit is non-zero if any check fails.
863
864#+begin_src sh
865gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
866gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill
867#+end_src
868
869=admin backup --verify= extracts the archive under =$TMPDIR=. A host
870whose =/tmp= is a tmpfs smaller than a full archive (bay1's is 3.9 GB
871of memory) fails with "no space left on device"; point =TMPDIR= at a
872directory on disk that the backup's user owns:
873
874#+begin_src sh
875install -d -o gitbay -g gitbay -m 700 /var/backups/gitbay/verify-tmp
876cd /var/backups/gitbay
877sudo -u gitbay env TMPDIR=/var/backups/gitbay/verify-tmp gitbayd admin backup --verify <archive>
878#+end_src
879
880What to restore, and from where:
881
882| Item | Source | Path on the drill host |
883|-------------------------------+-------------------------------------------------------------+------------------------------------------------|
884| Database | staged copy in =/var/lib/gitbay-stage= (restic), or an archive | =<root>/gitbay.db= |
885| Repositories | =/var/lib/gitbay/repos= (restic), or an archive | =<root>/repos= |
886| LFS objects | =/var/lib/gitbay/lfs= (restic), or an archive | =<root>/lfs= (or =[lfs] root=) |
887| Release assets | inside each repository (=gitbay-releases/=) | with the repositories |
888| Host keys | =/var/lib/gitbay/ssh= (restic), or an archive | =<root>/ssh= (or =[ssh] host_keys=) |
889| =config.toml= | =/var/lib/gitbay-stage/config.toml= (restic) | =/etc/gitbay/config.toml= |
890| =secret.key=, =apns.p8= | the off-host copy; no archive or main snapshot carries them | =/etc/gitbay/=, mode 0600, owned by =gitbay= |
891
892From the offsite path (restic is not run by any gitbay command):
893
894#+begin_src sh
895restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
896cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
897gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
898gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root
899#+end_src
900
901The second archive is how the restic tree gets the same checks and
902timestamps; =/tmp/drill-root= is discarded afterwards.
903
904What to check, each a column below:
905
906- DB integrity, connectivity, release assets, LFS: =restore-drill=
907 prints =integrity ok=, =connectivity ok on N repositories=, =release
908 assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin
909 stats --json= on the source at the snapshot time.
910- Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets
911 check= opens every value.
912- Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's
913 fingerprint.
914- Config: =gitbayd --config /etc/gitbay/config.toml check-config=.
915- Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over
916 SSH succeed.
917
918Time to service runs from the clean host's first root login to the
919first successful =git clone= over SSH from it. The recovery point is
920the time of the newest restic snapshot restored; record beside it the
921newest issue, comment and push =restore-drill= printed, which show how
922much activity the restore carries.
923
924The first drill ran on the operator's laptop rather than a provisioned
925host, so its time to service has no provisioning in it, and it could
926not check secrets: no copy of =secret.key= was on the machine, and
927gitbayd refuses to start while any sealed value does not open. The
928sealed values were cleared in the drill copy to reach service. The
929off-host copy of the key is confirmed to exist (#305); the next drill
930restores it with the snapshot and runs =admin secrets check= on the
931restored copy.
932
933| Date | Host | Snapshot restored (UTC) | Newest issue / comment / push | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
934|------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
935| 2026-09-29 | laptop (macOS), restic from offsite | 2026-09-29 00:19:15 (ccd646cf) | 00:05:17 / 00:09:44 / 00:15:31 | 8m43s (restore 1m49s, checks 33s) | ok | ok, 69/69 | ok, 2 | ok, 529 | matches | not checked (#305) | 4.99 GiB restored; 71 sealed values cleared in the drill copy to start |
936
937* Upgrades
938
939Replace the binary, restart the unit. Migrations apply automatically and
940are transactional; hook scripts under =<root>/hooks= are rewritten at
941startup to point at the current binary path.
942
943* CI runner
944
945=gitbay-runner= executes builds queued by pushes and merge requests. It
946polls over SSH with a key of scope =runner=, which reaches only the
947runner protocol and read-only git (a runner executes arbitrary
948repository code, so the key it holds must not do more). A runner key
949claims builds only for the repositories it is attached to, by =repo
950runner add= from a repository admin or an instance admin; an admin key
951claims any. Users attach their own runners: see the Users page. For an
952instance runner, run it as a dedicated unprivileged user on a non-admin
953account. =admin user create --key= registers a full-scope key, so the
954runner key is added afterwards through a bootstrap key that is then
955removed, and attached to each repository it should build:
956
957#+begin_src sh
958useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
959sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
960ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
961gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
962ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
963ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
964rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
965gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
966#+end_src
967
968#+begin_src sh
969gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
970#+end_src
971
972=-jobs N= runs N builds at once. Claiming is one transaction that
973selects and updates, and each build works in its own =build-<id>=
974directory, so workers do not collide; idle polls are staggered across
975the interval so N of them do not wake together. The drop-in's weights
976below are per service, not per build, so raising =-jobs= divides them
977rather than multiplying the host's load.
978
979=admin runners= shows which account each runner polls as, and what each
980may claim. A runner key claims builds only for the repositories it is
981attached to: with none attached it claims nothing, and =-repos= may only
982narrow within them. An admin's full-scope key claims any repository —
983that is what =-repos= was for — and still works for the protocol during
984a rotation. A merge request head from a fork is built in the target
985repository as untrusted: the claim carries no secrets, and only a runner
986started with =-untrusted= takes it. Same-repository heads were built by
987their branch push and are not built again.
988
989=make deploy-runner= also installs
990=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
991=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
992the daemon or the backup timers, and =NoNewPrivileges=,
993=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
994and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
995and the runner's home. The e2e suite alone starts sixty
996daemon instances; without the drop-in a deploy's copy over the admin
997sshd stalled. Both deploy targets copy with =rsync --partial=, which
998resumes a stalled transfer.
999
1000Under =-isolation podman=, the default and what bay1 runs, each build
1001is confined to a container (see Container isolation below). Under
1002=-isolation none= steps run directly on the host as the runner's user,
1003so treat that machine as executing whatever your users push, and
1004install the toolchains your builds need on it.
1005
1006A runner claims the oldest pending build among the repositories its key
1007is attached to — for an admin key, the oldest in the instance. =-repos=
1008narrows within that set, which is what makes a runner outside the server
1009practical: one on a machine that should build a single project, or that
1010holds credentials for one deployment, stays on it.
1011
1012Oldest-first is across everything the key may claim, so a repository
1013with a deep queue holds every other repository the same runner serves;
1014bay1 measured a 15-minute average wait on a day of merge request
1015stacks from one repository. A runner attached to one repository cannot
1016be starved. That is the rule, decided in krz/gitbay#207: a runner
1017serving several repositories takes them oldest-first, and an operator
1018who wants one repository never to wait on another runs a second
1019runner attached to it alone. Nothing caps what an account queues,
1020and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
1021while the job's last build is pending or running, so a repository
1022with no runner holds one row per scheduled job rather than one per
1023tick, and a build runs only on a runner its owner attaches, so a busy
1024schedule spends the owner's compute. Pushes are bounded by what an
1025account can push.
1026
1027#+begin_src sh
1028gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
1029 -workdir /var/lib/gitbay-runner/work
1030#+end_src
1031
1032Add =-untrusted= only with =-isolation podman=.
1033
1034gitbay.org's runner is attached to the forge's own repositories and
1035the isolation canary, nothing else, because it shares the host with
1036the forge; its unit names no =-repos=, the attachments are the
1037boundary. Any other repository builds on a runner its owner attaches.
1038
1039=-repos= narrows an admin runner; for a runner key the attachments are
1040the boundary, held by the server, and =-repos= may only name
1041repositories among them. =-untrusted= makes a runner claim merge
1042request heads from forks; the bay1 unit sets it because it isolates in
1043podman. A runner without it builds trusted commits only.
1044
1045=gitbay dashboard= and =ssh git@<host> admin runners= list every key
1046that has polled as a runner: the account, the key's fingerprint, when it
1047last polled, the repositories it may claim — its attachments for a runner
1048key, the =-repos= it asked for or =any= for an admin key — and the build
1049it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
1050that polled by mistake, the key itself untouched. =admin runners= also
1051heads the list with the queue: builds
1052pending now, and over the last day how many were claimed, how long they
1053waited to be claimed (average and worst), and
1054how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
1055reported is failed by the scheduler's minute tick, whether or not any
1056runner is still alive: within about two minutes of its log stream ending
1057with no outcome reported — the runner reports right after closing the
1058stream, retrying for half a minute if gitbayd is unreachable — or, if no
1059stream was ever seen, at the deadline.
1060
1061Instance admin on the runner account only authorizes the claim/report
1062protocol; it grants no repo access. A build that pushes back — a pages
1063deploy, an archive publish, an automated MR branch — needs an explicit
1064grant on that repo: =repo access grant <owner/name> ci write=. Private
1065repos likewise need at least read for the clone.
1066
1067** Container isolation
1068
1069Builds run in a rootless podman container, one per job, with the
1070workspace bind mounted and nothing else: the clone happens outside with
1071the runner's key, so a step cannot read it. =-isolation none= keeps the
1072old behaviour — steps on the host as the runner's user — for an instance
1073where every repository is trusted. There is no automatic fallback: a
1074runner started with =-isolation podman= that cannot find a working
1075podman exits rather than running a build unsandboxed.
1076
1077gitbay's own jobs name =localhost/gitbay-ci:2=, built from
1078=deploy/Containerfile.ci= on the runner host. A job's image must carry
1079what its steps need: the suite drives real git, git-lfs, gpg and sshd and
1080asserts they exist before running, so the stock runner default would fail
1081it immediately. Build or rebuild it with:
1082
1083#+begin_src sh
1084ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
1085ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
1086 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
1087#+end_src
1088
1089The tag is deliberate rather than =:latest=: changing the file means
1090bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
1091change under it.
1092
1093=-image= names the image a job runs in when it declares none, and is
1094required under =-isolation podman=: there is no built-in default,
1095because an image this host does not have would fail every build. A job
1096overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
1097so a config file cannot turn it into podman arguments.
1098
1099=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
1100-memory 4g=); unset means uncapped. The runner applies them itself: it
1101creates a cgroup per build under its own delegated service cgroup,
1102writes the limits there, and starts every podman process for the build
1103inside it, with podman's cgroup handling off. Podman's own =--memory=
1104and =--cpus= never applied under rootless cgroupfs, which is what a
1105system service gets (krz/gitbay#188). The unit therefore needs
1106=Delegate=yes=, which the drop-in sets; without it the runner refuses
1107to start when a limit is set, and logs that builds run unconfined when
1108none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
1109and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
1110swap: the memory cap is what keeps the forge alive when a build
1111allocates without bound, and it sits above the e2e suite's 5GB peak
1112rather than at a fair share. =OOMPolicy=continue= keeps systemd from
1113stopping the runner when a build is OOM-killed.
1114
1115A trusted build's home is its repository's, under
1116=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
1117=HOME=: caches persist between trusted builds of one repository and are
1118never read by another's. An untrusted build — a merge request head from
1119a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
1120the build ends. Homes under =<workdir>/home= are from runners before
1121krz/gitbay#255, which shared them with untrusted builds; nothing reads
1122them any more, and they can be deleted.
1123
1124*Images are provisioned, never pulled by a build.* The runner passes
1125=--pull=never=. Two reasons, and the second is the better one: the
1126service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
1127holding a setuid file, which is nearly every distribution image; and on
1128an instance where anyone can push a =ci.yml=, =image:= would otherwise
1129mean "fetch and run anything from the internet". An operator pulls or
1130builds what is allowed and a build picks among those. A job naming an
1131image the host does not have fails with a message saying so.
1132
1133#+begin_src sh
1134su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
1135su - ci-runner -s /bin/sh -c "podman images"
1136#+end_src
1137
1138Prepare a host before pointing an isolating runner at it:
1139
1140#+begin_src sh
1141ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
1142make deploy-runner
1143#+end_src
1144
1145The script installs podman, delegates a subuid/subgid range to
1146=ci-runner=, checks that user namespaces are enabled rather than
1147assuming, enables lingering, and verifies rootless podman actually runs
1148as that user. It is idempotent.
1149
1150It also installs nftables. =make deploy-runner= ships
1151=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
1152with =gitbay-runner-egress.service=, which loads it and which the
1153runner's unit requires; it checks the file with =nft -c=, reloads the
1154unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
1155restarting the runner: =127.0.0.1:22= and the public 22 must answer,
11562222 must not. The table limits the runner's user to =127.0.0.1:22=,
1157DNS on loopback, and 22, 80 and 443 on the host's public address; the
1158Threat-Model page says why.
1159
1160That table cannot tell a build from the runner, since both run as
1161=ci-runner=. =deploy/gitbay-runner-builds.nft=, shipped as
1162=/etc/gitbay-runner/builds.nft=, matches by cgroup instead: the runner
1163starts each build under =builds/trusted= or =builds/untrusted= in its
1164service cgroup, and the table closes the host's loopback to every
1165build, limits an untrusted build to TCP 80 and 443 and DNS, and closes
1166private ranges to both (the CI page has the table). nftables resolves a
1167cgroup path to its id when the table loads, and the service cgroup is
1168new on every start, so the drop-in's =ExecStartPre= creates the two
1169cgroups, hands them to =ci-runner= and loads the table before the
1170runner starts; a table that does not load stops the start. =make
1171deploy-runner= checks the file's syntax before the restart and that the
1172table is loaded after it. If =/etc/resolv.conf= names a nameserver in a
1173private range, allow it in the file first or builds resolve nothing.
1174
1175A restart of =nftables.service= flushes both tables;
1176=systemctl reload gitbay-runner-egress= restores both.
1177
1178The egress unit's stop and the rollback below use =nft destroy=,
1179which needs nftables 1.0.8 or later; check =nft --version= on a new
1180host.
1181
1182Checking the tables on a running host, during a build (the flood test
1183below waits a minute before its first probe, for this):
1184
1185#+begin_src sh
1186nft list table inet gitbay_builds # counters on the reject rules
1187for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
1188# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>
1189#+end_src
1190
1191A pasta process anywhere else — the runner's own =runner= cgroup, a
1192user slice — means the builds table does not see that build's traffic
1193and only the first table applies. Run
1194=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
1195once as a push and once with =--untrusted=. It fails if the runner was
1196locked out or if the build log lacks the lines only a working table
1197produces. The authoritative proof that pasta's sockets are in the
1198build's cgroup is the untrusted run: =169.254.1.2:22= and
1199=github.com:22= refused, all twelve logins refused, and the counters on
1200the =untrusted= chain's rejects rising. The first table lets
1201=ci-runner= reach both of those addresses.
1202
1203To take the builds table out, on the host:
1204
1205#+begin_src sh
1206sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
1207rm /etc/gitbay-runner/builds.nft
1208systemctl daemon-reload
1209nft destroy table inet gitbay_builds
1210#+end_src
1211
1212The runner needs no restart. It still places builds under
1213=builds/trusted= and =builds/untrusted=, but with the file gone neither
1214a runner start nor =systemctl reload gitbay-runner-egress= loads the
1215table again, and builds keep the first table and =--no-map-gw=. A
1216runner that takes =-untrusted= or polls over loopback refuses to start
1217without build cgroups at all, since the table would match nothing. The
1218next =make deploy-runner= installs the file and the lines again.
1219
1220The drop-in sets =NoNewPrivileges=no=, without which rootless podman
1221cannot call =newuidmap= and the runner refuses to start. That is a
1222considered trade, explained in the file and in the Threat-Model; if you
1223run with =-isolation none=, set it back to =yes=.
1224
1225*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
1226the build in flight, reports it, and exits; the drop-in's
1227=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
1228is what makes the signal reach the runner alone — under systemd's default
1229the build's container and the log session are signalled with it, and the
1230runner drains a build that is already dead. So =make deploy-runner=
1231waits for a running build rather than orphaning it, and a build's result
1232is retried for half a minute if gitbayd is restarting at that moment. A
1233second SIGTERM ends the runner at once, abandoning the build to the
1234reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
1235the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
1236=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
1237the runner as a transient user unit under =systemd-run= and stops it
1238under both kill modes. That last one needs a systemd user manager, so it
1239skips in the container CI runs in; run it on a Linux host with
1240=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
1241
1242*Validate podman mode on a scratch repository before pointing the runner
1243at real ones.* Every deploy that switched the whole instance to
1244containers and failed took CI down with it. Instead: create a throwaway
1245repository the runner account can read (public, or granted read — a
1246private one is "not found" to the runner and the build stays pending),
1247give it one job that names the CI image, attach the runner's key to it,
1248and deploy the runner with =-repos= naming only that repository. The
1249production unit, with its real hardening, then claims nothing else;
1250other repositories' builds queue until =-repos= is removed again, which
1251is a pause, not an outage.
1252
1253#+begin_src sh
1254# on a machine with an admin gitbay identity (root on the host has none),
1255# with the runner's public key copied from the host:
1256gitbay repo create cmc/runner-scratch # then push a .gitbay/ci.yml naming the image
1257scp -P 2222 root@<host>:/var/lib/gitbay-runner/.ssh/id_ed25519.pub runner.pub
1258gitbay repo runner add cmc/runner-scratch < runner.pub
1259# on the host:
1260sed -i 's#^ExecStart=/usr/local/bin/gitbay-runner #&-repos cmc/runner-scratch #' /etc/systemd/system/gitbay-runner.service.d/override.conf
1261systemctl daemon-reload && systemctl restart gitbay-runner
1262# back on the admin machine:
1263gitbay build log cmc/runner-scratch 1 # green: remove -repos, redeploy, delete the scratch repository
1264#+end_src
1265
1266*Do not deploy an isolating runner to a host that has not been
1267prepared.* The runner is specified to refuse to start without a working
1268podman rather than fall back to running builds unsandboxed — a fallback
1269that silently drops isolation is worse than a stopped runner, because
1270nothing surfaces it. On an unprepared host that refusal stops every
1271build on the instance.
1272
1273The service drop-in carries =Delegate=yes= for rootless cgroup
1274management and =ReadWritePaths= for podman's store under
1275=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
1276make read-only. Those paths are prefixed =-= so they are ignored when
1277absent: the drop-in installs on unprepared hosts too, and a unit that
1278refused to start would stop every build.
1279=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
1280user: rootless storage belongs to that user, and root's prune would not
1281see it. An unpruned image store on a 40GB host is a slow outage.
1282
1283* LFS storage
1284
1285Objects live content-addressed under =[lfs] root= (default
1286=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
1287(512MB default). Storage sits behind a small interface — an
1288S3-compatible backend is a drop-in with the server proxying, and
1289presigned URLs a later optimization. LFS objects do not travel with
1290push mirrors (mirrors move git refs only), and gc does not yet collect
1291orphaned objects.
1292
1293* Pages
1294
1295=[pages] domain = "example.site"= serves public repos' =pages= branches
1296on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
1297server; ACME issues per-subdomain certificates on demand (only for
1298owners that exist). The domain must not be the site host or a parent of
1299it — pages content runs its own scripts and must stay off the forge's
1300origin.
1301
1302Users with repo admin claim custom domains with =repo domain add=.
1303Claims activate only after a DNS TXT challenge proves control of the
1304domain (=repo domain verify=, audit-logged); pending claims serve
1305nothing, get no certificates, and expire after 7 days. ACME issues
1306certificates only for verified hosts, so stray DNS pointed at the
1307server gets nothing.
1308
1309* Security
1310
1311The [[Threat-Model]] file is the reference for what the forge
1312trusts and refuses to do. Operational checklist:
1313
1314- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
1315 (the module list is deliberately short — review it on each release),
1316 and a short fuzz pass over every attacker-facing parser (pkt-line,
1317 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
1318 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
1319 against main rather than per push, because =@latest= scans today's
1320 advisory database and an advisory lands without anyone pushing;
1321 =build trigger krz/gitbay vuln= runs it on demand.
1322- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
1323 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
1324 configuration needed.
1325- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
1326 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
1327 =LockPersonality=, =MemoryDenyWriteExecute=,
1328 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
1329 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
1330- *OS patches* apply via =unattended-upgrades= (security origins,
1331 auto-reboot 04:30 if required).
1332- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
1333 watched by =fail2ban=; gitbayd's own port 22 is throttled by
1334 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
1335 account's writes by =limits.write_rate=.
1336- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
1337journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
1338webhook: disk, service, the daemon's own =/healthz= answer, certificate
1339expiry, and the age of the newest full backup and database snapshot.
1340It exits non-zero on an alert so the unit shows in =systemctl
1341--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
1342a certificate under 21 days, a full backup over 25 hours old, or a
1343database snapshot over 2 hours old.
1344
1345=GET /healthz= is unauthenticated and cache-free: whether the database
1346answers and which commit serves, 503 when it does not.
1347- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
1348 0750, owned by =gitbay=). The nightly archive plus provider snapshots
1349 are the recovery path; for tighter RPO, add continuous replication
1350 (litestream) against the same file — it coexists with the WAL.
1351
1352* Odds and ends
1353
1354- deleting a fork marks MRs sourced from it =source_gone=; their diffs
1355 remain viewable and mergeable because the target repo owns the
1356 objects.
1357- =refs/merge-requests/*= is server-owned and unpushable by clients;
1358 only =admin mr prune= removes one (see Maintenance).
1359- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
1360 the =events= table, which also feeds webhooks.
1361- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.