.gitbay/wiki/Admin.org
1367 lines · 73691 bytes
1#+title: gitbay admin guide
2
3One static binary (=gitbayd=), one SQLite file, bare repositories on
4disk, and the system =git=. Schema migrations run automatically on
5startup and on every admin command.
6
7* Install
8
9Build from source (=go build ./cmd/gitbayd=), install via the vanity
10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
13manifest (CGO off, trimpath, stripped — byte-identical per commit and
14toolchain).
15
16#+begin_src sh
17install -m 755 gitbayd /usr/local/bin/
18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
20gitbayd --config /etc/gitbay/config.toml check-config
21gitbayd --config /etc/gitbay/config.toml admin secrets init
22chown gitbay:gitbay /etc/gitbay/secret.key
23#+end_src
24
25=admin secrets init= refuses when the key file already exists, so a
26reinstall on the same host should skip it — =deploy/install.sh= does
27this with a file check before running it.
28
29=deploy/= in the source tree has a cloud-init file, a hardened systemd
30unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
31the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
3222/80/443 without root.
33
34** The SSH port decision
35
36- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
37 22 — move the host's admin sshd to another port. Remotes read
38 =git@host:owner/repo= with no port gymnastics.
39- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
40 =AuthorizedKeysCommand=:
41 #+begin_example
42 AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
43 AuthorizedKeysCommandUser gitbay
44 #+end_example
45 sshd requires that binary to be root-owned and not group/world
46 writable. Unknown keys fail authentication inside sshd, so system mode
47 requires =registration.mode = "closed"= (check-config enforces this).
48 Under the forced command the CLI's leading =--term=<cols>[,color]=
49 argument works as is; a client that sets =GITBAY_TERM= instead needs
50 =AcceptEnv GITBAY_TERM= in =sshd_config=, since that env request is
51 handled by the host's sshd, not gitbayd.
52
53* Configuration reference
54
55=/etc/gitbay/config.toml=. =check-config= validates and names every
56contradiction; =--no-host-checks= skips port/path probes.
57=gitbayd admin config show= prints the configuration in effect as TOML,
58every default filled in and =smtp_pass= redacted. A file that fails
59validation still prints, followed by the contradiction.
60
61** [server]
62- =root= (default =/var/lib/gitbay=) — repositories, database, host
63 keys, ACME cache all live here.
64- =site_url= (required) — canonical =https://host=; drives ACME, clone
65 URLs, mail links.
66- =secret_key_file= (default =/etc/gitbay/secret.key=) — the keys that
67 seal CI secrets, webhook secrets, mirror tokens and push device
68 tokens in the database. Must be outside =root=, mode 0600, readable
69 by the daemon's user. See "Secret key" below.
70- =source_repo= (optional, =owner/name=) — the repository this instance
71 develops itself in. Startup warns when the running build's commit is
72 not on that repository's default branch, which is how a binary built
73 from an unmerged branch stops being invisible. Leave it unset unless
74 the instance hosts its own source.
75
76** [ssh]
77- =mode= — =embedded= | =system= (above).
78- =port= (22) — embedded listener port.
79- =host_keys= — list of private key paths; empty generates an ed25519
80 key at =<root>/ssh/host_ed25519=.
81
82** [http]
83- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
84- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
85 =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
86 (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
87 failing to bind it is a warning, not fatal. Requires an =https://=
88 site_url with a public DNS name.
89- =files=: =cert_file= + =key_file=.
90- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
91- The HTTPS listener accepts TLS 1.2 and 1.3 only (=serverTLS= in
92 =cmd/gitbayd/tls.go=), with the default cipher suites of the Go
93 release the binary was built with. =openssl s_client -connect
94 <host>:443 -tls1_1= fails the handshake.
95
96=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
97front of the daemon. A request from one of them is attributed, for API
98rate limiting, to the last =X-Forwarded-For= hop that is not itself a
99trusted proxy; from anyone else the header is ignored. Empty, the
100default, is right when gitbayd terminates TLS itself.
101
102A reverse proxy in front of gitbay must not buffer responses, or the
103build page's live log arrives only when the build ends;
104=X-Accel-Buffering: no= covers nginx.
105
106** [web]
107- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
108 web routes are never registered; in accounts, browser sessions are
109 minted over SSH (=web login=), and users with write access can create
110 repos, comment, and make simple file edits (which commit unsigned,
111 honestly). =password_auth= is reserved and currently rejected.
112- =title= — the instance's display name in the rail and page titles;
113 empty falls back to the site host. Lower case is the convention for
114 gitbay itself.
115- =privacy_notice= — operator text shown on =/privacy= under the fixed
116 statement.
117- =apple_app_ids= (empty) — iOS app IDs (=TEAMID.bundle.id=) allowed to
118 open this instance's links, served at
119 =/.well-known/apple-app-site-association=. Empty leaves the path a
120 404. The gitbay iOS app is =ZCNAX3VL9D.org.gitbay.gitbay=; its
121 entitlement names gitbay.org only, so listing it elsewhere does
122 nothing.
123
124** [registration]
125- =mode= — =closed= (default) | =invite= | =open=. invite/open require
126 [mail]. See the user guide for the flows.
127
128- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
129 self-registered account still unverified after that long is removed,
130 hourly and at start, audited as =pending.expired=.
131
132- =notify_admin= (false) — mail every instance admin when an account
133 becomes active: an invite redeemed, or an open-mode signup that
134 verified its address. The unverified row an open signup creates is
135 not reported, because anyone can post the form and mailing on that
136 would point a flood at the admins. Recipients are the verified
137 primary addresses of active admins who have activity mail on, the
138 same rule any other notice follows, so an admin with no verified
139 address hears nothing. The notice is queued, so a dead SMTP host
140 shows up in the admin page's Mail table instead of failing the
141 registration. Requires [mail].
142
143** [mail]
144- =smtp_host= (host:port; 587 assumed, 465 with =tls = "implicit"=),
145 =from=, optional =smtp_user= / =smtp_pass=. Required for invite/open
146 registration and self-service =email add=; in closed mode you may omit
147 it entirely and assert addresses by hand (below).
148- =tls= — =starttls= (default) or =implicit= (TLS from the first byte,
149 for relays on 465).
150- =require_tls= — with =starttls=, a relay that does not offer STARTTLS
151 gets no mail: delivery fails and retries, and the admin page's Mail
152 table shows the error. Unset, it is on for every relay except
153 =localhost= and loopback addresses; set =false= to allow plaintext
154 to a remote relay.
155
156** [mail.inbound]
157Reply by mail: a reply to issue or merge request mail posts a comment
158(#295). gitbayd polls a mailbox over IMAP; no port is opened for it.
159Off unless =enabled=, and it requires =[mail] smtp_host=, since the
160replies answer mail gitbayd sends.
161
162#+begin_src toml
163[mail.inbound]
164enabled = true
165imap_host = "imap.example.org" # host:port; 993, or 143 with tls = "starttls"
166tls = "implicit" # or "starttls"; there is no plaintext setting
167user = "reply@gitbay.example"
168password_file = "/etc/gitbay/imap.pass" # one line, mode 0600, owned by gitbayd's user
169mailbox = "INBOX" # default
170poll_interval = "1m" # default; at least 10s
171reply_address = "reply@gitbay.example"
172require_dkim = true # recommended; see below
173trusted_authserv_id = "mx.example.org" # the mail host's Authentication-Results id
174#+end_src
175
176- The password is read from =password_file= at every connection and
177 never appears in the config, argv or the log. An unreadable file, or
178 one readable by group or others, stops the daemon at start.
179- =reply_address= is what each Reply-To is built from:
180 =reply@gitbay.example= becomes =reply+<token>@gitbay.example=. The
181 mailbox must receive that. Hosts that deliver subaddresses to the
182 mailbox need nothing more; others need a wildcard rule. Migadu, for
183 one, sent =threads+x@gitbay.org= nowhere until a rewrite from
184 =threads+*= to =threads@gitbay.org= was added. A domain catch-all only
185 works if it points at this mailbox, which then also holds everything
186 else sent to the domain. Send a test to =<reply_address>+test= and
187 expect a =refused mail reply= audit row with "malformed reply token"
188 within a poll interval (=gitbay audit --action 'refused mail'=). Point
189 the domain's MX at the mail host as for any other mailbox; nothing
190 about it involves gitbayd.
191- Use a mailbox that holds nothing else. gitbayd reads every unseen
192 message in it and marks each one =\Seen= when it is handled, posted
193 or refused. A message that fails for a reason that may pass (the
194 database busy, the server refusing that one fetch) stays unseen and is
195 tried again on the next poll, up to five times, then is marked seen
196 and audited. A dropped connection or a timeout ends the poll and
197 counts against no message. A
198 message over 10 MiB is refused by its size without being fetched. A
199 server that sends more than about 11 MiB, or more than a thousand
200 untagged responses, for one command has its connection closed; one
201 poll handles at most ten thousand messages.
202- Whatever the settings, a message is refused when a header field name
203 is not RFC 5322 =ftext= (=From : x=, a space or a non-ASCII byte in a
204 name), when it does not have exactly one =From=, or when it has more
205 than one =To=, =Cc=, =Message-ID=, =Content-Type= or
206 =Content-Transfer-Encoding=.
207- =require_dkim= (default false) should be on for any instance
208 reachable from the internet. With it, a reply is posted only when
209 gitbayd itself verifies one of its DKIM signatures (RFC 6376) with a
210 =d== in relaxed alignment with the From domain (the same
211 organizational domain by the public suffix list; =d=github.io= aligns
212 with nothing) and an =h== that covers =From=, the =To= or =Cc= holding
213 the reply address, =Content-Type=, and =Message-ID= when the message
214 has one. A =Content-Transfer-Encoding= outside =h== is accepted only
215 when it is =7bit=, =8bit= or =binary=, which leave the decoded body as
216 it is; Thunderbird, for one, does not sign it. An unsigned
217 =quoted-printable= or =base64= is refused. The reply
218 address is read only from =To= or =Cc=: a reply that reached the
219 mailbox by Bcc, with the address only in =Delivered-To=, is refused
220 ("reply address not in To or Cc"). It needs nothing from the mail
221 host, so it works where the host adds no =Authentication-Results=.
222 rsa-sha256 (keys of 1024 bits or more) and ed25519-sha256 are
223 accepted, with simple or relaxed canonicalization; rsa-sha1, a body
224 length tag (=l==), an expired =x== and a =t== more than fifteen
225 minutes ahead are refused. Only the first five signatures are
226 checked. The key is looked up at =<s>._domainkey.<d>= with a
227 five-second timeout and cached for fifteen minutes (the resolver does
228 not report the record's TTL); a lookup that fails for a reason that
229 may pass (a timeout, SERVFAIL) leaves the message for the next poll,
230 up to the five tries above, while a missing key refuses it.
231 Signatures are checked on the message as fetched. Each passing
232 signature is recorded with the =Message-ID=, so a copy of the same
233 signed message posts once even with unsigned fields changed.
234- Either =require_dkim= or =trusted_authserv_id= passing is enough to
235 authenticate =From=; when both are set, a reply needs only one of
236 them, and a refusal names both reasons. With only
237 =trusted_authserv_id=, the reply address may come from any recipient
238 field (=Delivered-To=, =X-Original-To=, =Envelope-To=, =To=, =Cc=),
239 since the mail host vouches for the sender and not for the fields.
240 Set both when the mail host adds =Authentication-Results= for most
241 senders but not all.
242- =trusted_authserv_id= names the authserv-id the mail host writes at
243 the start of its =Authentication-Results= header (Gmail's is =mx.google.com=).
244 With it set, a reply is posted only when the topmost header with that
245 id shows =dmarc=pass= with =header.from= equal to the From domain, or
246 =dkim=pass= with a =header.d= in relaxed alignment with it (the same
247 organizational domain by the public suffix list; a public suffix such
248 as =github.io= aligns with nothing). The header is parsed per RFC
249 8601, so text inside a quoted string or a comment (a quoted MAIL FROM
250 local part, a reason) is never read as a result. Lower headers
251 claiming the same id are the sender's and are not read. This is only safe when the mail host
252 removes incoming =Authentication-Results= headers that claim its id,
253 as RFC 8601 asks; Gmail, Fastmail and Migadu do. Check yours before
254 relying on it. With neither this nor =require_dkim= set, the daemon
255 logs a warning at start and =admin mail inbound check= repeats it:
256 =From= is then whatever the sender wrote. Mail between two addresses
257 at the same host may carry no =Authentication-Results= at all: at
258 Migadu, mail from another
259 Migadu-hosted domain is delivered through its outbound path and gets
260 none, so setting the id alone refuses every reply from such users.
261 That mail does carry a DKIM signature aligned with =From= (for
262 example =d=cleberg.net; s=key1; a=rsa-sha256; c=simple/simple;
263 h=from:to:subject:date:message-id:mime-version:content-type=), which
264 is why gitbay.org sets =require_dkim= instead.
265- =gitbay admin mail inbound check= logs in, opens the mailbox
266 read-only (EXAMINE) and reports the message and unseen counts and how
267 =From= is authenticated (=require_dkim=, =trusted_authserv_id=), so a
268 check never marks a reply seen before the poller reads it. It warns
269 when neither is set. With inbound off it says so and exits 0. Poll
270 failures are logged as =mail reply: poll failed= with the server and
271 the IMAP error.
272
273A reply is posted when all of these hold, checked when it is read:
274
2751. It is not an automatic reply (=Auto-Submitted=, =Precedence: bulk=
276 and the like).
2772. A recipient header carries a reply address whose token verifies
278 and has not expired (thirty days from the mail it came on).
2793. The token's account exists, is active and not disabled, still has
280 reply by mail on, and was created before the token was: an id freed
281 by a delete and reused is not the account the token named. The same
282 holds for the repository.
2834. =From= is one of that account's verified addresses, and, with
284 =require_dkim= or =trusted_authserv_id= set, a DKIM signature
285 gitbayd verified or the mail host's result authenticated it.
2865. The message has a =text/plain= part (HTML-only mail is refused, not
287 converted), and what is left after quoted text and the signature
288 are removed is not empty and fits a comment (64 KiB).
2896. No earlier copy of the message (same =Message-ID=, account and
290 thread) posted.
2917. =issue comment= or =mr comment=, dispatched as the account, accepts
292 it: the account can still read the repository, the thread exists,
293 the repository is not archived, the write budget is not spent.
294
295The comment's audit row is =cmd issue comment= (or =mr comment=) with
296=source: mail=. A refusal writes a =refused mail reply= row with the
297reason and the =Message-ID=, never the message's content, at most sixty
298a minute, and sends nothing back. See Threat-Model for why the token
299and the sender address are both required.
300
301** [push]
302Push notifications to Apple devices, delivered by gitbayd talking to
303APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an
304operator-supplied provider key. Off unless configured.
305
306- =enabled= (false).
307- =key_file= — path to the =.p8= provider key from Apple's developer
308 portal (Certificates, Identifiers & Profiles → Keys). It belongs at
309 =/etc/gitbay/apns.p8=, mode 0600, owned by the account gitbayd runs
310 as. Read and validated at startup: it must parse as a PEM-wrapped
311 PKCS#8 EC (P-256) private key, or the daemon refuses to start rather
312 than fill a queue nobody is watching.
313- =key_id=, =team_id= — the key's id and your Apple developer team id,
314 both from the same portal page.
315- =topic= — the app's bundle identifier. *An APNs key belongs to a
316 bundle ID.* gitbay.org pushes to the App Store build under its own
317 bundle id; a self-hoster who wants push ships their own iOS build
318 under their own bundle id, with its own =.p8= key from their own
319 developer account, and points =topic= at that id. There is no way to
320 push to someone else's build, by design — this is Apple's model, not
321 gitbay's.
322- =environment= — =production= or =sandbox=, naming the APNs host
323 rather than taking a URL, so a typo cannot aim the key at a host that
324 is not Apple's.
325
326All five of =key_file=, =key_id=, =team_id=, =topic= and =environment=
327are required when =enabled= is true; validation runs at config load,
328so a misconfigured =[push]= is caught before the daemon serves
329anything. The delivery queue (a device's undelivered and attempted
330pushes) is capped the same way the mail queue is, by =[retention]
331push=.
332
333** [backup]
334- =age_recipients= (optional) — age public keys (=age1...=). When set,
335 =admin backup= encrypts every archive to them and appends =.age= to
336 its name. Generate the pair off the host with =age-keygen=; only the
337 public key goes here, so the host writes archives it cannot read.
338 The restic copy is unaffected: the offsite job stages its own
339 =VACUUM INTO= of the live database and snapshots =/var/lib/gitbay=,
340 not the archives.
341
342** [api]
343- =enabled= (false) — the JSON API surface; see [[API]]. Off
344 means no credential-bearing HTTP endpoint exists at all.
345
346** [webhooks]
347- =allow_local= (false) — permit webhook, mirror and import targets on
348 loopback, private, shared (100.64.0.0/10), link-local or multicast
349 addresses. Leave off unless you know why you need it (SSRF).
350
351** [limits]
352- =clone_timeout= (3600s) — cap on =repo import= fetches. An import
353 takes http and https URLs only, passes the same address check as a
354 mirror sync and is pinned the same way, so it needs git 2.37 too.
355- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
356- =max_asset_bytes= (512MB) — cap per uploaded release asset.
357- =max_snippet_bytes= (1MB) — cap per snippet file.
358- =max_snippets_per_user= (0, unlimited) — snippets an account may own.
359- =max_repos_per_user= (0, unlimited) — repositories an account may own
360 directly; =repo create=, =fork= and =import= refuse past it.
361 Organizations are not capped.
362- =max_bytes_per_user= (0, unlimited) — disk the account's own
363 repositories may take; a push may be no larger than what is left.
364- =pack_concurrency= (3), =pack_per_principal= (2), =pack_queue= (32),
365 =pack_queue_wait= (="60s"=) — git pack generation (clones, fetches,
366 =git archive --remote=, web archive downloads, =repo download= over
367 SSH and the API) over SSH, smart HTTP and git:// shares one
368 budget: this many at once, this many per account (per client
369 address when anonymous: an IPv4 address, or an IPv6 /64), and this
370 many waiting for at most the wait. Anonymous clients together hold
371 at most =pack_concurrency= − 1 slots when it is above 1: an account
372 (SSH key, bearer token or web session) can take the last slot when it
373 is free, and anonymous clients cannot hold it.
374 Past that an SSH client gets "the server is busy…" and exit 1, HTTP
375 gets 503 with =Retry-After: 30=, git:// an =ERR= line; the daemon
376 logs a =pack limit= warning naming the transport and whether the
377 client was signed in, at most once a minute per transport. An SSH client
378 that disconnects while queued leaves the queue; an HTTP or git:// one
379 keeps its place until the wait runs out. A running clone is killed
380 when its client disconnects, or when no write to the client completes
381 for two minutes: a client reading below about 550 B/s, or an HTTP
382 request body that takes over two minutes with nothing written back,
383 is cut. Ref listings (info/refs, protocol v2
384 =ls-refs=) and pushes are outside the budget. A busy =repo download=
385 exits 1 with the same message; over the API it is a 503 with
386 =Retry-After: 60=. For the
387 three counts 0 means the default and a negative value turns that
388 bound off. The defaults suit a four-core host; see [[Performance]].
389 With =ssh.mode = "system"= each SSH session is its own process and
390 SSH clones are not counted.
391- =push_concurrency= (2), =push_per_principal= (1), =push_queue= (16),
392 =push_queue_wait= (="60s"=) — =git receive-pack= over SSH, on a
393 budget of its own so clones cannot starve pushes or the reverse:
394 this many at once, this many per account or deploy key, and this
395 many waiting for at most the wait. One principal may have up to half
396 of =push_queue= (at least one) waiting, so parallel pushes from one
397 account queue rather than being refused, but cannot fill the queue.
398 A deploy key is its own principal, apart from the account that
399 registered it, so an account with deploy keys can hold
400 =push_per_principal= slots per key;
401 the global cap still holds. A bot account such as a runner's counts
402 like any other account. The slot is taken after the access checks and held
403 until receive-pack exits, which is after =post-receive= (merge
404 detection, CI queueing, webhooks) has run. Past that the client gets
405 "the server is busy: it is at its limit of concurrent pushes…" and
406 exit 1, and the daemon logs a =push limit= warning at most once a
407 minute. A client that disconnects while queued leaves the queue; one
408 that disconnects mid-push ends receive-pack and frees its slot, and
409 a revoked key kills it. 0 and negative values read as for =pack_*=.
410 HTTP and git:// carry no pushes. With =ssh.mode = "system"= pushes
411 are not counted and the two timeouts below do not apply.
412- =push_idle= (="60s"=), =push_receive_timeout= (="15m"=) — a push
413 holding a slot is killed, its process group with it, when its
414 pre-receive has not started =push_receive_timeout= after it took the
415 slot, or when no byte has been read from the client and none written
416 to it for =push_idle=. The idle rule starts when the pack signature
417 arrives from the client or pre-receive starts, whichever is first
418 (a delete-only push sends no pack): before that the client may be
419 silent while its =pack-objects= counts and compresses, and only
420 =push_receive_timeout= applies. receive-pack runs with
421 =receive.keepAlive= set to a quarter of =push_idle= (1 to 5 seconds),
422 so the keepalives it sends to side-band clients while it indexes and
423 runs hooks keep a working push alive. Once pre-receive has started
424 only =push_idle= applies: post-receive is never cut short by the
425 clock. A push killed before pre-receive answers updates no refs.
426- =max_pack_bytes= (2 GiB) — the largest pack one push may send,
427 enforced as =receive.maxInputSize= and lowered to what an owner's
428 storage quota has left.
429- =ssh_auth_rate= (10) — SSH authentication failures per client address
430 per minute on the embedded listener; see below.
431
432** [git_daemon]
433- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
434 Serves only public repositories that additionally ran
435 =repo settings git-daemon <repo> on=.
436
437** [mirrors]
438- =pull_interval_minutes= (15) — how often pull mirrors fetch their
439 upstream. Push mirrors sync shortly after each local ref update.
440 Mirror URLs pass the same SSRF rules as webhook targets, when saved
441 and again before every sync; git then connects only to the addresses
442 that were checked (=http.curloptResolve=) and does not follow
443 redirects, so a mirror of a renamed repository fails until its URL
444 is updated. Needs git 2.37 or later on the server; with an older git
445 the worker logs an error at start and syncs no mirror, recording the
446 reason on each. Sync ignores the system and global gitconfig.
447
448** [go_import]
449Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
450Requests with =?go-get=1= at or under the module path answer with the
451go-import meta tag pointing at the repository's HTTPS clone URL, so
452=go install host/module/cmd/...@latest= resolves. The repository should
453be public (the module path itself confirms it exists).
454
455* Users, email, invites
456
457Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
458backfills is a wrapper that dispatches the registry command of the same
459name as the host: an admin context with no account behind it, so its
460audit rows carry no actor and =source: host=. The same commands run in an
461instance admin's SSH session (=ssh git@<host> admin ...=) and write the
462same rows with the key fingerprint as source. One implementation, two
463credentials.
464
465#+begin_src sh
466gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
467gitbayd admin email verify alice a@example.org # admin assertion, no SMTP needed
468gitbayd admin invite --email b@example.org # mails a code; prints it if no SMTP
469#+end_src
470
471"Verified" means SMTP-confirmed or host-admin-asserted; the database
472records which. Verified emails are what make commit signatures
473meaningful — an unverified address never produces a =verified= badge.
474
475* Audit and account control
476
477The audit log is the security feed (events are the product feed): every
478successful mutating command with its argv and source credential (SSH key
479fingerprint or API), every refused one (exit 3 or 4) as =refused
480<command>=, refused pushes as =refused git-receive-pack= (the access
481check) or =refused push= (a branch or tag rule, a release anchor or an
482unsigned commit, with the repository and ref names), hook socket
483requests failing the peer or push-token check as =refused hook= (never
484the token), registrations, admin actions, force-pushes, and auth
485failures/throttling. A refusal row
486keeps the flag names and the first positional, not the values.
487Refusals are recorded up to ten a minute per account and 600 a minute
488across the instance; past either, one =refused.throttled= row stands
489for the rest of that minute. The caps bound the embedded listener, the
490web and the API; under =ssh.mode = "system"= each =gitbayd shell=
491connection counts separately. Secrets never appear — they travel on
492stdin, never in argv.
493
494Each row carries the SHA-256 of the row before it. =gitbayd admin audit
495verify= opens the store as other admin commands do, applying pending
496migrations, so run it with the binary that matches the daemon. It
497recomputes the chain and exits 1 naming the first row that was
498edited or whose predecessor was removed. The chain is unkeyed: whoever
499can write the database can recompute every hash after an edit, and
500verify then finds nothing. It catches an edit only when the later
501hashes were not recomputed. Retention removing the oldest
502rows is not a break. Rows written before the chain existed are counted
503and skipped; when every row is such a row, verify warns and exits 1,
504since clearing the hash columns looks the same. After an upgrade that
505clears with the first new audit row.
506
507Removing the newest rows leaves no break, and neither do rows written
508afterwards under the freed ids. The database cannot show either. The
509daemon logs every row it writes to its journal, outside the database
510(=journalctl -u gitbayd -g 'INFO audit '=), and verify prints the last
511id and hash: comparing them with the newest journal line is the check
512for any change, recomputed hashes included. Rows written by
513host =gitbayd admin= commands, and by =gitbayd shell= when =ssh.mode =
514"system"=, are not copied to the journal.
515
516#+begin_src sh
517gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
518gitbayd admin audit verify # check the hash chain; exit 1 names the first bad row
519ssh git@<host> audit ... # the same, from an admin session
520ssh git@<host> admin user list [--state active|pending|disabled|admin]
521ssh git@<host> admin user show <name> # keys, emails, orgs, tokens, sessions
522ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default] # per-account caps
523ssh git@<host> admin user promote <name> # grant instance admin
524ssh git@<host> admin user demote <name> # remove it; the last admin is refused
525gitbayd admin user promote <name> # host-local: recovery when no admin key is reachable
526gitbayd admin user disable <name> # suspend: SSH, web, API all refused;
527gitbayd admin user enable <name> # sessions dropped, nothing deleted
528gitbayd admin user delete <name> --yes # only for accounts anchoring nothing:
529 # refused (with each blocker named) while
530 # the account owns repos, authored
531 # issues/MRs/comments/reviews, or is an
532 # org's only admin
533#+end_src
534
535=--actor= takes a username, or =-= for rows with no actor: host commands
536and auth failures. =--action= is a prefix, so =cmd repo= catches every
537repository command and =admin= every host or admin-session action.
538=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
539
540=admin user list= pages by username (=--limit=, =--cursor=) and carries
541each account's state and =last_seen=, the newest use of any of its SSH
542keys or API tokens. =admin user show= adds the keys with their last use,
543each address with how it was verified, PGP keys, org roles, the owned
544repository count, API token names, and live browser sessions. Both are
545Both are refused to non-admins, like =audit=, on every surface.
546
547=/admin/users= is the same list in a browser, linked from the admin
548page: the state filter the command takes, keyset paging on its cursor,
549and a row per account with promote, demote, disable and enable, each
550dispatching the command. Demote and disable ask for the username to be
551typed, since both take someone's access away. Creating and deleting an
552account, issuing an invite and asserting an address stay on the command
553line: each takes a key, mints a credential, or cannot be undone. A
554non-admin gets the 404 a missing page would, so the URL confirms
555nothing.
556
557Promotion needs an active account: a pending or disabled one is refused.
558Demotion is refused when it would leave no admin, over SSH, in the
559browser and on the host alike, so the host-local =promote= is the way
560back in when the only admin key is lost.
561
562Instance admin carries no right on anyone's repository: policy does not
563consult it, and a private repository still answers not-found to an
564admin. Moderation goes through explicit overrides that skip the access
565check and write their own audit row:
566
567#+begin_src sh
568ssh git@<host> admin repo list [--owner o] [--visibility public|private] # size, last push
569ssh git@<host> admin repo archive|unarchive <owner/name>
570ssh git@<host> admin repo visibility <owner/name> public|private
571ssh git@<host> admin repo delete <owner/name> --yes
572#+end_src
573
574Each lands in the audit log as =admin repo.<action>= naming the
575repository, on top of the =cmd= row every mutating command gets.
576
577=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
578per minute — successful auths never count and clear the slate.
579=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
580push.
581
582=limits.write_rate= (60) bounds *mutating commands per account per
583minute*. It is counted in the dispatcher, so SSH, the JSON API and the
584web spend one budget and a caller cannot refresh it by changing surface;
585=limits.api_rate= stays in front of it, bounding a network source rather
586than an account. A command is one token whatever it writes, so a bundle
587import costs one and only a loop of separate commands spends the budget.
588Read-only commands, the runner protocol (a build streams its log in many
589small writes) and the host CLI are exempt. Refusals exit 4 and say when
590to retry. A negative value turns the limit off; it matters most with
591=registration = "open"=, where every write also queues notification mail
592and webhook deliveries.
593
594* Queues
595
596Every background worker keeps a backlog and a failure state. An instance
597admin reads them all in one place:
598
599#+begin_src sh
600gitbay dashboard --json | jq .queues # webhooks, mail, push, mirrors, builds, deps
601#+end_src
602
603Per worker: pending, retrying (pending with a failed attempt) and
604dead-lettered counts with the oldest pending age, and the retrying or
605failed rows themselves, capped at twenty each. Builds list what is
606running and then what is pending, each since when, so a build no runner
607is scoped to claim is visible here rather than only in its repository;
608mirrors list the ones whose last sync failed;
609dependency checks list the ones whose last check errored. Non-admins get
610no =queues= key at all.
611
612Push rows name the device id, never the token. Watch this one after
613configuring =[push]=: a =key_id= or =team_id= Apple did not issue passes
614config validation, which can only check that the =.p8= parses, and then
615every send comes back =403 InvalidProviderToken= and dead-letters on its
616first attempt.
617
618In accounts mode the same read renders at =/admin=, linked from the rail
619for admins. Anyone else gets a 404 there.
620
621A dead-lettered mail is logged as =notification dead-lettered mail=<id>=,
622with the address redacted out of the relay's error. The id is the queue
623row: find it in the Mail table on =/admin=, or in =dashboard --json=,
624where the recipient and the unredacted error are. That is deliberate —
625see the Threat-Model page.
626
627* Maintenance
628
629#+begin_src sh
630gitbayd admin stats [--json] # counts, database size, per-repo disk
631ssh git@<host> admin stats [--json] # the same, from an admin session
632gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
633gitbayd admin gc --aggressive # thorough repack; slow, rarely needed
634gitbayd admin gc --lfs # also drop LFS objects no pointer names (older than a day)
635#+end_src
636
637A history rewrite leaves the commits it removed reachable through
638=refs/merge-requests/N/head= of the merge requests that landed them, so
639they stay fetchable by anyone who can read the repository. Nothing drops
640a head ref on its own — an open or source-gone MR is merged through it,
641and a merged or closed one keeps its diff readable through it — so the
642cleanup is a command an instance admin runs, naming the MRs:
643
644#+begin_src sh
645ssh git@<host> admin mr prune owner/name 1 2 3 --yes
646#+end_src
647
648It refuses an open or source-gone MR, deletes the named refs, runs
649=git gc --prune=now= on that one repository so the objects go at once
650rather than after git's two-week grace, leaves a system comment on each
651MR, and audits as =admin mr.prune=. The MR keeps its title, comments,
652reviews and head sha; =mr diff= and the MR page say the head is gone.
653Run it when nothing is pushing to that repository: without the grace, a
654push caught between leaving quarantine and writing its ref loses its
655objects. Objects also survive in offsite backups until those are
656rewritten; see "Removing a repository's history from every snapshot".
657
658=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
659weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
660layout the source sent, so a first manual =admin gc= after a bulk
661import is worthwhile.
662
663* Backup and restore
664
665#+begin_src sh
666gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
667gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz # read it back
668#+end_src
669
670One archive: a consistent SQLite snapshot (taken *before* the
671repositories are read, so the database never references objects the
672archive missed), every repository, and the SSH host keys. Excluded:
673hook socket, regenerated hook scripts, WAL files. Safe to run against a
674live daemon. =--out= must be outside =server.root=, or the next full
675backup would carry the archive. Each run removes the snapshot
676directories (=.gitbay-snap-*=) and temporary archives (=.*.tmp-*=) a
677killed run left beside its archive once they are a day old, and prints
678each one it removes.
679
680=--verify= reads an archive back: the snapshot must pass SQLite's
681integrity check, every repository the snapshot names must be in the
682archive, and each must pass =git fsck --connectivity-only=. Every
683release asset the snapshot names must be in its repository with its
684recorded size and sha256, and every LFS object in the archive must
685hash to its name. LFS objects are named by pointer files in git
686history rather than by the database, so a pointer whose object was
687never uploaded is not caught, and with =[lfs] root= outside
688=server.root= the archive holds no LFS objects at all. It extracts the
689repositories to a temporary directory for the checks, so it needs free
690space the size of the repositories. A database-only archive is checked
691for integrity and says so. Exit is non-zero on damage, a missing
692repository, a missing object, or a missing or altered asset or LFS
693object.
694
695A full backup holds =<root>/backup.lock= from its database snapshot to
696its last repository. While it runs, =repo delete=, =repo rename=,
697=repo transfer=, =admin repo delete=, =org rename=, =admin mr prune=
698and =gitbayd admin gc= refuse with "a backup is running"; retry when it
699finishes. Database-only backups take no lock. A pack that git's own
700automatic gc removes during the walk is skipped; each repository's refs
701are archived before its objects, so the refs still find their objects,
702and =--verify= reports it if one does not.
703
704Verifying an archive of unknown origin: run it as an unprivileged
705user. The connectivity check runs git with =--git-dir= on each
706extracted repository, so a directory that is not a repository fails
707rather than git checking an enclosing one. =objects/info/alternates=
708and a =commondir= directly in a =*.git= directory are not extracted,
709so an archive cannot use either to have git read another repository's
710objects or refs on the host. Git still reads each archived
711repository's own =config=.
712
713Archives carry a directory entry for every directory, including an
714empty one, so a bare repository whose refs are all packed restores as
715a repository. Archives written before this release do not: extracting
716one can leave a repository's =refs/= directory missing, which stops
717git from recognizing it as a repository at all. =gitbayd admin backup
718--verify <archive>= names the repositories this affects; the fix is
719=mkdir -p <root>/repos/<owner>/<name>.git/refs= for each one, after
720which it opens normally.
721
722With =[backup] age_recipients= set the archive is =<name>.tar.gz.age=
723and =--verify= needs the private key:
724
725#+begin_src sh
726gitbayd admin backup --verify gitbay-20260927-090000.tar.gz.age --identity ~/.config/gitbay/backup-identity.txt
727age -d -i ~/.config/gitbay/backup-identity.txt gitbay-20260927-090000.tar.gz.age | tar -xz -C /new/root
728#+end_src
729
730The identity lives off the host (with the secret key file and the
731restic credentials), so verifying an encrypted archive happens there
732or on a restore host.
733
734Restore: extract into an empty directory, point =server.root= at it,
735restore =server.secret_key_file= from its own copy (mode 0600, owned
736by the account gitbayd runs as), then start gitbayd. No archive carries
737the key file, and without it gitbayd refuses to start. Host keys are
738preserved, so clients keep their known_hosts entries; hooks regenerate
739at startup.
740
741** Schedule and recovery point
742
743Two timers, because the two halves of the data have different exposure.
744
745- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
746- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
747 writes the SQLite snapshot alone, last 48 kept. A few MB against the
748 full archive's hundreds, which is what makes the frequency affordable.
749
750The split follows what a loss would actually cost. Repositories are git,
751so a mirror or any clone is a second copy; the database is the only copy
752of issues, merge requests, comments and review state. So the recovery
753point is about an hour for the data that exists nowhere else, and a day
754for the data that does.
755
756Continuous replication (litestream and similar) was considered and not
757adopted. It would take the database's recovery point to seconds, but the
758repositories would still be on the nightly archive, so a restore could
759produce a database referencing commits the repository backup does not
760have. Consistency between the two halves is worth more here than latency
761on one of them. Revisit if repository replication becomes continuous
762too.
763
764** Offsite copies
765
766bay1 also takes a nightly restic snapshot of =/var/lib/gitbay= and
767=/var/lib/gitbay-stage= (a database copy and =config.toml=, staged
768there) to an S3 bucket at Scaleway, with a key that can only add
769snapshots. Nothing else under =/etc/gitbay= is in it: not the secret
770key file, not =apns.p8=. The key that can
771remove them lives on the operator's machine, in
772=~/.config/gitbay/offsite.env=, and never on bay1: a compromised host
773cannot destroy its own history. Forgetting, pruning and rewriting all
774run from there.
775
776*** Removing a repository's history from every snapshot
777
778A history rewrite plus =admin mr prune= takes commits off the server,
779but every snapshot taken before it still holds them, and the retention
780window is the only thing that ages them out. To remove them now,
781rewrite the snapshots without that repository rather than forgetting
782the snapshots: everything else in them stays restorable. The next
783nightly run adds the repository back in its current state.
784
785Repositories are stored under the name they had on disk when each
786snapshot was taken, so a renamed repository needs every name it has
787carried. Check what an older snapshot holds before choosing the paths:
788
789#+begin_src sh
790set -a; . ~/.config/gitbay/offsite.env; set +a
791restic $RESTIC_OPTS snapshots
792restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>
793#+end_src
794
795Then dry-run, apply, prune, and confirm nothing matches:
796
797#+begin_src sh
798EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
799restic $RESTIC_OPTS rewrite --dry-run $EXCL # "would modify N snapshots"
800restic $RESTIC_OPTS rewrite --forget $EXCL # new snapshots replace the originals
801restic $RESTIC_OPTS prune # drops the data nothing references
802restic $RESTIC_OPTS find <name>.git <old-name>.git # expect no output
803#+end_src
804
805=--forget= is what makes the originals go; without it the rewritten
806snapshots sit beside them and the data stays referenced. Snapshot IDs
807change; their times do not. Done for krz/keycask (formerly rust-pass)
808on 2026-09-18, across 22 snapshots.
809
810** Secret key
811
812CI secrets, webhook secrets, mirror tokens and APNs device tokens are
813stored sealed: AES-256-GCM under a key in =server.secret_key_file=,
814each value prefixed with the id of the key that sealed it
815(=gbs1:<id>:=). The key file is not in the database, not under
816=server.root=, and therefore in neither the local archives nor the
817main restic repository. It must be copied off the host separately;
818without it a restored database's secrets cannot be opened, and
819gitbayd refuses to start against them. A separate restic repository
820for it and =apns.p8= is planned (runbook D of the data-at-rest plan)
821and not yet in place, so today the only off-host copy is one the
822operator makes by hand after =init= and after every =rotate=.
823
824#+begin_src sh
825gitbayd admin secrets init # once; deploy/install.sh does it on first install
826gitbayd admin secrets check # open every value, count by key
827gitbayd admin secrets rotate # new key, reseal, retire the old one (as root)
828#+end_src
829
830- Missing file: every gitbayd process that opens the database refuses
831 to run and names the path, including =serve= and, in system mode,
832 =authorized-keys=. =migrate= does not need it.
833- Backups: =admin backup= and =admin backup --verify= do not need the
834 key file; a restore does.
835- Wrong key: =serve= stops at startup naming the first row that does
836 not open; =secrets check= does the same without starting anything.
837- Upgrade: the first start after the upgrade seals every value still
838 in clear and logs =sealed secret values=.
839- Rotation: =rotate= adds a key, reseals every value under it in one
840 transaction, then removes the old keys. Run it as root, since it
841 replaces the key file in =/etc/gitbay=; the file keeps its owner.
842 The daemon re-reads the file when it changes, so it needs no
843 restart. Copy the new file off the host afterwards. =init= and
844 =rotate= hold an flock on =<key file>.lock= while they run, so a
845 second run waits for the first.
846- Push devices are looked up by the SHA-256 of their token
847 (=push_devices.token_hash=), since two seals of one token differ.
848
849** Restore drill
850
851A restore onto a clean host, run quarterly (January, April, July,
852October) and after any change to the backup code
853(=cmd/gitbayd/backup.go=, =cmd/gitbayd/restoredrill.go=, the offsite
854job), and recorded below. The disaster it rehearses is losing bay1, so
855the local archives are gone with it and the sources are the main
856offsite restic repository (repositories, LFS, the staged database,
857=config.toml=), the off-host copy of =secret.key= and =apns.p8= (a keys
858repository once runbook D creates it; until then the operator's
859hand-made copy), and the operator's password manager (=offsite.env=,
860the keys repository's password and token once it exists,
861=backup-identity.txt=). The full steps are runbook C of the
862data-at-rest plan (=docs/plans/2026-09-27-data-at-rest-and-backup.md=).
863
864=gitbayd admin restore-drill= does the archive half. It extracts a
865full archive into an empty or absent directory, runs every =--verify=
866check on the extracted copy, and prints what was restored, the newest
867issue, issue comment, merge request comment and push in the restored
868database, and the elapsed time. Exit is non-zero if any check fails.
869
870#+begin_src sh
871gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
872gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill
873#+end_src
874
875=admin backup --verify= extracts the archive under =$TMPDIR=. A host
876whose =/tmp= is a tmpfs smaller than a full archive (bay1's is 3.9 GB
877of memory) fails with "no space left on device"; point =TMPDIR= at a
878directory on disk that the backup's user owns:
879
880#+begin_src sh
881install -d -o gitbay -g gitbay -m 700 /var/backups/gitbay/verify-tmp
882cd /var/backups/gitbay
883sudo -u gitbay env TMPDIR=/var/backups/gitbay/verify-tmp gitbayd admin backup --verify <archive>
884#+end_src
885
886What to restore, and from where:
887
888| Item | Source | Path on the drill host |
889|-------------------------------+-------------------------------------------------------------+------------------------------------------------|
890| Database | staged copy in =/var/lib/gitbay-stage= (restic), or an archive | =<root>/gitbay.db= |
891| Repositories | =/var/lib/gitbay/repos= (restic), or an archive | =<root>/repos= |
892| LFS objects | =/var/lib/gitbay/lfs= (restic), or an archive | =<root>/lfs= (or =[lfs] root=) |
893| Release assets | inside each repository (=gitbay-releases/=) | with the repositories |
894| Host keys | =/var/lib/gitbay/ssh= (restic), or an archive | =<root>/ssh= (or =[ssh] host_keys=) |
895| =config.toml= | =/var/lib/gitbay-stage/config.toml= (restic) | =/etc/gitbay/config.toml= |
896| =secret.key=, =apns.p8= | the off-host copy; no archive or main snapshot carries them | =/etc/gitbay/=, mode 0600, owned by =gitbay= |
897
898From the offsite path (restic is not run by any gitbay command):
899
900#+begin_src sh
901restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
902cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
903gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
904gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root
905#+end_src
906
907The second archive is how the restic tree gets the same checks and
908timestamps; =/tmp/drill-root= is discarded afterwards.
909
910What to check, each a column below:
911
912- DB integrity, connectivity, release assets, LFS: =restore-drill=
913 prints =integrity ok=, =connectivity ok on N repositories=, =release
914 assets ok: N=, =LFS objects ok: N=. Compare N with =gitbayd admin
915 stats --json= on the source at the snapshot time.
916- Secrets: =gitbayd --config /etc/gitbay/config.toml admin secrets
917 check= opens every value.
918- Host key: =ssh-keyscan -p 22 <drill-host>= matches the source's
919 fingerprint.
920- Config: =gitbayd --config /etc/gitbay/config.toml check-config=.
921- Service: =ssh -p 22 git@<drill-host> whoami= and a =git clone= over
922 SSH succeed.
923
924Time to service runs from the clean host's first root login to the
925first successful =git clone= over SSH from it. The recovery point is
926the time of the newest restic snapshot restored; record beside it the
927newest issue, comment and push =restore-drill= printed, which show how
928much activity the restore carries.
929
930The first drill ran on the operator's laptop rather than a provisioned
931host, so its time to service has no provisioning in it, and it could
932not check secrets: no copy of =secret.key= was on the machine, and
933gitbayd refuses to start while any sealed value does not open. The
934sealed values were cleared in the drill copy to reach service. The
935off-host copy of the key is confirmed to exist (#305); the next drill
936restores it with the snapshot and runs =admin secrets check= on the
937restored copy.
938
939| Date | Host | Snapshot restored (UTC) | Newest issue / comment / push | Time to service | DB integrity | Connectivity | LFS | Release assets | Host key | Secrets | Notes |
940|------+------+-------------------------+-------------------------------+-----------------+--------------+--------------+-----+----------------+----------+---------+-------|
941| 2026-09-29 | laptop (macOS), restic from offsite | 2026-09-29 00:19:15 (ccd646cf) | 00:05:17 / 00:09:44 / 00:15:31 | 8m43s (restore 1m49s, checks 33s) | ok | ok, 69/69 | ok, 2 | ok, 529 | matches | not checked (#305) | 4.99 GiB restored; 71 sealed values cleared in the drill copy to start |
942
943* Upgrades
944
945Replace the binary, restart the unit. Migrations apply automatically and
946are transactional; hook scripts under =<root>/hooks= are rewritten at
947startup to point at the current binary path.
948
949* CI runner
950
951=gitbay-runner= executes builds queued by pushes and merge requests. It
952polls over SSH with a key of scope =runner=, which reaches only the
953runner protocol and read-only git (a runner executes arbitrary
954repository code, so the key it holds must not do more). A runner key
955claims builds only for the repositories it is attached to, by =repo
956runner add= from a repository admin or an instance admin; an admin key
957claims any. Users attach their own runners: see the Users page. For an
958instance runner, run it as a dedicated unprivileged user on a non-admin
959account. =admin user create --key= registers a full-scope key, so the
960runner key is added afterwards through a bootstrap key that is then
961removed, and attached to each repository it should build:
962
963#+begin_src sh
964useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
965sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
966ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
967gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
968ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
969ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
970rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
971gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
972#+end_src
973
974#+begin_src sh
975gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
976#+end_src
977
978=-jobs N= runs N builds at once. Claiming is one transaction that
979selects and updates, and each build works in its own =build-<id>=
980directory, so workers do not collide; idle polls are staggered across
981the interval so N of them do not wake together. The drop-in's weights
982below are per service, not per build, so raising =-jobs= divides them
983rather than multiplying the host's load.
984
985=admin runners= shows which account each runner polls as, and what each
986may claim. A runner key claims builds only for the repositories it is
987attached to: with none attached it claims nothing, and =-repos= may only
988narrow within them. An admin's full-scope key claims any repository —
989that is what =-repos= was for — and still works for the protocol during
990a rotation. A merge request head from a fork is built in the target
991repository as untrusted: the claim carries no secrets, and only a runner
992started with =-untrusted= takes it. Same-repository heads were built by
993their branch push and are not built again.
994
995=make deploy-runner= also installs
996=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
997=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
998the daemon or the backup timers, and =NoNewPrivileges=,
999=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
1000and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
1001and the runner's home. The e2e suite alone starts sixty
1002daemon instances; without the drop-in a deploy's copy over the admin
1003sshd stalled. Both deploy targets copy with =rsync --partial=, which
1004resumes a stalled transfer.
1005
1006Under =-isolation podman=, the default and what bay1 runs, each build
1007is confined to a container (see Container isolation below). Under
1008=-isolation none= steps run directly on the host as the runner's user,
1009so treat that machine as executing whatever your users push, and
1010install the toolchains your builds need on it.
1011
1012A runner claims the oldest pending build among the repositories its key
1013is attached to — for an admin key, the oldest in the instance. =-repos=
1014narrows within that set, which is what makes a runner outside the server
1015practical: one on a machine that should build a single project, or that
1016holds credentials for one deployment, stays on it.
1017
1018Oldest-first is across everything the key may claim, so a repository
1019with a deep queue holds every other repository the same runner serves;
1020bay1 measured a 15-minute average wait on a day of merge request
1021stacks from one repository. A runner attached to one repository cannot
1022be starved. That is the rule, decided in krz/gitbay#207: a runner
1023serving several repositories takes them oldest-first, and an operator
1024who wants one repository never to wait on another runs a second
1025runner attached to it alone. Nothing caps what an account queues,
1026and nothing needs to (krz/gitbay#206): a schedule tick queues nothing
1027while the job's last build is pending or running, so a repository
1028with no runner holds one row per scheduled job rather than one per
1029tick, and a build runs only on a runner its owner attaches, so a busy
1030schedule spends the owner's compute. Pushes are bounded by what an
1031account can push.
1032
1033#+begin_src sh
1034gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
1035 -workdir /var/lib/gitbay-runner/work
1036#+end_src
1037
1038Add =-untrusted= only with =-isolation podman=.
1039
1040gitbay.org's runner is attached to the forge's own repositories and
1041the isolation canary, nothing else, because it shares the host with
1042the forge; its unit names no =-repos=, the attachments are the
1043boundary. Any other repository builds on a runner its owner attaches.
1044
1045=-repos= narrows an admin runner; for a runner key the attachments are
1046the boundary, held by the server, and =-repos= may only name
1047repositories among them. =-untrusted= makes a runner claim merge
1048request heads from forks; the bay1 unit sets it because it isolates in
1049podman. A runner without it builds trusted commits only.
1050
1051=gitbay dashboard= and =ssh git@<host> admin runners= list every key
1052that has polled as a runner: the account, the key's fingerprint, when it
1053last polled, the repositories it may claim — its attachments for a runner
1054key, the =-repos= it asked for or =any= for an admin key — and the build
1055it holds; =admin runners remove <fingerprint>= (=forget= until the next release) drops the row for a key
1056that polled by mistake, the key itself untouched. =admin runners= also
1057heads the list with the queue: builds
1058pending now, and over the last day how many were claimed, how long they
1059waited to be claimed (average and worst), and
1060how many the reaper ended instead of a runner reporting them. A build a runner claimed and never
1061reported is failed by the scheduler's minute tick, whether or not any
1062runner is still alive: within about two minutes of its log stream ending
1063with no outcome reported — the runner reports right after closing the
1064stream, retrying for half a minute if gitbayd is unreachable — or, if no
1065stream was ever seen, at the deadline.
1066
1067Instance admin on the runner account only authorizes the claim/report
1068protocol; it grants no repo access. A build that pushes back — a pages
1069deploy, an archive publish, an automated MR branch — needs an explicit
1070grant on that repo: =repo access grant <owner/name> ci write=. Private
1071repos likewise need at least read for the clone.
1072
1073** Container isolation
1074
1075Builds run in a rootless podman container, one per job, with the
1076workspace bind mounted and nothing else: the clone happens outside with
1077the runner's key, so a step cannot read it. =-isolation none= keeps the
1078old behaviour — steps on the host as the runner's user — for an instance
1079where every repository is trusted. There is no automatic fallback: a
1080runner started with =-isolation podman= that cannot find a working
1081podman exits rather than running a build unsandboxed.
1082
1083gitbay's own jobs name =localhost/gitbay-ci:2=, built from
1084=deploy/Containerfile.ci= on the runner host. A job's image must carry
1085what its steps need: the suite drives real git, git-lfs, gpg and sshd and
1086asserts they exist before running, so the stock runner default would fail
1087it immediately. Build or rebuild it with:
1088
1089#+begin_src sh
1090ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
1091ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
1092 "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'
1093#+end_src
1094
1095The tag is deliberate rather than =:latest=: changing the file means
1096bumping the tag in =.gitbay/ci.yml=, so a running branch's image does not
1097change under it.
1098
1099=-image= names the image a job runs in when it declares none, and is
1100required under =-isolation podman=: there is no built-in default,
1101because an image this host does not have would fail every build. A job
1102overrides it with =image:= in =.gitbay/ci.yml=, validated as a reference
1103so a config file cannot turn it into podman arguments.
1104
1105=-cpus= and =-memory= cap one build (podman's units, e.g. =-cpus 2
1106-memory 4g=); unset means uncapped. The runner applies them itself: it
1107creates a cgroup per build under its own delegated service cgroup,
1108writes the limits there, and starts every podman process for the build
1109inside it, with podman's cgroup handling off. Podman's own =--memory=
1110and =--cpus= never applied under rootless cgroupfs, which is what a
1111system service gets (krz/gitbay#188). The unit therefore needs
1112=Delegate=yes=, which the drop-in sets; without it the runner refuses
1113to start when a limit is set, and logs that builds run unconfined when
1114none is. bay1 runs =-cpus 3 -memory 6g= per build inside =MemoryMax=6G=
1115and =CPUQuota=300%= on the unit, on a 7.7GB four-core host with no
1116swap: the memory cap is what keeps the forge alive when a build
1117allocates without bound, and it sits above the e2e suite's 5GB peak
1118rather than at a fair share. =OOMPolicy=continue= keeps systemd from
1119stopping the runner when a build is OOM-killed.
1120
1121A trusted build's home is its repository's, under
1122=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
1123=HOME=: caches persist between trusted builds of one repository and are
1124never read by another's. An untrusted build — a merge request head from
1125a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
1126the build ends. Homes under =<workdir>/home= are from runners before
1127krz/gitbay#255, which shared them with untrusted builds; nothing reads
1128them any more, and they can be deleted.
1129
1130*Images are provisioned, never pulled by a build.* The runner passes
1131=--pull=never=. Two reasons, and the second is the better one: the
1132service runs with =RestrictSUIDSGID=yes= so podman cannot unpack a layer
1133holding a setuid file, which is nearly every distribution image; and on
1134an instance where anyone can push a =ci.yml=, =image:= would otherwise
1135mean "fetch and run anything from the internet". An operator pulls or
1136builds what is allowed and a build picks among those. A job naming an
1137image the host does not have fails with a message saying so.
1138
1139#+begin_src sh
1140su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
1141su - ci-runner -s /bin/sh -c "podman images"
1142#+end_src
1143
1144Prepare a host before pointing an isolating runner at it:
1145
1146#+begin_src sh
1147ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
1148make deploy-runner
1149#+end_src
1150
1151The script installs podman, delegates a subuid/subgid range to
1152=ci-runner=, checks that user namespaces are enabled rather than
1153assuming, enables lingering, and verifies rootless podman actually runs
1154as that user. It is idempotent.
1155
1156It also installs nftables. =make deploy-runner= ships
1157=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
1158with =gitbay-runner-egress.service=, which loads it and which the
1159runner's unit requires; it checks the file with =nft -c=, reloads the
1160unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
1161restarting the runner: =127.0.0.1:22= and the public 22 must answer,
11622222 must not. The table limits the runner's user to =127.0.0.1:22=,
1163DNS on loopback, and 22, 80 and 443 on the host's public address; the
1164Threat-Model page says why.
1165
1166That table cannot tell a build from the runner, since both run as
1167=ci-runner=. =deploy/gitbay-runner-builds.nft=, shipped as
1168=/etc/gitbay-runner/builds.nft=, matches by cgroup instead: the runner
1169starts each build under =builds/trusted= or =builds/untrusted= in its
1170service cgroup, and the table closes the host's loopback to every
1171build, limits an untrusted build to TCP 80 and 443 and DNS, and closes
1172private ranges to both (the CI page has the table). nftables resolves a
1173cgroup path to its id when the table loads, and the service cgroup is
1174new on every start, so the drop-in's =ExecStartPre= creates the two
1175cgroups, hands them to =ci-runner= and loads the table before the
1176runner starts; a table that does not load stops the start. =make
1177deploy-runner= checks the file's syntax before the restart and that the
1178table is loaded after it. If =/etc/resolv.conf= names a nameserver in a
1179private range, allow it in the file first or builds resolve nothing.
1180
1181A restart of =nftables.service= flushes both tables;
1182=systemctl reload gitbay-runner-egress= restores both.
1183
1184The egress unit's stop and the rollback below use =nft destroy=,
1185which needs nftables 1.0.8 or later; check =nft --version= on a new
1186host.
1187
1188Checking the tables on a running host, during a build (the flood test
1189below waits a minute before its first probe, for this):
1190
1191#+begin_src sh
1192nft list table inet gitbay_builds # counters on the reject rules
1193for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
1194# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>
1195#+end_src
1196
1197A pasta process anywhere else — the runner's own =runner= cgroup, a
1198user slice — means the builds table does not see that build's traffic
1199and only the first table applies. Run
1200=deploy/runner-auth-flood-test.sh= on the scratch repository (below),
1201once as a push and once with =--untrusted=. It fails if the runner was
1202locked out or if the build log lacks the lines only a working table
1203produces. The authoritative proof that pasta's sockets are in the
1204build's cgroup is the untrusted run: =169.254.1.2:22= and
1205=github.com:22= refused, all twelve logins refused, and the counters on
1206the =untrusted= chain's rejects rising. The first table lets
1207=ci-runner= reach both of those addresses.
1208
1209To take the builds table out, on the host:
1210
1211#+begin_src sh
1212sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
1213rm /etc/gitbay-runner/builds.nft
1214systemctl daemon-reload
1215nft destroy table inet gitbay_builds
1216#+end_src
1217
1218The runner needs no restart. It still places builds under
1219=builds/trusted= and =builds/untrusted=, but with the file gone neither
1220a runner start nor =systemctl reload gitbay-runner-egress= loads the
1221table again, and builds keep the first table and =--no-map-gw=. A
1222runner that takes =-untrusted= or polls over loopback refuses to start
1223without build cgroups at all, since the table would match nothing. The
1224next =make deploy-runner= installs the file and the lines again.
1225
1226The drop-in sets =NoNewPrivileges=no=, without which rootless podman
1227cannot call =newuidmap= and the runner refuses to start. That is a
1228considered trade, explained in the file and in the Threat-Model; if you
1229run with =-isolation none=, set it back to =yes=.
1230
1231*Restarting the runner is safe.* On SIGTERM it stops claiming, finishes
1232the build in flight, reports it, and exits; the drop-in's
1233=TimeoutStopSec=50min= covers the longest build, and its =KillMode=mixed=
1234is what makes the signal reach the runner alone — under systemd's default
1235the build's container and the log session are signalled with it, and the
1236runner drains a build that is already dead. So =make deploy-runner=
1237waits for a running build rather than orphaning it, and a build's result
1238is retried for half a minute if gitbayd is restarting at that moment. A
1239second SIGTERM ends the runner at once, abandoning the build to the
1240reaper. The suite checks all three: =TestRunnerDrainsOnSIGTERM= signals
1241the process, =TestRunnerDropInLetsTheDrainHappen= reads the drop-in's
1242=KillMode= and =TimeoutStopSec=, and =TestRunnerDrainUnderSystemd= runs
1243the runner as a transient user unit under =systemd-run= and stops it
1244under both kill modes. That last one needs a systemd user manager, so it
1245skips in the container CI runs in; run it on a Linux host with
1246=go test ./e2e -run TestRunnerDrainUnderSystemd -v=.
1247
1248*Validate podman mode on a scratch repository before pointing the runner
1249at real ones.* Every deploy that switched the whole instance to
1250containers and failed took CI down with it. Instead: create a throwaway
1251repository the runner account can read (public, or granted read — a
1252private one is "not found" to the runner and the build stays pending),
1253give it one job that names the CI image, attach the runner's key to it,
1254and deploy the runner with =-repos= naming only that repository. The
1255production unit, with its real hardening, then claims nothing else;
1256other repositories' builds queue until =-repos= is removed again, which
1257is a pause, not an outage.
1258
1259#+begin_src sh
1260# on a machine with an admin gitbay identity (root on the host has none),
1261# with the runner's public key copied from the host:
1262gitbay repo create cmc/runner-scratch # then push a .gitbay/ci.yml naming the image
1263scp -P 2222 root@<host>:/var/lib/gitbay-runner/.ssh/id_ed25519.pub runner.pub
1264gitbay repo runner add cmc/runner-scratch < runner.pub
1265# on the host:
1266sed -i 's#^ExecStart=/usr/local/bin/gitbay-runner #&-repos cmc/runner-scratch #' /etc/systemd/system/gitbay-runner.service.d/override.conf
1267systemctl daemon-reload && systemctl restart gitbay-runner
1268# back on the admin machine:
1269gitbay build log cmc/runner-scratch 1 # green: remove -repos, redeploy, delete the scratch repository
1270#+end_src
1271
1272*Do not deploy an isolating runner to a host that has not been
1273prepared.* The runner is specified to refuse to start without a working
1274podman rather than fall back to running builds unsandboxed — a fallback
1275that silently drops isolation is worse than a stopped runner, because
1276nothing surfaces it. On an unprepared host that refusal stops every
1277build on the instance.
1278
1279The service drop-in carries =Delegate=yes= for rootless cgroup
1280management and =ReadWritePaths= for podman's store under
1281=/var/lib/gitbay-runner=, which =ProtectSystem=full= would otherwise
1282make read-only. Those paths are prefixed =-= so they are ignored when
1283absent: the drop-in installs on unprepared hosts too, and a unit that
1284refused to start would stop every build.
1285=gitbay-runner-prune.timer= prunes unused images weekly, as the runner's
1286user: rootless storage belongs to that user, and root's prune would not
1287see it. An unpruned image store on a 40GB host is a slow outage.
1288
1289* LFS storage
1290
1291Objects live content-addressed under =[lfs] root= (default
1292=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
1293(512MB default). Storage sits behind a small interface — an
1294S3-compatible backend is a drop-in with the server proxying, and
1295presigned URLs a later optimization. LFS objects do not travel with
1296push mirrors (mirrors move git refs only), and gc does not yet collect
1297orphaned objects.
1298
1299* Pages
1300
1301=[pages] domain = "example.site"= serves public repos' =pages= branches
1302on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
1303server; ACME issues per-subdomain certificates on demand (only for
1304owners that exist). The domain must not be the site host or a parent of
1305it — pages content runs its own scripts and must stay off the forge's
1306origin.
1307
1308Users with repo admin claim custom domains with =repo domain add=.
1309Claims activate only after a DNS TXT challenge proves control of the
1310domain (=repo domain verify=, audit-logged); pending claims serve
1311nothing, get no certificates, and expire after 7 days. ACME issues
1312certificates only for verified hosts, so stray DNS pointed at the
1313server gets nothing.
1314
1315* Security
1316
1317The [[Threat-Model]] file is the reference for what the forge
1318trusts and refuses to do. Operational checklist:
1319
1320- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
1321 (the module list is deliberately short — review it on each release),
1322 and a short fuzz pass over every attacker-facing parser (pkt-line,
1323 commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
1324 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
1325 against main rather than per push, because =@latest= scans today's
1326 advisory database and an advisory lands without anyone pushing;
1327 =build trigger krz/gitbay vuln= runs it on demand.
1328- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
1329 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
1330 configuration needed.
1331- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
1332 gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
1333 =LockPersonality=, =MemoryDenyWriteExecute=,
1334 =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
1335 INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
1336- *OS patches* apply via =unattended-upgrades= (security origins,
1337 auto-reboot 04:30 if required).
1338- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
1339 watched by =fail2ban=; gitbayd's own port 22 is throttled by
1340 =limits.ssh_auth_rate= (auth failures per IP per minute), and every
1341 account's writes by =limits.write_rate=.
1342- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
1343journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
1344webhook: disk, service, the daemon's own =/healthz= answer, certificate
1345expiry, and the age of the newest full backup and database snapshot.
1346It exits non-zero on an alert so the unit shows in =systemctl
1347--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
1348a certificate under 21 days, a full backup over 25 hours old, or a
1349database snapshot over 2 hours old.
1350
1351=GET /healthz= is unauthenticated and cache-free: whether the database
1352answers and which commit serves, 503 when it does not.
1353- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
1354 0750, owned by =gitbay=). The nightly archive plus provider snapshots
1355 are the recovery path; for tighter RPO, add continuous replication
1356 (litestream) against the same file — it coexists with the WAL.
1357
1358* Odds and ends
1359
1360- deleting a fork marks MRs sourced from it =source_gone=; their diffs
1361 remain viewable and mergeable because the target repo owns the
1362 objects.
1363- =refs/merge-requests/*= is server-owned and unpushable by clients;
1364 only =admin mr prune= removes one (see Maintenance).
1365- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
1366 the =events= table, which also feeds webhooks.
1367- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.