.gitbay/wiki/Admin.org

  1#+title: gitbay admin guide
  2
  3One static binary (=gitbayd=), one SQLite file, bare repositories on
  4disk, and the system =git=. Schema migrations run automatically on
  5startup and on every admin command.
  6
  7* Install
  8
  9Build from source (=go build ./cmd/gitbayd=), install via the vanity
 10module path (=go install gitbay.org/gitbay/cmd/gitbayd@latest=), or use
 11a release build: =deploy/release.sh <tag>= cross-compiles reproducible
 12linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS
 13manifest (CGO off, trimpath, stripped — byte-identical per commit and
 14toolchain).
 15
 16#+begin_src sh
 17install -m 755 gitbayd /usr/local/bin/
 18adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
 19install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
 20gitbayd --config /etc/gitbay/config.toml check-config
 21#+end_src
 22
 23=deploy/= in the source tree has a cloud-init file, a hardened systemd
 24unit, and a nightly backup timer. Run as the unprivileged =gitbay= user;
 25the unit's =AmbientCapabilities=CAP_NET_BIND_SERVICE= covers ports
 2622/80/443 without root.
 27
 28** The SSH port decision
 29
 30- =ssh.mode = "embedded"= (default): gitbayd itself listens, normally on
 31  22 — move the host's admin sshd to another port. Remotes read
 32  =git@host:owner/repo= with no port gymnastics.
 33- =ssh.mode = "system"=: the host sshd owns 22 and invokes gitbayd via
 34  =AuthorizedKeysCommand=:
 35  #+begin_example
 36  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
 37  AuthorizedKeysCommandUser gitbay
 38  #+end_example
 39  sshd requires that binary to be root-owned and not group/world
 40  writable. Unknown keys fail authentication inside sshd, so system mode
 41  requires =registration.mode = "closed"= (check-config enforces this).
 42
 43* Configuration reference
 44
 45=/etc/gitbay/config.toml=. =check-config= validates and names every
 46contradiction; =--no-host-checks= skips port/path probes.
 47=gitbayd admin config show= prints the configuration in effect as TOML,
 48every default filled in and =smtp_pass= redacted. A file that fails
 49validation still prints, followed by the contradiction.
 50
 51** [server]
 52- =root= (default =/var/lib/gitbay=) — repositories, database, host
 53  keys, ACME cache all live here.
 54- =site_url= (required) — canonical =https://host=; drives ACME, clone
 55  URLs, mail links.
 56- =source_repo= (optional, =owner/name=) — the repository this instance
 57  develops itself in. Startup warns when the running build's commit is
 58  not on that repository's default branch, which is how a binary built
 59  from an unmerged branch stops being invisible. Leave it unset unless
 60  the instance hosts its own source.
 61
 62** [ssh]
 63- =mode= — =embedded= | =system= (above).
 64- =port= (22) — embedded listener port.
 65- =host_keys= — list of private key paths; empty generates an ed25519
 66  key at =<root>/ssh/host_ed25519=.
 67
 68** [http]
 69- =addr= (=:443=), =tls= — =acme= | =files= | =off=.
 70- =acme=: certificates via TLS-ALPN-01 on the HTTPS port, cached at
 71  =<root>/acme=; =acme_email= for the CA account; =acme_http_addr=
 72  (=:80=, ="off"= to disable) adds HTTP-01 and an https redirect —
 73  failing to bind it is a warning, not fatal. Requires an =https://=
 74  site_url with a public DNS name.
 75- =files=: =cert_file= + =key_file=.
 76- =off=: plain HTTP — development, or behind a TLS-terminating proxy.
 77
 78=trusted_proxies= lists the addresses or CIDRs of reverse proxies in
 79front of the daemon. A request from one of them is attributed, for API
 80rate limiting, to the last =X-Forwarded-For= hop that is not itself a
 81trusted proxy; from anyone else the header is ignored. Empty, the
 82default, is right when gitbayd terminates TLS itself.
 83
 84** [web]
 85- =mode= — =view_only= (default) | =accounts=. In view_only the mutating
 86  web routes are never registered; in accounts, browser sessions are
 87  minted over SSH (=web login=), and users with write access can create
 88  repos, comment, and make simple file edits (which commit unsigned,
 89  honestly). =password_auth= is reserved and currently rejected.
 90
 91** [registration]
 92- =mode= — =closed= (default) | =invite= | =open=. invite/open require
 93  [mail]. See the user guide for the flows.
 94
 95- =pending_expiry= (empty, never) — a duration such as ="168h"=; a
 96  self-registered account still unverified after that long is removed,
 97  hourly and at start, audited as =pending.expired=.
 98
 99** [mail]
100- =smtp_host= (host:port, 587 assumed), =from=, optional =smtp_user= /
101  =smtp_pass=. STARTTLS when offered. Required for invite/open
102  registration and self-service =email add=; in closed mode you may omit
103  it entirely and assert addresses by hand (below).
104
105** [api]
106- =enabled= (false) — the JSON API surface; see [[API]]. Off
107  means no credential-bearing HTTP endpoint exists at all.
108
109** [webhooks]
110- =allow_local= (false) — permit webhook targets on loopback/private
111  addresses. Leave off unless you know why you need it (SSRF).
112
113** [limits]
114- =clone_timeout= (3600s) — cap on =repo import= fetches.
115- =max_blob_bytes= (100MB) — cap on raw file serving over the web.
116- =max_asset_bytes= (512MB) — cap per uploaded release asset.
117- =max_repos_per_user= (0, unlimited) — repositories an account may own
118  directly; =repo create=, =fork= and =import= refuse past it.
119  Organizations are not capped.
120- =max_bytes_per_user= (0, unlimited) — disk the account's own
121  repositories may take; a push may be no larger than what is left.
122- =max_pack_bytes=, =ssh_auth_rate= — reserved, not yet enforced.
123
124** [git_daemon]
125- =enabled= (false), =port= (9418) — the anonymous =git://= listener.
126  Serves only public repositories that additionally ran
127  =repo settings git-daemon <repo> on=.
128
129** [mirrors]
130- =pull_interval_minutes= (15) — how often pull mirrors fetch their
131  upstream. Push mirrors sync shortly after each local ref update.
132  Mirror URLs pass the same SSRF rules as webhook targets.
133
134** [go_import]
135Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
136Requests with =?go-get=1= at or under the module path answer with the
137go-import meta tag pointing at the repository's HTTPS clone URL, so
138=go install host/module/cmd/...@latest= resolves. The repository should
139be public (the module path itself confirms it exists).
140
141* Users, email, invites
142
143Every =gitbayd admin= subcommand except =backup=, =gc= and the one-shot
144backfills is a wrapper that dispatches the registry command of the same
145name as the host: an admin context with no account behind it, so its
146audit rows carry no actor and =source: host=. The same commands run in an
147instance admin's SSH session (=ssh git@<host> admin ...=) and write the
148same rows with the key fingerprint as source. One implementation, two
149credentials.
150
151#+begin_src sh
152gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
153gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
154gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP
155#+end_src
156
157"Verified" means SMTP-confirmed or host-admin-asserted; the database
158records which. Verified emails are what make commit signatures
159meaningful — an unverified address never produces a =verified= badge.
160
161* Audit and account control
162
163The audit log is the security feed (events are the product feed): every
164successful mutating command with its argv and source credential (SSH key
165fingerprint or API), registrations, admin actions, force-pushes, and
166auth failures/throttling. Secrets never appear — they travel on stdin,
167never in argv.
168
169#+begin_src sh
170gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
171ssh git@<host> audit ...             # the same, from an admin session (SSH only)
172ssh git@<host> admin user list [--state active|pending|disabled|admin]
173ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
174ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default]   # per-account caps
175ssh git@<host> admin user promote <name>   # grant instance admin
176ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
177gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
178gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
179gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
180gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
181                                     #   refused (with each blocker named) while
182                                     #   the account owns repos, authored
183                                     #   issues/MRs/comments/reviews, or is an
184                                     #   org's only admin
185#+end_src
186
187=--actor= takes a username, or =-= for rows with no actor: host commands
188and auth failures. =--action= is a prefix, so =cmd repo= catches every
189repository command and =admin= every host or admin-session action.
190=--since= is a duration back from now (=30m=, =24h=, =7d=) or a date.
191
192=admin user list= pages by username (=--limit=, =--cursor=) and carries
193each account's state and =last_seen=, the newest use of any of its SSH
194keys or API tokens. =admin user show= adds the keys with their last use,
195each address with how it was verified, PGP keys, org roles, the owned
196repository count, API token names, and live browser sessions. Both are
197SSH-only and refused to non-admins, like =audit=.
198
199Promotion needs an active account: a pending or disabled one is refused.
200Demotion is refused when it would leave no admin, over SSH and on the
201host alike, so the host-local =promote= is the way back in when the only
202admin key is lost.
203
204Instance admin carries no right on anyone's repository: policy does not
205consult it, and a private repository still answers not-found to an
206admin. Moderation goes through explicit overrides that skip the access
207check and write their own audit row:
208
209#+begin_src sh
210ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
211ssh git@<host> admin repo archive|unarchive <owner/name>
212ssh git@<host> admin repo visibility <owner/name> public|private
213ssh git@<host> admin repo delete <owner/name> --yes
214#+end_src
215
216Each lands in the audit log as =admin repo.<action>= naming the
217repository, on top of the =cmd= row every mutating command gets.
218
219=limits.ssh_auth_rate= (10) throttles per-IP authentication *failures*
220per minute — successful auths never count and clear the slate.
221=limits.max_pack_bytes= is enforced as =receive.maxInputSize= on every
222push.
223
224* Queues
225
226Every background worker keeps a backlog and a failure state. An instance
227admin reads them all in one place:
228
229#+begin_src sh
230gitbay dashboard --json | jq .queues   # webhooks, mail, mirrors, builds, deps
231#+end_src
232
233Per worker: pending, retrying (pending with a failed attempt) and
234dead-lettered counts with the oldest pending age, and the retrying or
235failed rows themselves, capped at twenty each. Builds list what is
236running and since when; mirrors list the ones whose last sync failed;
237dependency checks list the ones whose last check errored. Non-admins get
238no =queues= key at all.
239
240In accounts mode the same read renders at =/admin=, linked from the rail
241for admins. Anyone else gets a 404 there.
242
243* Maintenance
244
245#+begin_src sh
246gitbayd admin stats [--json]         # counts, database size, per-repo disk
247ssh git@<host> admin stats [--json]  # the same, from an admin session
248gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
249gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
250gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)
251#+end_src
252
253=deploy/cloud-init.yaml= ships a =gitbay-gc.timer= that runs =admin gc=
254weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack
255layout the source sent, so a first manual =admin gc= after a bulk
256import is worthwhile.
257
258* Backup and restore
259
260#+begin_src sh
261gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
262gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back
263#+end_src
264
265One archive: a consistent SQLite snapshot (taken *before* the
266repositories are read, so the database never references objects the
267archive missed), every repository, and the SSH host keys. Excluded:
268hook socket, regenerated hook scripts, WAL files. Safe to run against a
269live daemon.
270
271=--verify= reads an archive back: the snapshot must pass SQLite's
272integrity check, and every repository the snapshot names must be in the
273archive. A database-only archive is checked for integrity and says so.
274Exit is non-zero on damage or a missing repository.
275
276Restore: extract into an empty directory, point =server.root= at it,
277start gitbayd. Host keys are preserved, so clients keep their
278known_hosts entries; hooks regenerate at startup.
279
280** Schedule and recovery point
281
282Two timers, because the two halves of the data have different exposure.
283
284- =gitbay-backup.timer=, nightly. The full archive above, last 7 kept.
285- =gitbay-db-backup.timer=, hourly. =admin backup --db-only=, which
286  writes the SQLite snapshot alone, last 48 kept. A few MB against the
287  full archive's hundreds, which is what makes the frequency affordable.
288
289The split follows what a loss would actually cost. Repositories are git,
290so a mirror or any clone is a second copy; the database is the only copy
291of issues, merge requests, comments and review state. So the recovery
292point is about an hour for the data that exists nowhere else, and a day
293for the data that does.
294
295Continuous replication (litestream and similar) was considered and not
296adopted. It would take the database's recovery point to seconds, but the
297repositories would still be on the nightly archive, so a restore could
298produce a database referencing commits the repository backup does not
299have. Consistency between the two halves is worth more here than latency
300on one of them. Revisit if repository replication becomes continuous
301too.
302
303* Upgrades
304
305Replace the binary, restart the unit. Migrations apply automatically and
306are transactional; hook scripts under =<root>/hooks= are rewritten at
307startup to point at the current binary path.
308
309* CI runner
310
311=gitbay-runner= executes builds queued by pushes and merge requests. It
312polls over SSH with a key added by =keys add --scope runner=, which
313reaches only the runner protocol and read-only git (a runner executes
314arbitrary repository code, so the key it holds must not do more), then
315clones, runs the steps, streams the log back and resolves the commit
316status. Run it as a dedicated unprivileged user on a non-admin account.
317=admin user create --key= registers a full-scope key, so the runner key
318is added afterwards through a bootstrap key that is then removed:
319
320#+begin_src sh
321useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
322sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
323ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
324gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
325ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
326ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
327rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
328gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work
329#+end_src
330
331=-jobs N= runs N builds at once. Claiming is one transaction that
332selects and updates, and each build works in its own =build-<id>=
333directory, so workers do not collide; idle polls are staggered across
334the interval so N of them do not wake together. The drop-in's weights
335below are per service, not per build, so raising =-jobs= divides them
336rather than multiplying the host's load.
337
338=admin runners= shows which account each runner polls as, and what each
339is scoped to. A runner with no scope claims builds for *any*
340repository, which on an instance with open registration means running a
341stranger's steps; scope one with =-repos owner/name=. An admin key
342still works for the protocol during a rotation. A merge request head
343from a fork is built in the target repository as untrusted: the claim
344carries no secrets. Same-repository heads were built by their branch
345push and are not built again.
346
347=make deploy-runner= also installs
348=deploy/gitbay-runner.override.conf= as a systemd drop-in: =Nice=10=,
349=CPUWeight=30=, =IOWeight=30=, so a build never starves the host's sshd,
350the daemon or the backup timers, and =NoNewPrivileges=,
351=ProtectSystem=full=, =ProtectKernelTunables=, =ProtectControlGroups=
352and =RestrictSUIDSGID=, so a step cannot reach outside its workspace
353and the runner's home. The e2e suite alone starts sixty
354daemon instances; without the drop-in a deploy's copy over the admin
355sshd stalled. Both deploy targets copy with =rsync --partial=, which
356resumes a stalled transfer.
357
358v1 runs steps directly on the host — no containers — so treat the
359runner machine as executing whatever your users push. Install the
360toolchains your builds need on it.
361
362A runner claims the oldest pending build in the queue, whichever
363repository it belongs to. =-repos= narrows that to named repositories,
364which is what makes a runner outside the server practical — one on a
365machine that should build a single project, or that holds credentials for
366one deployment, no longer picks up a build belonging to someone else. With
367open registration that someone need not be anyone you know.
368
369#+begin_src sh
370gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
371  -workdir /var/lib/gitbay-runner/work
372#+end_src
373
374Naming no repositories is the old behaviour and stays the right choice for
375the runner on the server itself. The scoping is what the runner asks for,
376not an ACL the server holds over it: a runner account is admin by
377necessity, so the boundary is you choosing how to start it.
378
379=gitbay dashboard= and =ssh git@<host> admin runners= list every account
380that has polled as a runner: when it last polled, the =-repos= scope it
381asked for, and the build it holds. A build a runner claimed and never
382reported is failed by the scheduler's minute tick, whether or not any
383runner is still alive.
384
385Instance admin on the runner account only authorizes the claim/report
386protocol; it grants no repo access. A build that pushes back — a pages
387deploy, an archive publish, an automated MR branch — needs an explicit
388grant on that repo: =repo access grant <owner/name> ci write=. Private
389repos likewise need at least read for the clone.
390
391* LFS storage
392
393Objects live content-addressed under =[lfs] root= (default
394=<server.root>/lfs=); =[lfs] max_object_bytes= caps a single object
395(512MB default). Storage sits behind a small interface — an
396S3-compatible backend is a drop-in with the server proxying, and
397presigned URLs a later optimization. LFS objects do not travel with
398push mirrors (mirrors move git refs only), and gc does not yet collect
399orphaned objects.
400
401* Pages
402
403=[pages] domain = "example.site"= serves public repos' =pages= branches
404on =<owner>.<domain>=. DNS needs a wildcard record =*.<domain>= to the
405server; ACME issues per-subdomain certificates on demand (only for
406owners that exist). The domain must not be the site host or a parent of
407it — pages content runs its own scripts and must stay off the forge's
408origin.
409
410Users with repo admin claim custom domains with =repo domain add=.
411Claims activate only after a DNS TXT challenge proves control of the
412domain (=repo domain verify=, audit-logged); pending claims serve
413nothing, get no certificates, and expire after 7 days. ACME issues
414certificates only for verified hosts, so stray DNS pointed at the
415server gets nothing.
416
417* Security
418
419The [[Threat-Model]] file is the reference for what the forge
420trusts and refuses to do. Operational checklist:
421
422- *Software checks.* =deploy/audit.sh= runs =go vet=, =govulncheck=
423  (the module list is deliberately short — review it on each release),
424  and a short fuzz pass over every attacker-facing parser (pkt-line,
425  commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before
426  tagging a release.
427- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
428  DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
429  configuration needed.
430- *Host sandboxing.* The systemd unit in =deploy/cloud-init.yaml= runs
431  gitbayd unprivileged with =ProtectSystem=strict=, =PrivateDevices=,
432  =LockPersonality=, =MemoryDenyWriteExecute=,
433  =SystemCallFilter=@system-service=, and =RestrictAddressFamilies= to
434  INET/INET6/UNIX. It keeps =CAP_NET_BIND_SERVICE= only, to bind 22/80/443.
435- *OS patches* apply via =unattended-upgrades= (security origins,
436  auto-reboot 04:30 if required).
437- *Admin sshd (2222)* is throttled by =MaxStartups=/=MaxAuthTries= and
438  watched by =fail2ban=; gitbayd's own port 22 is throttled by
439  =limits.ssh_auth_rate= (auth failures per IP per minute).
440- *Monitoring.* =gitbay-monitor.timer= writes a reading hourly to
441journald and, when =/etc/gitbay/monitor.url= exists, posts it to that
442webhook: disk, service, the daemon's own =/healthz= answer, certificate
443expiry, and the age of the newest full backup and database snapshot.
444It exits non-zero on an alert so the unit shows in =systemctl
445--failed=: a stopped service, =/healthz= not answering =ok=, disk ≥ 85%,
446a certificate under 21 days, a full backup over 25 hours old, or a
447database snapshot over 2 hours old.
448
449=GET /healthz= is unauthenticated and cache-free: whether the database
450answers and which commit serves, 503 when it does not.
451- *Database.* =gitbay.db= and its WAL live under =/var/lib/gitbay= (mode
452  0750, owned by =gitbay=). The nightly archive plus provider snapshots
453  are the recovery path; for tighter RPO, add continuous replication
454  (litestream) against the same file — it coexists with the WAL.
455
456* Odds and ends
457
458- deleting a fork marks MRs sourced from it =source_gone=; their diffs
459  remain viewable and mergeable because the target repo owns the
460  objects.
461- =refs/merge-requests/*= is server-owned and unpushable by clients.
462- audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in
463  the =events= table, which also feeds webhooks.
464- the daemon idles under 10MB RSS; the smallest VPS tier is adequate.