.gitbay/wiki/Admin.org

Build from source (go build ./cmd/gitbayd), install via the vanity module path (go install gitbay.org/gitbay/cmd/gitbayd@latest), or use a release build: deploy/release.sh <tag> cross-compiles reproducible linux/amd64, linux/arm64, and darwin/arm64 binaries with a SHA256SUMS manifest (CGO off, trimpath, stripped — byte-identical per commit and toolchain).

install -m 755 gitbayd /usr/local/bin/
adduser --system --group --home /var/lib/gitbay --shell /usr/sbin/nologin gitbay
install -d -o gitbay -g gitbay -m 750 /var/lib/gitbay
gitbayd --config /etc/gitbay/config.toml check-config

deploy/ in the source tree has a cloud-init file, a hardened systemd unit, and a nightly backup timer. Run as the unprivileged gitbay user; the unit's AmbientCapabilities=CAP_NET_BIND_SERVICE covers ports 22/80/443 without root.

/etc/gitbay/config.toml. check-config validates and names every contradiction; --no-host-checks skips port/path probes. gitbayd admin config show prints the configuration in effect as TOML, every default filled in and smtp_pass redacted. A file that fails validation still prints, followed by the contradiction.

Every gitbayd admin subcommand except backup, gc and the one-shot backfills is a wrapper that dispatches the registry command of the same name as the host: an admin context with no account behind it, so its audit rows carry no actor and source: host. The same commands run in an instance admin's SSH session (ssh git@<host> admin ...) and write the same rows with the key fingerprint as source. One implementation, two credentials.

gitbayd admin user create alice --key alice.pub --email a@example.org --verified [--admin]
gitbayd admin email verify alice a@example.org   # admin assertion, no SMTP needed
gitbayd admin invite --email b@example.org       # mails a code; prints it if no SMTP

"Verified" means SMTP-confirmed or host-admin-asserted; the database records which. Verified emails are what make commit signatures meaningful — an unverified address never produces a verified badge.

The audit log is the security feed (events are the product feed): every successful mutating command with its argv and source credential (SSH key fingerprint or API), every refused one (exit 3 or 4) as refused <command>, refused pushes as refused git-receive-pack, registrations, admin actions, force-pushes, and auth failures/throttling. A refusal row keeps the flag names and the first positional, not the values. Refusals are recorded up to ten a minute per account and 600 a minute across the instance; past either, one refused.throttled row stands for the rest of that minute. The caps bound the embedded listener, the web and the API; under ssh.mode = "system" each gitbayd shell connection counts separately. Secrets never appear — they travel on stdin, never in argv.

Each row carries the SHA-256 of the row before it. gitbayd admin audit verify opens the store as other admin commands do, applying pending migrations, so run it with the binary that matches the daemon. It recomputes the chain and exits 1 naming the first row that was edited or whose predecessor was removed. Retention removing the oldest rows is not a break. Rows written before the chain existed are counted and skipped; when every row is such a row, verify warns and exits 1, since clearing the hash columns looks the same. After an upgrade that clears with the first new audit row.

Removing the newest rows leaves no break, and neither do rows written afterwards under the freed ids. The database cannot show either. The daemon logs every row it writes to its journal, outside the database (journalctl -u gitbayd -g 'INFO audit '), and verify prints the last id and hash: compare them with the newest journal line. Rows written by host gitbayd admin commands, and by gitbayd shell when ssh.mode = "system", are not copied to the journal.

gitbayd admin audit [--actor u|-] [--action prefix] [--since 24h|7d|date] [--limit n] [--json]
gitbayd admin audit verify           # check the hash chain; exit 1 names the first bad row
ssh git@<host> audit ...             # the same, from an admin session
ssh git@<host> admin user list [--state active|pending|disabled|admin]
ssh git@<host> admin user show <name>   # keys, emails, orgs, tokens, sessions
ssh git@<host> admin user limits <name> [--repos n|default] [--bytes n|default]   # per-account caps
ssh git@<host> admin user promote <name>   # grant instance admin
ssh git@<host> admin user demote <name>    # remove it; the last admin is refused
gitbayd admin user promote <name>    # host-local: recovery when no admin key is reachable
gitbayd admin user disable <name>    # suspend: SSH, web, API all refused;
gitbayd admin user enable <name>     #   sessions dropped, nothing deleted
gitbayd admin user delete <name> --yes  # only for accounts anchoring nothing:
                                     #   refused (with each blocker named) while
                                     #   the account owns repos, authored
                                     #   issues/MRs/comments/reviews, or is an
                                     #   org's only admin

--actor takes a username, or - for rows with no actor: host commands and auth failures. --action is a prefix, so cmd repo catches every repository command and admin every host or admin-session action. --since is a duration back from now (30m, 24h, 7d) or a date.

admin user list pages by username (--limit, --cursor) and carries each account's state and last_seen, the newest use of any of its SSH keys or API tokens. admin user show adds the keys with their last use, each address with how it was verified, PGP keys, org roles, the owned repository count, API token names, and live browser sessions. Both are Both are refused to non-admins, like audit, on every surface.

/admin/users is the same list in a browser, linked from the admin page: the state filter the command takes, keyset paging on its cursor, and a row per account with promote, demote, disable and enable, each dispatching the command. Demote and disable ask for the username to be typed, since both take someone's access away. Creating and deleting an account, issuing an invite and asserting an address stay on the command line: each takes a key, mints a credential, or cannot be undone. A non-admin gets the 404 a missing page would, so the URL confirms nothing.

Promotion needs an active account: a pending or disabled one is refused. Demotion is refused when it would leave no admin, over SSH, in the browser and on the host alike, so the host-local promote is the way back in when the only admin key is lost.

Instance admin carries no right on anyone's repository: policy does not consult it, and a private repository still answers not-found to an admin. Moderation goes through explicit overrides that skip the access check and write their own audit row:

ssh git@<host> admin repo list [--owner o] [--visibility public|private]  # size, last push
ssh git@<host> admin repo archive|unarchive <owner/name>
ssh git@<host> admin repo visibility <owner/name> public|private
ssh git@<host> admin repo delete <owner/name> --yes

Each lands in the audit log as admin repo.<action> naming the repository, on top of the cmd row every mutating command gets.

limits.ssh_auth_rate (10) throttles per-IP authentication failures per minute — successful auths never count and clear the slate. limits.max_pack_bytes is enforced as receive.maxInputSize on every push.

limits.write_rate (60) bounds mutating commands per account per minute. It is counted in the dispatcher, so SSH, the JSON API and the web spend one budget and a caller cannot refresh it by changing surface; limits.api_rate stays in front of it, bounding a network source rather than an account. A command is one token whatever it writes, so a bundle import costs one and only a loop of separate commands spends the budget. Read-only commands, the runner protocol (a build streams its log in many small writes) and the host CLI are exempt. Refusals exit 4 and say when to retry. A negative value turns the limit off; it matters most with registration = "open", where every write also queues notification mail and webhook deliveries.

Every background worker keeps a backlog and a failure state. An instance admin reads them all in one place:

gitbay dashboard --json | jq .queues   # webhooks, mail, push, mirrors, builds, deps

Per worker: pending, retrying (pending with a failed attempt) and dead-lettered counts with the oldest pending age, and the retrying or failed rows themselves, capped at twenty each. Builds list what is running and then what is pending, each since when, so a build no runner is scoped to claim is visible here rather than only in its repository; mirrors list the ones whose last sync failed; dependency checks list the ones whose last check errored. Non-admins get no queues key at all.

Push rows name the device id, never the token. Watch this one after configuring [push]: a key_id or team_id Apple did not issue passes config validation, which can only check that the .p8 parses, and then every send comes back 403 InvalidProviderToken and dead-letters on its first attempt.

In accounts mode the same read renders at /admin, linked from the rail for admins. Anyone else gets a 404 there.

A dead-lettered mail is logged as notification dead-lettered mail=<id>, with the address redacted out of the relay's error. The id is the queue row: find it in the Mail table on /admin, or in dashboard --json, where the recipient and the unredacted error are. That is deliberate — see the Threat-Model page.

gitbayd admin stats [--json]         # counts, database size, per-repo disk
ssh git@<host> admin stats [--json]  # the same, from an admin session
gitbayd admin gc [--repo owner/name] # git gc: repack and prune; per-repo sizes
gitbayd admin gc --aggressive        # thorough repack; slow, rarely needed
gitbayd admin gc --lfs               # also drop LFS objects no pointer names (older than a day)

A history rewrite leaves the commits it removed reachable through refs/merge-requests/N/head of the merge requests that landed them, so they stay fetchable by anyone who can read the repository. Nothing drops a head ref on its own — an open or source-gone MR is merged through it, and a merged or closed one keeps its diff readable through it — so the cleanup is a command an instance admin runs, naming the MRs:

ssh git@<host> admin mr prune owner/name 1 2 3 --yes

It refuses an open or source-gone MR, deletes the named refs, runs git gc --prune=now on that one repository so the objects go at once rather than after git's two-week grace, leaves a system comment on each MR, and audits as admin mr.prune. The MR keeps its title, comments, reviews and head sha; mr diff and the MR page say the head is gone. Run it when nothing is pushing to that repository: without the grace, a push caught between leaving quarantine and writing its ref loses its objects. Objects also survive in offsite backups until those are rewritten; see "Removing a repository's history from every snapshot".

deploy/cloud-init.yaml ships a gitbay-gc.timer that runs admin gc weekly (Sunday 07:00 UTC). Imported repositories keep whatever pack layout the source sent, so a first manual admin gc after a bulk import is worthwhile.

gitbayd admin backup --out /var/backups/gitbay/backup.tar.gz
gitbayd admin backup --verify /var/backups/gitbay/backup.tar.gz   # read it back

One archive: a consistent SQLite snapshot (taken before the repositories are read, so the database never references objects the archive missed), every repository, and the SSH host keys. Excluded: hook socket, regenerated hook scripts, WAL files. Safe to run against a live daemon.

--verify reads an archive back: the snapshot must pass SQLite's integrity check, and every repository the snapshot names must be in the archive. A database-only archive is checked for integrity and says so. Exit is non-zero on damage or a missing repository.

Restore: extract into an empty directory, point server.root at it, start gitbayd. Host keys are preserved, so clients keep their known_hosts entries; hooks regenerate at startup.

set -a; . ~/.config/gitbay/offsite.env; set +a
restic $RESTIC_OPTS snapshots
restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>

EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
restic $RESTIC_OPTS rewrite --dry-run $EXCL     # "would modify N snapshots"
restic $RESTIC_OPTS rewrite --forget $EXCL      # new snapshots replace the originals
restic $RESTIC_OPTS prune                       # drops the data nothing references
restic $RESTIC_OPTS find <name>.git <old-name>.git   # expect no output

Replace the binary, restart the unit. Migrations apply automatically and are transactional; hook scripts under <root>/hooks are rewritten at startup to point at the current binary path.

gitbay-runner executes builds queued by pushes and merge requests. It polls over SSH with a key of scope runner, which reaches only the runner protocol and read-only git (a runner executes arbitrary repository code, so the key it holds must not do more). A runner key claims builds only for the repositories it is attached to, by repo runner add from a repository admin or an instance admin; an admin key claims any. Users attach their own runners: see the Users page. For an instance runner, run it as a dedicated unprivileged user on a non-admin account. admin user create --key registers a full-scope key, so the runner key is added afterwards through a bootstrap key that is then removed, and attached to each repository it should build:

useradd --system --create-home --home-dir /var/lib/gitbay-runner ci-runner
sudo -u ci-runner ssh-keygen -t ed25519 -N "" -f /var/lib/gitbay-runner/.ssh/id_ed25519
ssh-keygen -t ed25519 -N "" -f /tmp/ci-bootstrap
gitbayd --config /etc/gitbay/config.toml admin user create ci --key /tmp/ci-bootstrap.pub
ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys add --scope runner < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
ssh -i /tmp/ci-bootstrap git@127.0.0.1 keys remove "$(ssh-keygen -lf /tmp/ci-bootstrap.pub | awk '{print $2}')"
rm /tmp/ci-bootstrap /tmp/ci-bootstrap.pub
gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work

gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub

-jobs N runs N builds at once. Claiming is one transaction that selects and updates, and each build works in its own build-<id> directory, so workers do not collide; idle polls are staggered across the interval so N of them do not wake together. The drop-in's weights below are per service, not per build, so raising -jobs divides them rather than multiplying the host's load.

admin runners shows which account each runner polls as, and what each may claim. A runner key claims builds only for the repositories it is attached to: with none attached it claims nothing, and -repos may only narrow within them. An admin's full-scope key claims any repository — that is what -repos was for — and still works for the protocol during a rotation. A merge request head from a fork is built in the target repository as untrusted: the claim carries no secrets, and only a runner started with -untrusted takes it. Same-repository heads were built by their branch push and are not built again.

make deploy-runner also installs deploy/gitbay-runner.override.conf as a systemd drop-in: Nice=10, CPUWeight=30, IOWeight=30, so a build never starves the host's sshd, the daemon or the backup timers, and NoNewPrivileges, ProtectSystem=full, ProtectKernelTunables, ProtectControlGroups and RestrictSUIDSGID, so a step cannot reach outside its workspace and the runner's home. The e2e suite alone starts sixty daemon instances; without the drop-in a deploy's copy over the admin sshd stalled. Both deploy targets copy with rsync --partial, which resumes a stalled transfer.

Under -isolation podman, the default and what bay1 runs, each build is confined to a container (see Container isolation below). Under -isolation none steps run directly on the host as the runner's user, so treat that machine as executing whatever your users push, and install the toolchains your builds need on it.

A runner claims the oldest pending build among the repositories its key is attached to — for an admin key, the oldest in the instance. -repos narrows within that set, which is what makes a runner outside the server practical: one on a machine that should build a single project, or that holds credentials for one deployment, stays on it.

Oldest-first is across everything the key may claim, so a repository with a deep queue holds every other repository the same runner serves; bay1 measured a 15-minute average wait on a day of merge request stacks from one repository. A runner attached to one repository cannot be starved. That is the rule, decided in krz/gitbay#207: a runner serving several repositories takes them oldest-first, and an operator who wants one repository never to wait on another runs a second runner attached to it alone. Nothing caps what an account queues, and nothing needs to (krz/gitbay#206): a schedule tick queues nothing while the job's last build is pending or running, so a repository with no runner holds one row per scheduled job rather than one per tick, and a build runs only on a runner its owner attaches, so a busy schedule spends the owner's compute. Pushes are bounded by what an account can push.

gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs \
  -workdir /var/lib/gitbay-runner/work

Add -untrusted only with -isolation podman.

gitbay.org's runner is attached to the forge's own repositories and the isolation canary, nothing else, because it shares the host with the forge; its unit names no -repos, the attachments are the boundary. Any other repository builds on a runner its owner attaches.

-repos narrows an admin runner; for a runner key the attachments are the boundary, held by the server, and -repos may only name repositories among them. -untrusted makes a runner claim merge request heads from forks; the bay1 unit sets it because it isolates in podman. A runner without it builds trusted commits only.

gitbay dashboard and ssh git@<host> admin runners list every key that has polled as a runner: the account, the key's fingerprint, when it last polled, the repositories it may claim — its attachments for a runner key, the -repos it asked for or any for an admin key — and the build it holds; admin runners remove <fingerprint> (forget until the next release) drops the row for a key that polled by mistake, the key itself untouched. admin runners also heads the list with the queue: builds pending now, and over the last day how many were claimed, how long they waited to be claimed (average and worst), and how many the reaper ended instead of a runner reporting them. A build a runner claimed and never reported is failed by the scheduler's minute tick, whether or not any runner is still alive: within about two minutes of its log stream ending with no outcome reported — the runner reports right after closing the stream, retrying for half a minute if gitbayd is unreachable — or, if no stream was ever seen, at the deadline.

Instance admin on the runner account only authorizes the claim/report protocol; it grants no repo access. A build that pushes back — a pages deploy, an archive publish, an automated MR branch — needs an explicit grant on that repo: repo access grant <owner/name> ci write. Private repos likewise need at least read for the clone.

ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'

su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
su - ci-runner -s /bin/sh -c "podman images"

ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
make deploy-runner

gitbay repo create cmc/ci-smoke          # then push a .gitbay/ci.yml naming the image
sed -i 's#-repos krz/gitbay #-repos cmc/ci-smoke #' /etc/systemd/system/gitbay-runner.service.d/override.conf
systemctl daemon-reload && systemctl restart gitbay-runner
gitbay build log cmc/ci-smoke 1         # green: switch -repos back, redeploy

Objects live content-addressed under [lfs] root (default <server.root>/lfs); [lfs] max_object_bytes caps a single object (512MB default). Storage sits behind a small interface — an S3-compatible backend is a drop-in with the server proxying, and presigned URLs a later optimization. LFS objects do not travel with push mirrors (mirrors move git refs only), and gc does not yet collect orphaned objects.

[pages] domain = "example.site" serves public repos' pages branches on <owner>.<domain>. DNS needs a wildcard record *.<domain> to the server; ACME issues per-subdomain certificates on demand (only for owners that exist). The domain must not be the site host or a parent of it — pages content runs its own scripts and must stay off the forge's origin.

Users with repo admin claim custom domains with repo domain add. Claims activate only after a DNS TXT challenge proves control of the domain (repo domain verify, audit-logged); pending claims serve nothing, get no certificates, and expire after 7 days. ACME issues certificates only for verified hosts, so stray DNS pointed at the server gets nothing.

The Threat-Model file is the reference for what the forge trusts and refuses to do. Operational checklist:

• Software checks. deploy/audit.sh runs go vet, govulncheck (the module list is deliberately short — review it on each release), and a short fuzz pass over every attacker-facing parser (pkt-line, commit, SSHSIG armor, OpenPGP key, SSH tokenizer). Run it before tagging a release. CI's own vuln job runs govulncheck nightly against main rather than per push, because @latest scans today's advisory database and an advisory lands without anyone pushing; build trigger krz/gitbay vuln runs it on demand.
• Web responses carry a scripts-forbidden CSP, X-Frame-Options: DENY, nosniff, no-referrer, and HSTS when TLS is on — no configuration needed.
• Host sandboxing. The systemd unit in deploy/cloud-init.yaml runs gitbayd unprivileged with ProtectSystem=strict, PrivateDevices, LockPersonality, MemoryDenyWriteExecute, SystemCallFilter=@system-service, and RestrictAddressFamilies to INET/INET6/UNIX. It keeps CAP_NET_BIND_SERVICE only, to bind 22/80/443.
• OS patches apply via unattended-upgrades (security origins, auto-reboot 04:30 if required).
• Admin sshd (2222) is throttled by MaxStartups=/=MaxAuthTries and watched by fail2ban; gitbayd's own port 22 is throttled by limits.ssh_auth_rate (auth failures per IP per minute), and every account's writes by limits.write_rate.
• Monitoring. gitbay-monitor.timer writes a reading hourly to

journald and, when /etc/gitbay/monitor.url exists, posts it to that webhook: disk, service, the daemon's own /healthz answer, certificate expiry, and the age of the newest full backup and database snapshot. It exits non-zero on an alert so the unit shows in systemctl --failed: a stopped service, /healthz not answering ok, disk ≥ 85%, a certificate under 21 days, a full backup over 25 hours old, or a database snapshot over 2 hours old.

GET /healthz is unauthenticated and cache-free: whether the database answers and which commit serves, 503 when it does not.

• Database. gitbay.db and its WAL live under /var/lib/gitbay (mode 0750, owned by gitbay). The nightly archive plus provider snapshots are the recovery path; for tighter RPO, add continuous replication (litestream) against the same file — it coexists with the WAL.

• deleting a fork marks MRs sourced from it source_gone; their diffs remain viewable and mergeable because the target repo owns the objects.
• refs/merge-requests/* is server-owned and unpushable by clients; only admin mr prune removes one (see Maintenance).
• audit-relevant activity (issue/MR lifecycle, imports, pushes) lands in the events table, which also feeds webhooks.
• the daemon idles under 10MB RSS; the smallest VPS tier is adequate.