.gitbay/wiki/Admin.org

• ssh.mode = "embedded" (default): gitbayd itself listens, normally on 22 — move the host's admin sshd to another port. Remotes read git@host:owner/repo with no port gymnastics.

• ssh.mode = "system": the host sshd owns 22 and invokes gitbayd via AuthorizedKeysCommand:

  AuthorizedKeysCommand /usr/local/bin/gitbayd --config /etc/gitbay/config.toml authorized-keys %t %k
  AuthorizedKeysCommandUser gitbay
  

  sshd requires that binary to be root-owned and not group/world writable. Unknown keys fail authentication inside sshd, so system mode requires registration.mode = "closed" (check-config enforces this). Under the forced command the CLI's leading --term=<cols>[,color] argument works as is; a client that sets GITBAY_TERM instead needs AcceptEnv GITBAY_TERM in sshd_config, since that env request is handled by the host's sshd, not gitbayd.

• root (default /var/lib/gitbay) — repositories, database, host keys, ACME cache all live here.
• site_url (required) — canonical https://host; drives ACME, clone URLs, mail links.
• secret_key_file (default /etc/gitbay/secret.key) — the keys that seal CI secrets, webhook secrets, mirror tokens and push device tokens in the database. Must be outside root, mode 0600, readable by the daemon's user. See "Secret key" below.
• source_repo (optional, owner/name) — the repository this instance develops itself in. Startup warns when the running build's commit is not on that repository's default branch, which is how a binary built from an unmerged branch stops being invisible. Leave it unset unless the instance hosts its own source.

• mode — embedded | system (above).
• port (22) — embedded listener port.
• host_keys — list of private key paths; empty generates an ed25519 key at <root>/ssh/host_ed25519.

• addr (:443), tls — acme | files | off.
• acme: certificates via TLS-ALPN-01 on the HTTPS port, cached at <root>/acme; acme_email for the CA account; acme_http_addr (:80, "off" to disable) adds HTTP-01 and an https redirect — failing to bind it is a warning, not fatal. Requires an https:// site_url with a public DNS name.
• files: cert_file + key_file.
• off: plain HTTP — development, or behind a TLS-terminating proxy.
• The HTTPS listener accepts TLS 1.2 and 1.3 only (serverTLS in cmd/gitbayd/tls.go), with the default cipher suites of the Go release the binary was built with. openssl s_client -connect <host>:443 -tls1_1 fails the handshake.

trusted_proxies lists the addresses or CIDRs of reverse proxies in front of the daemon. A request from one of them is attributed, for API rate limiting, to the last X-Forwarded-For hop that is not itself a trusted proxy; from anyone else the header is ignored. Empty, the default, is right when gitbayd terminates TLS itself.

A reverse proxy in front of gitbay must not buffer responses, or the build page's live log arrives only when the build ends; X-Accel-Buffering: no covers nginx.

• mode — view_only (default) | accounts. In view_only the mutating web routes are never registered; in accounts, browser sessions are minted over SSH (web login), and users with write access can create repos, comment, and make simple file edits (which commit unsigned, honestly). password_auth is reserved and currently rejected.
• title — the instance's display name in the rail and page titles; empty falls back to the site host. Lower case is the convention for gitbay itself.
• privacy_notice — operator text shown on /privacy under the fixed statement.

• mode — closed (default) | invite | open. invite/open require [mail]. See the user guide for the flows.
• pending_expiry (empty, never) — a duration such as "168h"; a self-registered account still unverified after that long is removed, hourly and at start, audited as pending.expired.
• notify_admin (false) — mail every instance admin when an account becomes active: an invite redeemed, or an open-mode signup that verified its address. The unverified row an open signup creates is not reported, because anyone can post the form and mailing on that would point a flood at the admins. Recipients are the verified primary addresses of active admins who have activity mail on, the same rule any other notice follows, so an admin with no verified address hears nothing. The notice is queued, so a dead SMTP host shows up in the admin page's Mail table instead of failing the registration. Requires [mail].

• smtp_host (host:port; 587 assumed, 465 with tls = "implicit"), from, optional smtp_user / smtp_pass. Required for invite/open registration and self-service email add; in closed mode you may omit it entirely and assert addresses by hand (below).
• tls — starttls (default) or implicit (TLS from the first byte, for relays on 465).
• require_tls — with starttls, a relay that does not offer STARTTLS gets no mail: delivery fails and retries, and the admin page's Mail table shows the error. Unset, it is on for every relay except localhost and loopback addresses; set false to allow plaintext to a remote relay.

Reply by mail: a reply to issue or merge request mail posts a comment (#295). gitbayd polls a mailbox over IMAP; no port is opened for it. Off unless enabled, and it requires [mail] smtp_host, since the replies answer mail gitbayd sends.

[mail.inbound]
enabled       = true
imap_host     = "imap.example.org"       # host:port; 993, or 143 with tls = "starttls"
tls           = "implicit"               # or "starttls"; there is no plaintext setting
user          = "reply@gitbay.example"
password_file = "/etc/gitbay/imap.pass"  # one line, mode 0600, owned by gitbayd's user
mailbox       = "INBOX"                  # default
poll_interval = "1m"                     # default; at least 10s
reply_address = "reply@gitbay.example"
require_dkim  = true                     # recommended; see below
trusted_authserv_id = "mx.example.org"   # the mail host's Authentication-Results id

• The password is read from password_file at every connection and never appears in the config, argv or the log. An unreadable file, or one readable by group or others, stops the daemon at start.
• reply_address is what each Reply-To is built from: reply@gitbay.example becomes reply+<token>@gitbay.example. The mailbox must receive that. Hosts that deliver subaddresses to the mailbox need nothing more; others need a wildcard rule. Migadu, for one, sent threads+x@gitbay.org nowhere until a rewrite from threads+* to threads@gitbay.org was added. A domain catch-all only works if it points at this mailbox, which then also holds everything else sent to the domain. Send a test to <reply_address>+test and expect a refused mail reply audit row with "malformed reply token" within a poll interval (gitbay audit --action 'refused mail'). Point the domain's MX at the mail host as for any other mailbox; nothing about it involves gitbayd.
• Use a mailbox that holds nothing else. gitbayd reads every unseen message in it and marks each one \Seen when it is handled, posted or refused. A message that fails for a reason that may pass (the database busy, the server refusing that one fetch) stays unseen and is tried again on the next poll, up to five times, then is marked seen and audited. A dropped connection or a timeout ends the poll and counts against no message. A message over 10 MiB is refused by its size without being fetched. A server that sends more than about 11 MiB, or more than a thousand untagged responses, for one command has its connection closed; one poll handles at most ten thousand messages.
• Whatever the settings, a message is refused when a header field name is not RFC 5322 ftext (From : x, a space or a non-ASCII byte in a name), when it does not have exactly one From, or when it has more than one To, Cc, Message-ID, Content-Type or Content-Transfer-Encoding.
• require_dkim (default false) should be on for any instance reachable from the internet. With it, a reply is posted only when gitbayd itself verifies one of its DKIM signatures (RFC 6376) with a d= in relaxed alignment with the From domain (the same organizational domain by the public suffix list; d=github.io aligns with nothing) and an h= that covers From, the To or Cc holding the reply address, Content-Type, and Message-ID when the message has one. A Content-Transfer-Encoding outside h= is accepted only when it is 7bit, 8bit or binary, which leave the decoded body as it is; Thunderbird, for one, does not sign it. An unsigned quoted-printable or base64 is refused. The reply address is read only from To or Cc: a reply that reached the mailbox by Bcc, with the address only in Delivered-To, is refused ("reply address not in To or Cc"). It needs nothing from the mail host, so it works where the host adds no Authentication-Results. rsa-sha256 (keys of 1024 bits or more) and ed25519-sha256 are accepted, with simple or relaxed canonicalization; rsa-sha1, a body length tag (l=), an expired x= and a t= more than fifteen minutes ahead are refused. Only the first five signatures are checked. The key is looked up at <s>._domainkey.<d> with a five-second timeout and cached for fifteen minutes (the resolver does not report the record's TTL); a lookup that fails for a reason that may pass (a timeout, SERVFAIL) leaves the message for the next poll, up to the five tries above, while a missing key refuses it. Signatures are checked on the message as fetched. Each passing signature is recorded with the Message-ID, so a copy of the same signed message posts once even with unsigned fields changed.
• Either require_dkim or trusted_authserv_id passing is enough to authenticate From; when both are set, a reply needs only one of them, and a refusal names both reasons. With only trusted_authserv_id, the reply address may come from any recipient field (Delivered-To, X-Original-To, Envelope-To, To, Cc), since the mail host vouches for the sender and not for the fields. Set both when the mail host adds Authentication-Results for most senders but not all.
• trusted_authserv_id names the authserv-id the mail host writes at the start of its Authentication-Results header (Gmail's is mx.google.com). With it set, a reply is posted only when the topmost header with that id shows dmarc=pass with header.from equal to the From domain, or dkim=pass with a header.d in relaxed alignment with it (the same organizational domain by the public suffix list; a public suffix such as github.io aligns with nothing). The header is parsed per RFC 8601, so text inside a quoted string or a comment (a quoted MAIL FROM local part, a reason) is never read as a result. Lower headers claiming the same id are the sender's and are not read. This is only safe when the mail host removes incoming Authentication-Results headers that claim its id, as RFC 8601 asks; Gmail, Fastmail and Migadu do. Check yours before relying on it. With neither this nor require_dkim set, the daemon logs a warning at start and admin mail inbound check repeats it: From is then whatever the sender wrote. Mail between two addresses at the same host may carry no Authentication-Results at all: at Migadu, mail from another Migadu-hosted domain is delivered through its outbound path and gets none, so setting the id alone refuses every reply from such users. That mail does carry a DKIM signature aligned with From (for example d=cleberg.net; s=key1; a=rsa-sha256; c=simple/simple; h=from:to:subject:date:message-id:mime-version:content-type), which is why gitbay.org sets require_dkim instead.
• gitbay admin mail inbound check logs in, opens the mailbox read-only (EXAMINE) and reports the message and unseen counts and how From is authenticated (require_dkim, trusted_authserv_id), so a check never marks a reply seen before the poller reads it. It warns when neither is set. With inbound off it says so and exits 0. Poll failures are logged as mail reply: poll failed with the server and the IMAP error.

A reply is posted when all of these hold, checked when it is read:

1. It is not an automatic reply (Auto-Submitted, Precedence: bulk and the like).
2. A recipient header carries a reply address whose token verifies and has not expired (thirty days from the mail it came on).
3. The token's account exists, is active and not disabled, still has reply by mail on, and was created before the token was: an id freed by a delete and reused is not the account the token named. The same holds for the repository.
4. From is one of that account's verified addresses, and, with require_dkim or trusted_authserv_id set, a DKIM signature gitbayd verified or the mail host's result authenticated it.
5. The message has a text/plain part (HTML-only mail is refused, not converted), and what is left after quoted text and the signature are removed is not empty and fits a comment (64 KiB).
6. No earlier copy of the message (same Message-ID, account and thread) posted.
7. issue comment or mr comment, dispatched as the account, accepts it: the account can still read the repository, the thread exists, the repository is not archived, the write budget is not spent.

The comment's audit row is cmd issue comment (or mr comment) with source: mail. A refusal writes a refused mail reply row with the reason and the Message-ID, never the message's content, at most sixty a minute, and sends nothing back. See Threat-Model for why the token and the sender address are both required.

Push notifications to Apple devices, delivered by gitbayd talking to APNs directly over HTTP/2, authenticated by an ES256 JWT signed with an operator-supplied provider key. Off unless configured.

• enabled (false).
• key_file — path to the .p8 provider key from Apple's developer portal (Certificates, Identifiers & Profiles → Keys). It belongs at /etc/gitbay/apns.p8, mode 0600, owned by the account gitbayd runs as. Read and validated at startup: it must parse as a PEM-wrapped PKCS#8 EC (P-256) private key, or the daemon refuses to start rather than fill a queue nobody is watching.
• key_id, team_id — the key's id and your Apple developer team id, both from the same portal page.
• topic — the app's bundle identifier. An APNs key belongs to a bundle ID. gitbay.org pushes to the App Store build under its own bundle id; a self-hoster who wants push ships their own iOS build under their own bundle id, with its own .p8 key from their own developer account, and points topic at that id. There is no way to push to someone else's build, by design — this is Apple's model, not gitbay's.
• environment — production or sandbox, naming the APNs host rather than taking a URL, so a typo cannot aim the key at a host that is not Apple's.

All five of key_file, key_id, team_id, topic and environment are required when enabled is true; validation runs at config load, so a misconfigured [push] is caught before the daemon serves anything. The delivery queue (a device's undelivered and attempted pushes) is capped the same way the mail queue is, by [retention] push.

• age_recipients (optional) — age public keys (age1...). When set, admin backup encrypts every archive to them and appends .age to its name. Generate the pair off the host with age-keygen; only the public key goes here, so the host writes archives it cannot read. The restic copy is unaffected: the offsite job stages its own VACUUM INTO of the live database and snapshots /var/lib/gitbay, not the archives.

• enabled (false) — the JSON API surface; see API. Off means no credential-bearing HTTP endpoint exists at all.

• allow_local (false) — permit webhook, mirror and import targets on loopback, private, shared (100.64.0.0/10), link-local or multicast addresses. Leave off unless you know why you need it (SSRF).

• clone_timeout (3600s) — cap on repo import fetches. An import takes http and https URLs only, passes the same address check as a mirror sync and is pinned the same way, so it needs git 2.37 too.
• max_blob_bytes (100MB) — cap on raw file serving over the web.
• max_asset_bytes (512MB) — cap per uploaded release asset.
• max_snippet_bytes (1MB) — cap per snippet file.
• max_snippets_per_user (0, unlimited) — snippets an account may own.
• max_repos_per_user (0, unlimited) — repositories an account may own directly; repo create, fork and import refuse past it. Organizations are not capped.
• max_bytes_per_user (0, unlimited) — disk the account's own repositories may take; a push may be no larger than what is left.
• pack_concurrency (3), pack_per_principal (2), pack_queue (32), pack_queue_wait ("60s") — git pack generation (clones, fetches, git archive --remote, web archive downloads, repo download over SSH and the API) over SSH, smart HTTP and git:// shares one budget: this many at once, this many per account (per client address when anonymous: an IPv4 address, or an IPv6 /64), and this many waiting for at most the wait. Anonymous clients together hold at most pack_concurrency − 1 slots when it is above 1: an account (SSH key, bearer token or web session) can take the last slot when it is free, and anonymous clients cannot hold it. Past that an SSH client gets "the server is busy…" and exit 1, HTTP gets 503 with Retry-After: 30, git:// an ERR line; the daemon logs a pack limit warning naming the transport and whether the client was signed in, at most once a minute per transport. An SSH client that disconnects while queued leaves the queue; an HTTP or git:// one keeps its place until the wait runs out. A running clone is killed when its client disconnects, or when no write to the client completes for two minutes: a client reading below about 550 B/s, or an HTTP request body that takes over two minutes with nothing written back, is cut. Ref listings (info/refs, protocol v2 ls-refs) and pushes are outside the budget. A busy repo download exits 1 with the same message; over the API it is a 503 with Retry-After: 60. For the three counts 0 means the default and a negative value turns that bound off. The defaults suit a four-core host; see Performance. With ssh.mode = "system" each SSH session is its own process and SSH clones are not counted.
• push_concurrency (2), push_per_principal (1), push_queue (16), push_queue_wait ("60s") — git receive-pack over SSH, on a budget of its own so clones cannot starve pushes or the reverse: this many at once, this many per account or deploy key, and this many waiting for at most the wait. One principal may have up to half of push_queue (at least one) waiting, so parallel pushes from one account queue rather than being refused, but cannot fill the queue. A deploy key is its own principal, apart from the account that registered it, so an account with deploy keys can hold push_per_principal slots per key; the global cap still holds. A bot account such as a runner's counts like any other account. The slot is taken after the access checks and held until receive-pack exits, which is after post-receive (merge detection, CI queueing, webhooks) has run. Past that the client gets "the server is busy: it is at its limit of concurrent pushes…" and exit 1, and the daemon logs a push limit warning at most once a minute. A client that disconnects while queued leaves the queue; one that disconnects mid-push ends receive-pack and frees its slot, and a revoked key kills it. 0 and negative values read as for pack_*. HTTP and git:// carry no pushes. With ssh.mode = "system" pushes are not counted and the two timeouts below do not apply.
• push_idle ("60s"), push_receive_timeout ("15m") — a push holding a slot is killed, its process group with it, when its pre-receive has not started push_receive_timeout after it took the slot, or when no byte has been read from the client and none written to it for push_idle. The idle rule starts when the pack signature arrives from the client or pre-receive starts, whichever is first (a delete-only push sends no pack): before that the client may be silent while its pack-objects counts and compresses, and only push_receive_timeout applies. receive-pack runs with receive.keepAlive set to a quarter of push_idle (1 to 5 seconds), so the keepalives it sends to side-band clients while it indexes and runs hooks keep a working push alive. Once pre-receive has started only push_idle applies: post-receive is never cut short by the clock. A push killed before pre-receive answers updates no refs.
• max_pack_bytes (2 GiB) — the largest pack one push may send, enforced as receive.maxInputSize and lowered to what an owner's storage quota has left.
• ssh_auth_rate (10) — SSH authentication failures per client address per minute on the embedded listener; see below.

• enabled (false), port (9418) — the anonymous git:// listener. Serves only public repositories that additionally ran repo settings git-daemon <repo> on.

• pull_interval_minutes (15) — how often pull mirrors fetch their upstream. Push mirrors sync shortly after each local ref update. Mirror URLs pass the same SSRF rules as webhook targets, when saved and again before every sync; git then connects only to the addresses that were checked (http.curloptResolve) and does not follow redirects, so a mirror of a renamed repository fails until its URL is updated. Needs git 2.37 or later on the server; with an older git the worker logs an error at start and syncs no mirror, recording the reason on each. Sync ignores the system and global gitconfig.

Vanity Go module paths, one per line: "host/module" = "owner/repo". Requests with ?go-get=1 at or under the module path answer with the go-import meta tag pointing at the repository's HTTPS clone URL, so go install host/module/cmd/...@latest resolves. The repository should be public (the module path itself confirms it exists).

Two timers, because the two halves of the data have different exposure.

• gitbay-backup.timer, nightly. The full archive above, last 7 kept.
• gitbay-db-backup.timer, hourly. admin backup --db-only, which writes the SQLite snapshot alone, last 48 kept. A few MB against the full archive's hundreds, which is what makes the frequency affordable.

The split follows what a loss would actually cost. Repositories are git, so a mirror or any clone is a second copy; the database is the only copy of issues, merge requests, comments and review state. So the recovery point is about an hour for the data that exists nowhere else, and a day for the data that does.

Continuous replication (litestream and similar) was considered and not adopted. It would take the database's recovery point to seconds, but the repositories would still be on the nightly archive, so a restore could produce a database referencing commits the repository backup does not have. Consistency between the two halves is worth more here than latency on one of them. Revisit if repository replication becomes continuous too.

bay1 also takes a nightly restic snapshot of /var/lib/gitbay and /var/lib/gitbay-stage (a database copy and config.toml, staged there) to an S3 bucket at Scaleway, with a key that can only add snapshots. Nothing else under /etc/gitbay is in it: not the secret key file, not apns.p8. The key that can remove them lives on the operator's machine, in ~/.config/gitbay/offsite.env, and never on bay1: a compromised host cannot destroy its own history. Forgetting, pruning and rewriting all run from there.

set -a; . ~/.config/gitbay/offsite.env; set +a
restic $RESTIC_OPTS snapshots
restic $RESTIC_OPTS ls <old-snapshot> /var/lib/gitbay/repos/<owner>

EXCL="--exclude /var/lib/gitbay/repos/<owner>/<name>.git --exclude /var/lib/gitbay/repos/<owner>/<old-name>.git"
restic $RESTIC_OPTS rewrite --dry-run $EXCL     # "would modify N snapshots"
restic $RESTIC_OPTS rewrite --forget $EXCL      # new snapshots replace the originals
restic $RESTIC_OPTS prune                       # drops the data nothing references
restic $RESTIC_OPTS find <name>.git <old-name>.git   # expect no output

CI secrets, webhook secrets, mirror tokens and APNs device tokens are stored sealed: AES-256-GCM under a key in server.secret_key_file, each value prefixed with the id of the key that sealed it (gbs1:<id>:). The key file is not in the database, not under server.root, and therefore in neither the local archives nor the main restic repository. It must be copied off the host separately; without it a restored database's secrets cannot be opened, and gitbayd refuses to start against them. A separate restic repository for it and apns.p8 is planned (runbook D of the data-at-rest plan) and not yet in place, so today the only off-host copy is one the operator makes by hand after init and after every rotate.

gitbayd admin secrets init     # once; deploy/install.sh does it on first install
gitbayd admin secrets check    # open every value, count by key
gitbayd admin secrets rotate   # new key, reseal, retire the old one (as root)

• Missing file: every gitbayd process that opens the database refuses to run and names the path, including serve and, in system mode, authorized-keys. migrate does not need it.
• Backups: admin backup and admin backup --verify do not need the key file; a restore does.
• Wrong key: serve stops at startup naming the first row that does not open; secrets check does the same without starting anything.
• Upgrade: the first start after the upgrade seals every value still in clear and logs sealed secret values.
• Rotation: rotate adds a key, reseals every value under it in one transaction, then removes the old keys. Run it as root, since it replaces the key file in /etc/gitbay; the file keeps its owner. The daemon re-reads the file when it changes, so it needs no restart. Copy the new file off the host afterwards. init and rotate hold an flock on <key file>.lock while they run, so a second run waits for the first.
• Push devices are looked up by the SHA-256 of their token (push_devices.token_hash), since two seals of one token differ.

A restore onto a clean host, run quarterly (January, April, July, October) and after any change to the backup code (cmd/gitbayd/backup.go, cmd/gitbayd/restoredrill.go, the offsite job), and recorded below. The disaster it rehearses is losing bay1, so the local archives are gone with it and the sources are the main offsite restic repository (repositories, LFS, the staged database, config.toml), the off-host copy of secret.key and apns.p8 (a keys repository once runbook D creates it; until then the operator's hand-made copy), and the operator's password manager (offsite.env, the keys repository's password and token once it exists, backup-identity.txt). The full steps are runbook C of the data-at-rest plan (docs/plans/2026-09-27-data-at-rest-and-backup.md).

gitbayd admin restore-drill does the archive half. It extracts a full archive into an empty or absent directory, runs every --verify check on the extracted copy, and prints what was restored, the newest issue, issue comment, merge request comment and push in the restored database, and the elapsed time. Exit is non-zero if any check fails.

gitbayd admin restore-drill /var/backups/gitbay/gitbay-20260927-090000.tar.gz --into /srv/drill
gitbayd admin restore-drill <archive>.tar.gz.age --identity backup-identity.txt --into /srv/drill

admin backup --verify extracts the archive under $TMPDIR. A host whose /tmp is a tmpfs smaller than a full archive (bay1's is 3.9 GB of memory) fails with "no space left on device"; point TMPDIR at a directory on disk that the backup's user owns:

install -d -o gitbay -g gitbay -m 700 /var/backups/gitbay/verify-tmp
cd /var/backups/gitbay
sudo -u gitbay env TMPDIR=/var/backups/gitbay/verify-tmp gitbayd admin backup --verify <archive>

What to restore, and from where:

From the offsite path (restic is not run by any gitbay command):

restic restore latest --target / --include /var/lib/gitbay --include /var/lib/gitbay-stage
cp /var/lib/gitbay-stage/gitbay.db /var/lib/gitbay/gitbay.db
gitbayd --config /etc/gitbay/config.toml admin backup --out /tmp/drill.tar.gz
gitbayd admin restore-drill /tmp/drill.tar.gz --into /tmp/drill-root

The second archive is how the restic tree gets the same checks and timestamps; /tmp/drill-root is discarded afterwards.

What to check, each a column below:

• DB integrity, connectivity, release assets, LFS: restore-drill prints integrity ok, connectivity ok on N repositories, release assets ok: N, LFS objects ok: N. Compare N with gitbayd admin stats --json on the source at the snapshot time.
• Secrets: gitbayd --config /etc/gitbay/config.toml admin secrets check opens every value.
• Host key: ssh-keyscan -p 22 <drill-host> matches the source's fingerprint.
• Config: gitbayd --config /etc/gitbay/config.toml check-config.
• Service: ssh -p 22 git@<drill-host> whoami and a git clone over SSH succeed.

Time to service runs from the clean host's first root login to the first successful git clone over SSH from it. The recovery point is the time of the newest restic snapshot restored; record beside it the newest issue, comment and push restore-drill printed, which show how much activity the restore carries.

The first drill ran on the operator's laptop rather than a provisioned host, so its time to service has no provisioning in it, and it could not check secrets: no copy of secret.key was on the machine, and gitbayd refuses to start while any sealed value does not open. The sealed values were cleared in the drill copy to reach service. The off-host copy of the key is confirmed to exist (#305); the next drill restores it with the snapshot and runs admin secrets check on the restored copy.

Builds run in a rootless podman container, one per job, with the workspace bind mounted and nothing else: the clone happens outside with the runner's key, so a step cannot read it. -isolation none keeps the old behaviour — steps on the host as the runner's user — for an instance where every repository is trusted. There is no automatic fallback: a runner started with -isolation podman that cannot find a working podman exits rather than running a build unsandboxed.

gitbay's own jobs name localhost/gitbay-ci:2, built from deploy/Containerfile.ci on the runner host. A job's image must carry what its steps need: the suite drives real git, git-lfs, gpg and sshd and asserts they exist before running, so the stock runner default would fail it immediately. Build or rebuild it with:

ssh -p 2222 root@<host> 'cat > /tmp/Containerfile.ci' < deploy/Containerfile.ci
ssh -p 2222 root@<host> 'su - ci-runner -s /bin/sh -c \
  "podman build -t localhost/gitbay-ci:2 -f /tmp/Containerfile.ci /tmp"'

The tag is deliberate rather than :latest: changing the file means bumping the tag in .gitbay/ci.yml, so a running branch's image does not change under it.

-image names the image a job runs in when it declares none, and is required under -isolation podman: there is no built-in default, because an image this host does not have would fail every build. A job overrides it with image: in .gitbay/ci.yml, validated as a reference so a config file cannot turn it into podman arguments.

-cpus and -memory cap one build (podman's units, e.g. -cpus 2 -memory 4g); unset means uncapped. The runner applies them itself: it creates a cgroup per build under its own delegated service cgroup, writes the limits there, and starts every podman process for the build inside it, with podman's cgroup handling off. Podman's own --memory and --cpus never applied under rootless cgroupfs, which is what a system service gets (krz/gitbay#188). The unit therefore needs Delegate=yes, which the drop-in sets; without it the runner refuses to start when a limit is set, and logs that builds run unconfined when none is. bay1 runs -cpus 3 -memory 6g per build inside MemoryMax=6G and CPUQuota=300% on the unit, on a 7.7GB four-core host with no swap: the memory cap is what keeps the forge alive when a build allocates without bound, and it sits above the e2e suite's 5GB peak rather than at a fair share. OOMPolicy=continue keeps systemd from stopping the runner when a build is OOM-killed.

A trusted build's home is its repository's, under <workdir>/trusted-home/<owner>/<name>, mounted into its containers as HOME: caches persist between trusted builds of one repository and are never read by another's. An untrusted build — a merge request head from a fork — gets <workdir>/build-<id>-home, new and empty, removed when the build ends. Homes under <workdir>/home are from runners before krz/gitbay#255, which shared them with untrusted builds; nothing reads them any more, and they can be deleted.

Images are provisioned, never pulled by a build. The runner passes --pull=never. Two reasons, and the second is the better one: the service runs with RestrictSUIDSGID=yes so podman cannot unpack a layer holding a setuid file, which is nearly every distribution image; and on an instance where anyone can push a ci.yml, image: would otherwise mean "fetch and run anything from the internet". An operator pulls or builds what is allowed and a build picks among those. A job naming an image the host does not have fails with a message saying so.

su - ci-runner -s /bin/sh -c "podman pull docker.io/library/alpine:3.20"
su - ci-runner -s /bin/sh -c "podman images"

Prepare a host before pointing an isolating runner at it:

ssh -p 2222 root@<host> 'sh -s' < deploy/runner-podman-setup.sh
make deploy-runner

The script installs podman, delegates a subuid/subgid range to ci-runner, checks that user namespaces are enabled rather than assuming, enables lingering, and verifies rootless podman actually runs as that user. It is idempotent.

It also installs nftables. make deploy-runner ships deploy/gitbay-runner-egress.nft to /etc/gitbay-runner/egress.nft with gitbay-runner-egress.service, which loads it and which the runner's unit requires; it checks the file with nft -c, reloads the unit, and runs deploy/runner-egress-check.sh as ci-runner before restarting the runner: 127.0.0.1:22 and the public 22 must answer, 2222 must not. The table limits the runner's user to 127.0.0.1:22, DNS on loopback, and 22, 80 and 443 on the host's public address; the Threat-Model page says why.

That table cannot tell a build from the runner, since both run as ci-runner. deploy/gitbay-runner-builds.nft, shipped as /etc/gitbay-runner/builds.nft, matches by cgroup instead: the runner starts each build under builds/trusted or builds/untrusted in its service cgroup, and the table closes the host's loopback to every build, limits an untrusted build to TCP 80 and 443 and DNS, and closes private ranges to both (the CI page has the table). nftables resolves a cgroup path to its id when the table loads, and the service cgroup is new on every start, so the drop-in's ExecStartPre creates the two cgroups, hands them to ci-runner and loads the table before the runner starts; a table that does not load stops the start. make deploy-runner checks the file's syntax before the restart and that the table is loaded after it. If /etc/resolv.conf names a nameserver in a private range, allow it in the file first or builds resolve nothing.

A restart of nftables.service flushes both tables; systemctl reload gitbay-runner-egress restores both.

The egress unit's stop and the rollback below use nft destroy, which needs nftables 1.0.8 or later; check nft --version on a new host.

Checking the tables on a running host, during a build (the flood test below waits a minute before its first probe, for this):

nft list table inet gitbay_builds          # counters on the reject rules
for p in $(pgrep -u ci-runner pasta); do cat /proc/$p/cgroup; done
# 0::/system.slice/gitbay-runner.service/builds/trusted/build-<id>

A pasta process anywhere else — the runner's own runner cgroup, a user slice — means the builds table does not see that build's traffic and only the first table applies. Run deploy/runner-auth-flood-test.sh on the scratch repository (below), once as a push and once with --untrusted. It fails if the runner was locked out or if the build log lacks the lines only a working table produces. The authoritative proof that pasta's sockets are in the build's cgroup is the untrusted run: 169.254.1.2:22 and github.com:22 refused, all twelve logins refused, and the counters on the untrusted chain's rejects rising. The first table lets ci-runner reach both of those addresses.

To take the builds table out, on the host:

sed -i '/^ExecStartPre=+.*builds/d' /etc/systemd/system/gitbay-runner.service.d/override.conf
rm /etc/gitbay-runner/builds.nft
systemctl daemon-reload
nft destroy table inet gitbay_builds

The runner needs no restart. It still places builds under builds/trusted and builds/untrusted, but with the file gone neither a runner start nor systemctl reload gitbay-runner-egress loads the table again, and builds keep the first table and --no-map-gw. A runner that takes -untrusted or polls over loopback refuses to start without build cgroups at all, since the table would match nothing. The next make deploy-runner installs the file and the lines again.

The drop-in sets NoNewPrivileges=no, without which rootless podman cannot call newuidmap and the runner refuses to start. That is a considered trade, explained in the file and in the Threat-Model; if you run with -isolation none, set it back to yes.

Restarting the runner is safe. On SIGTERM it stops claiming, finishes the build in flight, reports it, and exits; the drop-in's TimeoutStopSec=50min covers the longest build, and its KillMode=mixed is what makes the signal reach the runner alone — under systemd's default the build's container and the log session are signalled with it, and the runner drains a build that is already dead. So make deploy-runner waits for a running build rather than orphaning it, and a build's result is retried for half a minute if gitbayd is restarting at that moment. A second SIGTERM ends the runner at once, abandoning the build to the reaper. The suite checks all three: TestRunnerDrainsOnSIGTERM signals the process, TestRunnerDropInLetsTheDrainHappen reads the drop-in's KillMode and TimeoutStopSec, and TestRunnerDrainUnderSystemd runs the runner as a transient user unit under systemd-run and stops it under both kill modes. That last one needs a systemd user manager, so it skips in the container CI runs in; run it on a Linux host with go test ./e2e -run TestRunnerDrainUnderSystemd -v.

Validate podman mode on a scratch repository before pointing the runner at real ones. Every deploy that switched the whole instance to containers and failed took CI down with it. Instead: create a throwaway repository the runner account can read (public, or granted read — a private one is "not found" to the runner and the build stays pending), give it one job that names the CI image, attach the runner's key to it, and deploy the runner with -repos naming only that repository. The production unit, with its real hardening, then claims nothing else; other repositories' builds queue until -repos is removed again, which is a pause, not an outage.

# on a machine with an admin gitbay identity (root on the host has none),
# with the runner's public key copied from the host:
gitbay repo create cmc/runner-scratch   # then push a .gitbay/ci.yml naming the image
scp -P 2222 root@<host>:/var/lib/gitbay-runner/.ssh/id_ed25519.pub runner.pub
gitbay repo runner add cmc/runner-scratch < runner.pub
# on the host:
sed -i 's#^ExecStart=/usr/local/bin/gitbay-runner #&-repos cmc/runner-scratch #' /etc/systemd/system/gitbay-runner.service.d/override.conf
systemctl daemon-reload && systemctl restart gitbay-runner
# back on the admin machine:
gitbay build log cmc/runner-scratch 1   # green: remove -repos, redeploy, delete the scratch repository

Do not deploy an isolating runner to a host that has not been prepared. The runner is specified to refuse to start without a working podman rather than fall back to running builds unsandboxed — a fallback that silently drops isolation is worse than a stopped runner, because nothing surfaces it. On an unprepared host that refusal stops every build on the instance.

The service drop-in carries Delegate=yes for rootless cgroup management and ReadWritePaths for podman's store under /var/lib/gitbay-runner, which ProtectSystem=full would otherwise make read-only. Those paths are prefixed - so they are ignored when absent: the drop-in installs on unprepared hosts too, and a unit that refused to start would stop every build. gitbay-runner-prune.timer prunes unused images weekly, as the runner's user: rootless storage belongs to that user, and root's prune would not see it. An unpruned image store on a 40GB host is a slow outage.