docs/plans/2026-09-27-ci-trust-and-build-reporting.md
3870 lines · 147357 bytes
34 symbols in this file
CI trust and build reporting implementation planGlobal ConstraintsDecisionsOrder and dependenciesFile mapPart 1: disposable home for untrusted builds (branch `ci-untrusted-home`, #255)Task 1.1: the claim says whether a build is trustedTask 1.2: the runner's build home follows trustTask 1.3: wiki, and the MRPart 2: `ci/` statuses, trusted reuse, required contexts (branch `ci-status-trust`, #258)Task 2.1: `status set` refuses `ci/`Task 2.2: reuse only trusted results on the same imageTask 2.3: required contextsTask 2.4: wiki, and the MRPart 3: separate the runner's source address from its builds, and limit what builds reach on its host (branch `runner-source-address`, #260)Task 3.1: the claim carries the instance's public ssh destinationTask 3.2: a loopback runner's builds get no host loopbackTask 3.3: the limiter's behaviour, by registration modeTask 3.4: host egress rule for the runner's uidTask 3.5: egress policy in the wiki, and the MRPart 4: failed step and duration (branch `build-failure-report`, #266)Task 4.1: store the failed stepTask 4.2: `runner done --step --reason`Task 4.3: the runner names the failed stepTask 4.4: `build show`, `build log --step/--tail`Task 4.5: the build page folds by stepTask 4.6: e2e, wiki, and the MROpen questionsOperator runbook (cmc)R1. Scratch repository and scoped runner (once, before Part 1's runner deploy)R2. Part 1 (#255): validate, then discard the old homesR3. Part 3 (#260): pasta check, egress rule, measurement from a buildR4. Part 4 (#266): validate the failure reportSelf-review
CI trust and build reporting implementation plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Untrusted builds get a disposable home and never feed a
trusted build (#255, #258); ci/* statuses belong to the build
subsystem and merges can wait on named contexts (#258); a build can no
longer share the runner's source address, and reaches no port on the
runner's host but the forge's public 22, 80 and 443 (#260); a failed
build names its step, exit and duration on the CLI and the web (#266).
Architecture: The claim payload gains an explicit trusted flag and
the instance's public ssh destination (with the port when it is not
22). The runner picks the build home by trust (persistent per
repository for trusted builds, fresh and removed for untrusted ones),
keeps a loopback runner's builds off the host's loopback, and reports
the failed step on runner done. An nftables table on the runner host,
loaded by a oneshot unit the runner service requires, limits what the
runner's uid may reach on the host itself. The server refuses ci/
contexts in status set, restricts tree and commit reuse to trusted
builds on the same declared image, adds a required_contexts
repository setting that turns require_checks on and that MergeGates
treats as pending until reported, stores the failed step and reason on
the build, and cuts the log at its $ <step> lines for build log --step and the build page.
Tech Stack: Go, SQLite (hand-written SQL), html/template, rootless
podman with pasta, nftables, systemd.
Spec: the issues themselves: krz/gitbay#255, #258, #260, #266 (texts
in the session's issues.txt), plus the decisions recorded under
"Decisions" below.
Global Constraints
- Each MR on its own branch off
main. Commits are signed (the repo refuses unsigned), messages reference issues (Ref #N, andCloses #Non the commit that finishes one). No attribution to any assistant, model or AI anywhere: commits, MR bodies, comments. NoCo-Authored-Bytrailer. - MR:
gitbay mr create --source <branch> --target main --title "..."; merge withgitbay mr merge <n> --strategy ffonce CI is green, then delete the branch locally and on the remote. Behind main → rebase, force-push, merge again. - Locally:
go build ./...,go vet ./..., unit tests of touched packages, and at most the one e2e test being written (go test ./e2e -run TestName -count=1). CI on bay1 runs the full suite. - Registries that fail CI when a new thing lacks its row: top-level route
word in
internal/policy/names.go; new page template in the width map ofTestMainWidthClass(internal/web/web_test.go); newReadOnlycommand inreadArgsine2e/readonly_test.go; new control command needs apass()entry incmd/gitbay/main.go(coverage test); a command reading stdin needsReadsStdin: true. A new or changed commandSummaryneedsgo test ./cmd/gitbay -run TestSummariesAreCurrent -update. - New migrations: the highest today is 0059. Six plans are written in
parallel, so numbers are pre-assigned: plan 1 uses 0060–0064, plan 2
(this one) 0065–0068, plan 3 0069–0071, plan 4 0072–0074, plan 5
0075–0077, plan 6 0078–0079. Whoever lands second renumbers to the
next free number at execution time. Migrations come in
.up.sql/.down.sqlpairs. Hand-written SQL, no ORM. This plan uses one: 0065. - Secrets travel on stdin, never argv; never logged or echoed.
- Wiki pages live in
.gitbay/wiki/(Parity, API, Admin, Threat-Model, CI, Users, Performance, and theArchitecture/folder with its Known-Gaps table and controls matrix). Update the page in the same MR that changes the behaviour it describes, and close the matching Known-Gaps row. - Writing style: plain, direct, no hype; code comments match the surrounding density. Comments and docs state facts, never before/after narration.
- Deploy order for Parts 1, 3 and 4:
make deploy(gitbayd) beforemake deploy-runner. A new runner readstrustedandsshfrom the claim and sends--step/--reasontorunner done; an older server omits the first two (the runner then treats every build as untrusted: no secrets, disposable home) and refuses the flags with exit 2 (the build stays running until the reaper fails it). An older runner against a newer server works unchanged. The laptop runner is a brew bottle built from a release tag, so it always trails the server. - Runner changes are validated on a scratch repository before the bay1 runner touches real repositories (CLAUDE.md: every deploy that skipped this took CI down). The procedure is in the runbook at the end of this plan; Parts 1, 3 and 4 each have a validation section there.
cmd/gitbay-runnertests run on macOS and Linux; nothing here may need podman to pass (e2e/isolation_podman_test.gois the only podman test and skips without it).
Decisions
- #255. The claim carries
"trusted": true|falsewith noomitempty; a runner that finds the field absent treats the build as untrusted. Trusted homes move from<workdir>/home/<owner>/<name>to<workdir>/trusted-home/<owner>/<name>, so a home written before this change — which untrusted builds could write — is never read again, even before the operator deletes it. An untrusted build's home is<workdir>/build-<id>-home, created withos.Mkdir(so it is new and empty) and removed after the build with a helper that first makes every directory writable (the Go module cache leaves them 0555). No persistent cache for untrusted builds: a fork's build downloads its modules each time. The runner also drops secrets for an untrusted build, though the server never sends any. - #258.
status setrefuses any context starting withci/, case-folded, with exit 4, before resolving the repository. Tree reuse (SuccessBuildForTree) and the cancelled-build fallback (SuccessBuildFor) consider trusted builds only, and tree reuse also requires the sameimage:as the job declares. The same-commit dedupe inqueueJobslets an untrusted build stand only for another untrusted queue: a fork head that lands on a branch by fast-forward is built again as trusted.required_contextsis a list in the repository's settings JSON (no migration), set byrepo settings require-contexts <owner/name> [<context>...]. Setting a non-empty list also turnsrequire_checkson, in the same settings update; setting it empty clears the list and leavesrequire_checksas it was.require-checks offkeeps the list, which then does nothing until the gate is on again.repo settings showprintsrequire checksandrequired contextsside by side (today it prints neither), and on the web settings page the required-checks box is ticked after contexts are saved and its hint lists them. The gate itself only ever readsrequire_checks. A missing required context makes the combined checkpendingand appears as<context>=missingin the unmet sentence and inchecks_missing. - #258, default image. Tree reuse keys on the job's declared
image:. A job naming none is stored withimage = ''and matches other such builds whatever the runner defaulted to; bumping a runner's-imagedoes not invalidate them. This is documented on the CI page rather than fixed. Reuse is decided at queue time inqueueJobs, before any runner is chosen, and the default image belongs to whichever runner claims the build: bay1 and the laptop runner already have different defaults. Reporting the resolved image on claim orrunner donewould record it after the fact, but a queue-time comparison would still have nothing to compare against, so jobs without an image would never be reused at all. The remedy is on the repository's side (name the image inci.yml) or the operator's (build trigger, which never reuses, after bumping-image). - #260. Reading the code: the bay1 runner polls
git@127.0.0.1(deploy/gitbay-runner.override.conf:78);buildSSH(cmd/gitbay-runner/main.go:471-486) sends podman builds to169.254.1.2, and pasta's default gateway mapping lets a build reach the host's loopback, where its connections arrive from127.0.0.1. The limiter keys on the remote IP (internal/sshd/ratelimit.go:86, used atinternal/sshd/sshd.go:122). "Runner on the public address" does not separate anything: a build can connect to the public address too, and would then share the runner's source there instead. So the runner stays on loopback and its builds lose loopback: under podman, when the runner's remote is loopback, containers run with--network pasta:--no-map-gw, andGITBAY_SSHis the instance's public destination, which the server sends in the claim asssh. A build then reaches the host only as an internet client does. - #260,
GITBAY_SSHform.git@<site host>when[ssh] portis 22 (or unset, which config validation treats as 22), andgit@<site host>:<port>otherwise, built withnet.JoinHostPortso an IPv6 literal is bracketed. hutch and orgo buildssh://$GITBAY_SSH/<owner>/<name>.git, which is a valid URL in both forms, so nothing changes for them on 22 and they work unchanged on another port. A script that runs a command usesssh ssh://$GITBAY_SSH …, which OpenSSH accepts with or without the port; the Users page says so. The port is the daemon's[ssh] port, the one it listens on; an instance behind a port-mapping NAT is not modelled. - #260, host egress. Under rootless podman with pasta, a build's
connections are made by pasta on the host from sockets owned by the
runner's uid (
ci-runner), the same uid the runner's own ssh runs as. An nftables table (deploy/gitbay-runner-egress.nft, loaded bygitbay-runner-egress.service, whichgitbay-runner.servicerequires) matches output packets withmeta skuid "ci-runner"that leave throughlo— every packet to one of the host's own addresses, loopback or public, does — and allows only 127.0.0.1:22 (the runner's poll, clone and log stream), port 53 on loopback (the host resolver pasta forwards a build's DNS to), and 22, 80 and 443 on the public addresses. Everything else on the host is rejected: the admin sshd on 2222 on every address, and every service bound to loopback. Traffic to other hosts is not matched, so outbound internet stays open (a fork's merge request to a Go repository must fetch its modules). The rule applies to all builds, trusted and untrusted, and to-isolation nonebuilds too, since they run as the same uid. uid alone cannot tell a build from its runner, so 127.0.0.1:22 stays open to the uid;--no-map-gwis what keeps builds off loopback. The two are separate layers and both ship. The runner does not start without the rule (Requires=), followingrunner-podman-setup.sh's rule that a host that is not ready fails rather than runs builds unconfined;make deploy-runnerloads the rule and checks, asci-runner, that 127.0.0.1:22 answers and 2222 does not, before restarting the runner. The laptop runner (macOS, brew) is not covered. - Finding for #260. On gitbay.org the limiter's failure count is
unreachable by an unknown key:
authenticateadmits an unknown key as an anonymousregistersession wheneverregistration.modeis notclosed(internal/sshd/sshd.go:139-144), andfailis only called on the closed path (sshd.go:145). With registration closed,authenticatechecksallowbefore it looks at the key (sshd.go:123-129), so once an address hasssh_auth_ratefailures in the window every key from it is refused, the runner's included, andsuccess(sshd.go:149) is never reached to clear the count; below the limit a success clears it. A unit test ininternal/sshdrecords both modes (Task 3.3). No throttling test runs on production; the runbook measures, from inside a scratch build, the source address the forge sees and that 2222 and 127.0.0.1 are unreachable, and #260 closes when that is recorded on the CI wiki page. - #266. Duration is not stored:
Build.Elapsed()(internal/store/builds.go:420) already derives it fromstarted_atandfinished_at. Migration 0065 addsfailed_step(1-based, 0 for "no step": success, or a failure before the first step) andfailed_reason(one line, at most 200 bytes). The runner writesstep 3/3 failed: exit 1and reportsrunner done <id> failure --step 3 --reason 'exit 1'; the reason isexit <code>for a command that exited and the error text otherwise (build timed out after 45m0s,cancelled,git clone: exit 128). The log format is otherwise unchanged: sections are cut at the$ <step>line the runner already writes before each step (isolate.go:88,:186), matched against the build's ownstepsin order and only at a line start, so logs of builds that ran before this change fold too.build log --steptakes0(setup, before the first step), a step number, orfailed. An invalid--steponrunner doneis recorded as 0 rather than refused, so a runner/server mismatch never loses an outcome.
Order and dependencies
| # | Branch | Closes | Migration | Needs |
|---|---|---|---|---|
| 1 | ci-untrusted-home |
#255 | — | — |
| 2 | ci-status-trust |
#258 | — | — |
| 3 | runner-source-address |
Ref #260 (closed by the runbook result commit on the CI page) | — | Part 1 merged (both edit runRunnerNext's payload and stepEnv) |
| 4 | build-failure-report |
#266 | 0065 | Part 1 merged (both change run()); Part 3 merged (both change runStepsPodman) |
#255 goes first. Parts 1–4 land and deploy in order; each runner deploy follows the scratch validation in the runbook.
Other plans (all docs/plans/2026-09-27-*.md):
- Plan 4 (data-at-rest-and-backup, #273) encrypts
build_secrets; if it changesStore.BuildSecrets, Part 1's edit torunRunnerNext(internal/control/build.go:543-564) conflicts textually. Whoever lands second rebases; no behavioural dependency. - Plan 3 (server-hardening, #275) audits refused mutating commands; the
status setrefusal from Part 2 is one of them and needs nothing from this plan. - Plan 5 (web-ux, #261) covers documentation drift. Two items seen here
and left alone:
deploy/gitbay-runner.override.conf:27-30namescmc/ci-smoke, which no longer exists;cmd/gitbay-runner/main.go:513repeats its comment line. Part 3 adds a[Unit]section at the top of the same drop-in; if plan 5 edits its comment, whoever lands second rebases. - Plan 1 (credentials-and-sessions, #256) closes a removed key's
connections; the runner's
runner logsession is one such connection, and no code here depends on it. Plan 1's key expiry (#277) may add a check toauthenticate; Part 3's sshd test uses an unexpiring key and asserts only the limiter's behaviour, so it holds either way.
File map
| File | Part | Responsibility |
|---|---|---|
internal/control/build.go |
1, 3, 4 | claim payload trusted, ssh; runner done flags; build show fields; build log --step/--tail; queueJobs trust rule (2) |
internal/control/buildlog.go (create) |
4 | LogSection, SplitBuildLog, FailedSection, tailLines |
internal/control/status.go |
2 | ci/ refusal |
internal/control/mr.go, output.go |
2 | require-contexts, MergeGates, GatesOut.ChecksMissing |
internal/control/repo.go |
2 | repo settings show prints require checks and required contexts |
internal/store/builds.go |
2, 4 | trust and image on reuse; failed step columns |
internal/store/repos.go |
2 | RepoSettings.RequiredContexts |
internal/store/migrations/0065_build_failure.{up,down}.sql |
4 | columns |
cmd/gitbay-runner/main.go |
1, 3, 4 | job fields, buildHome, removeTree, stepEnv, loopbackRemote, buildSSH, buildNetwork, failure, exitReason |
cmd/gitbay-runner/isolate.go |
3, 4 | network flag; runSteps returns *failure |
cmd/gitbay-runner/report.go |
4 | doneArgs |
cmd/gitbay/main.go, summaries_gen.go |
2 | require-contexts pass-through |
internal/httpd/builds.go, settings.go |
2, 4 | step view; settings form mapping |
internal/web/templates/build.html, mr.html, settings.html |
2, 4 | steps, gates row, form |
internal/web/static/style.css |
4 | pre.buildlog wraps; step folds |
e2e/readonly_test.go, e2e/mrweb_test.go, e2e/status_test.go, e2e/settingsweb_test.go, e2e/ci_test.go |
2, 4 | contexts off ci/; refusal; settings; failed step |
internal/sshd/sshd_test.go |
3 | limiter behaviour by registration mode |
deploy/gitbay-runner-egress.nft (create), deploy/gitbay-runner-egress.service (create), deploy/runner-egress-check.sh (create) |
3 | host egress rule, its unit, the post-load check |
deploy/gitbay-runner.override.conf, deploy/runner-podman-setup.sh, Makefile |
3 | runner requires the rule; nftables installed; deploy-runner ships, loads and checks it |
.gitbay/wiki/… |
all | as listed per task |
Part 1: disposable home for untrusted builds (branch ci-untrusted-home, #255)
Task 1.1: the claim says whether a build is trusted
Files:
- Modify:
internal/control/build.go:554-564(the claim payload inrunRunnerNext) - Test:
internal/control/runnernext_test.go(append)
Interfaces:
-
Produces: the
runner next --jsonpayload gains"trusted": <bool>, always present. -
Step 1: Write the failing test
Append to internal/control/runnernext_test.go:
// The claim says whether a build is trusted in so many words. A runner
// must not infer it from secrets being absent: a trusted repository with
// no secrets looks the same (#255).
func TestRunnerNextSaysWhetherTrusted(t *testing.T) {
st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
for _, trusted := range []bool{true, false} {
if _, err := st.CreateBuild(repo.ID, "unit", baseSHA, "main", "[]", "", "", trusted); err != nil {
t.Fatal(err)
}
c, out := runnerCtx(st, uid, root)
c.JSON = true
if code := runRunnerNext(c, []string{"--untrusted"}); code != protocol.ExitOK {
t.Fatalf("runner next: exit %d, output:\n%s", code, out.String())
}
want := fmt.Sprintf(`"trusted":%v`, trusted)
if !strings.Contains(out.String(), want) {
t.Fatalf("claim of a trusted=%v build lacks %s:\n%s", trusted, want, out.String())
}
}
}
The first loop creates and claims the trusted build; the second creates the untrusted one, which is then the only pending build.
- Step 2: Run it and see it fail
Run: go test ./internal/control -run TestRunnerNextSaysWhetherTrusted -count=1
Expected: FAIL, claim of a trusted=true build lacks "trusted":true.
- Step 3: Implement
Replace the payload at internal/control/build.go:554-564 with:
d := struct {
ID int64 `json:"id"`
Repo string `json:"repo"`
Number int64 `json:"number"`
Job string `json:"job"`
SHA string `json:"sha"`
Ref string `json:"ref"`
Steps []string `json:"steps"`
Image string `json:"image,omitempty"`
// Trusted is always sent: a runner decides a build's home and
// secrets from it, and reads a missing field as untrusted (#255).
Trusted bool `json:"trusted"`
Secrets map[string]string `json:"secrets,omitempty"`
}{ID: b.ID, Repo: repo.Path(), Number: b.Number, Job: b.Job, SHA: b.SHA, Ref: b.Ref,
Steps: steps, Image: b.Image, Trusted: b.Trusted, Secrets: secrets}
- Step 4: Run it and see it pass
Run: go test ./internal/control -run 'TestRunnerNext' -count=1
Expected: PASS.
- Step 5: Commit
git add internal/control/build.go internal/control/runnernext_test.go
git commit -S -m "runner next: say whether the build is trusted
Ref #255"
Task 1.2: the runner's build home follows trust
Files:
- Modify:
cmd/gitbay-runner/main.go:12-30(imports),:32-42(job),:315-330(run),:437-463(comment andbuildHomeFor),:488-511(stepEnv) - Modify:
cmd/gitbay-runner/home_test.go(rewrite),cmd/gitbay-runner/env_test.go:47-60
Interfaces:
-
Consumes: the
trustedclaim field from Task 1.1. -
Produces:
job.Trusted bool(json:"trusted")func buildHome(workdir string, j job) (string, func(), error)— the home and a cleanup to defer; replacesbuildHomeFor.func removeTree(dir string) error
-
Step 1: Write the failing tests
Replace cmd/gitbay-runner/home_test.go with:
package main
import (
"os"
"path/filepath"
"strings"
"testing"
)
// A trusted build's home is its repository's, kept between builds so
// tool caches survive: the same repository gets the same directory back,
// another repository a different one (#184).
func TestTrustedHomeIsPerRepositoryAndKept(t *testing.T) {
work := t.TempDir()
a, done, err := buildHome(work, job{ID: 1, Repo: "alice/app", Trusted: true})
if err != nil {
t.Fatal(err)
}
done()
if _, err := os.Stat(a); err != nil {
t.Fatalf("trusted home removed after its build: %v", err)
}
b, done, err := buildHome(work, job{ID: 2, Repo: "bob/app", Trusted: true})
if err != nil {
t.Fatal(err)
}
done()
if a == b {
t.Fatalf("two repositories share a build home: %s", a)
}
again, done, _ := buildHome(work, job{ID: 3, Repo: "alice/app", Trusted: true})
done()
if again != a {
t.Fatalf("build home moved between builds: %s then %s", a, again)
}
for _, dir := range []string{a, b} {
rel, err := filepath.Rel(filepath.Join(work, "trusted-home"), dir)
if err != nil || rel == "." || strings.HasPrefix(rel, "..") {
t.Fatalf("build home %s is not under %s/trusted-home", dir, work)
}
st, err := os.Stat(dir)
if err != nil {
t.Fatal(err)
}
if st.Mode().Perm() != 0o700 {
t.Fatalf("build home mode %o, want 0700", st.Mode().Perm())
}
}
}
// An untrusted build gets a home of its own, outside the trusted root,
// removed when the build ends: nothing a fork's build writes reaches a
// later build of the repository (#255).
func TestUntrustedHomeIsDisposable(t *testing.T) {
work := t.TempDir()
trusted, done, err := buildHome(work, job{ID: 1, Repo: "alice/app", Trusted: true})
if err != nil {
t.Fatal(err)
}
done()
home, done, err := buildHome(work, job{ID: 2, Repo: "alice/app"})
if err != nil {
t.Fatal(err)
}
if home == trusted || strings.HasPrefix(home, filepath.Join(work, "trusted-home")) {
t.Fatalf("untrusted build got a trusted home: %s", home)
}
// What the Go module cache leaves behind: read-only directories.
cache := filepath.Join(home, "go", "pkg", "mod", "example.com", "m@v1")
if err := os.MkdirAll(cache, 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(cache, "go.mod"), []byte("module m\n"), 0o444); err != nil {
t.Fatal(err)
}
os.Chmod(cache, 0o555)
os.Chmod(filepath.Dir(cache), 0o555)
done()
if _, err := os.Stat(home); !os.IsNotExist(err) {
t.Fatalf("untrusted home left behind: %v", err)
}
}
// A repository path is server-validated, but a trusted home must still
// never resolve outside the runner's home root.
func TestBuildHomeRefusesTraversal(t *testing.T) {
if _, _, err := buildHome(t.TempDir(), job{Repo: "../../etc", Trusted: true}); err == nil {
t.Fatal("a traversing repository path produced a build home")
}
}
In cmd/gitbay-runner/env_test.go, replace TestStepEnvCarriesSecrets
(lines 47-60) with:
// Secrets reach a trusted build's steps and never an untrusted one's,
// whatever the claim carried: the trust flag decides, not whether any
// secrets arrived (#255).
func TestStepEnvCarriesSecrets(t *testing.T) {
secrets := map[string]string{"TOKEN": "s3cret"}
env := stepEnv(job{Trusted: true, Secrets: secrets}, "/tmp/buildhome", "git@x.test")
if !containsEnv(env, "TOKEN=s3cret") {
t.Error("a trusted build's secret did not reach the step")
}
for _, j := range []job{{}, {Secrets: secrets}} {
for _, e := range stepEnv(j, "/tmp/buildhome", "git@x.test") {
if strings.HasPrefix(e, "TOKEN=") {
t.Errorf("a secret reached an untrusted build: %q", e)
}
}
}
}
- Step 2: Run them and see them fail
Run: go test ./cmd/gitbay-runner -count=1
Expected: build failure, undefined: buildHome and unknown field Trusted in struct literal of type job.
- Step 3: Implement
In cmd/gitbay-runner/main.go:
Add "io/fs" to the imports (between "io" and "log").
Add the field to job (after Image):
Image string `json:"image"`
// Trusted is false for a merge request head from a fork, and when the
// server did not say: such a build gets no secrets and a home of its
// own (#255).
Trusted bool `json:"trusted"`
Secrets map[string]string `json:"secrets"`
Replace lines 315-330 of run (the build-home comment and the
buildHomeFor call) with:
home, doneHome, err := buildHome(r.workdir, j)
if err != nil {
log.Printf("build %d: build home: %v", j.ID, err)
return false
}
defer doneHome()
and change line 433 to env := stepEnv(j, home, r.buildSSH()).
Replace lines 437-463 (the stepEnv comment that sits above
buildHomeFor, and buildHomeFor) with buildHome and removeTree;
the stepEnv comment moves to stepEnv in the next block:
// buildHome is a build's HOME and what to do with it when the build ends.
//
// Not the workspace, which is removed after every build: the Go module
// cache and every other tool cache live under HOME. Not the runner's own
// home either, where its SSH key and credential dotfiles are.
//
// A trusted build gets its repository's home,
// <workdir>/trusted-home/<owner>/<name>, kept between builds so the
// caches survive. One per repository: shared across repositories, a step
// could poison a cache or plant a .gitconfig that another repository's
// build would honour (#184). The root is not <workdir>/home, where homes
// that untrusted builds could write were kept before #255, so none of
// those is read again.
//
// An untrusted build gets <workdir>/build-<id>-home, new and empty,
// removed when the build ends. The container mounts HOME read-write, so
// a home a fork's build could write is a cache a stranger controls
// (#255).
func buildHome(workdir string, j job) (string, func(), error) {
if !j.Trusted {
dir := filepath.Join(workdir, fmt.Sprintf("build-%d-home", j.ID))
if err := os.Mkdir(dir, 0o700); err != nil {
return "", nil, err
}
return dir, func() {
if err := removeTree(dir); err != nil {
log.Printf("build %d: removing its home: %v", j.ID, err)
}
}, nil
}
root := filepath.Join(workdir, "trusted-home")
dir := filepath.Join(root, filepath.FromSlash(j.Repo))
if rel, err := filepath.Rel(root, dir); err != nil || rel == "." || strings.HasPrefix(rel, "..") {
return "", nil, fmt.Errorf("repository path %q escapes the build home root", j.Repo)
}
if err := os.MkdirAll(dir, 0o700); err != nil {
return "", nil, err
}
return dir, func() {}, nil
}
// removeTree deletes dir and everything under it. os.RemoveAll alone
// fails on a directory without write permission, and the Go module cache
// makes every directory it fills read-only.
func removeTree(dir string) error {
filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
if err == nil && d.IsDir() {
os.Chmod(p, 0o700)
}
return nil
})
return os.RemoveAll(dir)
}
Replace stepEnv (lines 488-511) with the function and the comment
that belongs to it:
// stepEnv builds the environment a build step runs with. It is
// constructed, not inherited: os.Environ() would hand repository content
// the runner's entire environment, including anything an operator set on
// the service (#144).
//
// HOME is the build's home (buildHome), not the runner's own: tools read
// credentials out of dotfiles — .netrc, .npmrc, .gitconfig — and a build
// has no business finding the runner's.
//
// PATH is the one thing carried over: without it a step cannot find the
// tools the host was provisioned with.
func stepEnv(j job, home, sshDest string) []string {
path := os.Getenv("PATH")
if path == "" {
path = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
}
env := []string{
"PATH=" + path,
"HOME=" + home,
"LANG=C.UTF-8",
"CI=true",
"GITBAY_REPO=" + j.Repo,
"GITBAY_SHA=" + j.SHA,
"GITBAY_REF=" + j.Ref,
"GITBAY_JOB=" + j.Job,
"GITBAY_SSH=" + sshDest,
}
// The server sends secrets only for a trusted build. The claim's
// trust flag decides here as well, not whether any arrived (#255).
if j.Trusted {
for name, value := range j.Secrets {
env = append(env, name+"="+value)
}
}
return env
}
- Step 4: Run the tests and see them pass
Run: go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner -count=1
Expected: PASS. (TestStepEnvHomeIsNotTheWorkspace and the other
stepEnv tests pass unchanged.)
- Step 5: Commit
git add cmd/gitbay-runner/main.go cmd/gitbay-runner/home_test.go cmd/gitbay-runner/env_test.go
git commit -S -m "runner: disposable home for untrusted builds
A trusted build keeps its repository's home, now under
<workdir>/trusted-home; an untrusted build gets a new home removed with
the build, and no secrets whatever the claim carries.
Ref #255"
Task 1.3: wiki, and the MR
Files:
-
Modify:
.gitbay/wiki/Threat-Model.org:135-147,:172-180 -
Modify:
.gitbay/wiki/Admin.org:640-642 -
Modify:
.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org:30-33,:67 -
Modify:
.gitbay/wiki/Architecture/09-Controls.org:87 -
Modify:
.gitbay/wiki/Architecture/04-Trust-Boundaries.org(TB7 row) -
Modify:
.gitbay/wiki/Architecture/10-Known-Gaps.org:13(remove the #255 row) -
Step 1: Threat-Model
In "The CI runner", replace the sentences of the "What a build sees" bullet from "=HOME= is a build home" to the end of the bullet with:
secrets, and nothing the operator set on the service. =HOME= is a build
home under the runner's =-workdir=, not the runner's own home, so a
build cannot read the =.netrc=, =.npmrc= or =.gitconfig= where tools
keep credentials. A trusted build's home belongs to its repository
and persists, so caches survive; an untrusted build's home is new,
empty and removed when the build ends, so nothing a fork's build
writes is read by a later build (krz/gitbay#255). The claim names a
build's trust explicitly, and a runner that finds no trust flag treats
the build as untrusted.
Replace the paragraph after the bullets ("Under =-isolation none=, anything a step can do…") with:
Under =-isolation none=, anything a step can do as the runner's user a
pushed =ci.yml= can do. Under podman a step is confined to its
container, the bind-mounted workspace and its build home: a trusted
build's cache is read only by later trusted builds of the same
repository, and an untrusted build's home is discarded with it. Treat
the runner host as executing untrusted code all the same: keep it off
the daemon's host where the database lives, or scope it to repositories
whose writers you trust. gitbay.org does the latter — its runner builds
only the repositories the operator names.
- Step 2: Admin
Replace Admin.org:640-642 ("Each repository gets its own build home…")
with:
A trusted build's home is its repository's, under
=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
=HOME=: caches persist between trusted builds of one repository and are
never read by another's. An untrusted build — a merge request head from
a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
the build ends. Homes under =<workdir>/home= are from runners before
krz/gitbay#255, which shared them with untrusted builds; nothing reads
them any more, and they can be deleted.
- Step 3: Architecture pages
07-CI-and-Supply-Chain.org, lifecycle step 2, replace "The claim
returns id, repository, job, commit, ref, steps, image and — for
trusted builds only — the repository's secrets (=build.go=)." with
"The claim returns id, repository, job, commit, ref, steps, image, the
build's trust, and — for trusted builds only — the repository's secrets
(=build.go=)."
Replace the Build home row (line 67) with:
| Build home | trusted: =<workdir>/trusted-home/<owner>/<name>=, one per repository, persistent; untrusted: =<workdir>/build-<id>-home=, removed with the build (=main.go=) |
09-Controls.org line 87:
| Untrusted code runs isolated | in place | rootless podman, cgroup limits; untrusted builds get a disposable home (=cmd/gitbay-runner/main.go=) |
04-Trust-Boundaries.org TB7 row, last column:
| TB7 | Z5 → Z4 container | build steps, workspace, build home | rootless podman, operator-provisioned image, cgroup limits; a trusted build's home is its repository's, an untrusted build's is discarded with it; the network is open (#260) |
10-Known-Gaps.org: delete the #255 row.
- Step 4: Verify and commit
Run: go build ./... && go vet ./... && go test ./cmd/gitbay-runner ./internal/control -count=1
Expected: PASS.
git add .gitbay/wiki
git commit -S -m "wiki: trusted and untrusted build homes
Closes #255"
- Step 5: MR
git push -u origin ci-untrusted-home
gitbay mr create --source ci-untrusted-home --target main --title "runner: disposable home for untrusted builds"
Body (via --file - from a file written with the Write tool): what
changed, the deploy order (gitbayd before the runner), and a pointer to
runbook sections R1 and R2. After CI is green and the runbook's R2
validation passed on the scratch repository:
gitbay mr merge <n> --strategy ff, delete the branch both places.
Part 2: ci/ statuses, trusted reuse, required contexts (branch ci-status-trust, #258)
Task 2.1: status set refuses ci/
Files:
- Modify:
internal/control/status.go:15-27(registration),:68-70(after the usage check) - Create:
internal/control/status_test.go - Modify:
e2e/readonly_test.go:75,e2e/mrweb_test.go:275,e2e/status_test.go(after line 49) - Modify:
cmd/gitbay/summaries_gen.goonly if the summary changes (it does not here)
Interfaces:
-
Produces:
status set … --context ci/…exits 4 (protocol.ExitDenied) with a message containingreserved. -
Step 1: Write the failing test
Create internal/control/status_test.go:
package control
import (
"strings"
"testing"
"gitbay.org/gitbay/internal/protocol"
"gitbay.org/gitbay/internal/store"
)
// ci/<job> statuses are the build subsystem's. A writer who could post
// one could mark ci/test green on their own head before, or instead of,
// the build (#258).
func TestStatusSetRefusesReservedContext(t *testing.T) {
st, repo, uid := newQueueTestRepo(t)
for _, ctx := range []string{"ci/test", "CI/test", "ci/"} {
c, errOut := pruneCtx(st, t.TempDir(), store.User{ID: uid, Username: "alice"})
code := Dispatch(c, []string{"status", "set", repo.Path(), "abc1234", "--context", ctx, "--state", "success"})
if code != protocol.ExitDenied || !strings.Contains(errOut.String(), "reserved") {
t.Errorf("--context %s: exit %d, %s", ctx, code, errOut.String())
}
}
if has, err := st.RepoHasStatuses(repo.ID); err != nil || has {
t.Fatalf("a refused status was stored: %v %v", has, err)
}
}
- Step 2: Run it and see it fail
Run: go test ./internal/control -run TestStatusSetRefusesReservedContext -count=1
Expected: FAIL, exit 3 (no commit abc1234), since today the context is
accepted and the missing commit is what stops it.
- Step 3: Implement
In the registration, change the --context flag and the example:
{"--context", "<c>", "the check this status reports for; ci/ is reserved for the instance's builds", ""},
Examples: []string{
"status set krz/gitbay a1b2c3d --context ext/lint --state success",
},
After the usage check at status.go:68-70, before the --url check:
// ci/<job> statuses are the build subsystem's: queued, reused,
// skipped and finished by the server itself. A writer who could post
// one could mark ci/test green on their own head before, or instead
// of, the build (#258). Case-folded, so CI/test is no way around it.
if strings.HasPrefix(strings.ToLower(context), "ci/") {
return c.fail(protocol.ExitDenied, "the ci/ prefix is reserved for the instance's builds; report under another name, such as ext/%s",
strings.TrimPrefix(strings.ToLower(context), "ci/"))
}
- Step 4: Run it and see it pass
Run: go test ./internal/control -run 'TestStatusSet|TestHelp' -count=1
Expected: PASS.
- Step 5: e2e callers off
ci/, and the refusal over SSH
e2e/readonly_test.go:75: "--context", "ci/x" → "--context", "ext/x".
e2e/mrweb_test.go:275: "--context", "ci/test" → "--context", "ext/test".
In e2e/status_test.go, after the reader-denied check (line 49), add:
// ci/ is the instance's own: a writer is refused it (#258).
if _, errOut, code := inst.ssh(t, bobKey, "", "status", "set", "alice/svc", head, "--context", "ci/build", "--state", "success"); code != 4 || !strings.Contains(errOut, "reserved") {
t.Fatalf("writer posted a ci/ status: exit %d, %s", code, errOut)
}
Run: go test ./e2e -run TestCommitStatuses -count=1
Expected: PASS.
- Step 6: Commit
git add internal/control/status.go internal/control/status_test.go e2e/readonly_test.go e2e/mrweb_test.go e2e/status_test.go
git commit -S -m "status set: ci/ is reserved for the instance's builds
Ref #258"
Task 2.2: reuse only trusted results on the same image
Files:
- Modify:
internal/store/builds.go:453-477(SuccessBuildForTree,SuccessBuildFor) - Modify:
internal/control/build.go:856-864(queueJobs) - Modify:
internal/store/builds_test.go:241-249(TestSuccessBuildForTreecall sites) and append a test - Test:
internal/control/build_test.go(append)
Interfaces:
-
Produces:
func (s *Store) SuccessBuildForTree(repoID int64, tree, job, image string) (Build, bool, error)— trusted builds only, same image.SuccessBuildForkeeps its signature and considers trusted builds only. -
Step 1: Write the failing tests
In internal/store/builds_test.go, add "" as the fourth argument to
the three SuccessBuildForTree calls in TestSuccessBuildForTree, then
append:
// A result stands for another commit only when it came from a trusted
// build on the same image: a fork's green build, or one on an image the
// job has since left, proves nothing about the repository's own (#258).
func TestSuccessReuseNeedsTrustAndImage(t *testing.T) {
s := open(t)
if err := s.MigrateUp(); err != nil {
t.Fatal(err)
}
uid, _ := s.CreateUser("cmc", true)
repoID, _ := s.CreateRepo("user", uid, "app", "public")
for _, b := range []struct {
sha, image string
trusted bool
}{
{"aaa", "", false},
{"bbb", "localhost/old:1", true},
} {
if _, err := s.CreateBuild(repoID, "unit", b.sha, "main", `["true"]`, b.image, "tree1", b.trusted); err != nil {
t.Fatal(err)
}
claimed, ok, err := s.ClaimBuild([]int64{repoID}, true)
if err != nil || !ok {
t.Fatalf("claim: ok=%v err=%v", ok, err)
}
if err := s.FinishBuild(claimed.ID, "success"); err != nil {
t.Fatal(err)
}
}
if prev, ok, _ := s.SuccessBuildForTree(repoID, "tree1", "unit", ""); ok {
t.Fatalf("reused build %d: untrusted, or on another image", prev.Number)
}
if prev, ok, _ := s.SuccessBuildForTree(repoID, "tree1", "unit", "localhost/old:1"); !ok || prev.SHA != "bbb" {
t.Fatalf("trusted build on the same image not found: ok=%v prev=%+v", ok, prev)
}
if _, ok, _ := s.SuccessBuildFor(repoID, "aaa", "unit"); ok {
t.Error("an untrusted success stood for its commit")
}
}
Append to internal/control/build_test.go:
// A fork's green build of a commit does not stand for the repository's
// own: the same commit landing on a branch, or a commit with the same
// tree, is built again as trusted (#258).
func TestQueueBranchBuildsRebuildsWhatOnlyAForkBuilt(t *testing.T) {
st, repo, uid := newQueueTestRepo(t)
git := gitRunner(t)
root := t.TempDir()
src := filepath.Join(root, "src")
os.MkdirAll(filepath.Join(src, ".gitbay"), 0o755)
os.WriteFile(filepath.Join(src, ".gitbay", "ci.yml"), []byte(
"jobs:\n unit:\n steps:\n - echo hi\n"), 0o644)
git(root, "init", "-q", "-b", "main", "src")
git(src, "add", ".")
git(src, "commit", "-q", "-m", "base")
first := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
git(src, "commit", "-q", "--allow-empty", "-m", "same tree")
second := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
dir := RepoDir(root, repo.OwnerName, repo.Name)
os.MkdirAll(filepath.Dir(dir), 0o755)
git(root, "clone", "-q", "--bare", src, dir)
// A fork's merge request head, built untrusted and green.
QueueMRBuilds(st, root, "https://x.test", repo, uid, 1, first)
b, ok, err := st.ClaimBuild([]int64{repo.ID}, true)
if err != nil || !ok || b.Trusted {
t.Fatalf("claim: ok=%v trusted=%v err=%v", ok, b.Trusted, err)
}
if err := st.FinishBuild(b.ID, "success"); err != nil {
t.Fatal(err)
}
// The same commit lands on main, then a commit with the same tree.
QueueBranchBuilds(st, root, "https://x.test", repo, uid, "main", "", first, time.Now())
QueueBranchBuilds(st, root, "https://x.test", repo, uid, "main", first, second, time.Now())
pending, _ := st.ListBuilds(repo.ID, store.BuildFilter{Status: "pending"}, 10)
if len(pending) != 2 {
t.Fatalf("queued %d builds, want 2 (one per commit): %+v", len(pending), pending)
}
for _, p := range pending {
if !p.Trusted {
t.Errorf("build %d queued untrusted on a branch push", p.Number)
}
}
}
- Step 2: Run them and see them fail
Run: go test ./internal/store -run 'TestSuccessBuildForTree|TestSuccessReuse' -count=1
Expected: build failure (too many arguments in call to s.SuccessBuildForTree).
Run: go test ./internal/control -run TestQueueBranchBuildsRebuildsWhatOnlyAForkBuilt -count=1
Expected: FAIL, queued 0 builds, want 2.
- Step 3: Store
Replace internal/store/builds.go:453-477 with:
// SuccessBuildForTree finds a passed build of the job for a tree rather
// than a commit: a rebase that changes nothing in the tree has already
// been built (#177). Only a trusted build on the image the job names
// counts: a fork's result, or one from an image the job has left, does
// not stand for the repository's own (#258). A job naming no image
// matches builds that named none, whichever default the runner used;
// the CI wiki page says so. An empty tree never matches.
func (s *Store) SuccessBuildForTree(repoID int64, tree, job, image string) (Build, bool, error) {
if tree == "" {
return Build{}, false, nil
}
b, err := scanBuild(s.DB.QueryRow(buildSelect+
" WHERE repo_id = ? AND tree = ? AND job = ? AND image = ? AND trusted = 1 AND status = 'success'"+
" ORDER BY number DESC LIMIT 1", repoID, tree, job, image))
if errors.Is(err, sql.ErrNoRows) {
return Build{}, false, nil
}
return b, err == nil, err
}
// SuccessBuildFor finds a passed trusted build of the commit for the job,
// on any ref: what a cancelled duplicate can point back at.
func (s *Store) SuccessBuildFor(repoID int64, sha, job string) (Build, bool, error) {
b, err := scanBuild(s.DB.QueryRow(buildSelect+
" WHERE repo_id = ? AND sha = ? AND job = ? AND trusted = 1 AND status = 'success' ORDER BY number DESC LIMIT 1", repoID, sha, job))
if errors.Is(err, sql.ErrNoRows) {
return Build{}, false, nil
}
return b, err == nil, err
}
- Step 4:
queueJobs
Replace internal/control/build.go:856-859 with:
// A build of this commit that passed, or is queued or running,
// stands for it — unless this queue is trusted and that build was
// not: a fork's head that lands on a branch is built again as the
// repository's own (#258).
if b, ok := built[j.Name]; ok && (b.Trusted || !trusted) &&
(b.Status == "success" || b.Status == "pending" || b.Status == "running") {
continue
}
if prev, ok, _ := st.SuccessBuildForTree(repo.ID, tree, j.Name, j.Image); ok && prev.SHA != sha {
(the body of the if prev, ok block is unchanged.)
- Step 5: Run the tests and see them pass
Run: go vet ./... && go test ./internal/store ./internal/control ./internal/hookd ./internal/ci -count=1
Expected: PASS. TestPushShapes (hookd) has no row where a fork's
build lands on a branch, so its table is unchanged.
- Step 6: Commit
git add internal/store/builds.go internal/store/builds_test.go internal/control/build.go internal/control/build_test.go
git commit -S -m "ci: reuse only trusted results on the job's image
Ref #258"
Task 2.3: required contexts
Files:
- Modify:
internal/store/repos.go:25-37(RepoSettings) - Modify:
internal/control/output.go:59-60(GatesOut) - Modify:
internal/control/mr.go:45-49(registration), afterrunRequireChecks(:326-341),:1579-1603(MergeGateschecks block) - Modify:
internal/control/repo.go:710-717(repo settings show) - Modify:
internal/control/checksgate_test.go(helper refactor, two tests) - Test:
internal/control/mr_test.go(append) - Modify:
cmd/gitbay/main.go:580(add apass),cmd/gitbay/summaries_gen.go(regenerated) - Modify:
internal/httpd/settings.go:106-107,:228-229;internal/web/templates/settings.html:73-78(the require-checks form, its hint at line 75);internal/web/templates/mr.html:124 - Modify:
internal/httpd/mrpage_test.go(TestMRGatesRender),e2e/settingsweb_test.go:58
Interfaces:
-
Produces:
RepoSettings.RequiredContexts []string(json:"required_contexts,omitempty")GatesOut.ChecksMissing []string(json:"checks_missing,omitempty")- command
repo settings require-contexts <owner/name> [<context>...]: a non-empty list is stored and setsRequireChecks = truein the same update; an empty list clears the contexts and leavesRequireChecksalone. repo settings showhuman output gainsrequire checksandrequired contextsrows (JSON already carriesrequire_checksand gainsrequired_contexts).
-
Step 1: Write the failing tests
In internal/control/checksgate_test.go, replace gatesForHeadSeeded
(lines 20-73) with a general helper and a thin wrapper:
func gatesForHeadSeeded(t *testing.T, ciYML string, seed bool) GatesOut {
return gatesFor(t, ciYML, nil, func(st *store.Store, repoID, uid int64, targetSHA, _ string) {
if !seed {
return
}
if err := st.SetCommitStatus(repoID, targetSHA, "lint", "success", "", "", uid); err != nil {
t.Fatal(err)
}
})
}
// gatesFor builds a repository with require_checks on and set applied to
// its settings, a bare dir holding the given .gitbay/ci.yml (empty
// string for none), and one MR; seed records statuses before the gates
// are computed.
func gatesFor(t *testing.T, ciYML string, set func(*store.RepoSettings),
seed func(st *store.Store, repoID, uid int64, targetSHA, headSHA string)) GatesOut {
t.Helper()
st, repo, uid := newQueueTestRepo(t)
if _, err := st.UpdateRepoSettings(repo.ID, func(s *store.RepoSettings) {
s.RequireChecks = true
if set != nil {
set(s)
}
}); err != nil {
t.Fatal(err)
}
repo, err := st.RepoByID(repo.ID)
if err != nil {
t.Fatal(err)
}
git := gitRunner(t)
root := t.TempDir()
src := filepath.Join(root, "src")
os.MkdirAll(src, 0o755)
git(root, "init", "-q", "-b", "main", "src")
os.WriteFile(filepath.Join(src, "README"), []byte("x\n"), 0o644)
git(src, "add", ".")
git(src, "commit", "-q", "-m", "base")
targetSHA := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
git(src, "checkout", "-q", "-b", "feature")
if ciYML != "" {
os.MkdirAll(filepath.Join(src, ".gitbay"), 0o755)
os.WriteFile(filepath.Join(src, ".gitbay", "ci.yml"), []byte(ciYML), 0o644)
}
os.WriteFile(filepath.Join(src, "README"), []byte("y\n"), 0o644)
git(src, "add", ".")
git(src, "commit", "-q", "-m", "change")
headSHA := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
dir := RepoDir(root, repo.OwnerName, repo.Name)
os.MkdirAll(filepath.Dir(dir), 0o755)
git(root, "clone", "-q", "--bare", src, dir)
if _, err := st.CreateMR(repo.ID, uid, repo.ID, "feature", "main", "t", "", headSHA, "md", false); err != nil {
t.Fatal(err)
}
mr, err := st.MRByNumber(repo.ID, 1)
if err != nil {
t.Fatal(err)
}
if seed != nil {
seed(st, repo.ID, uid, targetSHA, headSHA)
}
g, err := MergeGates(st, repo, mr, dir, targetSHA, headSHA)
if err != nil {
t.Fatal(err)
}
return g
}
Append to the same file (add "slices" to its imports):
// A required context that has not reported holds the merge as pending,
// even when every status that did report is green (#258).
func TestRequiredContextMissingIsPending(t *testing.T) {
g := gatesFor(t, "", func(s *store.RepoSettings) { s.RequiredContexts = []string{"ext/deploy", "lint"} },
func(st *store.Store, repoID, uid int64, _, headSHA string) {
if err := st.SetCommitStatus(repoID, headSHA, "lint", "success", "", "", uid); err != nil {
t.Fatal(err)
}
})
if g.Checks != "pending" || !slices.Equal(g.ChecksMissing, []string{"ext/deploy"}) {
t.Fatalf("checks %q, missing %v", g.Checks, g.ChecksMissing)
}
if !checksUnmet(g) || !strings.Contains(strings.Join(g.Unmet, "\n"), "ext/deploy=missing") {
t.Fatalf("unmet: %v", g.Unmet)
}
}
// Every required context reported green: nothing is held.
func TestRequiredContextsReportedPass(t *testing.T) {
g := gatesFor(t, "", func(s *store.RepoSettings) { s.RequiredContexts = []string{"lint"} },
func(st *store.Store, repoID, uid int64, _, headSHA string) {
if err := st.SetCommitStatus(repoID, headSHA, "lint", "success", "", "", uid); err != nil {
t.Fatal(err)
}
})
if checksUnmet(g) || len(g.ChecksMissing) != 0 || g.Checks != "success" {
t.Fatalf("checks %q, missing %v, unmet %v", g.Checks, g.ChecksMissing, g.Unmet)
}
}
Append to internal/control/mr_test.go (add "slices" to its imports;
mrTestCtx is that file's helper):
// require-contexts stores a deduplicated list and turns require_checks
// on with it; an empty list clears the contexts and leaves
// require_checks as it was. A context with whitespace is refused (#258).
func TestRequireContextsSetsAndClears(t *testing.T) {
st, repo, uid := newQueueTestRepo(t)
alice := store.User{ID: uid, Username: "alice"}
dispatch := func(args ...string) int {
t.Helper()
c, _, _ := mrTestCtx(st, alice)
return Dispatch(c, args)
}
contexts := func(names ...string) int {
t.Helper()
return dispatch(append([]string{"repo", "settings", "require-contexts", repo.Path()}, names...)...)
}
settings := func() store.RepoSettings {
t.Helper()
got, err := st.RepoByID(repo.ID)
if err != nil {
t.Fatal(err)
}
return got.Settings
}
if settings().RequireChecks {
t.Fatal("require_checks on in a new repository")
}
if code := contexts("lint", "ext/deploy", "lint"); code != protocol.ExitOK {
t.Fatalf("set: exit %d", code)
}
if s := settings(); !slices.Equal(s.RequiredContexts, []string{"lint", "ext/deploy"}) || !s.RequireChecks {
t.Fatalf("stored %v, require_checks %v; want [lint ext/deploy], on", s.RequiredContexts, s.RequireChecks)
}
if code := contexts("bad context"); code != protocol.ExitUsage {
t.Fatalf("a context with a space: exit %d", code)
}
if code := contexts(); code != protocol.ExitOK {
t.Fatalf("clear: exit %d", code)
}
if s := settings(); len(s.RequiredContexts) != 0 || !s.RequireChecks {
t.Fatalf("after clearing: contexts %v, require_checks %v; want none, still on", s.RequiredContexts, s.RequireChecks)
}
if code := dispatch("repo", "settings", "require-checks", repo.Path(), "off"); code != protocol.ExitOK {
t.Fatalf("require-checks off: exit %d", code)
}
if code := contexts(); code != protocol.ExitOK {
t.Fatalf("clear again: exit %d", code)
}
if settings().RequireChecks {
t.Fatal("clearing the list turned require_checks on")
}
}
// settings show prints the checks gate beside the contexts it waits for,
// so a list that turned the gate on is visible where the gate is (#258).
func TestSettingsShowRequiredContexts(t *testing.T) {
st, repo, uid := newQueueTestRepo(t)
alice := store.User{ID: uid, Username: "alice"}
c, _, _ := mrTestCtx(st, alice)
if code := Dispatch(c, []string{"repo", "settings", "require-contexts", repo.Path(), "ext/deploy", "lint"}); code != protocol.ExitOK {
t.Fatalf("require-contexts: exit %d", code)
}
c, out, _ := mrTestCtx(st, alice)
if code := Dispatch(c, []string{"repo", "settings", "show", repo.Path()}); code != protocol.ExitOK {
t.Fatalf("settings show: exit %d", code)
}
got := strings.Join(strings.Fields(out.String()), " ")
for _, want := range []string{"require checks true", "required contexts ext/deploy, lint"} {
if !strings.Contains(got, want) {
t.Errorf("settings show lacks %q:\n%s", want, out.String())
}
}
}
- Step 2: Run them and see them fail
Run: go test ./internal/control -run 'TestRequire|TestRequiredContext|TestSettingsShowRequiredContexts' -count=1
Expected: build failure (s.RequiredContexts undefined).
- Step 3: Store and output types
internal/store/repos.go, in RepoSettings after RequireChecks:
RequireChecks bool `json:"require_checks,omitempty"`
// RequiredContexts are statuses require_checks waits for whether or
// not they have reported; one that has not is pending. Setting a
// non-empty list turns RequireChecks on (#258).
RequiredContexts []string `json:"required_contexts,omitempty"`
internal/control/output.go, in GatesOut after Checks:
Checks string `json:"checks,omitempty"` // combined status; "" when none reported
ChecksMissing []string `json:"checks_missing,omitempty"` // required contexts not reported
- Step 4: The command
Registration, after require-checks at mr.go:49:
register(Command{Path: []string{"repo", "settings", "require-contexts"},
Summary: "name the statuses the checks gate waits for, and turn the gate on",
Usage: "repo settings require-contexts <owner/name> [<context>...] (none clears the list)",
Examples: []string{"repo settings require-contexts krz/gitbay ci/build ci/test"},
Run: runRequireContexts})
After runRequireChecks:
// maxRequiredContexts bounds the list: a gate naming more checks than
// this is a configuration mistake.
const maxRequiredContexts = 20
func runRequireContexts(c *Ctx, args []string) int {
if len(args) < 1 {
return c.usage()
}
var contexts []string
for _, ctx := range args[1:] {
if ctx == "" || len(ctx) > 100 || strings.ContainsAny(ctx, " \t\r\n") {
return c.fail(protocol.ExitUsage, "a context is 1 to 100 characters with no whitespace: %q", ctx)
}
if !slices.Contains(contexts, ctx) {
contexts = append(contexts, ctx)
}
}
if len(contexts) > maxRequiredContexts {
return c.fail(protocol.ExitUsage, "at most %d required contexts", maxRequiredContexts)
}
repo, code := resolveRepo(c, args[0], policy.CanAdmin)
if code >= 0 {
return code
}
// Naming contexts asks for the gate, so it turns require_checks on in
// the same update. Clearing the list leaves the gate as it was.
s, err := c.Store.UpdateRepoSettings(repo.ID, func(s *store.RepoSettings) {
s.RequiredContexts = contexts
if len(contexts) > 0 {
s.RequireChecks = true
}
})
if err != nil {
return c.fail(protocol.ExitFailure, "%v", err)
}
return c.emit(s, func(w io.Writer) {
if len(contexts) > 0 {
fmt.Fprintf(w, "required contexts on %s: %s; require_checks on\n", repo.Path(), strings.Join(contexts, ", "))
return
}
gate := "off"
if s.RequireChecks {
gate = "on"
}
fmt.Fprintf(w, "required contexts cleared on %s; require_checks %s\n", repo.Path(), gate)
})
}
- Step 5:
MergeGates
Replace mr.go:1579-1603 (the checks block, from the comment through
the end of if set.RequireChecks { … }) with:
// Checks: with require_checks, every status the head carries must be
// green, a head something was going to report on must carry some, and
// every required context must have reported: one that has not is
// pending whatever the others say (#258). Setting contexts turns
// require_checks on; turned off again, the list is kept and unread.
statuses, err := st.ListCommitStatuses(repo.ID, headSHA)
if err != nil {
return g, err
}
g.Checks = store.CombinedStatus(statuses)
if set.RequireChecks {
reported := map[string]bool{}
for _, s := range statuses {
reported[s.Context] = true
}
for _, want := range set.RequiredContexts {
if !reported[want] {
g.ChecksMissing = append(g.ChecksMissing, want)
}
}
if len(g.ChecksMissing) > 0 && (g.Checks == "" || g.Checks == "success") {
g.Checks = "pending"
}
switch g.Checks {
case "success":
case "":
if checksExpected(st, repo.ID, dir, headSHA) {
g.Unmet = append(g.Unmet, fmt.Sprintf("%s requires green checks and none were reported on %.10s", repo.Path(), headSHA))
}
default:
var bad []string
for _, st := range statuses {
if st.State != "success" {
bad = append(bad, st.Context+"="+st.State)
}
}
for _, m := range g.ChecksMissing {
bad = append(bad, m+"=missing")
}
g.Unmet = append(g.Unmet, fmt.Sprintf("%s requires green checks; %.10s has %s", repo.Path(), headSHA, strings.Join(bad, ", ")))
}
}
repo.go:710-717, the repo settings show fields: today they print
neither the checks gate nor anything about it. Add two rows after
"require mr" (line 713), so the gate and the list that turned it on
read together:
"require mr", strconv.FormatBool(repo.Settings.RequireMR),
"require checks", strconv.FormatBool(repo.Settings.RequireChecks),
"required contexts", strings.Join(repo.Settings.RequiredContexts, ", "),
(fields skips a row whose value is empty, so a repository with no
contexts shows no required contexts row.)
- Step 6: Run the control tests
Run: go vet ./... && go test ./internal/control ./internal/store -count=1
Expected: PASS.
- Step 7: CLI, web, summaries
cmd/gitbay/main.go, after the require-checks line (580):
pass("require-contexts", passOpts{server: []string{"repo", "settings", "require-contexts"}, needsRepo: true}),
Run: go test ./cmd/gitbay -run TestSummariesAreCurrent -update -count=1 && go test ./cmd/gitbay -count=1
Expected: PASS; summaries_gen.go gains the repo settings require-contexts line.
internal/httpd/settings.go, in settingsSubmit after the
require-checks case:
case "require-contexts":
argv = append([]string{"repo", "settings", "require-contexts", repo}, strings.Fields(v("contexts"))...)
and in fieldLabel after require-checks:
case "require-contexts":
return "required contexts"
internal/web/templates/settings.html: the require-checks hint (line
75) names the contexts the gate waits for, so a box ticked by saving
contexts says why:
<div><label for="require-checks">Required checks</label><p class="hint">Requires CI to succeed.{{with .Repo.Settings.RequiredContexts}} Also waits for {{range $i, $c := .}}{{if $i}}, {{end}}<code>{{$c}}</code>{{end}} until they report{{if not $.Repo.Settings.RequireChecks}}, once this is on{{end}}.{{end}}</p></div>
and after the require-checks form (line 78):
<form method="post" action="{{$base}}" class="setform">
<input type="hidden" name="field" value="require-contexts">
<div><label for="contexts">Required contexts</label><p class="hint">Statuses the checks gate waits for until they report, separated by spaces. Saving any turns required checks on; saving none leaves it as it is.</p></div>
<div><input type="text" id="contexts" name="contexts" value="{{range $i, $c := .Repo.Settings.RequiredContexts}}{{if $i}} {{end}}{{$c}}{{end}}" autocomplete="off"></div>
<div><button type="submit" class="btn">Save</button></div>
</form>
internal/web/templates/mr.html, after the OwnersOutstanding line
(124):
{{range .ChecksMissing}}<p class="row none">waiting on <code>{{.}}</code>, not yet reported</p>{{end}}
In internal/httpd/mrpage_test.go TestMRGatesRender, add after the
first for loop:
if out := render(&control.GatesOut{Checks: "pending", ChecksMissing: []string{"ext/deploy"}}); !strings.Contains(out, "waiting on <code>ext/deploy</code>") {
t.Errorf("missing required context not rendered:\n%s", out)
}
In e2e/settingsweb_test.go, replace line 58,
post(url.Values{"field": {"require-checks"}, "require-checks": {"on"}}),
with a contexts post, so the "require_checks":true the test already
expects from settings show --json (line 69) now comes from the
contexts turning the gate on:
// Saving required contexts turns the checks gate on, and the page
// shows it ticked with the contexts in its hint (#258).
if body := post(url.Values{"field": {"require-contexts"}, "contexts": {"ext/deploy lint"}}); !strings.Contains(body, `id="require-checks" name="require-checks" value="on" checked`) ||
!strings.Contains(body, `Also waits for <code>ext/deploy</code>, <code>lint</code> until they report.`) {
t.Fatalf("required contexts did not show as turning required checks on:\n%s", body)
}
and add `"required_contexts":["ext/deploy","lint"]` to the
settings show --json want list at line 69.
Run: go test ./internal/httpd ./internal/web -count=1 && go test ./e2e -run TestRepoSettingsWeb -count=1
Expected: PASS.
- Step 8: Commit
git add internal/store/repos.go internal/control cmd/gitbay internal/httpd internal/web e2e/settingsweb_test.go
git commit -S -m "repo settings: required contexts turn the checks gate on, pending until reported
Ref #258"
Task 2.4: wiki, and the MR
Files:
-
Modify:
.gitbay/wiki/API.org:100-122,.gitbay/wiki/CI.org:5-9,.gitbay/wiki/Users.org:542-545,.gitbay/wiki/Parity.org(settings table near line 194) -
Modify:
.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org:57,09-Controls.org:39,:92,10-Known-Gaps.org(remove the #258 row) -
Step 1: API.org
Change the two example lines to --context ext/build, and after the
paragraph ending "…Each report also emits a =status= event to
webhooks." add:
Contexts starting with =ci/= are the instance's own: its builds queue,
reuse, skip and finish them, and =status set= refuses them with exit 4,
so a writer cannot mark =ci/test= green on a head the build has not
passed. Report under another prefix, such as =ext/=.
=repo settings require-contexts <repo> ext/deploy ci/test= names
statuses the checks gate waits for whether or not they have reported:
one that has not is =pending=, and =mr show= lists it as
=ext/deploy=missing=. Naming any context turns =require-checks= on;
with no contexts the command clears the list and leaves
=require-checks= as it was. =require-checks off= keeps the list, which
waits for nothing until the gate is on again. =repo settings show=
prints both.
- Step 2: CI.org
In the Dedupe bullet, after "…naming the build it came from (#177).", add: "Only a trusted build counts, and for tree reuse only one on the image the job names: a fork's green build does not stand for the repository's own, so its commit is built again when it lands on a branch (#258). A job that names no =image:= is compared as naming none: its reuse does not notice the runner's default image changing, because reuse is decided when the push is queued, before any runner claims the build, and runners can differ in their default. Name the image in =ci.yml= to tie reuse to it; after an operator changes a runner's =-image=, =build trigger= builds a job afresh, since a triggered build is never reused."
- Step 3: Users.org
After "…a =ci/= commit status, which =repo settings require-checks= can gate merges on." add: "=repo settings require-contexts= names statuses the gate waits for until they report, and turns the gate on."
- Step 4: Parity.org
After the require codeowners row:
| require contexts | yes | yes | no |
- Step 5: Architecture
07-CI-and-Supply-Chain.org:57:
| =status set= | write on the repository; =ci/*= contexts refused (=status.go=) |
and in lifecycle step 1 replace "or =success= copied from an earlier build of the same tree (#177)." with "or =success= copied from an earlier trusted build of the same tree on the same image (#177, #258)."
09-Controls.org:39:
| Merge gates | in place | =MergeGates=; =ci/*= statuses written only by the build subsystem; required contexts |
09-Controls.org:92:
| Build results reused only across equal trust | in place | =SuccessBuildForTree=, =SuccessBuildFor= (=internal/store/builds.go=) |
10-Known-Gaps.org: delete the #258 row.
- Step 6: Commit and MR
git add .gitbay/wiki
git commit -S -m "wiki: reserved ci/ statuses, trusted reuse, required contexts
Closes #258"
git push -u origin ci-status-trust
gitbay mr create --source ci-status-trust --target main --title "ci: reserve ci/ statuses; reuse only trusted results; required contexts"
No runner change: this part deploys with make deploy alone. Merge
with --strategy ff once CI is green; delete the branch both places.
Part 3: separate the runner's source address from its builds, and limit what builds reach on its host (branch runner-source-address, #260)
Task 3.1: the claim carries the instance's public ssh destination
Files:
- Modify:
internal/control/build.go(imports; the payload from Task 1.1; a helper besiderunRunnerNext) - Test:
internal/control/runnernext_test.go(append)
Interfaces:
-
Produces:
runner next --jsonpayload"ssh": "git@<site host>", or"git@<site host>:<port>"when[ssh] portis neither 0 nor 22; omitted whensite_urlis empty. -
Step 1: Write the failing test
// A build on the daemon's own host is given the public destination, not
// the loopback address its runner polls. The port rides along only when
// it is not 22, so ssh://$GITBAY_SSH/<owner>/<name>.git is a valid URL
// either way (#260).
func TestRunnerNextCarriesPublicSSH(t *testing.T) {
st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
for _, tc := range []struct {
port int
want string
}{
{0, `"ssh":"git@x.test"`},
{22, `"ssh":"git@x.test"`},
{2022, `"ssh":"git@x.test:2022"`},
} {
if _, err := st.CreateBuild(repo.ID, "unit", baseSHA, "main", "[]", "", "", true); err != nil {
t.Fatal(err)
}
c, out := runnerCtx(st, uid, root) // site_url https://x.test
c.Cfg.SSH.Port = tc.port
c.JSON = true
if code := runRunnerNext(c, nil); code != protocol.ExitOK {
t.Fatalf("port %d: runner next: exit %d, output:\n%s", tc.port, code, out.String())
}
if !strings.Contains(out.String(), tc.want) {
t.Fatalf("port %d: claim lacks %s:\n%s", tc.port, tc.want, out.String())
}
}
}
Each pass queues one build and claims it, as in Task 1.1's test.
runnerCtx builds its config.Config by hand, so SSH.Port starts at
0; config validation refuses 0 on a real instance (config.go:367),
and 0 is read as 22.
- Step 2: Run it and see it fail
Run: go test ./internal/control -run TestRunnerNextCarriesPublicSSH -count=1
Expected: FAIL, port 0: claim lacks "ssh":"git@x.test".
- Step 3: Implement
Add "net" to build.go's imports (strconv is already there), and
after maxOrphanSkip:
// publicSSH is the instance's ssh destination as anyone outside reaches
// it. A runner on the daemon's own host polls over loopback and hands
// its builds this instead, so no build connects from the runner's source
// address (#260). The port is added only when it is not 22: hutch and
// orgo build ssh://$GITBAY_SSH/... URLs, valid in both forms. Empty when
// site_url is not set.
func publicSSH(c *Ctx) string {
host := c.Cfg.SiteHost()
if host == "" {
return ""
}
if p := c.Cfg.SSH.Port; p != 0 && p != 22 {
return "git@" + net.JoinHostPort(host, strconv.Itoa(p))
}
return "git@" + host
}
In the payload struct add, after Trusted:
// SSH is the instance's public destination for the build's
// GITBAY_SSH when its runner polls over loopback (#260).
SSH string `json:"ssh,omitempty"`
and SSH: publicSSH(c), in the literal.
- Step 4: Run it and see it pass
Run: go test ./internal/control -run TestRunnerNext -count=1
Expected: PASS.
- Step 5: Commit
git add internal/control/build.go internal/control/runnernext_test.go
git commit -S -m "runner next: send the instance's public ssh destination
Ref #260"
Task 3.2: a loopback runner's builds get no host loopback
Files:
- Modify:
cmd/gitbay-runner/main.go:32-42(job),:433(stepEnvcall),:465-486(buildSSH) - Modify:
cmd/gitbay-runner/isolate.go:150-154(podman run args) - Modify:
cmd/gitbay-runner/env_test.go:190-212
Interfaces:
-
Consumes: the
sshclaim field from Task 3.1. -
Produces:
job.SSH string(json:"ssh")func (r *runner) loopbackRemote() boolfunc (r *runner) buildSSH(public string) string(wasbuildSSH())func (r *runner) buildNetwork() []string
-
Step 1: Write the failing tests
Replace TestStepEnvCarriesInstanceAddress (env_test.go:190-212) with:
// A build that talks back to the instance needs an address that works
// from where it runs. A runner polling over loopback keeps its podman
// builds off the host's loopback, so they get the instance's public
// destination from the claim, port included when it is not 22; any other
// remote is used as it is (#260).
func TestStepEnvCarriesInstanceAddress(t *testing.T) {
env := stepEnv(job{}, "/tmp/buildhome", "git@gitbay.org")
if !containsEnv(env, "GITBAY_SSH=git@gitbay.org") {
t.Errorf("GITBAY_SSH missing: %q", env)
}
for _, tc := range []struct{ remote, isolation, public, want string }{
{"git@127.0.0.1", isolationNone, "git@gitbay.org", "git@127.0.0.1"},
{"git@127.0.0.1", isolationPodman, "git@gitbay.org", "git@gitbay.org"},
{"git@127.0.0.1", isolationPodman, "git@gitbay.test:2022", "git@gitbay.test:2022"},
{"git@localhost", isolationPodman, "git@gitbay.org", "git@gitbay.org"},
{"git@127.0.0.1", isolationPodman, "", "git@127.0.0.1"},
{"git@gitbay.org", isolationPodman, "git@other.test", "git@gitbay.org"},
{"gitbay.org", isolationPodman, "git@gitbay.org", "gitbay.org"},
} {
r := &runner{remote: tc.remote, isolation: tc.isolation}
if got := r.buildSSH(tc.public); got != tc.want {
t.Errorf("remote %s under %s, public %q: got %s want %s", tc.remote, tc.isolation, tc.public, got, tc.want)
}
}
}
// Only a runner that polls over loopback shares an address a build could
// connect from, so only its builds lose the host-loopback mapping (#260).
func TestBuildNetworkKeepsLoopbackRunnersBuildsOff(t *testing.T) {
for _, tc := range []struct {
remote string
want []string
}{
{"git@127.0.0.1", []string{"--network", "pasta:--no-map-gw"}},
{"localhost", []string{"--network", "pasta:--no-map-gw"}},
{"git@::1", []string{"--network", "pasta:--no-map-gw"}},
{"git@gitbay.org", nil},
} {
r := &runner{remote: tc.remote, isolation: isolationPodman}
if got := r.buildNetwork(); strings.Join(got, " ") != strings.Join(tc.want, " ") {
t.Errorf("remote %s: %q, want %q", tc.remote, got, tc.want)
}
}
}
- Step 2: Run them and see them fail
Run: go test ./cmd/gitbay-runner -count=1
Expected: build failure (too many arguments in call to r.buildSSH, r.buildNetwork undefined).
- Step 3: Implement
job, after Trusted:
// SSH is the instance's public ssh destination, for a build whose
// runner polls over loopback (#260).
SSH string `json:"ssh"`
Replace buildSSH (main.go:465-486) with:
// loopbackRemote reports whether the runner polls the daemon on its own
// host over loopback.
func (r *runner) loopbackRemote() bool {
_, host, ok := strings.Cut(r.remote, "@")
if !ok {
host = r.remote
}
return host == "127.0.0.1" || host == "localhost" || host == "::1"
}
// buildSSH is the instance's ssh destination as a build reaches it. A
// runner polling over loopback keeps its podman builds off the host's
// loopback (buildNetwork), so they get the instance's public destination
// from the claim. Any other remote is a real host elsewhere and works as
// it is, and under -isolation none a build runs on the host itself.
func (r *runner) buildSSH(public string) string {
if r.isolation == isolationPodman && r.loopbackRemote() && public != "" {
return public
}
return r.remote
}
// buildNetwork is the podman network option for a build. pasta maps the
// container's gateway address to the host's loopback, and a build's
// connection through it arrives from 127.0.0.1 — the address a runner on
// the daemon's host polls from. The SSH auth limiter counts failures per
// source address, so a build sharing the runner's could throttle its
// polling (#260). --no-map-gw removes the mapping: the build reaches the
// host only at its public address, as any client on the internet does,
// and keeps its outbound access. The host's nftables table
// (deploy/gitbay-runner-egress.nft) then limits it to 22, 80 and 443
// there; it cannot tell a build from the runner by uid, so it leaves
// 127.0.0.1:22 open, and this flag is what keeps builds off it.
func (r *runner) buildNetwork() []string {
if !r.loopbackRemote() {
return nil
}
return []string{"--network", "pasta:--no-map-gw"}
}
In run, the stepEnv call becomes env := stepEnv(j, home, r.buildSSH(j.SSH)).
In isolate.go, after the --env-file append (line 153):
args = append(args, r.buildNetwork()...)
- Step 4: Run the tests and see them pass
Run: go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner -count=1
Expected: PASS.
- Step 5: Commit
git add cmd/gitbay-runner
git commit -S -m "runner: builds off the host's loopback when the runner polls over it
Ref #260"
Task 3.3: the limiter's behaviour, by registration mode
This test records what the code does today (see "Finding for #260" under Decisions), so it passes on its first run. It is the evidence the issue's on-production throttling test was meant to give, without touching production. If it fails, the limiter differs from what Decisions and the wiki say: stop and correct that text, not the test.
Files:
- Test:
internal/sshd/sshd_test.go(append; every import it needs is already in the file)
Interfaces:
-
Consumes:
(*Server).authenticate,rateLimiter.seen(package-internal). -
Step 1: Write the test
Append to internal/sshd/sshd_test.go:
// authMeta is the connection metadata authenticate reads: only the
// remote address.
type authMeta struct {
ssh.ConnMetadata
addr net.Addr
}
func (m authMeta) RemoteAddr() net.Addr { return m.addr }
func authKey(t *testing.T) ssh.PublicKey {
t.Helper()
pub, _, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatal(err)
}
k, err := ssh.NewPublicKey(pub)
if err != nil {
t.Fatal(err)
}
return k
}
// authServer is a Server holding what authenticate uses: a store with a
// runner account's key, the registration mode, and a limiter of three
// failures a minute.
func authServer(t *testing.T, mode string) (*Server, ssh.PublicKey) {
t.Helper()
st, err := store.Open(filepath.Join(t.TempDir(), "gitbay.db"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { st.Close() })
if err := st.MigrateUp(); err != nil {
t.Fatal(err)
}
uid, err := st.CreateUser("ci", false)
if err != nil {
t.Fatal(err)
}
runner := authKey(t)
if err := st.AddSSHKey(uid, ssh.FingerprintSHA256(runner), runner.Type(), runner.Marshal(), "runner", ""); err != nil {
t.Fatal(err)
}
cfg := config.Default()
cfg.Registration.Mode = mode
return &Server{cfg: cfg, st: st, authLimiter: newRateLimiter(3, time.Minute)}, runner
}
var (
fromLoopback = authMeta{addr: &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 40000}}
fromPublic = authMeta{addr: &net.TCPAddr{IP: net.IPv4(203, 0, 113, 7), Port: 40000}}
)
// With registration closed an unknown key counts against its address.
// Below the limit a known key's success clears the count. At the limit
// authenticate refuses before it looks at the key, so the runner's own
// key from that address is refused too and its success never runs to
// clear anything, until the window passes. Another address is not
// affected. This is why a build must not share the runner's source
// address (#260).
func TestAuthLockoutHoldsAgainstTheRunnersKey(t *testing.T) {
s, runner := authServer(t, "closed")
stranger := authKey(t)
failTimes := func(n int) {
t.Helper()
for i := 0; i < n; i++ {
if _, err := s.authenticate(fromLoopback, stranger); err == nil {
t.Fatal("unknown key admitted with registration closed")
}
}
}
failTimes(2)
if _, err := s.authenticate(fromLoopback, runner); err != nil {
t.Fatalf("runner below the limit: %v", err)
}
failTimes(2)
if _, err := s.authenticate(fromLoopback, runner); err != nil {
t.Fatalf("runner after its success cleared the count: %v", err)
}
failTimes(3)
for i := 0; i < 2; i++ {
if _, err := s.authenticate(fromLoopback, runner); err == nil || !strings.Contains(err.Error(), "too many") {
t.Fatalf("attempt %d from a locked-out address: %v, want refused", i+1, err)
}
}
if _, err := s.authenticate(fromPublic, runner); err != nil {
t.Fatalf("another address was locked out too: %v", err)
}
s.authLimiter.seen["127.0.0.1"].start = time.Now().Add(-2 * time.Minute)
if _, err := s.authenticate(fromLoopback, runner); err != nil {
t.Fatalf("runner after the window passed: %v", err)
}
}
// With registration open or by invite, an unknown key is admitted to run
// register and never counts, so no number of unknown-key attempts locks
// the runner's address out. gitbay.org runs open registration (#260).
func TestAuthUnknownKeyCountsOnlyWhenClosed(t *testing.T) {
for _, mode := range []string{"open", "invite"} {
s, runner := authServer(t, mode)
for i := 0; i < 10; i++ {
p, err := s.authenticate(fromLoopback, authKey(t))
if err != nil || p.Extensions["anon-key"] == "" {
t.Fatalf("%s: unknown key %d: %v %+v", mode, i+1, err, p)
}
}
if _, err := s.authenticate(fromLoopback, runner); err != nil {
t.Fatalf("%s: runner refused after unknown keys: %v", mode, err)
}
}
}
- Step 2: Run it
Run: go vet ./internal/sshd && go test ./internal/sshd -run 'TestAuth' -count=1
Expected: PASS.
- Step 3: Commit
git add internal/sshd/sshd_test.go
git commit -S -m "sshd: test the auth limiter's lockout by registration mode
Ref #260"
Task 3.4: host egress rule for the runner's uid
No Go code. The check script is the test: make deploy-runner runs it
after loading the rule and before restarting the runner, and runbook R3
runs the whole path against the scratch repository before merge.
Files:
- Create:
deploy/gitbay-runner-egress.nft,deploy/gitbay-runner-egress.service,deploy/runner-egress-check.sh - Modify:
deploy/gitbay-runner.override.conf(a[Unit]section before[Service]at line 25) - Modify:
deploy/runner-podman-setup.sh(afterpodman --version, line 29) - Modify:
Makefile:69-85(deploy-runner)
Interfaces:
-
Produces: nftables table
inet gitbay_runner; unitgitbay-runner-egress.service, required bygitbay-runner.service; rule file at/etc/gitbay-runner/egress.nfton the runner host. -
Step 1: The rule
Create deploy/gitbay-runner-egress.nft:
#!/usr/sbin/nft -f
# Host egress for CI builds (#260). Loaded by gitbay-runner-egress.service,
# which gitbay-runner.service requires, so the runner does not start
# without it. `make deploy-runner` installs it as
# /etc/gitbay-runner/egress.nft.
#
# Under rootless podman with pasta, a build's connections are made by
# pasta on the host, from sockets owned by the runner's user, ci-runner.
# nftables sees them exactly as it sees the runner's own ssh, so this
# table cannot tell a build from its runner. It limits what that user
# reaches on this host, and the runner needs little: 127.0.0.1:22, to
# poll, clone and stream logs.
#
# Every packet to one of the host's own addresses, loopback or public,
# leaves through lo, so the output hook sees host-bound traffic as
# oifname "lo". Traffic to other hosts is not matched: builds keep
# outbound internet access, trusted or not (go mod download needs it).
#
# What ci-runner may reach on this host:
# 127.0.0.1:22 the forge over loopback, for the runner. Builds do
# not reach loopback at all: the runner starts them
# with pasta's gateway mapping off (--no-map-gw).
# loopback :53 the host's resolver, which pasta forwards a
# build's DNS to when the host's nameserver is a
# loopback address.
# public 22/80/443 the forge, as anyone on the internet reaches it.
# Everything else is rejected: the admin sshd on 2222 on every address,
# and any service bound to loopback. -isolation none builds run as the
# same user and get the same rule.
#
# The account name is resolved when the file is loaded. A restart of
# nftables.service (flush ruleset) removes this table; `systemctl
# reload gitbay-runner-egress` puts it back.
table inet gitbay_runner
delete table inet gitbay_runner
table inet gitbay_runner {
chain output {
type filter hook output priority filter; policy accept;
oifname "lo" meta skuid "ci-runner" jump host
}
chain host {
ip daddr 127.0.0.1 tcp dport 22 accept
ip daddr 127.0.0.0/8 meta l4proto { tcp, udp } th dport 53 accept
ip6 daddr ::1 meta l4proto { tcp, udp } th dport 53 accept
ip daddr != 127.0.0.0/8 tcp dport { 22, 80, 443 } accept
ip6 daddr != ::1 tcp dport { 22, 80, 443 } accept
counter reject
}
}
The first table line creates the table if it is missing so the
delete never fails; the file then replaces it in one transaction, so
a reload never leaves a moment without the rule. The uid match is in
the base chain's one rule rather than a != accept, because a packet
with no socket (a kernel-sent reset) matches neither == nor != on
skuid and would otherwise fall through to the reject.
- Step 2: The unit
Create deploy/gitbay-runner-egress.service:
# Loads the CI runner's host egress rule (#260,
# deploy/gitbay-runner-egress.nft). gitbay-runner.service requires this
# unit, so the runner starts only with the rule in force; stopping this
# unit removes the table and stops the runner with it.
#
# Ordered after nftables.service and ufw.service: either may rewrite the
# ruleset at boot, and nftables.service's default config starts with
# flush ruleset. A missing unit in After= is ignored.
#
# Reload re-reads the file and replaces the table in one transaction; it
# does not restart the runner, which a restart of this unit would
# (Requires= propagates restarts). `make deploy-runner` reloads.
[Unit]
Description=Host egress rule for CI builds
After=nftables.service ufw.service
Before=gitbay-runner.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
ExecReload=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
ExecStop=/usr/sbin/nft delete table inet gitbay_runner
[Install]
WantedBy=multi-user.target
- Step 3: The runner requires it
In deploy/gitbay-runner.override.conf, insert before [Service]
(line 25):
[Unit]
# The host egress rule (#260, gitbay-runner-egress.nft) limits what this
# unit's user reaches on the host: 127.0.0.1:22 for the runner, the
# forge's public 22, 80 and 443 for builds, nothing else. Required, so
# the runner does not start without it: a table that failed to load must
# not mean builds reach the admin sshd.
Requires=gitbay-runner-egress.service
After=gitbay-runner-egress.service
- Step 4: The check
Create deploy/runner-egress-check.sh:
#!/bin/sh
# Check the CI runner's host egress rule (#260) as the runner's user:
# the forge over loopback on 22 must answer (the runner polls there),
# and the admin sshd on 2222 must not, on loopback or the public
# address. `make deploy-runner` runs this after loading the rule and
# before restarting the runner, and stops on a failure.
#
# ssh -p 2222 root@bay1 'sh -s' < deploy/runner-egress-check.sh
set -eu
RUNNER_USER="${RUNNER_USER:-ci-runner}"
public=$(hostname -I | awk '{print $1}')
probe() {
su -s /bin/bash "$RUNNER_USER" -c "timeout 5 bash -c 'exec 3<>/dev/tcp/$1/$2'" 2>/dev/null
}
nft list table inet gitbay_runner >/dev/null
for dest in 127.0.0.1:22 "$public:22"; do
if ! probe "${dest%:*}" "${dest##*:}"; then
echo "$RUNNER_USER cannot reach $dest: the egress rule would stop the runner" >&2
exit 1
fi
done
for dest in 127.0.0.1:2222 "$public:2222"; do
if probe "${dest%:*}" "${dest##*:}"; then
echo "$RUNNER_USER reaches $dest: the egress rule is not in force" >&2
exit 1
fi
done
echo "egress for $RUNNER_USER: 127.0.0.1:22 and $public:22 open, 2222 refused"
hostname -I lists the host's addresses, IPv4 first on bay1; the
first is the public one there.
- Step 5: nftables on the host
In deploy/runner-podman-setup.sh, after the podman install block
(after podman --version, line 29):
# nft loads the runner's host egress rule (#260,
# deploy/gitbay-runner-egress.nft), which `make deploy-runner` ships and
# the runner's unit requires. Without nft the runner does not start.
echo "==> installing nftables"
if ! command -v nft >/dev/null 2>&1; then
apt-get update
DEBIAN_FRONTEND=noninteractive apt-get install -y nftables
fi
nft --version
- Step 6:
make deploy-runnerships, loads and checks it
Replace Makefile:69-85 with:
deploy-runner: preflight
@echo "==> building $(RUNNER_BIN)"
$(CROSS) go build -trimpath -ldflags='$(LDFLAGS)' -o $(RUNNER_BIN) ./cmd/gitbay-runner
@echo "==> pushing runner to $(HOST)"
./deploy/copy.sh $(HOST) $(PORT) $(RUNNER_BIN) /usr/local/bin/gitbay-runner.new
ssh -p $(PORT) root@$(HOST) 'mkdir -p /etc/systemd/system/gitbay-runner.service.d /etc/gitbay-runner'
./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner.override.conf /etc/systemd/system/gitbay-runner.service.d/override.conf
./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-prune.service /etc/systemd/system/gitbay-runner-prune.service
./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-prune.timer /etc/systemd/system/gitbay-runner-prune.timer
./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-egress.nft /etc/gitbay-runner/egress.nft
./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-egress.service /etc/systemd/system/gitbay-runner-egress.service
@echo "==> loading the egress rule"
ssh -p $(PORT) root@$(HOST) 'set -eu; \
nft -c -f /etc/gitbay-runner/egress.nft; \
systemctl daemon-reload; \
systemctl enable gitbay-runner-egress.service; \
systemctl reload-or-restart gitbay-runner-egress.service'
ssh -p $(PORT) root@$(HOST) 'sh -s' < deploy/runner-egress-check.sh
ssh -p $(PORT) root@$(HOST) 'set -eu; \
chmod 755 /usr/local/bin/gitbay-runner.new; \
mv /usr/local/bin/gitbay-runner.new /usr/local/bin/gitbay-runner; \
systemctl enable --now gitbay-runner-prune.timer; \
systemctl restart gitbay-runner; \
systemctl --no-pager --lines=3 status gitbay-runner; \
systemctl --no-pager list-timers gitbay-runner-prune.timer'
nft -c checks the file without applying it, so a syntax error stops
the deploy with the old table and the old runner in place. On the first
deploy reload-or-restart starts the unit, and starting a required unit
does not restart the runner that requires it; later deploys reload it.
The check runs before the runner restarts, so a failure stops the
deploy with the old binary in place, but the new table is already
loaded and applies to the running runner too. If the check says the
rule blocks 127.0.0.1:22, remove the table with ssh -p 2222 root@gitbay.org nft delete table inet gitbay_runner (the unit stays
active, so the runner is not stopped with it), fix the rule, and deploy
again. Runbook R3 runs this path on the scratch runner before merge.
- Step 7: Verify locally
Run: sh -n deploy/runner-egress-check.sh && sh -n deploy/runner-podman-setup.sh && make -n deploy-runner HOST=example.test
Expected: no syntax errors; the dry run lists the two new copies, the
nft -c / reload-or-restart block, the check, then the runner block.
- Step 8: Commit
git add deploy/gitbay-runner-egress.nft deploy/gitbay-runner-egress.service deploy/runner-egress-check.sh deploy/gitbay-runner.override.conf deploy/runner-podman-setup.sh Makefile
git commit -S -m "runner host: builds reach only the forge's public ports on it
Ref #260"
Task 3.5: egress policy in the wiki, and the MR
Files:
-
Modify:
.gitbay/wiki/Threat-Model.org("The CI runner", after the Images bullet at line 169) -
Modify:
.gitbay/wiki/CI.org(new section before "* The table", line 55) -
Modify:
.gitbay/wiki/Users.org:551-555(theGITBAY_SSHsentence) -
Modify:
.gitbay/wiki/Admin.org(after "It is idempotent.", line 668) -
Modify:
.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org:70(Network row),04-Trust-Boundaries.org:27(TB7),09-Controls.org:91 -
Step 1: Threat-Model
Add a bullet after Images are provisioned…:
- *What a build can reach.* Outbound internet, trusted or not: a fork's
merge request to a Go repository has to fetch its modules. On the
runner's host, only the forge's public ports 22, 80 and 443, exactly
as anyone on the internet reaches them. Two layers keep it there. A
runner that polls the daemon over loopback starts its containers with
pasta's gateway mapping off, so a build does not reach the host's
loopback, and =GITBAY_SSH= names the public address; that keeps the
runner's source address, =127.0.0.1=, one no build connects from. And
an nftables table (=deploy/gitbay-runner-egress.nft=) rejects every
connection the runner's user makes to the host's own addresses except
=127.0.0.1:22=, DNS on loopback, and 22, 80 and 443 on the public
address: the operator's sshd on 2222 and anything bound to loopback
are closed to builds. Under rootless podman a build's connections are
made by pasta as the runner's user, so the table cannot tell a build
from its runner and leaves =127.0.0.1:22= open; the gateway mapping
is what closes it to builds. The runner does not start without the
table. The SSH auth limiter counts failures per source address and,
once an address is over the limit, refuses every key from it until
the window passes, the runner's included; with registration open or
by invite an unknown key never counts (krz/gitbay#260). Under
=-isolation none= a build runs on the host and shares its loopback;
the table still applies, since it runs as the same user.
- Step 2: CI.org
Add before * The table:
* What a build can reach
Builds have outbound internet access, trusted and untrusted alike. On
the runner's host they reach only the forge's public ports 22, 80 and
443: not the host's loopback, not the operator's sshd. The forge is
reached at its public address, the one in =GITBAY_SSH=. See the
Threat-Model page, "What a build can reach", for how and why
(krz/gitbay#260).
- Step 3: Users.org
Replace "(=git@gitbay.org= from a runner elsewhere; inside a container on the server's own runner the host is at a private address the runner fills in)" with "(=git@gitbay.org=, the instance's public address, from a runner elsewhere and from a container on the server's own runner alike; =git@host:port= on an instance whose ssh is not on 22, so use it as =ssh://GITBAY_SSH …=, which work in both forms)".
- Step 4: Admin.org
After "…verifies rootless podman actually runs as that user. It is idempotent." add:
It also installs nftables. =make deploy-runner= ships
=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
with =gitbay-runner-egress.service=, which loads it and which the
runner's unit requires; it checks the file with =nft -c=, reloads the
unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
restarting the runner: =127.0.0.1:22= and the public 22 must answer,
2222 must not. The table limits the runner's user to =127.0.0.1:22=,
DNS on loopback, and 22, 80 and 443 on the host's public address; the
Threat-Model page says why. A restart of =nftables.service= flushes it;
=systemctl reload gitbay-runner-egress= restores it.
- Step 5: Architecture
07-CI-and-Supply-Chain.org:70 Network row:
| Network | pasta; outbound open; a loopback runner's builds run with =--no-map-gw= (=main.go=); on the host only public 22/80/443 (=gitbay-runner-egress.nft=, #260) |
04-Trust-Boundaries.org:27 TB7: replace "the network is open (#260)"
with "outbound is open; on the host only the forge's public ports
(#260)". 09-Controls.org:91:
| Build network egress restricted | partial | host: loopback closed, public 22/80/443 only (=gitbay-runner-egress.nft=); internet outbound open by decision (#260) |
The Known-Gaps row for #260 and its "What can a build reach…" question stay until the runbook's R3 results are recorded on the CI page.
- Step 6: Verify, commit, MR
Run: go build ./... && go vet ./... && go test ./cmd/gitbay-runner ./internal/control ./internal/sshd -count=1
Expected: PASS.
git add .gitbay/wiki
git commit -S -m "wiki: what a build can reach
Ref #260"
git push -u origin runner-source-address
gitbay mr create --source runner-source-address --target main --title "runner: keep builds off the runner's source address and the host's other ports"
Before merging, run the runbook's R3 on the scratch repository. Merge
with --strategy ff; delete the branch both places.
Part 4: failed step and duration (branch build-failure-report, #266)
Task 4.1: store the failed step
Files:
- Create:
internal/store/migrations/0065_build_failure.up.sql,0065_build_failure.down.sql - Modify:
internal/store/builds.go:12-33(Build),:67-78(buildSelect,scanBuild), afterFinishBuild(:251-264) - Test:
internal/store/builds_test.go(append; add"errors"to imports)
Interfaces:
-
Produces:
Build.FailedStep int,Build.FailedReason stringfunc (s *Store) SetBuildFailure(id int64, step int, reason string) error— only on a running build;ErrNotFoundotherwise.
-
Step 1: Write the failing test
// Where a failed build stopped is recorded while it runs, before the
// outcome, and a finished build is not rewritten (#266).
func TestSetBuildFailure(t *testing.T) {
s := open(t)
if err := s.MigrateUp(); err != nil {
t.Fatal(err)
}
uid, _ := s.CreateUser("cmc", true)
repoID, _ := s.CreateRepo("user", uid, "app", "public")
if _, err := s.CreateBuild(repoID, "unit", "abc", "main", `["true","false"]`, "", "", true); err != nil {
t.Fatal(err)
}
b, ok, err := s.ClaimBuild(nil, false)
if err != nil || !ok {
t.Fatalf("claim: ok=%v err=%v", ok, err)
}
if err := s.SetBuildFailure(b.ID, 2, "exit 1"); err != nil {
t.Fatal(err)
}
if err := s.FinishBuild(b.ID, "failure"); err != nil {
t.Fatal(err)
}
got, err := s.BuildByID(b.ID)
if err != nil {
t.Fatal(err)
}
if got.FailedStep != 2 || got.FailedReason != "exit 1" {
t.Fatalf("failed step %d reason %q", got.FailedStep, got.FailedReason)
}
if err := s.SetBuildFailure(b.ID, 1, "late"); !errors.Is(err, ErrNotFound) {
t.Fatalf("rewrote a finished build: %v", err)
}
}
- Step 2: Run it and see it fail
Run: go test ./internal/store -run TestSetBuildFailure -count=1
Expected: build failure (s.SetBuildFailure undefined).
- Step 3: Migration
0065_build_failure.up.sql:
-- Where a failed build stopped: the 1-based step, 0 when it stopped
-- before any step or did not fail, and the runner's one-line reason.
ALTER TABLE builds ADD COLUMN failed_step INTEGER NOT NULL DEFAULT 0;
ALTER TABLE builds ADD COLUMN failed_reason TEXT NOT NULL DEFAULT '';
0065_build_failure.down.sql:
ALTER TABLE builds DROP COLUMN failed_reason;
ALTER TABLE builds DROP COLUMN failed_step;
- Step 4: Store
In Build, after Trusted:
// FailedStep is the 1-based step a failed build stopped at, 0 when it
// stopped before its first step or did not fail. FailedReason is the
// runner's one line: "exit 1", "build timed out after 45m0s".
FailedStep int
FailedReason string
buildSelect and scanBuild:
const buildSelect = `
SELECT id, repo_id, number, job, sha, ref, steps, image, tree, status, created_at, started_at, finished_at, log_closed_at, trusted,
failed_step, failed_reason
FROM builds`
func scanBuild(row interface{ Scan(...any) error }) (Build, error) {
var b Build
var trusted int
err := row.Scan(&b.ID, &b.RepoID, &b.Number, &b.Job, &b.SHA, &b.Ref, &b.Steps, &b.Image, &b.Tree,
&b.Status, &b.CreatedAt, &b.StartedAt, &b.FinishedAt, &b.LogClosedAt, &trusted,
&b.FailedStep, &b.FailedReason)
b.Trusted = trusted != 0
return b, err
}
After FinishBuild:
// SetBuildFailure records where a running build failed. The runner
// reports it with the outcome; it is written first, so a reader woken
// by the finish sees both.
func (s *Store) SetBuildFailure(id int64, step int, reason string) error {
res, err := s.DB.Exec(`UPDATE builds SET failed_step = ?, failed_reason = ?
WHERE id = ? AND status = 'running'`, step, reason, id)
if err != nil {
return err
}
if n, _ := res.RowsAffected(); n == 0 {
return ErrNotFound
}
return nil
}
- Step 5: Run the tests and see them pass
Run: go test ./internal/store -count=1
Expected: PASS, TestMigrateUpDown included.
- Step 6: Commit
git add internal/store
git commit -S -m "store: failed step and reason on a build
Ref #266"
Task 4.2: runner done --step --reason
Files:
- Modify:
internal/control/build.go:102-106(registration),:648-713(runRunnerDone) - Test:
internal/control/runnernext_test.go(append)
Interfaces:
-
Consumes:
Store.SetBuildFailure. -
Produces:
runner done <build-id> success|failure [--step <n>] [--reason <text>];func failureReason(s string) string. -
Step 1: Write the failing tests
// runner done records the failed step and a one-line reason (#266).
func TestRunnerDoneRecordsFailedStep(t *testing.T) {
st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
n, err := st.CreateBuild(repo.ID, "unit", baseSHA, "main", `["go build ./...","go test ./..."]`, "", "", true)
if err != nil {
t.Fatal(err)
}
b, ok, err := st.ClaimBuild([]int64{repo.ID}, false)
if err != nil || !ok {
t.Fatalf("claim: ok=%v err=%v", ok, err)
}
c, out := runnerCtx(st, uid, root)
if code := runRunnerDone(c, []string{fmt.Sprint(b.ID), "failure", "--step", "2", "--reason", "exit 1\n"}); code != protocol.ExitOK {
t.Fatalf("runner done: exit %d\n%s", code, out.String())
}
got, _ := st.BuildByNumber(repo.ID, n)
if got.Status != "failure" || got.FailedStep != 2 || got.FailedReason != "exit 1" {
t.Fatalf("status %s step %d reason %q", got.Status, got.FailedStep, got.FailedReason)
}
}
// A report with no flags — an older runner — or with a step past the
// job's still finishes the build; the step is then recorded as 0.
func TestRunnerDoneToleratesMissingOrBadStep(t *testing.T) {
st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
for _, extra := range [][]string{nil, {"--step", "9"}} {
n, _ := st.CreateBuild(repo.ID, "unit", baseSHA, "main", `["true"]`, "", "", true)
b, ok, err := st.ClaimBuild([]int64{repo.ID}, false)
if err != nil || !ok {
t.Fatalf("claim: ok=%v err=%v", ok, err)
}
c, out := runnerCtx(st, uid, root)
if code := runRunnerDone(c, append([]string{fmt.Sprint(b.ID), "failure"}, extra...)); code != protocol.ExitOK {
t.Fatalf("runner done %v: exit %d\n%s", extra, code, out.String())
}
if got, _ := st.BuildByNumber(repo.ID, n); got.Status != "failure" || got.FailedStep != 0 {
t.Fatalf("%v: status %s step %d", extra, got.Status, got.FailedStep)
}
}
}
- Step 2: Run them and see them fail
Run: go test ./internal/control -run TestRunnerDone -count=1
Expected: FAIL, the first with exit 2 (usage: four arguments where two
are accepted).
- Step 3: Implement
Registration:
register(Command{Path: []string{"runner", "done"},
Summary: "finish a build",
Usage: "runner done <build-id> success|failure [--step <n>] [--reason <text>]",
Flags: []Flag{
{"--step", "<n>", "the 1-based step a failed build stopped at", ""},
{"--reason", "<text>", "how it failed, one line", ""},
},
Examples: []string{"runner done 431 success", "runner done 431 failure --step 3 --reason 'exit 1'"},
Run: runRunnerDone})
Replace runRunnerDone with:
func runRunnerDone(c *Ctx, args []string) int {
key, code := runnerSession(c)
if code >= 0 {
return code
}
f, err := parseFlags(args, flagSpec{Values: []string{"--step", "--reason"}, MaxPos: 2,
Usage: "runner done <build-id> success|failure [--step <n>] [--reason <text>]"})
if err != nil {
return c.fail(protocol.ExitUsage, "%v", err)
}
if len(f.Pos) != 2 || (f.Pos[1] != "success" && f.Pos[1] != "failure") {
return c.usage()
}
outcome := f.Pos[1]
id, err := strconv.ParseInt(f.Pos[0], 10, 64)
if err != nil {
return c.fail(protocol.ExitUsage, "bad build id %q", f.Pos[0])
}
b, err := c.Store.BuildByID(id)
if err != nil {
return c.fail(protocol.ExitNotFound, "no build %d", id)
}
if ok, err := runnerMayBuild(c, key, b.RepoID); err != nil {
return c.fail(protocol.ExitFailure, "%v", err)
} else if !ok {
return c.fail(protocol.ExitDenied, "this key is not attached to the build's repository; a repository admin attaches it with repo runner add")
}
// Cancelled underneath the runner: its report is late, not wrong.
// The row, the status and the log were settled by the cancel.
if b.Status == "cancelled" {
c.Store.RunnerDone(key.ID)
return c.emit(map[string]any{"build": b.Number, "status": "cancelled"}, func(w io.Writer) {
fmt.Fprintf(w, "build %d was cancelled\n", b.Number)
})
}
if outcome == "failure" {
// A step the job does not have is recorded as none rather than
// refused: refusing would lose the outcome over a detail (#266).
var steps []string
json.Unmarshal([]byte(b.Steps), &steps)
step, _ := strconv.Atoi(f.Value("--step"))
if step < 0 || step > len(steps) {
step = 0
}
if err := c.Store.SetBuildFailure(id, step, failureReason(f.Value("--reason"))); err != nil && !errors.Is(err, store.ErrNotFound) {
return c.fail(protocol.ExitFailure, "recording build %d's failure: %v", id, err)
}
}
if err := c.Store.FinishBuild(id, outcome); err != nil {
return c.fail(protocol.ExitFailure, "finishing build %d: %v", id, err)
}
c.Store.RunnerDone(key.ID)
repo, err := c.Store.RepoByID(b.RepoID)
if err != nil {
return c.fail(protocol.ExitFailure, "%v", err)
}
url := fmt.Sprintf("%s/%s/builds/%d", c.Cfg.Server.SiteURL, repo.Path(), b.Number)
desc := "build " + outcome
if err := c.Store.SetCommitStatus(repo.ID, b.SHA, "ci/"+b.Job, outcome, desc, url, c.User.ID); err != nil {
return c.fail(protocol.ExitFailure, "%v", err)
}
c.Store.RecordEvent(repo.ID, c.User.ID, "build."+outcome,
fmt.Sprintf(`{"number":%d,"job":%q,"sha":%q}`, b.Number, b.Job, b.SHA))
// A red build mails the repo's notify targets with the log tail — a
// failed scheduled job must not wait to be noticed.
if outcome == "failure" {
if targets, err := c.Store.RepoNotifyTargets(repo); err == nil {
tail := ""
if log, err := c.Store.BuildLog(id); err == nil && len(log) > 0 {
if len(log) > 2000 {
log = log[len(log)-2000:]
}
tail = string(log)
}
notify(c, targets, notice{repo: repo, kind: "build",
subject: fmt.Sprintf("[%s] build %d failed: %s on %s", repo.Path(), b.Number, b.Job, b.Ref),
action: fmt.Sprintf("build %d failed: %s on %s", b.Number, b.Job, b.Ref),
body: fmt.Sprintf("job %s failed at %.10s.\n\n…%s\n\n%s\n", b.Job, b.SHA, tail, url),
path: fmt.Sprintf("%s/builds/%d", repo.Path(), b.Number)})
}
}
return c.emit(map[string]any{"build": b.Number, "status": outcome}, func(w io.Writer) {
fmt.Fprintf(w, "build %d %s\n", b.Number, outcome)
})
}
// failureReason keeps a runner's reason to one line of at most 200
// bytes: it is shown on the build page and by build show.
func failureReason(s string) string {
s = strings.Join(strings.Fields(s), " ")
if len(s) > 200 {
s = s[:200]
}
return strings.ToValidUTF8(s, "")
}
- Step 4: Run the tests and see them pass
Run: go vet ./... && go test ./internal/control -count=1
Expected: PASS.
- Step 5: Commit
git add internal/control/build.go internal/control/runnernext_test.go
git commit -S -m "runner done: record the failed step and reason
Ref #266"
Task 4.3: the runner names the failed step
Files:
- Modify:
cmd/gitbay-runner/main.go:245-277(step),:309-435(run),:363-398(runStep) - Modify:
cmd/gitbay-runner/isolate.go:79-197(runSteps,runStepsPodman) - Modify:
cmd/gitbay-runner/report.go:19-23(reportDone), newdoneArgs - Test:
cmd/gitbay-runner/steps_test.go(create),cmd/gitbay-runner/report_test.go(append)
Interfaces:
-
Consumes:
runner done … --step <n> --reason <text>from Task 4.2. -
Produces:
type failure struct { Step int; Reason string }func exitReason(err error) stringrun(j job) *failure,runSteps(…) *failure,runStepsPodman(…) *failure(nil is success)func (r *runner) reportDone(id int64, status string, f *failure) errorfunc doneArgs(id int64, status string, f *failure) []string
-
Step 1: Write the failing tests
Create cmd/gitbay-runner/steps_test.go:
package main
import (
"os"
"os/exec"
"strings"
"testing"
"time"
)
// The failing step is named by number in the log and in the outcome
// reported to the server (#266).
func TestRunStepsNamesTheFailedStep(t *testing.T) {
r := &runner{isolation: isolationNone}
run := func(cmd *exec.Cmd, _ time.Time) (bool, string) {
if err := cmd.Run(); err != nil {
return false, exitReason(err)
}
return true, ""
}
env := []string{"PATH=" + os.Getenv("PATH")}
var log strings.Builder
f := r.runSteps(job{Steps: []string{"true", "exit 3", "true"}}, t.TempDir(), env, &log, time.Now().Add(time.Minute), run)
if f == nil || f.Step != 2 || f.Reason != "exit 3" {
t.Fatalf("failure %+v, want step 2, exit 3", f)
}
if !strings.Contains(log.String(), "step 2/3 failed: exit 3\n") {
t.Fatalf("log does not name the step:\n%s", log.String())
}
if f := r.runSteps(job{Steps: []string{"true"}}, t.TempDir(), env, &log, time.Now().Add(time.Minute), run); f != nil {
t.Fatalf("a passing job failed: %+v", f)
}
}
Append to cmd/gitbay-runner/report_test.go (add imports
"strings" and "gitbay.org/gitbay/internal/protocol"):
// The reason survives the trip: ssh joins arguments with spaces and the
// server splits the line again with POSIX rules (#266).
func TestDoneArgsNameTheFailedStep(t *testing.T) {
got := doneArgs(7, "failure", &failure{Step: 3, Reason: "can't: exit 1"})
argv, err := protocol.Tokenize(strings.Join(got, " "))
if err != nil {
t.Fatal(err)
}
want := []string{"runner", "done", "7", "failure", "--step", "3", "--reason", "can't: exit 1"}
if strings.Join(argv, "|") != strings.Join(want, "|") {
t.Fatalf("server reads %q, want %q", argv, want)
}
if got := doneArgs(7, "success", nil); strings.Join(got, " ") != "runner done 7 success" {
t.Fatalf("success: %q", got)
}
if got := doneArgs(7, "failure", &failure{Reason: "git clone: exit 128"}); strings.Contains(strings.Join(got, " "), "--step") {
t.Fatalf("a failure before any step sent a step: %q", got)
}
}
- Step 2: Run them and see them fail
Run: go test ./cmd/gitbay-runner -count=1
Expected: build failure (undefined: exitReason, undefined: doneArgs, undefined: failure).
- Step 3:
failureandexitReason
In main.go, before run:
// failure says where a build stopped: Step is the 1-based step that
// failed, 0 when the build stopped before its first step (the clone, the
// container), and Reason is one short line (#266).
type failure struct {
Step int
Reason string
}
// exitReason is how a finished command's failure reads in a build's log
// and on the build: "exit 1" for a command that exited, the error
// otherwise (a signal, a start failure).
func exitReason(err error) string {
var ee *exec.ExitError
if errors.As(err, &ee) && ee.ExitCode() >= 0 {
return fmt.Sprintf("exit %d", ee.ExitCode())
}
return err.Error()
}
Add "errors" to main.go's imports.
- Step 4:
runandstep
In run, the signature becomes func (r *runner) run(j job) *failure
with its comment "…Returns nil when every step succeeded, else where the
build stopped." Each early return false returns a failure instead:
home, doneHome, err := buildHome(r.workdir, j)
if err != nil {
log.Printf("build %d: build home: %v", j.ID, err)
return &failure{Reason: "preparing the build home failed"}
}
defer doneHome()
pipe, err := logCmd.StdinPipe()
if err != nil {
log.Printf("build %d: log pipe: %v", j.ID, err)
return &failure{Reason: "opening the log stream failed"}
}
if err := logCmd.Start(); err != nil {
log.Printf("build %d: log stream: %v", j.ID, err)
return &failure{Reason: "opening the log stream failed"}
}
In runStep, return false, fmt.Sprintf("step failed: %v", err)
becomes return false, exitReason(err).
The clone loop's failure:
if ok, why := runStep(cmd, deadline); !ok {
fmt.Fprintf(sink, "git %s: %s\n", args[0], why)
return &failure{Reason: "git " + args[0] + ": " + why}
}
step(), from status := "failure" to the report:
f := r.run(j)
status := "success"
if f != nil {
status = "failure"
}
if err := r.reportDone(j.ID, status, f); err != nil {
return true, err
}
- Step 5:
runSteps,runStepsPodman
In isolate.go, the runSteps comment ends "…Returns nil when every
step succeeded." and the function becomes:
func (r *runner) runSteps(j job, dir string, env []string, sink io.Writer, deadline time.Time, runStep stepRunner) *failure {
if r.isolation == isolationNone {
for i, step := range j.Steps {
fmt.Fprintf(sink, "$ %s\n", step)
cmd := exec.Command(toolpath.Look("sh"), "-c", step)
cmd.Dir, cmd.Env = dir, env
cmd.Stdout, cmd.Stderr = sink, sink
if ok, why := runStep(cmd, deadline); !ok {
fmt.Fprintf(sink, "step %d/%d failed: %s\n", i+1, len(j.Steps), why)
return &failure{Step: i + 1, Reason: why}
}
}
return nil
}
return r.runStepsPodman(j, dir, env, sink, deadline, runStep)
}
runStepsPodman returns *failure. Its setup failures (env file,
cgroup, container start) keep their log lines and return
&failure{Reason: "preparing the build environment failed"},
&failure{Reason: "preparing the build cgroup failed"} and
&failure{Reason: "starting the build container failed"}
respectively. The step loop:
for i, step := range j.Steps {
fmt.Fprintf(sink, "$ %s\n", step)
cmd := exec.Command(podman, append(r.podmanGlobal(), "exec", "--workdir", "/workspace", name, "sh", "-c", step)...)
cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "HOME=" + r.podmanHome()}
intoCgroup(cmd, cgroupFD)
cmd.Stdout, cmd.Stderr = sink, sink
if ok, why := runStep(cmd, deadline); !ok {
fmt.Fprintf(sink, "step %d/%d failed: %s\n", i+1, len(j.Steps), why)
return &failure{Step: i + 1, Reason: why}
}
}
return nil
podman exec exits with the step's own status, so exit 1 is the
step's.
- Step 6:
reportDone,doneArgs
In report.go (add "strconv" and "strings" to its imports):
func (r *runner) reportDone(id int64, status string, f *failure) error {
args := doneArgs(id, status, f)
return reportWithRetry(func() (string, error) {
return r.ssh(nil, args...)
}, id, retryDelays)
}
// doneArgs is the runner done command for a build's outcome. The reason
// is single-quoted: ssh joins arguments with spaces, and the server
// splits the line again with POSIX rules.
func doneArgs(id int64, status string, f *failure) []string {
args := []string{"runner", "done", fmt.Sprint(id), status}
if f == nil {
return args
}
if f.Step > 0 {
args = append(args, "--step", strconv.Itoa(f.Step))
}
if f.Reason != "" {
args = append(args, "--reason", "'"+strings.ReplaceAll(f.Reason, "'", `'\''`)+"'")
}
return args
}
(The comment above reportDone is unchanged.)
- Step 7: Run the tests and see them pass
Run: go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner -count=1
Expected: PASS.
- Step 8: Commit
git add cmd/gitbay-runner
git commit -S -m "runner: name the failed step and report it
Ref #266"
Task 4.4: build show, build log --step/--tail
Files:
- Create:
internal/control/buildlog.go,internal/control/buildlog_test.go - Modify:
internal/control/build.go:40-47(registration),:109-126(BuildOut,buildToOut),:221-238(runBuildShow),:240-258(runBuildLog)
Interfaces:
-
Consumes:
Build.FailedStep,Build.FailedReason,Build.Elapsed(). -
Produces (used by Task 4.5):
type LogSection struct { N int; Step string; Text string }func SplitBuildLog(log string, steps []string) []LogSectionfunc FailedSection(sections []LogSection, status string, failedStep int) intBuildOut.FailedStep int(failed_step),BuildOut.FailedReason string(failed_reason),BuildOut.DurationS int64(duration_s),BuildOut.Steps []string(steps, onbuild showonly)
-
Step 1: Write the failing tests
Create internal/control/buildlog_test.go:
package control
import (
"bytes"
"fmt"
"reflect"
"regexp"
"strings"
"testing"
"gitbay.org/gitbay/internal/protocol"
"gitbay.org/gitbay/internal/store"
)
func TestSplitBuildLog(t *testing.T) {
log := "$ git clone ssh://x/a.git (abc)\n" +
"$ go build ./...\n" +
"built\n" +
"$ go test ./...\n" +
"--- FAIL: TestX\n" +
"step 2/2 failed: exit 1\n"
got := SplitBuildLog(log, []string{"go build ./...", "go test ./..."})
want := []LogSection{
{N: 0, Text: "$ git clone ssh://x/a.git (abc)\n"},
{N: 1, Step: "go build ./...", Text: "built\n"},
{N: 2, Step: "go test ./...", Text: "--- FAIL: TestX\nstep 2/2 failed: exit 1\n"},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %+v\nwant %+v", got, want)
}
}
// A step's line inside other output, not at a line start, does not cut;
// a build that stopped before a step has no section for it; an empty
// setup is left out.
func TestSplitBuildLogStopsAtMissingStep(t *testing.T) {
got := SplitBuildLog("$ make\nrunning: $ make test\nerror\n", []string{"make", "make test"})
want := []LogSection{{N: 1, Step: "make", Text: "running: $ make test\nerror\n"}}
if !reflect.DeepEqual(got, want) {
t.Fatalf("got %+v\nwant %+v", got, want)
}
}
func TestTailLines(t *testing.T) {
for _, tc := range []struct {
in string
n int
want string
}{
{"a\nb\nc\n", 2, "b\nc\n"},
{"a\nb\nc\n", 5, "a\nb\nc\n"},
{"a\nb", 1, "b"},
} {
if got := string(tailLines([]byte(tc.in), tc.n)); got != tc.want {
t.Errorf("tailLines(%q, %d) = %q, want %q", tc.in, tc.n, got, tc.want)
}
}
}
// failedBuild is a finished failure whose second of two steps failed,
// having run 10m56s.
func failedBuild(t *testing.T) (*store.Store, store.Repo, int64, int64) {
t.Helper()
st, repo, uid := newQueueTestRepo(t)
n, err := st.CreateBuild(repo.ID, "unit", "abc", "main", `["go build ./...","go test ./..."]`, "", "", true)
if err != nil {
t.Fatal(err)
}
b, ok, err := st.ClaimBuild([]int64{repo.ID}, false)
if err != nil || !ok {
t.Fatalf("claim: ok=%v err=%v", ok, err)
}
st.AppendBuildLog(b.ID, []byte("$ git clone x (abc)\n$ go build ./...\nok\n$ go test ./...\none\n--- FAIL: TestX\nstep 2/2 failed: exit 1\n"))
if err := st.SetBuildFailure(b.ID, 2, "exit 1"); err != nil {
t.Fatal(err)
}
if err := st.FinishBuild(b.ID, "failure"); err != nil {
t.Fatal(err)
}
if _, err := st.DB.Exec(`UPDATE builds SET started_at = '2026-09-27T10:00:00Z', finished_at = '2026-09-27T10:10:56Z' WHERE id = ?`, b.ID); err != nil {
t.Fatal(err)
}
return st, repo, uid, n
}
func TestBuildLogStepAndTail(t *testing.T) {
st, repo, uid, n := failedBuild(t)
run := func(args ...string) (string, int) {
t.Helper()
c, errOut := pruneCtx(st, t.TempDir(), store.User{ID: uid})
code := Dispatch(c, append([]string{"build", "log", repo.Path(), fmt.Sprint(n)}, args...))
return c.Stdout.(*bytes.Buffer).String() + errOut.String(), code
}
if out, _ := run("--step", "1"); out != "ok\n" {
t.Errorf("--step 1: %q", out)
}
if out, _ := run("--step", "failed", "--tail", "2"); out != "--- FAIL: TestX\nstep 2/2 failed: exit 1\n" {
t.Errorf("--step failed --tail 2: %q", out)
}
if out, _ := run("--tail", "1"); out != "step 2/2 failed: exit 1\n" {
t.Errorf("--tail 1: %q", out)
}
if _, code := run("--step", "3"); code != protocol.ExitUsage {
t.Errorf("--step past the job: exit %d", code)
}
if _, code := run("--follow", "--tail", "1"); code != protocol.ExitUsage {
t.Errorf("--follow with --tail: exit %d", code)
}
}
func TestBuildShowNamesFailedStepAndDuration(t *testing.T) {
st, repo, uid, n := failedBuild(t)
c, errOut := pruneCtx(st, t.TempDir(), store.User{ID: uid})
if code := Dispatch(c, []string{"build", "show", repo.Path(), fmt.Sprint(n)}); code != protocol.ExitOK {
t.Fatalf("exit %d: %s", code, errOut)
}
out := c.Stdout.(*bytes.Buffer).String()
for _, re := range []string{`failed step\s+2/2 go test \./\.\.\. \(exit 1\)`, `duration\s+10m56s`} {
if !regexp.MustCompile(re).MatchString(out) {
t.Errorf("build show missing %s:\n%s", re, out)
}
}
c, _ = pruneCtx(st, t.TempDir(), store.User{ID: uid})
c.JSON = true
Dispatch(c, []string{"build", "show", repo.Path(), fmt.Sprint(n)})
for _, want := range []string{`"failed_step":2`, `"failed_reason":"exit 1"`, `"duration_s":656`, `"steps":["go build ./...","go test ./..."]`} {
if !strings.Contains(c.Stdout.(*bytes.Buffer).String(), want) {
t.Errorf("build show --json missing %s", want)
}
}
}
- Step 2: Run them and see them fail
Run: go test ./internal/control -run 'TestSplitBuildLog|TestTailLines|TestBuildLogStep|TestBuildShowNames' -count=1
Expected: build failure (undefined: SplitBuildLog).
- Step 3:
buildlog.go
package control
import "strings"
// LogSection is one part of a build log: the setup before the first
// step (N 0), or one step and its output.
type LogSection struct {
N int // 0 for the setup, else the 1-based step
Step string // the step's command; "" for the setup
Text string
}
// SplitBuildLog cuts a log at the "$ <step>" line the runner writes
// before each step, matching the build's steps in order and only at a
// line start. Output before the first step is the setup section, left
// out when empty. A step with no line in the log — the build stopped
// before it — has no section, and neither has any step after it.
func SplitBuildLog(log string, steps []string) []LogSection {
var out []LogSection
cur := LogSection{}
start := 0
for i, step := range steps {
marker := "$ " + step + "\n"
at := findLine(log, marker, start)
if at < 0 {
break
}
cur.Text = log[start:at]
if cur.N > 0 || cur.Text != "" {
out = append(out, cur)
}
cur = LogSection{N: i + 1, Step: step}
start = at + len(marker)
}
cur.Text = log[start:]
if cur.N > 0 || cur.Text != "" {
out = append(out, cur)
}
return out
}
// findLine is the index of line in log at or after from where it starts
// a line, or -1.
func findLine(log, line string, from int) int {
for i := from; i <= len(log)-len(line); {
j := strings.Index(log[i:], line)
if j < 0 {
return -1
}
at := i + j
if at == 0 || log[at-1] == '\n' {
return at
}
i = at + 1
}
return -1
}
// FailedSection is the index of the section a failed build stopped in:
// the step the runner named, or the last section when it named none (an
// older runner, or a failure the runner could not tie to a step). -1
// when the build did not fail or its log is empty.
func FailedSection(sections []LogSection, status string, failedStep int) int {
if status != "failure" || len(sections) == 0 {
return -1
}
for i, s := range sections {
if failedStep > 0 && s.N == failedStep {
return i
}
}
return len(sections) - 1
}
// tailLines is the last n lines of b; a final newline ends the last line
// rather than starting another.
func tailLines(b []byte, n int) []byte {
end := len(b)
if end > 0 && b[end-1] == '\n' {
end--
}
for i := end - 1; i >= 0; i-- {
if b[i] == '\n' {
n--
if n == 0 {
return b[i+1:]
}
}
}
return b
}
- Step 4:
BuildOut,build show
BuildOut, after Subject:
// FailedStep is the 1-based step a failed build stopped at, 0 when
// none; FailedReason says how ("exit 1") (#266).
FailedStep int `json:"failed_step,omitempty"`
FailedReason string `json:"failed_reason,omitempty"`
// DurationS is how long the build ran, once it has a start and a
// finish.
DurationS int64 `json:"duration_s,omitempty"`
// Steps are the job's commands; build show only.
Steps []string `json:"steps,omitempty"`
buildToOut:
func buildToOut(b store.Build) BuildOut {
return BuildOut{Number: b.Number, Job: b.Job, Status: b.Status, SHA: b.SHA,
Ref: b.Ref, CreatedAt: b.CreatedAt, FinishedAt: b.FinishedAt,
FailedStep: b.FailedStep, FailedReason: b.FailedReason,
DurationS: int64(b.Elapsed() / time.Second)}
}
runBuildShow:
func runBuildShow(c *Ctx, args []string) int {
repo, b, code := buildRef(c, args)
if code >= 0 {
return code
}
d := buildToOut(b)
json.Unmarshal([]byte(b.Steps), &d.Steps)
return c.emit(d, func(w io.Writer) {
failedStep, failed := "", ""
if d.FailedStep > 0 && d.FailedStep <= len(d.Steps) {
step, _, _ := strings.Cut(d.Steps[d.FailedStep-1], "\n")
failedStep = fmt.Sprintf("%d/%d %s", d.FailedStep, len(d.Steps), step)
if d.FailedReason != "" {
failedStep += " (" + d.FailedReason + ")"
}
} else {
failed = d.FailedReason
}
duration := ""
if d.DurationS > 0 {
duration = (time.Duration(d.DurationS) * time.Second).String()
}
v := c.view(w)
v.title(fmt.Sprintf("#%d", d.Number), d.Job, d.Status)
v.fields(
"sha", fmt.Sprintf("%.10s", d.SHA),
"ref", d.Ref,
"queued", c.when(d.CreatedAt),
"finished", c.when(d.FinishedAt),
"duration", duration,
"failed step", failedStep,
"failed", failed,
"url", c.siteURL(repo.Path(), "builds", strconv.FormatInt(d.Number, 10)),
)
})
}
- Step 5:
build log
Registration:
register(Command{Path: []string{"build", "log"},
Summary: "print a build's log, or follow it until the build ends",
Usage: "build log <owner/name> <n> [--follow] [--step <step>|failed] [--tail <lines>]",
Flags: []Flag{
{"--follow", "", "stream the log until the build ends", ""},
{"--step", "<step>|failed", "only one step's output: 0 for the setup, a step number, or the one that failed", ""},
{"--tail", "<lines>", "only the last lines", ""},
},
Examples: []string{"build log krz/gitbay 431 --follow", "build log krz/gitbay 431 --step failed --tail 40"},
ReadOnly: true, Run: runBuildLog})
runBuildLog:
func runBuildLog(c *Ctx, args []string) int {
f, err := parseFlags(args, flagSpec{Bools: []string{"--follow"}, Values: []string{"--step", "--tail"}, MaxPos: 2, Usage: c.Cmd.Usage})
if err != nil {
return c.fail(protocol.ExitUsage, "%v", err)
}
repo, b, code := buildRef(c, f.Pos)
if code >= 0 {
return code
}
if f.Has("--follow") {
if f.Has("--step") || f.Has("--tail") {
return c.fail(protocol.ExitUsage, "--step and --tail read the stored log; drop --follow")
}
return followBuildLog(c, repo, b)
}
tail := 0
if f.Has("--tail") {
if tail, err = strconv.Atoi(f.Value("--tail")); err != nil || tail < 1 {
return c.fail(protocol.ExitUsage, "--tail takes a number of lines, 1 or more")
}
}
log, err := c.Store.BuildLog(b.ID)
if err != nil {
return c.fail(protocol.ExitFailure, "%v", err)
}
if f.Has("--step") {
var steps []string
json.Unmarshal([]byte(b.Steps), &steps)
sections := SplitBuildLog(string(log), steps)
at := -1
if want := f.Value("--step"); want == "failed" {
if at = FailedSection(sections, b.Status, b.FailedStep); at < 0 {
return c.fail(protocol.ExitNotFound, "build %d did not fail", b.Number)
}
} else {
n, err := strconv.Atoi(want)
if err != nil || n < 0 || n > len(steps) {
return c.fail(protocol.ExitUsage, "--step takes 0 (the setup) to %d, or failed", len(steps))
}
for i, s := range sections {
if s.N == n {
at = i
}
}
if at < 0 {
return c.fail(protocol.ExitNotFound, "build %d has no output for step %d", b.Number, n)
}
}
log = []byte(sections[at].Text)
}
if tail > 0 {
log = tailLines(log, tail)
}
c.Stdout.Write(log)
return protocol.ExitOK
}
- Step 6: Run the tests and see them pass
Run: go vet ./... && go test ./internal/control -count=1 && go test ./cmd/gitbay -run TestSummariesAreCurrent -count=1
Expected: PASS (the build log summary is unchanged; no regeneration
needed).
- Step 7: Commit
git add internal/control
git commit -S -m "build show: failed step and duration; build log --step, --tail
Ref #266"
Task 4.5: the build page folds by step
Files:
- Modify:
internal/httpd/builds.go:1-18(imports: add"time"),:287-322(build,buildView) - Modify:
internal/web/templates/build.html:14-17 - Modify:
internal/web/static/style.css:1116 - Test:
internal/httpd/buildpages_test.go(append)
Interfaces:
-
Consumes:
control.SplitBuildLog,control.FailedSection,control.LogSection,BuildOutfields from Task 4.4. -
Produces:
buildView.Steps []logStep,buildView.Failed bool,buildView.Duration string;func logSteps(log string, b control.BuildOut) ([]logStep, bool). -
Step 1: Write the failing test
Append to internal/httpd/buildpages_test.go:
// A failed build's page folds its log by step, opens the step that
// failed and links to it; no JavaScript (#266).
func TestBuildPageFoldsStepsAndOpensFailure(t *testing.T) {
b := control.BuildOut{Number: 61, Job: "test", Status: "failure",
SHA: "ff6271a9d4570cd46f169091637a9d2e40ad5c2b", Ref: "main",
CreatedAt: "2026-08-28T04:42:54Z", FinishedAt: "2026-08-28T04:53:50Z", DurationS: 656,
Steps: []string{"go build ./...", "go test ./..."}, FailedStep: 2, FailedReason: "exit 1"}
log := "$ git clone x (ff6271a9d4)\n$ go build ./...\n$ go test ./...\n--- FAIL: TestCLI\nstep 2/2 failed: exit 1\n"
v := buildView{repoPage: testRepoPage(), Build: b, Log: log, Duration: "10m56s"}
v.Steps, v.Failed = logSteps(log, b)
var sb strings.Builder
if err := web.Render(&sb, "build.html", v); err != nil {
t.Fatalf("render: %v", err)
}
out := sb.String()
for _, want := range []string{
`<details class="difffold buildstep" id="failed" open>`,
"step 2/2", "<code>go test ./...</code>", `href="#failed"`, "Jump to failure",
"ran 10m56s", "--- FAIL: TestCLI",
} {
if !strings.Contains(out, want) {
t.Errorf("build.html missing %q", want)
}
}
if n := strings.Count(out, `class="difffold buildstep"`); n != 3 {
t.Errorf("%d step folds, want 3 (setup and two steps)", n)
}
if n := strings.Count(out, `id="failed"`); n != 1 {
t.Errorf("%d failed anchors, want 1", n)
}
}
- Step 2: Run it and see it fail
Run: go test ./internal/httpd -run TestBuildPageFoldsStepsAndOpensFailure -count=1
Expected: build failure (undefined: logSteps).
- Step 3: Handler
Replace buildView and add logStep and logSteps:
type buildView struct {
repoPage
Build control.BuildOut
Log string
// Steps is the finished log cut at its steps, nil when there is no
// step to cut at; Failed says whether one of them is marked failed.
Steps []logStep
Failed bool
Duration string
Live bool
CanWrite bool
Notice string
}
type logStep struct {
control.LogSection
Failed bool
}
// logSteps cuts a finished build's log at its steps and marks the one it
// failed at. Nil when no step's line is in the log — a build that
// stopped in the clone — which renders as one block.
func logSteps(log string, b control.BuildOut) ([]logStep, bool) {
sections := control.SplitBuildLog(log, b.Steps)
stepped := false
for _, s := range sections {
if s.N > 0 {
stepped = true
}
}
if !stepped {
return nil, false
}
failed := control.FailedSection(sections, b.Status, b.FailedStep)
out := make([]logStep, len(sections))
for i, s := range sections {
out[i] = logStep{LogSection: s, Failed: i == failed}
}
return out, failed >= 0
}
In build, replace the last two lines (v.Log, _, _ = … and
s.render(…)) with:
v.Log, _, _ = s.runControl(viewer, []string{"build", "log", p.Repo.Path(), n})
v.Steps, v.Failed = logSteps(v.Log, b)
if b.DurationS > 0 {
v.Duration = (time.Duration(b.DurationS) * time.Second).String()
}
s.render(w, "build.html", v)
Add "time" to the imports.
- Step 4: Template
Replace build.html:14-17 with:
<p class="meta">{{.Build.Job}} on {{.Build.Ref}} · <code><a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/commit/{{.Build.SHA}}">{{printf "%.10s" .Build.SHA}}</a></code> · queued {{when .Build.CreatedAt}}{{if .Build.FinishedAt}} · finished {{when .Build.FinishedAt}}{{with .Duration}} · ran {{.}}{{end}}{{end}}{{if .Failed}} · <a href="#failed">Jump to failure</a>{{end}}</p>
{{if .Live}}<p class="meta">Live: the log streams here until the build ends. If it stops without a “build finished” line, reload to pick it up again. <a href="?follow=0">Show it without updates</a></p>
<pre class="code buildlog" tabindex="0">{{.Log}}</pre>
{{else if .Steps}}{{$total := len .Build.Steps}}{{range .Steps}}
<details class="difffold buildstep"{{if .Failed}} id="failed" open{{end}}>
<summary>{{if .N}}<span>step {{.N}}/{{$total}}</span> <code>{{.Step}}</code>{{else}}<span>setup</span>{{end}}{{if .Failed}} <span class="chip check-failure">failed</span>{{end}}</summary>
<pre class="code buildlog" tabindex="0">{{.Text}}</pre>
</details>{{end}}
{{else if .Log}}<pre class="code buildlog" tabindex="0">{{.Log}}</pre>{{else}}<p class="empty-note">no log yet</p>{{end}}
The live branch keeps its single <pre> directly after the marker, which
streamBuild requires (builds.go:340).
- Step 5: CSS
Replace style.css:1116 with:
pre.buildlog { max-height: 40rem; overflow: auto; white-space: pre-wrap; overflow-wrap: anywhere; }
details.buildstep pre.buildlog { margin: 0; border: 0; border-radius: 0; }
details.buildstep summary code { overflow-wrap: anywhere; }
- Step 6: Run the tests
Run: go test ./internal/httpd ./internal/web -count=1
Expected: PASS (TestPreBlocksAreFocusable sees the new <pre> with
tabindex="0"; no new template, so TestMainWidthClass is unchanged).
- Step 7: Commit
git add internal/httpd internal/web
git commit -S -m "web: build log folded by step, failed step open
Ref #266"
Task 4.6: e2e, wiki, and the MR
Files:
-
Modify:
e2e/ci_test.go:145-149(TestCI) -
Modify:
.gitbay/wiki/CI.org(after thebuild log --followparagraph),.gitbay/wiki/Users.org:542-547,.gitbay/wiki/Parity.org:211-213,.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org(lifecycle step 5) -
Step 1: e2e
Replace e2e/ci_test.go:145-149 with:
out, _, _ = inst.ssh(t, aliceKey, "", "build", "log", "alice/app", brokenN)
if !strings.Contains(out, "step 1/1 failed: exit 1") {
t.Fatalf("broken log:\n%s", out)
}
out, _, _ = inst.ssh(t, aliceKey, "", "build", "show", "alice/app", brokenN)
if !strings.Contains(out, "1/1 false (exit 1)") {
t.Fatalf("build show does not name the failed step:\n%s", out)
}
if out, _, _ = inst.ssh(t, aliceKey, "", "build", "log", "alice/app", brokenN, "--step", "failed"); strings.Contains(out, "git clone") || !strings.Contains(out, "exit 1") {
t.Fatalf("build log --step failed:\n%s", out)
}
if _, body := inst.get(t, "/alice/app/builds/"+brokenN); !strings.Contains(body, `id="failed" open`) {
t.Fatalf("build page does not open the failed step:\n%s", body)
}
Before editing, check the broken job's step in TestCI's ci.yml
(line ~100) is "false"; the expected 1/1 false follows from it.
Run: go test ./e2e -run 'TestCI$' -count=1
Expected: PASS.
- Step 2: Wiki
CI.org, after the build log --follow paragraph:
A failed build names the step it stopped at: the log's last line reads
=step 3/3 failed: exit 1=, and =build show= prints =failed step= (=3/3
go test ./... (exit 1)=) and =duration=. =build log <owner/name> <n>
--step failed= prints only that step's output, =--step 2= another one
(=0= is the clone before the first step), and =--tail 40= the last forty
lines of whichever was chosen; neither combines with =--follow=. The
build page folds the finished log into one section per step, opens the
failed one and links to it from the top as "Jump to failure". Builds
from before this reported no step; their last section is taken as the
failed one.
Users.org, after "…=build list= takes =--ref=, =--status= and =--job= to narrow the listing, combinable;" sentence group, add: "=build show= names a failed build's step and how long it ran, and =build log= takes =--step |failed= and =--tail =."
Parity.org, after build log follow (until it ends):
| build failed step, duration | yes | yes | no |
| build log one step | yes | yes | no |
| build log tail | yes | no | no |
07-CI-and-Supply-Chain.org, lifecycle step 5, replace
"=runner done success|failure= sets the status," with "=runner
done success|failure [--step ] [--reason ]= records where
a failed build stopped, sets the status,".
- Step 3: Verify, commit, MR
Run: go build ./... && go vet ./... && go test ./cmd/gitbay-runner ./cmd/gitbay ./internal/control ./internal/store ./internal/httpd ./internal/web -count=1
Expected: PASS.
git add e2e/ci_test.go .gitbay/wiki
git commit -S -m "wiki: failed step, build log --step and --tail
Closes #266"
git push -u origin build-failure-report
gitbay mr create --source build-failure-report --target main --title "builds: name the failed step and duration; jump to failure"
Deploy gitbayd (schema 64→65 on the first start, or whatever the
number is after renumbering), then validate per runbook R4, then merge
with --strategy ff and delete the branch both places.
Open questions
- pasta on bay1. The plan relies on
--network pasta:--no-map-gwremoving the host-loopback path and on a pasta container reaching the host's public address. The code comment atmain.go:465-470says pasta exposes the host at169.254.1.2as its--map-host-loopbackdefault; newer podman instead passes--map-guest-addr 169.254.1.2, which maps to the host's public address. Which one bay1's podman does is not in the repository. Runbook R3 step 0 measures it before Part 3 deploys; if--no-map-gwis refused or leaves127.0.0.1reachable, stop and revisit Part 3 before merging. The nftables table does not cover this path: a build's connection to 127.0.0.1:22 through pasta is ci-runner's, like the runner's own poll.
Operator runbook (cmc)
Everything here runs from the laptop against bay1. One forge write per
Bash call; after make deploy the CLI's control master is gone, so do
not poll with several ssh calls a tick. Operator ssh is
ssh -p 2222 root@gitbay.org.
R1. Scratch repository and scoped runner (once, before Part 1's runner deploy)
-
Create the scratch repository and its fork, and attach the bay1 runner key to the scratch repository only:
gitbay repo create cmc/ci-scratch --private gitbay repo fork cmc/ci-scratch --name ci-scratch-fork ssh -p 2222 root@gitbay.org cat /var/lib/gitbay-runner/.ssh/id_ed25519.pub > /tmp/ci-runner.pub gitbay repo runner add cmc/ci-scratch < /tmp/ci-runner.pub -
Scope the service to it with a second drop-in that sorts after
override.conf. Copy the currentExecStartfromdeploy/gitbay-runner.override.confand add-repos cmc/ci-scratch:ssh -p 2222 root@gitbay.org 'cat > /etc/systemd/system/gitbay-runner.service.d/zz-scratch.conf' <<'EOF' [Service] ExecStart= ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -isolation podman -image localhost/gitbay-ci:2 -cpus 3 -memory 6g -untrusted -repos cmc/ci-scratch EOFmake deploy-runnerreloads and restarts the unit, so the scopedExecStartis live for each validation below. While it is in place builds of real repositories queue and wait. -
After each validation: remove the drop-in and restart.
ssh -p 2222 root@gitbay.org 'rm /etc/systemd/system/gitbay-runner.service.d/zz-scratch.conf && systemctl daemon-reload && systemctl restart gitbay-runner'
R2. Part 1 (#255): validate, then discard the old homes
-
make deployfrom the Part 1 branch;curl -s https://gitbay.org/healthznames its commit. Install the R1 drop-in, thenmake deploy-runner. -
On
cmc/ci-scratchmain, push a.gitbay/ci.yml:jobs: home: steps: - echo "HOME=$HOME" - test ! -e "$HOME/poison" || { echo "POISONED"; exit 1; } - touch "$HOME/trusted-marker"The build passes; its log shows
HOME=/var/lib/gitbay-runner/work/trusted-home/cmc/ci-scratch. -
In
cmc/ci-scratch-fork, change the step list toecho "HOME=$HOME",ls -a "$HOME",touch "$HOME/poison", push a branch and open an MR intocmc/ci-scratch. The untrusted build's log showsHOME=/var/lib/gitbay-runner/work/build-<id>-homeand an empty listing (notrusted-marker). -
On bay1:
ls /var/lib/gitbay-runner/workshows nobuild-<id>-homeleft behind. -
gitbay build trigger cmc/ci-scratch home: passes (noPOISONED). -
Remove the R1 drop-in (R1 step 3). Merge Part 1.
-
Discard the homes that trusted and untrusted builds shared:
ssh -p 2222 root@gitbay.org 'chmod -R u+w /var/lib/gitbay-runner/work/home && rm -rf /var/lib/gitbay-runner/work/home'The laptop runner (
~/Library/Caches/gitbay-runner/homeor its configured workdir) gets the same once its brew bottle carries Part 1. -
Record: the release's CHANGELOG upgrade note says to deploy gitbayd before runners, and that
<workdir>/homecan be deleted after the runner upgrade. Nothing further in the wiki; Part 1's MR updated it.
R3. Part 3 (#260): pasta check, egress rule, measurement from a build
No throttling test runs on production; Task 3.3's unit test covers the limiter. What is measured here is what a build sees.
-
Before merging Part 3, on bay1: re-run the host setup (idempotent; it now installs nftables), then check pasta as the runner user:
ssh -p 2222 root@gitbay.org 'sh -s' < deploy/runner-podman-setup.sh ssh -p 2222 root@gitbay.org "podman --version; pasta --version | head -1; nft --version; hostname -I" ssh -p 2222 root@gitbay.org "su - ci-runner -s /bin/sh -c 'podman --cgroup-manager=cgroupfs run --rm --pull=never --network pasta:--no-map-gw --entrypoint sh localhost/gitbay-ci:2 -c \"getent hosts proxy.golang.org; timeout 5 bash -c \\\"exec 3<>/dev/tcp/gitbay.org/22\\\" && echo public-ok; cat /proc/net/route\"'"Expected:
proxy.golang.orgresolves,public-okprints, andhostname -Ilists the public IPv4 address first (the check script takes the first). If podman rejects the option, stop (open question 1). -
make deployfrom the Part 3 branch, the R1 drop-in, thenmake deploy-runner. Its output shows==> loading the egress ruleand thenegress for ci-runner: 127.0.0.1:22 and <public>:22 open, 2222 refusedbefore the runner restarts. If the check fails, follow Task 3.4 Step 6 (remove the table, fix, deploy again) before anything else: the running runner is under the new table. -
On
cmc/ci-scratchmain, a probe job (keep each step under 4096 bytes):jobs: probe: steps: - echo "GITBAY_SSH=$GITBAY_SSH" - | host=${GITBAY_SSH#*@}; host=${host%:*} for t in 127.0.0.1:22 127.0.0.1:2222 169.254.1.2:22 $host:22 $host:80 $host:443 $host:2222 proxy.golang.org:443; do timeout 5 bash -c "exec 3<>/dev/tcp/${t%:*}/${t##*:}" 2>/dev/null && echo "open $t" || echo "closed $t" done - bash -c 'exec 3<>/dev/tcp/${GITBAY_SSH#*@}/22; sleep 90' -
While the last step holds its connection open, on bay1:
ssh -p 2222 root@gitbay.org "ss -tn state established '( sport = :22 )'"Note the peer address of the build's connection and of the runner's
runner logsession. Expected: the runner's is127.0.0.1, the build's is the host's public address, never127.0.0.1. -
Expected build log:
GITBAY_SSH=git@gitbay.org;closed 127.0.0.1:22,closed 127.0.0.1:2222;169.254.1.2:22as measured (open only if podman maps that address to the public one);open gitbay.org:22,:80,:443;closed gitbay.org:2222;open proxy.golang.org:443. -
The runner kept polling:
gitbay admin runnersshows a recent poll for the bay1 key, and the probe build finished with its log. -
Remove the R1 drop-in. Trigger one real trusted job that talks back (
gitbay build trigger krz/orgo <its release or pages job>only if one is due; otherwise wait for the next hutch/orgo scheduled job) and check it reachedgit@gitbay.org. -
Record the result, one commit on a branch
wiki-260-resultswithCloses #260: in CI.org under "What a build can reach", a dated paragraph with the podman, pasta and nft versions, the source address the forge saw for a build and for the runner (step 3), and the reachability list from step 4. InArchitecture/10-Known-Gaps.org, remove the#260row and answer "What can a build reach on the host's network?" with the date and a pointer to the CI page.09-Controlsstayspartial(outbound is open by decision). MR,--strategy ff, delete the branch.
R4. Part 4 (#266): validate the failure report
-
make deployfrom the Part 4 branch (migration 0065 runs on start), the R1 drop-in,make deploy-runner. -
On
cmc/ci-scratchmain:jobs: fail: steps: - echo one - echo two - echo about to fail; exit 7 -
Expected: the log ends
step 3/3 failed: exit 7;gitbay build show cmc/ci-scratch <n>printsfailed step 3/3 echo about to fail; exit 7 (exit 7)and aduration;gitbay build log cmc/ci-scratch <n> --step failedprintsabout to failand the failure line; the build page opens step 3 and "Jump to failure" scrolls to it; at phone width the log wraps with no sideways scroll. -
Remove the R1 drop-in; merge Part 4. Once all four parts are in, delete
cmc/ci-scratchand its fork, or keep them as the standing scratch pair for the next runner change.
Self-review
- Coverage. #255: explicit trust (1.1), disposable untrusted home
and trusted-only caches (1.2), the discard (R2.7), wiki (1.3). #258:
ci/refused (2.1), reuse by trust and declared image, the default image's limit documented (2.2, 2.4), required contexts turning the gate on, shown bysettings showand the web page, missing as pending inMergeGates(2.3), wiki (2.4). #260: public destination with the port off 22 (3.1), builds off loopback (3.2), the limiter's lockout as a unit test instead of a production test (3.3), the host egress table with its unit, check and deploy wiring (3.4), egress policy in Threat-Model, CI and Admin (3.5), the source address and reachability measured from a scratch build and recorded on the CI page (R3), with the scratch-repository rule (R1). #266: runner names the step (4.3), stored step and duration (4.1; duration derived),build show(4.4),build log --step/--tail(4.4), web<details>per step with the failed one open,id="failed", "Jump to failure", duration beside finished (4.5),pre.buildlogwraps (4.5). - Placeholders. Every code step carries the code. The one reference to "the number after renumbering" is the migration rule, not a gap.
- Names across tasks.
buildHome(1.2) is used byrunin 1.2 and 4.3.job.Trusted(1.2),job.SSH(3.2).buildSSH(public string)replacesbuildSSH()in 3.2 and the call site changes there.failure/exitReason/doneArgs(4.3).SetBuildFailure(4.1) is used in 4.2 and 4.4's test fixture.SplitBuildLog,FailedSection,LogSection(4.4) are used bylogSteps(4.5).SuccessBuildForTreegainsimagein 2.2 and its one caller changes there.GatesOut.ChecksMissing(2.3) is rendered inmr.html(2.3).publicSSH(3.1) fills the claim'sssh, read asjob.SSH(3.2). The table nameinet gitbay_runnerand the path/etc/gitbay-runner/egress.nftmatch across the rule, the unit, the check script and the Makefile (3.4).