docs/plans/2026-09-27-ci-trust-and-build-reporting.md
3870 lines · 147357 bytes
34 symbols in this file
CI trust and build reporting implementation planGlobal ConstraintsDecisionsOrder and dependenciesFile mapPart 1: disposable home for untrusted builds (branch `ci-untrusted-home`, #255)Task 1.1: the claim says whether a build is trustedTask 1.2: the runner's build home follows trustTask 1.3: wiki, and the MRPart 2: `ci/` statuses, trusted reuse, required contexts (branch `ci-status-trust`, #258)Task 2.1: `status set` refuses `ci/`Task 2.2: reuse only trusted results on the same imageTask 2.3: required contextsTask 2.4: wiki, and the MRPart 3: separate the runner's source address from its builds, and limit what builds reach on its host (branch `runner-source-address`, #260)Task 3.1: the claim carries the instance's public ssh destinationTask 3.2: a loopback runner's builds get no host loopbackTask 3.3: the limiter's behaviour, by registration modeTask 3.4: host egress rule for the runner's uidTask 3.5: egress policy in the wiki, and the MRPart 4: failed step and duration (branch `build-failure-report`, #266)Task 4.1: store the failed stepTask 4.2: `runner done --step --reason`Task 4.3: the runner names the failed stepTask 4.4: `build show`, `build log --step/--tail`Task 4.5: the build page folds by stepTask 4.6: e2e, wiki, and the MROpen questionsOperator runbook (cmc)R1. Scratch repository and scoped runner (once, before Part 1's runner deploy)R2. Part 1 (#255): validate, then discard the old homesR3. Part 3 (#260): pasta check, egress rule, measurement from a buildR4. Part 4 (#266): validate the failure reportSelf-review
1# CI trust and build reporting implementation plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Untrusted builds get a disposable home and never feed a
6trusted build (#255, #258); `ci/*` statuses belong to the build
7subsystem and merges can wait on named contexts (#258); a build can no
8longer share the runner's source address, and reaches no port on the
9runner's host but the forge's public 22, 80 and 443 (#260); a failed
10build names its step, exit and duration on the CLI and the web (#266).
11
12**Architecture:** The claim payload gains an explicit `trusted` flag and
13the instance's public ssh destination (with the port when it is not
1422). The runner picks the build home by trust (persistent per
15repository for trusted builds, fresh and removed for untrusted ones),
16keeps a loopback runner's builds off the host's loopback, and reports
17the failed step on `runner done`. An nftables table on the runner host,
18loaded by a oneshot unit the runner service requires, limits what the
19runner's uid may reach on the host itself. The server refuses `ci/`
20contexts in `status set`, restricts tree and commit reuse to trusted
21builds on the same declared image, adds a `required_contexts`
22repository setting that turns `require_checks` on and that `MergeGates`
23treats as pending until reported, stores the failed step and reason on
24the build, and cuts the log at its `$ <step>` lines for `build log
25--step` and the build page.
26
27**Tech Stack:** Go, SQLite (hand-written SQL), `html/template`, rootless
28podman with pasta, nftables, systemd.
29
30**Spec:** the issues themselves: krz/gitbay#255, #258, #260, #266 (texts
31in the session's `issues.txt`), plus the decisions recorded under
32"Decisions" below.
33
34## Global Constraints
35
36- Each MR on its own branch off `main`. Commits are signed (the repo
37 refuses unsigned), messages reference issues (`Ref #N`, and
38 `Closes #N` on the commit that finishes one). No attribution to any
39 assistant, model or AI anywhere: commits, MR bodies, comments. No
40 `Co-Authored-By` trailer.
41- MR: `gitbay mr create --source <branch> --target main --title "..."`;
42 merge with `gitbay mr merge <n> --strategy ff` once CI is green, then
43 delete the branch locally and on the remote. Behind main → rebase,
44 force-push, merge again.
45- Locally: `go build ./...`, `go vet ./...`, unit tests of touched
46 packages, and at most the one e2e test being written
47 (`go test ./e2e -run TestName -count=1`). CI on bay1 runs the full suite.
48- Registries that fail CI when a new thing lacks its row: top-level route
49 word in `internal/policy/names.go`; new page template in the width map
50 of `TestMainWidthClass` (`internal/web/web_test.go`); new `ReadOnly`
51 command in `readArgs` in `e2e/readonly_test.go`; new control command
52 needs a `pass()` entry in `cmd/gitbay/main.go` (coverage test);
53 a command reading stdin needs `ReadsStdin: true`. A new or changed
54 command `Summary` needs `go test ./cmd/gitbay -run TestSummariesAreCurrent -update`.
55- New migrations: the highest today is 0059. Six plans are written in
56 parallel, so numbers are pre-assigned: plan 1 uses 0060–0064, plan 2
57 (this one) 0065–0068, plan 3 0069–0071, plan 4 0072–0074, plan 5
58 0075–0077, plan 6 0078–0079. Whoever lands second renumbers to the
59 next free number at execution time. Migrations come in
60 `.up.sql`/`.down.sql` pairs. Hand-written SQL, no ORM. This plan uses
61 one: 0065.
62- Secrets travel on stdin, never argv; never logged or echoed.
63- Wiki pages live in `.gitbay/wiki/` (Parity, API, Admin, Threat-Model,
64 CI, Users, Performance, and the `Architecture/` folder with its
65 Known-Gaps table and controls matrix). Update the page in the same MR
66 that changes the behaviour it describes, and close the matching
67 Known-Gaps row.
68- Writing style: plain, direct, no hype; code comments match the
69 surrounding density. Comments and docs state facts, never
70 before/after narration.
71- **Deploy order for Parts 1, 3 and 4: `make deploy` (gitbayd) before
72 `make deploy-runner`.** A new runner reads `trusted` and `ssh` from the
73 claim and sends `--step`/`--reason` to `runner done`; an older server
74 omits the first two (the runner then treats every build as untrusted:
75 no secrets, disposable home) and refuses the flags with exit 2 (the
76 build stays running until the reaper fails it). An older runner
77 against a newer server works unchanged. The laptop runner is a brew
78 bottle built from a release tag, so it always trails the server.
79- **Runner changes are validated on a scratch repository before the
80 bay1 runner touches real repositories** (CLAUDE.md: every deploy that
81 skipped this took CI down). The procedure is in the runbook at the end
82 of this plan; Parts 1, 3 and 4 each have a validation section there.
83- `cmd/gitbay-runner` tests run on macOS and Linux; nothing here may
84 need podman to pass (`e2e/isolation_podman_test.go` is the only podman
85 test and skips without it).
86
87## Decisions
88
89- **#255.** The claim carries `"trusted": true|false` with no
90 `omitempty`; a runner that finds the field absent treats the build as
91 untrusted. Trusted homes move from `<workdir>/home/<owner>/<name>` to
92 `<workdir>/trusted-home/<owner>/<name>`, so a home written before this
93 change — which untrusted builds could write — is never read again,
94 even before the operator deletes it. An untrusted build's home is
95 `<workdir>/build-<id>-home`, created with `os.Mkdir` (so it is new and
96 empty) and removed after the build with a helper that first makes
97 every directory writable (the Go module cache leaves them 0555). No
98 persistent cache for untrusted builds: a fork's build downloads its
99 modules each time. The runner also drops secrets for an untrusted
100 build, though the server never sends any.
101- **#258.** `status set` refuses any context starting with `ci/`,
102 case-folded, with exit 4, before resolving the repository. Tree reuse
103 (`SuccessBuildForTree`) and the cancelled-build fallback
104 (`SuccessBuildFor`) consider trusted builds only, and tree reuse also
105 requires the same `image:` as the job declares. The same-commit dedupe
106 in `queueJobs` lets an untrusted build stand only for another
107 untrusted queue: a fork head that lands on a branch by fast-forward is
108 built again as trusted. `required_contexts` is a list in the
109 repository's settings JSON (no migration), set by
110 `repo settings require-contexts <owner/name> [<context>...]`. Setting
111 a non-empty list also turns `require_checks` on, in the same settings
112 update; setting it empty clears the list and leaves `require_checks`
113 as it was. `require-checks off` keeps the list, which then does
114 nothing until the gate is on again. `repo settings show` prints
115 `require checks` and `required contexts` side by side (today it prints
116 neither), and on the web settings page the required-checks box is
117 ticked after contexts are saved and its hint lists them. The gate
118 itself only ever reads `require_checks`. A missing required context
119 makes the combined check `pending` and appears as `<context>=missing`
120 in the unmet sentence and in `checks_missing`.
121- **#258, default image.** Tree reuse keys on the job's declared
122 `image:`. A job naming none is stored with `image = ''` and matches
123 other such builds whatever the runner defaulted to; bumping a runner's
124 `-image` does not invalidate them. This is documented on the CI page
125 rather than fixed. Reuse is decided at queue time in `queueJobs`,
126 before any runner is chosen, and the default image belongs to
127 whichever runner claims the build: bay1 and the laptop runner already
128 have different defaults. Reporting the resolved image on claim or
129 `runner done` would record it after the fact, but a queue-time
130 comparison would still have nothing to compare against, so jobs
131 without an image would never be reused at all. The remedy is on the
132 repository's side (name the image in `ci.yml`) or the operator's
133 (`build trigger`, which never reuses, after bumping `-image`).
134- **#260.** Reading the code: the bay1 runner polls `git@127.0.0.1`
135 (`deploy/gitbay-runner.override.conf:78`); `buildSSH`
136 (`cmd/gitbay-runner/main.go:471-486`) sends podman builds to
137 `169.254.1.2`, and pasta's default gateway mapping lets a build reach
138 the host's loopback, where its connections arrive from `127.0.0.1`.
139 The limiter keys on the remote IP (`internal/sshd/ratelimit.go:86`,
140 used at `internal/sshd/sshd.go:122`). "Runner on the public address"
141 does not separate anything: a build can connect to the public address
142 too, and would then share the runner's source there instead. So the
143 runner stays on loopback and its builds lose loopback: under podman,
144 when the runner's remote is loopback, containers run with
145 `--network pasta:--no-map-gw`, and `GITBAY_SSH` is the instance's
146 public destination, which the server sends in the claim as `ssh`. A
147 build then reaches the host only as an internet client does.
148- **#260, `GITBAY_SSH` form.** `git@<site host>` when `[ssh] port` is 22
149 (or unset, which config validation treats as 22), and
150 `git@<site host>:<port>` otherwise, built with `net.JoinHostPort` so
151 an IPv6 literal is bracketed. hutch and orgo build
152 `ssh://$GITBAY_SSH/<owner>/<name>.git`, which is a valid URL in both
153 forms, so nothing changes for them on 22 and they work unchanged on
154 another port. A script that runs a command uses `ssh
155 ssh://$GITBAY_SSH …`, which OpenSSH accepts with or without the port;
156 the Users page says so. The port is the daemon's `[ssh] port`, the
157 one it listens on; an instance behind a port-mapping NAT is not
158 modelled.
159- **#260, host egress.** Under rootless podman with pasta, a build's
160 connections are made by pasta on the host from sockets owned by the
161 runner's uid (`ci-runner`), the same uid the runner's own ssh runs as.
162 An nftables table (`deploy/gitbay-runner-egress.nft`, loaded by
163 `gitbay-runner-egress.service`, which `gitbay-runner.service`
164 requires) matches output packets with `meta skuid "ci-runner"` that
165 leave through `lo` — every packet to one of the host's own addresses,
166 loopback or public, does — and allows only 127.0.0.1:22 (the runner's
167 poll, clone and log stream), port 53 on loopback (the host resolver
168 pasta forwards a build's DNS to), and 22, 80 and 443 on the public
169 addresses. Everything else on the host is rejected: the admin sshd on
170 2222 on every address, and every service bound to loopback. Traffic
171 to other hosts is not matched, so outbound internet stays open (a
172 fork's merge request to a Go repository must fetch its modules). The
173 rule applies to all builds, trusted and untrusted, and to
174 `-isolation none` builds too, since they run as the same uid. uid
175 alone cannot tell a build from its runner, so 127.0.0.1:22 stays open
176 to the uid; `--no-map-gw` is what keeps builds off loopback. The two
177 are separate layers and both ship. The runner does not start without
178 the rule (`Requires=`), following `runner-podman-setup.sh`'s rule that
179 a host that is not ready fails rather than runs builds unconfined;
180 `make deploy-runner` loads the rule and checks, as `ci-runner`, that
181 127.0.0.1:22 answers and 2222 does not, before restarting the runner.
182 The laptop runner (macOS, brew) is not covered.
183- **Finding for #260.** On gitbay.org the limiter's failure count is
184 unreachable by an unknown key: `authenticate` admits an unknown key as
185 an anonymous `register` session whenever `registration.mode` is not
186 `closed` (`internal/sshd/sshd.go:139-144`), and `fail` is only called
187 on the closed path (`sshd.go:145`). With registration closed,
188 `authenticate` checks `allow` before it looks at the key
189 (`sshd.go:123-129`), so once an address has `ssh_auth_rate` failures
190 in the window every key from it is refused, the runner's included, and
191 `success` (`sshd.go:149`) is never reached to clear the count; below
192 the limit a success clears it. A unit test in `internal/sshd` records
193 both modes (Task 3.3). No throttling test runs on production; the
194 runbook measures, from inside a scratch build, the source address the
195 forge sees and that 2222 and 127.0.0.1 are unreachable, and #260
196 closes when that is recorded on the CI wiki page.
197- **#266.** Duration is not stored: `Build.Elapsed()`
198 (`internal/store/builds.go:420`) already derives it from `started_at`
199 and `finished_at`. Migration 0065 adds `failed_step` (1-based, 0 for
200 "no step": success, or a failure before the first step) and
201 `failed_reason` (one line, at most 200 bytes). The runner writes
202 `step 3/3 failed: exit 1` and reports `runner done <id> failure --step
203 3 --reason 'exit 1'`; the reason is `exit <code>` for a command that
204 exited and the error text otherwise (`build timed out after 45m0s`,
205 `cancelled`, `git clone: exit 128`). The log format is otherwise
206 unchanged: sections are cut at the `$ <step>` line the runner already
207 writes before each step (`isolate.go:88`, `:186`), matched against the
208 build's own `steps` in order and only at a line start, so logs of
209 builds that ran before this change fold too. `build log --step`
210 takes `0` (setup, before the first step), a step number, or `failed`.
211 An invalid `--step` on `runner done` is recorded as 0 rather than
212 refused, so a runner/server mismatch never loses an outcome.
213
214## Order and dependencies
215
216| # | Branch | Closes | Migration | Needs |
217|---|---|---|---|---|
218| 1 | `ci-untrusted-home` | #255 | — | — |
219| 2 | `ci-status-trust` | #258 | — | — |
220| 3 | `runner-source-address` | Ref #260 (closed by the runbook result commit on the CI page) | — | Part 1 merged (both edit `runRunnerNext`'s payload and `stepEnv`) |
221| 4 | `build-failure-report` | #266 | 0065 | Part 1 merged (both change `run()`); Part 3 merged (both change `runStepsPodman`) |
222
223#255 goes first. Parts 1–4 land and deploy in order; each runner deploy
224follows the scratch validation in the runbook.
225
226Other plans (all `docs/plans/2026-09-27-*.md`):
227
228- Plan 4 (data-at-rest-and-backup, #273) encrypts `build_secrets`; if it
229 changes `Store.BuildSecrets`, Part 1's edit to `runRunnerNext`
230 (`internal/control/build.go:543-564`) conflicts textually. Whoever
231 lands second rebases; no behavioural dependency.
232- Plan 3 (server-hardening, #275) audits refused mutating commands; the
233 `status set` refusal from Part 2 is one of them and needs nothing
234 from this plan.
235- Plan 5 (web-ux, #261) covers documentation drift. Two items seen here
236 and left alone: `deploy/gitbay-runner.override.conf:27-30` names
237 `cmc/ci-smoke`, which no longer exists; `cmd/gitbay-runner/main.go:513`
238 repeats its comment line. Part 3 adds a `[Unit]` section at the top of
239 the same drop-in; if plan 5 edits its comment, whoever lands second
240 rebases.
241- Plan 1 (credentials-and-sessions, #256) closes a removed key's
242 connections; the runner's `runner log` session is one such
243 connection, and no code here depends on it. Plan 1's key expiry
244 (#277) may add a check to `authenticate`; Part 3's sshd test uses an
245 unexpiring key and asserts only the limiter's behaviour, so it holds
246 either way.
247
248## File map
249
250| File | Part | Responsibility |
251|---|---|---|
252| `internal/control/build.go` | 1, 3, 4 | claim payload `trusted`, `ssh`; `runner done` flags; `build show` fields; `build log --step/--tail`; `queueJobs` trust rule (2) |
253| `internal/control/buildlog.go` (create) | 4 | `LogSection`, `SplitBuildLog`, `FailedSection`, `tailLines` |
254| `internal/control/status.go` | 2 | `ci/` refusal |
255| `internal/control/mr.go`, `output.go` | 2 | `require-contexts`, `MergeGates`, `GatesOut.ChecksMissing` |
256| `internal/control/repo.go` | 2 | `repo settings show` prints require checks and required contexts |
257| `internal/store/builds.go` | 2, 4 | trust and image on reuse; failed step columns |
258| `internal/store/repos.go` | 2 | `RepoSettings.RequiredContexts` |
259| `internal/store/migrations/0065_build_failure.{up,down}.sql` | 4 | columns |
260| `cmd/gitbay-runner/main.go` | 1, 3, 4 | `job` fields, `buildHome`, `removeTree`, `stepEnv`, `loopbackRemote`, `buildSSH`, `buildNetwork`, `failure`, `exitReason` |
261| `cmd/gitbay-runner/isolate.go` | 3, 4 | network flag; `runSteps` returns `*failure` |
262| `cmd/gitbay-runner/report.go` | 4 | `doneArgs` |
263| `cmd/gitbay/main.go`, `summaries_gen.go` | 2 | `require-contexts` pass-through |
264| `internal/httpd/builds.go`, `settings.go` | 2, 4 | step view; settings form mapping |
265| `internal/web/templates/build.html`, `mr.html`, `settings.html` | 2, 4 | steps, gates row, form |
266| `internal/web/static/style.css` | 4 | `pre.buildlog` wraps; step folds |
267| `e2e/readonly_test.go`, `e2e/mrweb_test.go`, `e2e/status_test.go`, `e2e/settingsweb_test.go`, `e2e/ci_test.go` | 2, 4 | contexts off `ci/`; refusal; settings; failed step |
268| `internal/sshd/sshd_test.go` | 3 | limiter behaviour by registration mode |
269| `deploy/gitbay-runner-egress.nft` (create), `deploy/gitbay-runner-egress.service` (create), `deploy/runner-egress-check.sh` (create) | 3 | host egress rule, its unit, the post-load check |
270| `deploy/gitbay-runner.override.conf`, `deploy/runner-podman-setup.sh`, `Makefile` | 3 | runner requires the rule; nftables installed; `deploy-runner` ships, loads and checks it |
271| `.gitbay/wiki/…` | all | as listed per task |
272
273---
274
275# Part 1: disposable home for untrusted builds (branch `ci-untrusted-home`, #255)
276
277### Task 1.1: the claim says whether a build is trusted
278
279**Files:**
280- Modify: `internal/control/build.go:554-564` (the claim payload in `runRunnerNext`)
281- Test: `internal/control/runnernext_test.go` (append)
282
283**Interfaces:**
284- Produces: the `runner next --json` payload gains `"trusted": <bool>`, always present.
285
286- [ ] **Step 1: Write the failing test**
287
288Append to `internal/control/runnernext_test.go`:
289
290```go
291// The claim says whether a build is trusted in so many words. A runner
292// must not infer it from secrets being absent: a trusted repository with
293// no secrets looks the same (#255).
294func TestRunnerNextSaysWhetherTrusted(t *testing.T) {
295 st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
296 for _, trusted := range []bool{true, false} {
297 if _, err := st.CreateBuild(repo.ID, "unit", baseSHA, "main", "[]", "", "", trusted); err != nil {
298 t.Fatal(err)
299 }
300 c, out := runnerCtx(st, uid, root)
301 c.JSON = true
302 if code := runRunnerNext(c, []string{"--untrusted"}); code != protocol.ExitOK {
303 t.Fatalf("runner next: exit %d, output:\n%s", code, out.String())
304 }
305 want := fmt.Sprintf(`"trusted":%v`, trusted)
306 if !strings.Contains(out.String(), want) {
307 t.Fatalf("claim of a trusted=%v build lacks %s:\n%s", trusted, want, out.String())
308 }
309 }
310}
311```
312
313The first loop creates and claims the trusted build; the second creates
314the untrusted one, which is then the only pending build.
315
316- [ ] **Step 2: Run it and see it fail**
317
318Run: `go test ./internal/control -run TestRunnerNextSaysWhetherTrusted -count=1`
319Expected: FAIL, `claim of a trusted=true build lacks "trusted":true`.
320
321- [ ] **Step 3: Implement**
322
323Replace the payload at `internal/control/build.go:554-564` with:
324
325```go
326 d := struct {
327 ID int64 `json:"id"`
328 Repo string `json:"repo"`
329 Number int64 `json:"number"`
330 Job string `json:"job"`
331 SHA string `json:"sha"`
332 Ref string `json:"ref"`
333 Steps []string `json:"steps"`
334 Image string `json:"image,omitempty"`
335 // Trusted is always sent: a runner decides a build's home and
336 // secrets from it, and reads a missing field as untrusted (#255).
337 Trusted bool `json:"trusted"`
338 Secrets map[string]string `json:"secrets,omitempty"`
339 }{ID: b.ID, Repo: repo.Path(), Number: b.Number, Job: b.Job, SHA: b.SHA, Ref: b.Ref,
340 Steps: steps, Image: b.Image, Trusted: b.Trusted, Secrets: secrets}
341```
342
343- [ ] **Step 4: Run it and see it pass**
344
345Run: `go test ./internal/control -run 'TestRunnerNext' -count=1`
346Expected: PASS.
347
348- [ ] **Step 5: Commit**
349
350```bash
351git add internal/control/build.go internal/control/runnernext_test.go
352git commit -S -m "runner next: say whether the build is trusted
353
354Ref #255"
355```
356
357### Task 1.2: the runner's build home follows trust
358
359**Files:**
360- Modify: `cmd/gitbay-runner/main.go:12-30` (imports), `:32-42` (`job`), `:315-330` (`run`), `:437-463` (comment and `buildHomeFor`), `:488-511` (`stepEnv`)
361- Modify: `cmd/gitbay-runner/home_test.go` (rewrite), `cmd/gitbay-runner/env_test.go:47-60`
362
363**Interfaces:**
364- Consumes: the `trusted` claim field from Task 1.1.
365- Produces:
366 - `job.Trusted bool` (`json:"trusted"`)
367 - `func buildHome(workdir string, j job) (string, func(), error)` — the home and a cleanup to defer; replaces `buildHomeFor`.
368 - `func removeTree(dir string) error`
369
370- [ ] **Step 1: Write the failing tests**
371
372Replace `cmd/gitbay-runner/home_test.go` with:
373
374```go
375package main
376
377import (
378 "os"
379 "path/filepath"
380 "strings"
381 "testing"
382)
383
384// A trusted build's home is its repository's, kept between builds so
385// tool caches survive: the same repository gets the same directory back,
386// another repository a different one (#184).
387func TestTrustedHomeIsPerRepositoryAndKept(t *testing.T) {
388 work := t.TempDir()
389 a, done, err := buildHome(work, job{ID: 1, Repo: "alice/app", Trusted: true})
390 if err != nil {
391 t.Fatal(err)
392 }
393 done()
394 if _, err := os.Stat(a); err != nil {
395 t.Fatalf("trusted home removed after its build: %v", err)
396 }
397 b, done, err := buildHome(work, job{ID: 2, Repo: "bob/app", Trusted: true})
398 if err != nil {
399 t.Fatal(err)
400 }
401 done()
402 if a == b {
403 t.Fatalf("two repositories share a build home: %s", a)
404 }
405 again, done, _ := buildHome(work, job{ID: 3, Repo: "alice/app", Trusted: true})
406 done()
407 if again != a {
408 t.Fatalf("build home moved between builds: %s then %s", a, again)
409 }
410 for _, dir := range []string{a, b} {
411 rel, err := filepath.Rel(filepath.Join(work, "trusted-home"), dir)
412 if err != nil || rel == "." || strings.HasPrefix(rel, "..") {
413 t.Fatalf("build home %s is not under %s/trusted-home", dir, work)
414 }
415 st, err := os.Stat(dir)
416 if err != nil {
417 t.Fatal(err)
418 }
419 if st.Mode().Perm() != 0o700 {
420 t.Fatalf("build home mode %o, want 0700", st.Mode().Perm())
421 }
422 }
423}
424
425// An untrusted build gets a home of its own, outside the trusted root,
426// removed when the build ends: nothing a fork's build writes reaches a
427// later build of the repository (#255).
428func TestUntrustedHomeIsDisposable(t *testing.T) {
429 work := t.TempDir()
430 trusted, done, err := buildHome(work, job{ID: 1, Repo: "alice/app", Trusted: true})
431 if err != nil {
432 t.Fatal(err)
433 }
434 done()
435 home, done, err := buildHome(work, job{ID: 2, Repo: "alice/app"})
436 if err != nil {
437 t.Fatal(err)
438 }
439 if home == trusted || strings.HasPrefix(home, filepath.Join(work, "trusted-home")) {
440 t.Fatalf("untrusted build got a trusted home: %s", home)
441 }
442 // What the Go module cache leaves behind: read-only directories.
443 cache := filepath.Join(home, "go", "pkg", "mod", "example.com", "m@v1")
444 if err := os.MkdirAll(cache, 0o755); err != nil {
445 t.Fatal(err)
446 }
447 if err := os.WriteFile(filepath.Join(cache, "go.mod"), []byte("module m\n"), 0o444); err != nil {
448 t.Fatal(err)
449 }
450 os.Chmod(cache, 0o555)
451 os.Chmod(filepath.Dir(cache), 0o555)
452 done()
453 if _, err := os.Stat(home); !os.IsNotExist(err) {
454 t.Fatalf("untrusted home left behind: %v", err)
455 }
456}
457
458// A repository path is server-validated, but a trusted home must still
459// never resolve outside the runner's home root.
460func TestBuildHomeRefusesTraversal(t *testing.T) {
461 if _, _, err := buildHome(t.TempDir(), job{Repo: "../../etc", Trusted: true}); err == nil {
462 t.Fatal("a traversing repository path produced a build home")
463 }
464}
465```
466
467In `cmd/gitbay-runner/env_test.go`, replace `TestStepEnvCarriesSecrets`
468(lines 47-60) with:
469
470```go
471// Secrets reach a trusted build's steps and never an untrusted one's,
472// whatever the claim carried: the trust flag decides, not whether any
473// secrets arrived (#255).
474func TestStepEnvCarriesSecrets(t *testing.T) {
475 secrets := map[string]string{"TOKEN": "s3cret"}
476 env := stepEnv(job{Trusted: true, Secrets: secrets}, "/tmp/buildhome", "git@x.test")
477 if !containsEnv(env, "TOKEN=s3cret") {
478 t.Error("a trusted build's secret did not reach the step")
479 }
480 for _, j := range []job{{}, {Secrets: secrets}} {
481 for _, e := range stepEnv(j, "/tmp/buildhome", "git@x.test") {
482 if strings.HasPrefix(e, "TOKEN=") {
483 t.Errorf("a secret reached an untrusted build: %q", e)
484 }
485 }
486 }
487}
488```
489
490- [ ] **Step 2: Run them and see them fail**
491
492Run: `go test ./cmd/gitbay-runner -count=1`
493Expected: build failure, `undefined: buildHome` and `unknown field Trusted in struct literal of type job`.
494
495- [ ] **Step 3: Implement**
496
497In `cmd/gitbay-runner/main.go`:
498
499Add `"io/fs"` to the imports (between `"io"` and `"log"`).
500
501Add the field to `job` (after `Image`):
502
503```go
504 Image string `json:"image"`
505 // Trusted is false for a merge request head from a fork, and when the
506 // server did not say: such a build gets no secrets and a home of its
507 // own (#255).
508 Trusted bool `json:"trusted"`
509 Secrets map[string]string `json:"secrets"`
510```
511
512Replace lines 315-330 of `run` (the build-home comment and the
513`buildHomeFor` call) with:
514
515```go
516 home, doneHome, err := buildHome(r.workdir, j)
517 if err != nil {
518 log.Printf("build %d: build home: %v", j.ID, err)
519 return false
520 }
521 defer doneHome()
522```
523
524and change line 433 to `env := stepEnv(j, home, r.buildSSH())`.
525
526Replace lines 437-463 (the `stepEnv` comment that sits above
527`buildHomeFor`, and `buildHomeFor`) with `buildHome` and `removeTree`;
528the `stepEnv` comment moves to `stepEnv` in the next block:
529
530```go
531// buildHome is a build's HOME and what to do with it when the build ends.
532//
533// Not the workspace, which is removed after every build: the Go module
534// cache and every other tool cache live under HOME. Not the runner's own
535// home either, where its SSH key and credential dotfiles are.
536//
537// A trusted build gets its repository's home,
538// <workdir>/trusted-home/<owner>/<name>, kept between builds so the
539// caches survive. One per repository: shared across repositories, a step
540// could poison a cache or plant a .gitconfig that another repository's
541// build would honour (#184). The root is not <workdir>/home, where homes
542// that untrusted builds could write were kept before #255, so none of
543// those is read again.
544//
545// An untrusted build gets <workdir>/build-<id>-home, new and empty,
546// removed when the build ends. The container mounts HOME read-write, so
547// a home a fork's build could write is a cache a stranger controls
548// (#255).
549func buildHome(workdir string, j job) (string, func(), error) {
550 if !j.Trusted {
551 dir := filepath.Join(workdir, fmt.Sprintf("build-%d-home", j.ID))
552 if err := os.Mkdir(dir, 0o700); err != nil {
553 return "", nil, err
554 }
555 return dir, func() {
556 if err := removeTree(dir); err != nil {
557 log.Printf("build %d: removing its home: %v", j.ID, err)
558 }
559 }, nil
560 }
561 root := filepath.Join(workdir, "trusted-home")
562 dir := filepath.Join(root, filepath.FromSlash(j.Repo))
563 if rel, err := filepath.Rel(root, dir); err != nil || rel == "." || strings.HasPrefix(rel, "..") {
564 return "", nil, fmt.Errorf("repository path %q escapes the build home root", j.Repo)
565 }
566 if err := os.MkdirAll(dir, 0o700); err != nil {
567 return "", nil, err
568 }
569 return dir, func() {}, nil
570}
571
572// removeTree deletes dir and everything under it. os.RemoveAll alone
573// fails on a directory without write permission, and the Go module cache
574// makes every directory it fills read-only.
575func removeTree(dir string) error {
576 filepath.WalkDir(dir, func(p string, d fs.DirEntry, err error) error {
577 if err == nil && d.IsDir() {
578 os.Chmod(p, 0o700)
579 }
580 return nil
581 })
582 return os.RemoveAll(dir)
583}
584```
585
586Replace `stepEnv` (lines 488-511) with the function and the comment
587that belongs to it:
588
589```go
590// stepEnv builds the environment a build step runs with. It is
591// constructed, not inherited: os.Environ() would hand repository content
592// the runner's entire environment, including anything an operator set on
593// the service (#144).
594//
595// HOME is the build's home (buildHome), not the runner's own: tools read
596// credentials out of dotfiles — .netrc, .npmrc, .gitconfig — and a build
597// has no business finding the runner's.
598//
599// PATH is the one thing carried over: without it a step cannot find the
600// tools the host was provisioned with.
601func stepEnv(j job, home, sshDest string) []string {
602 path := os.Getenv("PATH")
603 if path == "" {
604 path = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
605 }
606 env := []string{
607 "PATH=" + path,
608 "HOME=" + home,
609 "LANG=C.UTF-8",
610 "CI=true",
611 "GITBAY_REPO=" + j.Repo,
612 "GITBAY_SHA=" + j.SHA,
613 "GITBAY_REF=" + j.Ref,
614 "GITBAY_JOB=" + j.Job,
615 "GITBAY_SSH=" + sshDest,
616 }
617 // The server sends secrets only for a trusted build. The claim's
618 // trust flag decides here as well, not whether any arrived (#255).
619 if j.Trusted {
620 for name, value := range j.Secrets {
621 env = append(env, name+"="+value)
622 }
623 }
624 return env
625}
626```
627
628- [ ] **Step 4: Run the tests and see them pass**
629
630Run: `go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner -count=1`
631Expected: PASS. (`TestStepEnvHomeIsNotTheWorkspace` and the other
632`stepEnv` tests pass unchanged.)
633
634- [ ] **Step 5: Commit**
635
636```bash
637git add cmd/gitbay-runner/main.go cmd/gitbay-runner/home_test.go cmd/gitbay-runner/env_test.go
638git commit -S -m "runner: disposable home for untrusted builds
639
640A trusted build keeps its repository's home, now under
641<workdir>/trusted-home; an untrusted build gets a new home removed with
642the build, and no secrets whatever the claim carries.
643
644Ref #255"
645```
646
647### Task 1.3: wiki, and the MR
648
649**Files:**
650- Modify: `.gitbay/wiki/Threat-Model.org:135-147`, `:172-180`
651- Modify: `.gitbay/wiki/Admin.org:640-642`
652- Modify: `.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org:30-33`, `:67`
653- Modify: `.gitbay/wiki/Architecture/09-Controls.org:87`
654- Modify: `.gitbay/wiki/Architecture/04-Trust-Boundaries.org` (TB7 row)
655- Modify: `.gitbay/wiki/Architecture/10-Known-Gaps.org:13` (remove the #255 row)
656
657- [ ] **Step 1: Threat-Model**
658
659In "The CI runner", replace the sentences of the "What a build sees"
660bullet from "=HOME= is a build home" to the end of the bullet with:
661
662```org
663 secrets, and nothing the operator set on the service. =HOME= is a build
664 home under the runner's =-workdir=, not the runner's own home, so a
665 build cannot read the =.netrc=, =.npmrc= or =.gitconfig= where tools
666 keep credentials. A trusted build's home belongs to its repository
667 and persists, so caches survive; an untrusted build's home is new,
668 empty and removed when the build ends, so nothing a fork's build
669 writes is read by a later build (krz/gitbay#255). The claim names a
670 build's trust explicitly, and a runner that finds no trust flag treats
671 the build as untrusted.
672```
673
674Replace the paragraph after the bullets ("Under =-isolation none=,
675anything a step can do…") with:
676
677```org
678Under =-isolation none=, anything a step can do as the runner's user a
679pushed =ci.yml= can do. Under podman a step is confined to its
680container, the bind-mounted workspace and its build home: a trusted
681build's cache is read only by later trusted builds of the same
682repository, and an untrusted build's home is discarded with it. Treat
683the runner host as executing untrusted code all the same: keep it off
684the daemon's host where the database lives, or scope it to repositories
685whose writers you trust. gitbay.org does the latter — its runner builds
686only the repositories the operator names.
687```
688
689- [ ] **Step 2: Admin**
690
691Replace `Admin.org:640-642` ("Each repository gets its own build home…")
692with:
693
694```org
695A trusted build's home is its repository's, under
696=<workdir>/trusted-home/<owner>/<name>=, mounted into its containers as
697=HOME=: caches persist between trusted builds of one repository and are
698never read by another's. An untrusted build — a merge request head from
699a fork — gets =<workdir>/build-<id>-home=, new and empty, removed when
700the build ends. Homes under =<workdir>/home= are from runners before
701krz/gitbay#255, which shared them with untrusted builds; nothing reads
702them any more, and they can be deleted.
703```
704
705- [ ] **Step 3: Architecture pages**
706
707`07-CI-and-Supply-Chain.org`, lifecycle step 2, replace "The claim
708returns id, repository, job, commit, ref, steps, image and — for
709trusted builds only — the repository's secrets (=build.go=)." with
710"The claim returns id, repository, job, commit, ref, steps, image, the
711build's trust, and — for trusted builds only — the repository's secrets
712(=build.go=)."
713
714Replace the Build home row (line 67) with:
715
716```org
717| Build home | trusted: =<workdir>/trusted-home/<owner>/<name>=, one per repository, persistent; untrusted: =<workdir>/build-<id>-home=, removed with the build (=main.go=) |
718```
719
720`09-Controls.org` line 87:
721
722```org
723| Untrusted code runs isolated | in place | rootless podman, cgroup limits; untrusted builds get a disposable home (=cmd/gitbay-runner/main.go=) |
724```
725
726`04-Trust-Boundaries.org` TB7 row, last column:
727
728```org
729| TB7 | Z5 → Z4 container | build steps, workspace, build home | rootless podman, operator-provisioned image, cgroup limits; a trusted build's home is its repository's, an untrusted build's is discarded with it; the network is open (#260) |
730```
731
732`10-Known-Gaps.org`: delete the `#255` row.
733
734- [ ] **Step 4: Verify and commit**
735
736Run: `go build ./... && go vet ./... && go test ./cmd/gitbay-runner ./internal/control -count=1`
737Expected: PASS.
738
739```bash
740git add .gitbay/wiki
741git commit -S -m "wiki: trusted and untrusted build homes
742
743Closes #255"
744```
745
746- [ ] **Step 5: MR**
747
748```bash
749git push -u origin ci-untrusted-home
750gitbay mr create --source ci-untrusted-home --target main --title "runner: disposable home for untrusted builds"
751```
752
753Body (via `--file -` from a file written with the Write tool): what
754changed, the deploy order (gitbayd before the runner), and a pointer to
755runbook sections R1 and R2. After CI is green and the runbook's R2
756validation passed on the scratch repository:
757`gitbay mr merge <n> --strategy ff`, delete the branch both places.
758
759---
760
761# Part 2: `ci/` statuses, trusted reuse, required contexts (branch `ci-status-trust`, #258)
762
763### Task 2.1: `status set` refuses `ci/`
764
765**Files:**
766- Modify: `internal/control/status.go:15-27` (registration), `:68-70` (after the usage check)
767- Create: `internal/control/status_test.go`
768- Modify: `e2e/readonly_test.go:75`, `e2e/mrweb_test.go:275`, `e2e/status_test.go` (after line 49)
769- Modify: `cmd/gitbay/summaries_gen.go` only if the summary changes (it does not here)
770
771**Interfaces:**
772- Produces: `status set … --context ci/…` exits 4 (`protocol.ExitDenied`) with a message containing `reserved`.
773
774- [ ] **Step 1: Write the failing test**
775
776Create `internal/control/status_test.go`:
777
778```go
779package control
780
781import (
782 "strings"
783 "testing"
784
785 "gitbay.org/gitbay/internal/protocol"
786 "gitbay.org/gitbay/internal/store"
787)
788
789// ci/<job> statuses are the build subsystem's. A writer who could post
790// one could mark ci/test green on their own head before, or instead of,
791// the build (#258).
792func TestStatusSetRefusesReservedContext(t *testing.T) {
793 st, repo, uid := newQueueTestRepo(t)
794 for _, ctx := range []string{"ci/test", "CI/test", "ci/"} {
795 c, errOut := pruneCtx(st, t.TempDir(), store.User{ID: uid, Username: "alice"})
796 code := Dispatch(c, []string{"status", "set", repo.Path(), "abc1234", "--context", ctx, "--state", "success"})
797 if code != protocol.ExitDenied || !strings.Contains(errOut.String(), "reserved") {
798 t.Errorf("--context %s: exit %d, %s", ctx, code, errOut.String())
799 }
800 }
801 if has, err := st.RepoHasStatuses(repo.ID); err != nil || has {
802 t.Fatalf("a refused status was stored: %v %v", has, err)
803 }
804}
805```
806
807- [ ] **Step 2: Run it and see it fail**
808
809Run: `go test ./internal/control -run TestStatusSetRefusesReservedContext -count=1`
810Expected: FAIL, exit 3 (`no commit abc1234`), since today the context is
811accepted and the missing commit is what stops it.
812
813- [ ] **Step 3: Implement**
814
815In the registration, change the `--context` flag and the example:
816
817```go
818 {"--context", "<c>", "the check this status reports for; ci/ is reserved for the instance's builds", ""},
819```
820
821```go
822 Examples: []string{
823 "status set krz/gitbay a1b2c3d --context ext/lint --state success",
824 },
825```
826
827After the usage check at `status.go:68-70`, before the `--url` check:
828
829```go
830 // ci/<job> statuses are the build subsystem's: queued, reused,
831 // skipped and finished by the server itself. A writer who could post
832 // one could mark ci/test green on their own head before, or instead
833 // of, the build (#258). Case-folded, so CI/test is no way around it.
834 if strings.HasPrefix(strings.ToLower(context), "ci/") {
835 return c.fail(protocol.ExitDenied, "the ci/ prefix is reserved for the instance's builds; report under another name, such as ext/%s",
836 strings.TrimPrefix(strings.ToLower(context), "ci/"))
837 }
838```
839
840- [ ] **Step 4: Run it and see it pass**
841
842Run: `go test ./internal/control -run 'TestStatusSet|TestHelp' -count=1`
843Expected: PASS.
844
845- [ ] **Step 5: e2e callers off `ci/`, and the refusal over SSH**
846
847`e2e/readonly_test.go:75`: `"--context", "ci/x"` → `"--context", "ext/x"`.
848`e2e/mrweb_test.go:275`: `"--context", "ci/test"` → `"--context", "ext/test"`.
849
850In `e2e/status_test.go`, after the reader-denied check (line 49), add:
851
852```go
853 // ci/ is the instance's own: a writer is refused it (#258).
854 if _, errOut, code := inst.ssh(t, bobKey, "", "status", "set", "alice/svc", head, "--context", "ci/build", "--state", "success"); code != 4 || !strings.Contains(errOut, "reserved") {
855 t.Fatalf("writer posted a ci/ status: exit %d, %s", code, errOut)
856 }
857```
858
859Run: `go test ./e2e -run TestCommitStatuses -count=1`
860Expected: PASS.
861
862- [ ] **Step 6: Commit**
863
864```bash
865git add internal/control/status.go internal/control/status_test.go e2e/readonly_test.go e2e/mrweb_test.go e2e/status_test.go
866git commit -S -m "status set: ci/ is reserved for the instance's builds
867
868Ref #258"
869```
870
871### Task 2.2: reuse only trusted results on the same image
872
873**Files:**
874- Modify: `internal/store/builds.go:453-477` (`SuccessBuildForTree`, `SuccessBuildFor`)
875- Modify: `internal/control/build.go:856-864` (`queueJobs`)
876- Modify: `internal/store/builds_test.go:241-249` (`TestSuccessBuildForTree` call sites) and append a test
877- Test: `internal/control/build_test.go` (append)
878
879**Interfaces:**
880- Produces: `func (s *Store) SuccessBuildForTree(repoID int64, tree, job, image string) (Build, bool, error)` — trusted builds only, same image. `SuccessBuildFor` keeps its signature and considers trusted builds only.
881
882- [ ] **Step 1: Write the failing tests**
883
884In `internal/store/builds_test.go`, add `""` as the fourth argument to
885the three `SuccessBuildForTree` calls in `TestSuccessBuildForTree`, then
886append:
887
888```go
889// A result stands for another commit only when it came from a trusted
890// build on the same image: a fork's green build, or one on an image the
891// job has since left, proves nothing about the repository's own (#258).
892func TestSuccessReuseNeedsTrustAndImage(t *testing.T) {
893 s := open(t)
894 if err := s.MigrateUp(); err != nil {
895 t.Fatal(err)
896 }
897 uid, _ := s.CreateUser("cmc", true)
898 repoID, _ := s.CreateRepo("user", uid, "app", "public")
899 for _, b := range []struct {
900 sha, image string
901 trusted bool
902 }{
903 {"aaa", "", false},
904 {"bbb", "localhost/old:1", true},
905 } {
906 if _, err := s.CreateBuild(repoID, "unit", b.sha, "main", `["true"]`, b.image, "tree1", b.trusted); err != nil {
907 t.Fatal(err)
908 }
909 claimed, ok, err := s.ClaimBuild([]int64{repoID}, true)
910 if err != nil || !ok {
911 t.Fatalf("claim: ok=%v err=%v", ok, err)
912 }
913 if err := s.FinishBuild(claimed.ID, "success"); err != nil {
914 t.Fatal(err)
915 }
916 }
917 if prev, ok, _ := s.SuccessBuildForTree(repoID, "tree1", "unit", ""); ok {
918 t.Fatalf("reused build %d: untrusted, or on another image", prev.Number)
919 }
920 if prev, ok, _ := s.SuccessBuildForTree(repoID, "tree1", "unit", "localhost/old:1"); !ok || prev.SHA != "bbb" {
921 t.Fatalf("trusted build on the same image not found: ok=%v prev=%+v", ok, prev)
922 }
923 if _, ok, _ := s.SuccessBuildFor(repoID, "aaa", "unit"); ok {
924 t.Error("an untrusted success stood for its commit")
925 }
926}
927```
928
929Append to `internal/control/build_test.go`:
930
931```go
932// A fork's green build of a commit does not stand for the repository's
933// own: the same commit landing on a branch, or a commit with the same
934// tree, is built again as trusted (#258).
935func TestQueueBranchBuildsRebuildsWhatOnlyAForkBuilt(t *testing.T) {
936 st, repo, uid := newQueueTestRepo(t)
937 git := gitRunner(t)
938 root := t.TempDir()
939
940 src := filepath.Join(root, "src")
941 os.MkdirAll(filepath.Join(src, ".gitbay"), 0o755)
942 os.WriteFile(filepath.Join(src, ".gitbay", "ci.yml"), []byte(
943 "jobs:\n unit:\n steps:\n - echo hi\n"), 0o644)
944 git(root, "init", "-q", "-b", "main", "src")
945 git(src, "add", ".")
946 git(src, "commit", "-q", "-m", "base")
947 first := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
948 git(src, "commit", "-q", "--allow-empty", "-m", "same tree")
949 second := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
950
951 dir := RepoDir(root, repo.OwnerName, repo.Name)
952 os.MkdirAll(filepath.Dir(dir), 0o755)
953 git(root, "clone", "-q", "--bare", src, dir)
954
955 // A fork's merge request head, built untrusted and green.
956 QueueMRBuilds(st, root, "https://x.test", repo, uid, 1, first)
957 b, ok, err := st.ClaimBuild([]int64{repo.ID}, true)
958 if err != nil || !ok || b.Trusted {
959 t.Fatalf("claim: ok=%v trusted=%v err=%v", ok, b.Trusted, err)
960 }
961 if err := st.FinishBuild(b.ID, "success"); err != nil {
962 t.Fatal(err)
963 }
964
965 // The same commit lands on main, then a commit with the same tree.
966 QueueBranchBuilds(st, root, "https://x.test", repo, uid, "main", "", first, time.Now())
967 QueueBranchBuilds(st, root, "https://x.test", repo, uid, "main", first, second, time.Now())
968 pending, _ := st.ListBuilds(repo.ID, store.BuildFilter{Status: "pending"}, 10)
969 if len(pending) != 2 {
970 t.Fatalf("queued %d builds, want 2 (one per commit): %+v", len(pending), pending)
971 }
972 for _, p := range pending {
973 if !p.Trusted {
974 t.Errorf("build %d queued untrusted on a branch push", p.Number)
975 }
976 }
977}
978```
979
980- [ ] **Step 2: Run them and see them fail**
981
982Run: `go test ./internal/store -run 'TestSuccessBuildForTree|TestSuccessReuse' -count=1`
983Expected: build failure (`too many arguments in call to s.SuccessBuildForTree`).
984
985Run: `go test ./internal/control -run TestQueueBranchBuildsRebuildsWhatOnlyAForkBuilt -count=1`
986Expected: FAIL, `queued 0 builds, want 2`.
987
988- [ ] **Step 3: Store**
989
990Replace `internal/store/builds.go:453-477` with:
991
992```go
993// SuccessBuildForTree finds a passed build of the job for a tree rather
994// than a commit: a rebase that changes nothing in the tree has already
995// been built (#177). Only a trusted build on the image the job names
996// counts: a fork's result, or one from an image the job has left, does
997// not stand for the repository's own (#258). A job naming no image
998// matches builds that named none, whichever default the runner used;
999// the CI wiki page says so. An empty tree never matches.
1000func (s *Store) SuccessBuildForTree(repoID int64, tree, job, image string) (Build, bool, error) {
1001 if tree == "" {
1002 return Build{}, false, nil
1003 }
1004 b, err := scanBuild(s.DB.QueryRow(buildSelect+
1005 " WHERE repo_id = ? AND tree = ? AND job = ? AND image = ? AND trusted = 1 AND status = 'success'"+
1006 " ORDER BY number DESC LIMIT 1", repoID, tree, job, image))
1007 if errors.Is(err, sql.ErrNoRows) {
1008 return Build{}, false, nil
1009 }
1010 return b, err == nil, err
1011}
1012
1013// SuccessBuildFor finds a passed trusted build of the commit for the job,
1014// on any ref: what a cancelled duplicate can point back at.
1015func (s *Store) SuccessBuildFor(repoID int64, sha, job string) (Build, bool, error) {
1016 b, err := scanBuild(s.DB.QueryRow(buildSelect+
1017 " WHERE repo_id = ? AND sha = ? AND job = ? AND trusted = 1 AND status = 'success' ORDER BY number DESC LIMIT 1", repoID, sha, job))
1018 if errors.Is(err, sql.ErrNoRows) {
1019 return Build{}, false, nil
1020 }
1021 return b, err == nil, err
1022}
1023```
1024
1025- [ ] **Step 4: `queueJobs`**
1026
1027Replace `internal/control/build.go:856-859` with:
1028
1029```go
1030 // A build of this commit that passed, or is queued or running,
1031 // stands for it — unless this queue is trusted and that build was
1032 // not: a fork's head that lands on a branch is built again as the
1033 // repository's own (#258).
1034 if b, ok := built[j.Name]; ok && (b.Trusted || !trusted) &&
1035 (b.Status == "success" || b.Status == "pending" || b.Status == "running") {
1036 continue
1037 }
1038 if prev, ok, _ := st.SuccessBuildForTree(repo.ID, tree, j.Name, j.Image); ok && prev.SHA != sha {
1039```
1040
1041(the body of the `if prev, ok` block is unchanged.)
1042
1043- [ ] **Step 5: Run the tests and see them pass**
1044
1045Run: `go vet ./... && go test ./internal/store ./internal/control ./internal/hookd ./internal/ci -count=1`
1046Expected: PASS. `TestPushShapes` (hookd) has no row where a fork's
1047build lands on a branch, so its table is unchanged.
1048
1049- [ ] **Step 6: Commit**
1050
1051```bash
1052git add internal/store/builds.go internal/store/builds_test.go internal/control/build.go internal/control/build_test.go
1053git commit -S -m "ci: reuse only trusted results on the job's image
1054
1055Ref #258"
1056```
1057
1058### Task 2.3: required contexts
1059
1060**Files:**
1061- Modify: `internal/store/repos.go:25-37` (`RepoSettings`)
1062- Modify: `internal/control/output.go:59-60` (`GatesOut`)
1063- Modify: `internal/control/mr.go:45-49` (registration), after `runRequireChecks` (`:326-341`), `:1579-1603` (`MergeGates` checks block)
1064- Modify: `internal/control/repo.go:710-717` (`repo settings show`)
1065- Modify: `internal/control/checksgate_test.go` (helper refactor, two tests)
1066- Test: `internal/control/mr_test.go` (append)
1067- Modify: `cmd/gitbay/main.go:580` (add a `pass`), `cmd/gitbay/summaries_gen.go` (regenerated)
1068- Modify: `internal/httpd/settings.go:106-107`, `:228-229`; `internal/web/templates/settings.html:73-78` (the require-checks form, its hint at line 75); `internal/web/templates/mr.html:124`
1069- Modify: `internal/httpd/mrpage_test.go` (`TestMRGatesRender`), `e2e/settingsweb_test.go:58`
1070
1071**Interfaces:**
1072- Produces:
1073 - `RepoSettings.RequiredContexts []string` (`json:"required_contexts,omitempty"`)
1074 - `GatesOut.ChecksMissing []string` (`json:"checks_missing,omitempty"`)
1075 - command `repo settings require-contexts <owner/name> [<context>...]`:
1076 a non-empty list is stored and sets `RequireChecks = true` in the
1077 same update; an empty list clears the contexts and leaves
1078 `RequireChecks` alone.
1079 - `repo settings show` human output gains `require checks` and
1080 `required contexts` rows (JSON already carries `require_checks` and
1081 gains `required_contexts`).
1082
1083- [ ] **Step 1: Write the failing tests**
1084
1085In `internal/control/checksgate_test.go`, replace `gatesForHeadSeeded`
1086(lines 20-73) with a general helper and a thin wrapper:
1087
1088```go
1089func gatesForHeadSeeded(t *testing.T, ciYML string, seed bool) GatesOut {
1090 return gatesFor(t, ciYML, nil, func(st *store.Store, repoID, uid int64, targetSHA, _ string) {
1091 if !seed {
1092 return
1093 }
1094 if err := st.SetCommitStatus(repoID, targetSHA, "lint", "success", "", "", uid); err != nil {
1095 t.Fatal(err)
1096 }
1097 })
1098}
1099
1100// gatesFor builds a repository with require_checks on and set applied to
1101// its settings, a bare dir holding the given .gitbay/ci.yml (empty
1102// string for none), and one MR; seed records statuses before the gates
1103// are computed.
1104func gatesFor(t *testing.T, ciYML string, set func(*store.RepoSettings),
1105 seed func(st *store.Store, repoID, uid int64, targetSHA, headSHA string)) GatesOut {
1106 t.Helper()
1107 st, repo, uid := newQueueTestRepo(t)
1108 if _, err := st.UpdateRepoSettings(repo.ID, func(s *store.RepoSettings) {
1109 s.RequireChecks = true
1110 if set != nil {
1111 set(s)
1112 }
1113 }); err != nil {
1114 t.Fatal(err)
1115 }
1116 repo, err := st.RepoByID(repo.ID)
1117 if err != nil {
1118 t.Fatal(err)
1119 }
1120
1121 git := gitRunner(t)
1122 root := t.TempDir()
1123 src := filepath.Join(root, "src")
1124 os.MkdirAll(src, 0o755)
1125 git(root, "init", "-q", "-b", "main", "src")
1126 os.WriteFile(filepath.Join(src, "README"), []byte("x\n"), 0o644)
1127 git(src, "add", ".")
1128 git(src, "commit", "-q", "-m", "base")
1129 targetSHA := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
1130 git(src, "checkout", "-q", "-b", "feature")
1131 if ciYML != "" {
1132 os.MkdirAll(filepath.Join(src, ".gitbay"), 0o755)
1133 os.WriteFile(filepath.Join(src, ".gitbay", "ci.yml"), []byte(ciYML), 0o644)
1134 }
1135 os.WriteFile(filepath.Join(src, "README"), []byte("y\n"), 0o644)
1136 git(src, "add", ".")
1137 git(src, "commit", "-q", "-m", "change")
1138 headSHA := strings.TrimSpace(git(src, "rev-parse", "HEAD"))
1139
1140 dir := RepoDir(root, repo.OwnerName, repo.Name)
1141 os.MkdirAll(filepath.Dir(dir), 0o755)
1142 git(root, "clone", "-q", "--bare", src, dir)
1143
1144 if _, err := st.CreateMR(repo.ID, uid, repo.ID, "feature", "main", "t", "", headSHA, "md", false); err != nil {
1145 t.Fatal(err)
1146 }
1147 mr, err := st.MRByNumber(repo.ID, 1)
1148 if err != nil {
1149 t.Fatal(err)
1150 }
1151 if seed != nil {
1152 seed(st, repo.ID, uid, targetSHA, headSHA)
1153 }
1154 g, err := MergeGates(st, repo, mr, dir, targetSHA, headSHA)
1155 if err != nil {
1156 t.Fatal(err)
1157 }
1158 return g
1159}
1160```
1161
1162Append to the same file (add `"slices"` to its imports):
1163
1164```go
1165// A required context that has not reported holds the merge as pending,
1166// even when every status that did report is green (#258).
1167func TestRequiredContextMissingIsPending(t *testing.T) {
1168 g := gatesFor(t, "", func(s *store.RepoSettings) { s.RequiredContexts = []string{"ext/deploy", "lint"} },
1169 func(st *store.Store, repoID, uid int64, _, headSHA string) {
1170 if err := st.SetCommitStatus(repoID, headSHA, "lint", "success", "", "", uid); err != nil {
1171 t.Fatal(err)
1172 }
1173 })
1174 if g.Checks != "pending" || !slices.Equal(g.ChecksMissing, []string{"ext/deploy"}) {
1175 t.Fatalf("checks %q, missing %v", g.Checks, g.ChecksMissing)
1176 }
1177 if !checksUnmet(g) || !strings.Contains(strings.Join(g.Unmet, "\n"), "ext/deploy=missing") {
1178 t.Fatalf("unmet: %v", g.Unmet)
1179 }
1180}
1181
1182// Every required context reported green: nothing is held.
1183func TestRequiredContextsReportedPass(t *testing.T) {
1184 g := gatesFor(t, "", func(s *store.RepoSettings) { s.RequiredContexts = []string{"lint"} },
1185 func(st *store.Store, repoID, uid int64, _, headSHA string) {
1186 if err := st.SetCommitStatus(repoID, headSHA, "lint", "success", "", "", uid); err != nil {
1187 t.Fatal(err)
1188 }
1189 })
1190 if checksUnmet(g) || len(g.ChecksMissing) != 0 || g.Checks != "success" {
1191 t.Fatalf("checks %q, missing %v, unmet %v", g.Checks, g.ChecksMissing, g.Unmet)
1192 }
1193}
1194```
1195
1196Append to `internal/control/mr_test.go` (add `"slices"` to its imports;
1197`mrTestCtx` is that file's helper):
1198
1199```go
1200// require-contexts stores a deduplicated list and turns require_checks
1201// on with it; an empty list clears the contexts and leaves
1202// require_checks as it was. A context with whitespace is refused (#258).
1203func TestRequireContextsSetsAndClears(t *testing.T) {
1204 st, repo, uid := newQueueTestRepo(t)
1205 alice := store.User{ID: uid, Username: "alice"}
1206 dispatch := func(args ...string) int {
1207 t.Helper()
1208 c, _, _ := mrTestCtx(st, alice)
1209 return Dispatch(c, args)
1210 }
1211 contexts := func(names ...string) int {
1212 t.Helper()
1213 return dispatch(append([]string{"repo", "settings", "require-contexts", repo.Path()}, names...)...)
1214 }
1215 settings := func() store.RepoSettings {
1216 t.Helper()
1217 got, err := st.RepoByID(repo.ID)
1218 if err != nil {
1219 t.Fatal(err)
1220 }
1221 return got.Settings
1222 }
1223
1224 if settings().RequireChecks {
1225 t.Fatal("require_checks on in a new repository")
1226 }
1227 if code := contexts("lint", "ext/deploy", "lint"); code != protocol.ExitOK {
1228 t.Fatalf("set: exit %d", code)
1229 }
1230 if s := settings(); !slices.Equal(s.RequiredContexts, []string{"lint", "ext/deploy"}) || !s.RequireChecks {
1231 t.Fatalf("stored %v, require_checks %v; want [lint ext/deploy], on", s.RequiredContexts, s.RequireChecks)
1232 }
1233 if code := contexts("bad context"); code != protocol.ExitUsage {
1234 t.Fatalf("a context with a space: exit %d", code)
1235 }
1236 if code := contexts(); code != protocol.ExitOK {
1237 t.Fatalf("clear: exit %d", code)
1238 }
1239 if s := settings(); len(s.RequiredContexts) != 0 || !s.RequireChecks {
1240 t.Fatalf("after clearing: contexts %v, require_checks %v; want none, still on", s.RequiredContexts, s.RequireChecks)
1241 }
1242 if code := dispatch("repo", "settings", "require-checks", repo.Path(), "off"); code != protocol.ExitOK {
1243 t.Fatalf("require-checks off: exit %d", code)
1244 }
1245 if code := contexts(); code != protocol.ExitOK {
1246 t.Fatalf("clear again: exit %d", code)
1247 }
1248 if settings().RequireChecks {
1249 t.Fatal("clearing the list turned require_checks on")
1250 }
1251}
1252
1253// settings show prints the checks gate beside the contexts it waits for,
1254// so a list that turned the gate on is visible where the gate is (#258).
1255func TestSettingsShowRequiredContexts(t *testing.T) {
1256 st, repo, uid := newQueueTestRepo(t)
1257 alice := store.User{ID: uid, Username: "alice"}
1258 c, _, _ := mrTestCtx(st, alice)
1259 if code := Dispatch(c, []string{"repo", "settings", "require-contexts", repo.Path(), "ext/deploy", "lint"}); code != protocol.ExitOK {
1260 t.Fatalf("require-contexts: exit %d", code)
1261 }
1262 c, out, _ := mrTestCtx(st, alice)
1263 if code := Dispatch(c, []string{"repo", "settings", "show", repo.Path()}); code != protocol.ExitOK {
1264 t.Fatalf("settings show: exit %d", code)
1265 }
1266 got := strings.Join(strings.Fields(out.String()), " ")
1267 for _, want := range []string{"require checks true", "required contexts ext/deploy, lint"} {
1268 if !strings.Contains(got, want) {
1269 t.Errorf("settings show lacks %q:\n%s", want, out.String())
1270 }
1271 }
1272}
1273```
1274
1275- [ ] **Step 2: Run them and see them fail**
1276
1277Run: `go test ./internal/control -run 'TestRequire|TestRequiredContext|TestSettingsShowRequiredContexts' -count=1`
1278Expected: build failure (`s.RequiredContexts undefined`).
1279
1280- [ ] **Step 3: Store and output types**
1281
1282`internal/store/repos.go`, in `RepoSettings` after `RequireChecks`:
1283
1284```go
1285 RequireChecks bool `json:"require_checks,omitempty"`
1286 // RequiredContexts are statuses require_checks waits for whether or
1287 // not they have reported; one that has not is pending. Setting a
1288 // non-empty list turns RequireChecks on (#258).
1289 RequiredContexts []string `json:"required_contexts,omitempty"`
1290```
1291
1292`internal/control/output.go`, in `GatesOut` after `Checks`:
1293
1294```go
1295 Checks string `json:"checks,omitempty"` // combined status; "" when none reported
1296 ChecksMissing []string `json:"checks_missing,omitempty"` // required contexts not reported
1297```
1298
1299- [ ] **Step 4: The command**
1300
1301Registration, after `require-checks` at `mr.go:49`:
1302
1303```go
1304 register(Command{Path: []string{"repo", "settings", "require-contexts"},
1305 Summary: "name the statuses the checks gate waits for, and turn the gate on",
1306 Usage: "repo settings require-contexts <owner/name> [<context>...] (none clears the list)",
1307 Examples: []string{"repo settings require-contexts krz/gitbay ci/build ci/test"},
1308 Run: runRequireContexts})
1309```
1310
1311After `runRequireChecks`:
1312
1313```go
1314// maxRequiredContexts bounds the list: a gate naming more checks than
1315// this is a configuration mistake.
1316const maxRequiredContexts = 20
1317
1318func runRequireContexts(c *Ctx, args []string) int {
1319 if len(args) < 1 {
1320 return c.usage()
1321 }
1322 var contexts []string
1323 for _, ctx := range args[1:] {
1324 if ctx == "" || len(ctx) > 100 || strings.ContainsAny(ctx, " \t\r\n") {
1325 return c.fail(protocol.ExitUsage, "a context is 1 to 100 characters with no whitespace: %q", ctx)
1326 }
1327 if !slices.Contains(contexts, ctx) {
1328 contexts = append(contexts, ctx)
1329 }
1330 }
1331 if len(contexts) > maxRequiredContexts {
1332 return c.fail(protocol.ExitUsage, "at most %d required contexts", maxRequiredContexts)
1333 }
1334 repo, code := resolveRepo(c, args[0], policy.CanAdmin)
1335 if code >= 0 {
1336 return code
1337 }
1338 // Naming contexts asks for the gate, so it turns require_checks on in
1339 // the same update. Clearing the list leaves the gate as it was.
1340 s, err := c.Store.UpdateRepoSettings(repo.ID, func(s *store.RepoSettings) {
1341 s.RequiredContexts = contexts
1342 if len(contexts) > 0 {
1343 s.RequireChecks = true
1344 }
1345 })
1346 if err != nil {
1347 return c.fail(protocol.ExitFailure, "%v", err)
1348 }
1349 return c.emit(s, func(w io.Writer) {
1350 if len(contexts) > 0 {
1351 fmt.Fprintf(w, "required contexts on %s: %s; require_checks on\n", repo.Path(), strings.Join(contexts, ", "))
1352 return
1353 }
1354 gate := "off"
1355 if s.RequireChecks {
1356 gate = "on"
1357 }
1358 fmt.Fprintf(w, "required contexts cleared on %s; require_checks %s\n", repo.Path(), gate)
1359 })
1360}
1361```
1362
1363- [ ] **Step 5: `MergeGates`**
1364
1365Replace `mr.go:1579-1603` (the checks block, from the comment through
1366the end of `if set.RequireChecks { … }`) with:
1367
1368```go
1369 // Checks: with require_checks, every status the head carries must be
1370 // green, a head something was going to report on must carry some, and
1371 // every required context must have reported: one that has not is
1372 // pending whatever the others say (#258). Setting contexts turns
1373 // require_checks on; turned off again, the list is kept and unread.
1374 statuses, err := st.ListCommitStatuses(repo.ID, headSHA)
1375 if err != nil {
1376 return g, err
1377 }
1378 g.Checks = store.CombinedStatus(statuses)
1379 if set.RequireChecks {
1380 reported := map[string]bool{}
1381 for _, s := range statuses {
1382 reported[s.Context] = true
1383 }
1384 for _, want := range set.RequiredContexts {
1385 if !reported[want] {
1386 g.ChecksMissing = append(g.ChecksMissing, want)
1387 }
1388 }
1389 if len(g.ChecksMissing) > 0 && (g.Checks == "" || g.Checks == "success") {
1390 g.Checks = "pending"
1391 }
1392 switch g.Checks {
1393 case "success":
1394 case "":
1395 if checksExpected(st, repo.ID, dir, headSHA) {
1396 g.Unmet = append(g.Unmet, fmt.Sprintf("%s requires green checks and none were reported on %.10s", repo.Path(), headSHA))
1397 }
1398 default:
1399 var bad []string
1400 for _, st := range statuses {
1401 if st.State != "success" {
1402 bad = append(bad, st.Context+"="+st.State)
1403 }
1404 }
1405 for _, m := range g.ChecksMissing {
1406 bad = append(bad, m+"=missing")
1407 }
1408 g.Unmet = append(g.Unmet, fmt.Sprintf("%s requires green checks; %.10s has %s", repo.Path(), headSHA, strings.Join(bad, ", ")))
1409 }
1410 }
1411```
1412
1413`repo.go:710-717`, the `repo settings show` fields: today they print
1414neither the checks gate nor anything about it. Add two rows after
1415`"require mr"` (line 713), so the gate and the list that turned it on
1416read together:
1417
1418```go
1419 "require mr", strconv.FormatBool(repo.Settings.RequireMR),
1420 "require checks", strconv.FormatBool(repo.Settings.RequireChecks),
1421 "required contexts", strings.Join(repo.Settings.RequiredContexts, ", "),
1422```
1423
1424(`fields` skips a row whose value is empty, so a repository with no
1425contexts shows no `required contexts` row.)
1426
1427- [ ] **Step 6: Run the control tests**
1428
1429Run: `go vet ./... && go test ./internal/control ./internal/store -count=1`
1430Expected: PASS.
1431
1432- [ ] **Step 7: CLI, web, summaries**
1433
1434`cmd/gitbay/main.go`, after the `require-checks` line (580):
1435
1436```go
1437 pass("require-contexts", passOpts{server: []string{"repo", "settings", "require-contexts"}, needsRepo: true}),
1438```
1439
1440Run: `go test ./cmd/gitbay -run TestSummariesAreCurrent -update -count=1 && go test ./cmd/gitbay -count=1`
1441Expected: PASS; `summaries_gen.go` gains the `repo settings require-contexts` line.
1442
1443`internal/httpd/settings.go`, in `settingsSubmit` after the
1444`require-checks` case:
1445
1446```go
1447 case "require-contexts":
1448 argv = append([]string{"repo", "settings", "require-contexts", repo}, strings.Fields(v("contexts"))...)
1449```
1450
1451and in `fieldLabel` after `require-checks`:
1452
1453```go
1454 case "require-contexts":
1455 return "required contexts"
1456```
1457
1458`internal/web/templates/settings.html`: the require-checks hint (line
145975) names the contexts the gate waits for, so a box ticked by saving
1460contexts says why:
1461
1462```html
1463 <div><label for="require-checks">Required checks</label><p class="hint">Requires CI to succeed.{{with .Repo.Settings.RequiredContexts}} Also waits for {{range $i, $c := .}}{{if $i}}, {{end}}<code>{{$c}}</code>{{end}} until they report{{if not $.Repo.Settings.RequireChecks}}, once this is on{{end}}.{{end}}</p></div>
1464```
1465
1466and after the `require-checks` form (line 78):
1467
1468```html
1469<form method="post" action="{{$base}}" class="setform">
1470 <input type="hidden" name="field" value="require-contexts">
1471 <div><label for="contexts">Required contexts</label><p class="hint">Statuses the checks gate waits for until they report, separated by spaces. Saving any turns required checks on; saving none leaves it as it is.</p></div>
1472 <div><input type="text" id="contexts" name="contexts" value="{{range $i, $c := .Repo.Settings.RequiredContexts}}{{if $i}} {{end}}{{$c}}{{end}}" autocomplete="off"></div>
1473 <div><button type="submit" class="btn">Save</button></div>
1474</form>
1475```
1476
1477`internal/web/templates/mr.html`, after the `OwnersOutstanding` line
1478(124):
1479
1480```html
1481 {{range .ChecksMissing}}<p class="row none">waiting on <code>{{.}}</code>, not yet reported</p>{{end}}
1482```
1483
1484In `internal/httpd/mrpage_test.go` `TestMRGatesRender`, add after the
1485first `for` loop:
1486
1487```go
1488 if out := render(&control.GatesOut{Checks: "pending", ChecksMissing: []string{"ext/deploy"}}); !strings.Contains(out, "waiting on <code>ext/deploy</code>") {
1489 t.Errorf("missing required context not rendered:\n%s", out)
1490 }
1491```
1492
1493In `e2e/settingsweb_test.go`, replace line 58,
1494`post(url.Values{"field": {"require-checks"}, "require-checks": {"on"}})`,
1495with a contexts post, so the `"require_checks":true` the test already
1496expects from `settings show --json` (line 69) now comes from the
1497contexts turning the gate on:
1498
1499```go
1500 // Saving required contexts turns the checks gate on, and the page
1501 // shows it ticked with the contexts in its hint (#258).
1502 if body := post(url.Values{"field": {"require-contexts"}, "contexts": {"ext/deploy lint"}}); !strings.Contains(body, `id="require-checks" name="require-checks" value="on" checked`) ||
1503 !strings.Contains(body, `Also waits for <code>ext/deploy</code>, <code>lint</code> until they report.`) {
1504 t.Fatalf("required contexts did not show as turning required checks on:\n%s", body)
1505 }
1506```
1507
1508and add `` `"required_contexts":["ext/deploy","lint"]` `` to the
1509`settings show --json` want list at line 69.
1510
1511Run: `go test ./internal/httpd ./internal/web -count=1 && go test ./e2e -run TestRepoSettingsWeb -count=1`
1512Expected: PASS.
1513
1514- [ ] **Step 8: Commit**
1515
1516```bash
1517git add internal/store/repos.go internal/control cmd/gitbay internal/httpd internal/web e2e/settingsweb_test.go
1518git commit -S -m "repo settings: required contexts turn the checks gate on, pending until reported
1519
1520Ref #258"
1521```
1522
1523### Task 2.4: wiki, and the MR
1524
1525**Files:**
1526- Modify: `.gitbay/wiki/API.org:100-122`, `.gitbay/wiki/CI.org:5-9`, `.gitbay/wiki/Users.org:542-545`, `.gitbay/wiki/Parity.org` (settings table near line 194)
1527- Modify: `.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org:57`, `09-Controls.org:39`, `:92`, `10-Known-Gaps.org` (remove the #258 row)
1528
1529- [ ] **Step 1: API.org**
1530
1531Change the two example lines to `--context ext/build`, and after the
1532paragraph ending "…Each report also emits a =status= event to
1533webhooks." add:
1534
1535```org
1536Contexts starting with =ci/= are the instance's own: its builds queue,
1537reuse, skip and finish them, and =status set= refuses them with exit 4,
1538so a writer cannot mark =ci/test= green on a head the build has not
1539passed. Report under another prefix, such as =ext/=.
1540
1541=repo settings require-contexts <repo> ext/deploy ci/test= names
1542statuses the checks gate waits for whether or not they have reported:
1543one that has not is =pending=, and =mr show= lists it as
1544=ext/deploy=missing=. Naming any context turns =require-checks= on;
1545with no contexts the command clears the list and leaves
1546=require-checks= as it was. =require-checks off= keeps the list, which
1547waits for nothing until the gate is on again. =repo settings show=
1548prints both.
1549```
1550
1551- [ ] **Step 2: CI.org**
1552
1553In the *Dedupe* bullet, after "…naming the build it came from (#177).",
1554add: "Only a trusted build counts, and for tree reuse only one on the
1555image the job names: a fork's green build does not stand for the
1556repository's own, so its commit is built again when it lands on a
1557branch (#258). A job that names no =image:= is compared as naming
1558none: its reuse does not notice the runner's default image changing,
1559because reuse is decided when the push is queued, before any runner
1560claims the build, and runners can differ in their default. Name the
1561image in =ci.yml= to tie reuse to it; after an operator changes a
1562runner's =-image=, =build trigger= builds a job afresh, since a
1563triggered build is never reused."
1564
1565- [ ] **Step 3: Users.org**
1566
1567After "…a =ci/<job>= commit status, which =repo settings
1568require-checks= can gate merges on." add: "=repo settings
1569require-contexts= names statuses the gate waits for until they report,
1570and turns the gate on."
1571
1572- [ ] **Step 4: Parity.org**
1573
1574After the `require codeowners` row:
1575
1576```org
1577| require contexts | yes | yes | no |
1578```
1579
1580- [ ] **Step 5: Architecture**
1581
1582`07-CI-and-Supply-Chain.org:57`:
1583
1584```org
1585| =status set= | write on the repository; =ci/*= contexts refused (=status.go=) |
1586```
1587
1588and in lifecycle step 1 replace "or =success= copied from an earlier
1589build of the same tree (#177)." with "or =success= copied from an
1590earlier trusted build of the same tree on the same image (#177, #258)."
1591
1592`09-Controls.org:39`:
1593
1594```org
1595| Merge gates | in place | =MergeGates=; =ci/*= statuses written only by the build subsystem; required contexts |
1596```
1597
1598`09-Controls.org:92`:
1599
1600```org
1601| Build results reused only across equal trust | in place | =SuccessBuildForTree=, =SuccessBuildFor= (=internal/store/builds.go=) |
1602```
1603
1604`10-Known-Gaps.org`: delete the `#258` row.
1605
1606- [ ] **Step 6: Commit and MR**
1607
1608```bash
1609git add .gitbay/wiki
1610git commit -S -m "wiki: reserved ci/ statuses, trusted reuse, required contexts
1611
1612Closes #258"
1613git push -u origin ci-status-trust
1614gitbay mr create --source ci-status-trust --target main --title "ci: reserve ci/ statuses; reuse only trusted results; required contexts"
1615```
1616
1617No runner change: this part deploys with `make deploy` alone. Merge
1618with `--strategy ff` once CI is green; delete the branch both places.
1619
1620---
1621
1622# Part 3: separate the runner's source address from its builds, and limit what builds reach on its host (branch `runner-source-address`, #260)
1623
1624### Task 3.1: the claim carries the instance's public ssh destination
1625
1626**Files:**
1627- Modify: `internal/control/build.go` (imports; the payload from Task 1.1; a helper beside `runRunnerNext`)
1628- Test: `internal/control/runnernext_test.go` (append)
1629
1630**Interfaces:**
1631- Produces: `runner next --json` payload `"ssh": "git@<site host>"`, or
1632 `"git@<site host>:<port>"` when `[ssh] port` is neither 0 nor 22;
1633 omitted when `site_url` is empty.
1634
1635- [ ] **Step 1: Write the failing test**
1636
1637```go
1638// A build on the daemon's own host is given the public destination, not
1639// the loopback address its runner polls. The port rides along only when
1640// it is not 22, so ssh://$GITBAY_SSH/<owner>/<name>.git is a valid URL
1641// either way (#260).
1642func TestRunnerNextCarriesPublicSSH(t *testing.T) {
1643 st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
1644 for _, tc := range []struct {
1645 port int
1646 want string
1647 }{
1648 {0, `"ssh":"git@x.test"`},
1649 {22, `"ssh":"git@x.test"`},
1650 {2022, `"ssh":"git@x.test:2022"`},
1651 } {
1652 if _, err := st.CreateBuild(repo.ID, "unit", baseSHA, "main", "[]", "", "", true); err != nil {
1653 t.Fatal(err)
1654 }
1655 c, out := runnerCtx(st, uid, root) // site_url https://x.test
1656 c.Cfg.SSH.Port = tc.port
1657 c.JSON = true
1658 if code := runRunnerNext(c, nil); code != protocol.ExitOK {
1659 t.Fatalf("port %d: runner next: exit %d, output:\n%s", tc.port, code, out.String())
1660 }
1661 if !strings.Contains(out.String(), tc.want) {
1662 t.Fatalf("port %d: claim lacks %s:\n%s", tc.port, tc.want, out.String())
1663 }
1664 }
1665}
1666```
1667
1668Each pass queues one build and claims it, as in Task 1.1's test.
1669`runnerCtx` builds its `config.Config` by hand, so `SSH.Port` starts at
16700; config validation refuses 0 on a real instance (`config.go:367`),
1671and 0 is read as 22.
1672
1673- [ ] **Step 2: Run it and see it fail**
1674
1675Run: `go test ./internal/control -run TestRunnerNextCarriesPublicSSH -count=1`
1676Expected: FAIL, `port 0: claim lacks "ssh":"git@x.test"`.
1677
1678- [ ] **Step 3: Implement**
1679
1680Add `"net"` to `build.go`'s imports (`strconv` is already there), and
1681after `maxOrphanSkip`:
1682
1683```go
1684// publicSSH is the instance's ssh destination as anyone outside reaches
1685// it. A runner on the daemon's own host polls over loopback and hands
1686// its builds this instead, so no build connects from the runner's source
1687// address (#260). The port is added only when it is not 22: hutch and
1688// orgo build ssh://$GITBAY_SSH/... URLs, valid in both forms. Empty when
1689// site_url is not set.
1690func publicSSH(c *Ctx) string {
1691 host := c.Cfg.SiteHost()
1692 if host == "" {
1693 return ""
1694 }
1695 if p := c.Cfg.SSH.Port; p != 0 && p != 22 {
1696 return "git@" + net.JoinHostPort(host, strconv.Itoa(p))
1697 }
1698 return "git@" + host
1699}
1700```
1701
1702In the payload struct add, after `Trusted`:
1703
1704```go
1705 // SSH is the instance's public destination for the build's
1706 // GITBAY_SSH when its runner polls over loopback (#260).
1707 SSH string `json:"ssh,omitempty"`
1708```
1709
1710and `SSH: publicSSH(c),` in the literal.
1711
1712- [ ] **Step 4: Run it and see it pass**
1713
1714Run: `go test ./internal/control -run TestRunnerNext -count=1`
1715Expected: PASS.
1716
1717- [ ] **Step 5: Commit**
1718
1719```bash
1720git add internal/control/build.go internal/control/runnernext_test.go
1721git commit -S -m "runner next: send the instance's public ssh destination
1722
1723Ref #260"
1724```
1725
1726### Task 3.2: a loopback runner's builds get no host loopback
1727
1728**Files:**
1729- Modify: `cmd/gitbay-runner/main.go:32-42` (`job`), `:433` (`stepEnv` call), `:465-486` (`buildSSH`)
1730- Modify: `cmd/gitbay-runner/isolate.go:150-154` (podman run args)
1731- Modify: `cmd/gitbay-runner/env_test.go:190-212`
1732
1733**Interfaces:**
1734- Consumes: the `ssh` claim field from Task 3.1.
1735- Produces:
1736 - `job.SSH string` (`json:"ssh"`)
1737 - `func (r *runner) loopbackRemote() bool`
1738 - `func (r *runner) buildSSH(public string) string` (was `buildSSH()`)
1739 - `func (r *runner) buildNetwork() []string`
1740
1741- [ ] **Step 1: Write the failing tests**
1742
1743Replace `TestStepEnvCarriesInstanceAddress` (`env_test.go:190-212`) with:
1744
1745```go
1746// A build that talks back to the instance needs an address that works
1747// from where it runs. A runner polling over loopback keeps its podman
1748// builds off the host's loopback, so they get the instance's public
1749// destination from the claim, port included when it is not 22; any other
1750// remote is used as it is (#260).
1751func TestStepEnvCarriesInstanceAddress(t *testing.T) {
1752 env := stepEnv(job{}, "/tmp/buildhome", "git@gitbay.org")
1753 if !containsEnv(env, "GITBAY_SSH=git@gitbay.org") {
1754 t.Errorf("GITBAY_SSH missing: %q", env)
1755 }
1756 for _, tc := range []struct{ remote, isolation, public, want string }{
1757 {"git@127.0.0.1", isolationNone, "git@gitbay.org", "git@127.0.0.1"},
1758 {"git@127.0.0.1", isolationPodman, "git@gitbay.org", "git@gitbay.org"},
1759 {"git@127.0.0.1", isolationPodman, "git@gitbay.test:2022", "git@gitbay.test:2022"},
1760 {"git@localhost", isolationPodman, "git@gitbay.org", "git@gitbay.org"},
1761 {"git@127.0.0.1", isolationPodman, "", "git@127.0.0.1"},
1762 {"git@gitbay.org", isolationPodman, "git@other.test", "git@gitbay.org"},
1763 {"gitbay.org", isolationPodman, "git@gitbay.org", "gitbay.org"},
1764 } {
1765 r := &runner{remote: tc.remote, isolation: tc.isolation}
1766 if got := r.buildSSH(tc.public); got != tc.want {
1767 t.Errorf("remote %s under %s, public %q: got %s want %s", tc.remote, tc.isolation, tc.public, got, tc.want)
1768 }
1769 }
1770}
1771
1772// Only a runner that polls over loopback shares an address a build could
1773// connect from, so only its builds lose the host-loopback mapping (#260).
1774func TestBuildNetworkKeepsLoopbackRunnersBuildsOff(t *testing.T) {
1775 for _, tc := range []struct {
1776 remote string
1777 want []string
1778 }{
1779 {"git@127.0.0.1", []string{"--network", "pasta:--no-map-gw"}},
1780 {"localhost", []string{"--network", "pasta:--no-map-gw"}},
1781 {"git@::1", []string{"--network", "pasta:--no-map-gw"}},
1782 {"git@gitbay.org", nil},
1783 } {
1784 r := &runner{remote: tc.remote, isolation: isolationPodman}
1785 if got := r.buildNetwork(); strings.Join(got, " ") != strings.Join(tc.want, " ") {
1786 t.Errorf("remote %s: %q, want %q", tc.remote, got, tc.want)
1787 }
1788 }
1789}
1790```
1791
1792- [ ] **Step 2: Run them and see them fail**
1793
1794Run: `go test ./cmd/gitbay-runner -count=1`
1795Expected: build failure (`too many arguments in call to r.buildSSH`, `r.buildNetwork undefined`).
1796
1797- [ ] **Step 3: Implement**
1798
1799`job`, after `Trusted`:
1800
1801```go
1802 // SSH is the instance's public ssh destination, for a build whose
1803 // runner polls over loopback (#260).
1804 SSH string `json:"ssh"`
1805```
1806
1807Replace `buildSSH` (`main.go:465-486`) with:
1808
1809```go
1810// loopbackRemote reports whether the runner polls the daemon on its own
1811// host over loopback.
1812func (r *runner) loopbackRemote() bool {
1813 _, host, ok := strings.Cut(r.remote, "@")
1814 if !ok {
1815 host = r.remote
1816 }
1817 return host == "127.0.0.1" || host == "localhost" || host == "::1"
1818}
1819
1820// buildSSH is the instance's ssh destination as a build reaches it. A
1821// runner polling over loopback keeps its podman builds off the host's
1822// loopback (buildNetwork), so they get the instance's public destination
1823// from the claim. Any other remote is a real host elsewhere and works as
1824// it is, and under -isolation none a build runs on the host itself.
1825func (r *runner) buildSSH(public string) string {
1826 if r.isolation == isolationPodman && r.loopbackRemote() && public != "" {
1827 return public
1828 }
1829 return r.remote
1830}
1831
1832// buildNetwork is the podman network option for a build. pasta maps the
1833// container's gateway address to the host's loopback, and a build's
1834// connection through it arrives from 127.0.0.1 — the address a runner on
1835// the daemon's host polls from. The SSH auth limiter counts failures per
1836// source address, so a build sharing the runner's could throttle its
1837// polling (#260). --no-map-gw removes the mapping: the build reaches the
1838// host only at its public address, as any client on the internet does,
1839// and keeps its outbound access. The host's nftables table
1840// (deploy/gitbay-runner-egress.nft) then limits it to 22, 80 and 443
1841// there; it cannot tell a build from the runner by uid, so it leaves
1842// 127.0.0.1:22 open, and this flag is what keeps builds off it.
1843func (r *runner) buildNetwork() []string {
1844 if !r.loopbackRemote() {
1845 return nil
1846 }
1847 return []string{"--network", "pasta:--no-map-gw"}
1848}
1849```
1850
1851In `run`, the `stepEnv` call becomes `env := stepEnv(j, home, r.buildSSH(j.SSH))`.
1852
1853In `isolate.go`, after the `--env-file` append (line 153):
1854
1855```go
1856 args = append(args, r.buildNetwork()...)
1857```
1858
1859- [ ] **Step 4: Run the tests and see them pass**
1860
1861Run: `go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner -count=1`
1862Expected: PASS.
1863
1864- [ ] **Step 5: Commit**
1865
1866```bash
1867git add cmd/gitbay-runner
1868git commit -S -m "runner: builds off the host's loopback when the runner polls over it
1869
1870Ref #260"
1871```
1872
1873### Task 3.3: the limiter's behaviour, by registration mode
1874
1875This test records what the code does today (see "Finding for #260"
1876under Decisions), so it passes on its first run. It is the evidence the
1877issue's on-production throttling test was meant to give, without
1878touching production. If it fails, the limiter differs from what
1879Decisions and the wiki say: stop and correct that text, not the test.
1880
1881**Files:**
1882- Test: `internal/sshd/sshd_test.go` (append; every import it needs is already in the file)
1883
1884**Interfaces:**
1885- Consumes: `(*Server).authenticate`, `rateLimiter.seen` (package-internal).
1886
1887- [ ] **Step 1: Write the test**
1888
1889Append to `internal/sshd/sshd_test.go`:
1890
1891```go
1892// authMeta is the connection metadata authenticate reads: only the
1893// remote address.
1894type authMeta struct {
1895 ssh.ConnMetadata
1896 addr net.Addr
1897}
1898
1899func (m authMeta) RemoteAddr() net.Addr { return m.addr }
1900
1901func authKey(t *testing.T) ssh.PublicKey {
1902 t.Helper()
1903 pub, _, err := ed25519.GenerateKey(rand.Reader)
1904 if err != nil {
1905 t.Fatal(err)
1906 }
1907 k, err := ssh.NewPublicKey(pub)
1908 if err != nil {
1909 t.Fatal(err)
1910 }
1911 return k
1912}
1913
1914// authServer is a Server holding what authenticate uses: a store with a
1915// runner account's key, the registration mode, and a limiter of three
1916// failures a minute.
1917func authServer(t *testing.T, mode string) (*Server, ssh.PublicKey) {
1918 t.Helper()
1919 st, err := store.Open(filepath.Join(t.TempDir(), "gitbay.db"))
1920 if err != nil {
1921 t.Fatal(err)
1922 }
1923 t.Cleanup(func() { st.Close() })
1924 if err := st.MigrateUp(); err != nil {
1925 t.Fatal(err)
1926 }
1927 uid, err := st.CreateUser("ci", false)
1928 if err != nil {
1929 t.Fatal(err)
1930 }
1931 runner := authKey(t)
1932 if err := st.AddSSHKey(uid, ssh.FingerprintSHA256(runner), runner.Type(), runner.Marshal(), "runner", ""); err != nil {
1933 t.Fatal(err)
1934 }
1935 cfg := config.Default()
1936 cfg.Registration.Mode = mode
1937 return &Server{cfg: cfg, st: st, authLimiter: newRateLimiter(3, time.Minute)}, runner
1938}
1939
1940var (
1941 fromLoopback = authMeta{addr: &net.TCPAddr{IP: net.IPv4(127, 0, 0, 1), Port: 40000}}
1942 fromPublic = authMeta{addr: &net.TCPAddr{IP: net.IPv4(203, 0, 113, 7), Port: 40000}}
1943)
1944
1945// With registration closed an unknown key counts against its address.
1946// Below the limit a known key's success clears the count. At the limit
1947// authenticate refuses before it looks at the key, so the runner's own
1948// key from that address is refused too and its success never runs to
1949// clear anything, until the window passes. Another address is not
1950// affected. This is why a build must not share the runner's source
1951// address (#260).
1952func TestAuthLockoutHoldsAgainstTheRunnersKey(t *testing.T) {
1953 s, runner := authServer(t, "closed")
1954 stranger := authKey(t)
1955 failTimes := func(n int) {
1956 t.Helper()
1957 for i := 0; i < n; i++ {
1958 if _, err := s.authenticate(fromLoopback, stranger); err == nil {
1959 t.Fatal("unknown key admitted with registration closed")
1960 }
1961 }
1962 }
1963
1964 failTimes(2)
1965 if _, err := s.authenticate(fromLoopback, runner); err != nil {
1966 t.Fatalf("runner below the limit: %v", err)
1967 }
1968 failTimes(2)
1969 if _, err := s.authenticate(fromLoopback, runner); err != nil {
1970 t.Fatalf("runner after its success cleared the count: %v", err)
1971 }
1972
1973 failTimes(3)
1974 for i := 0; i < 2; i++ {
1975 if _, err := s.authenticate(fromLoopback, runner); err == nil || !strings.Contains(err.Error(), "too many") {
1976 t.Fatalf("attempt %d from a locked-out address: %v, want refused", i+1, err)
1977 }
1978 }
1979 if _, err := s.authenticate(fromPublic, runner); err != nil {
1980 t.Fatalf("another address was locked out too: %v", err)
1981 }
1982
1983 s.authLimiter.seen["127.0.0.1"].start = time.Now().Add(-2 * time.Minute)
1984 if _, err := s.authenticate(fromLoopback, runner); err != nil {
1985 t.Fatalf("runner after the window passed: %v", err)
1986 }
1987}
1988
1989// With registration open or by invite, an unknown key is admitted to run
1990// register and never counts, so no number of unknown-key attempts locks
1991// the runner's address out. gitbay.org runs open registration (#260).
1992func TestAuthUnknownKeyCountsOnlyWhenClosed(t *testing.T) {
1993 for _, mode := range []string{"open", "invite"} {
1994 s, runner := authServer(t, mode)
1995 for i := 0; i < 10; i++ {
1996 p, err := s.authenticate(fromLoopback, authKey(t))
1997 if err != nil || p.Extensions["anon-key"] == "" {
1998 t.Fatalf("%s: unknown key %d: %v %+v", mode, i+1, err, p)
1999 }
2000 }
2001 if _, err := s.authenticate(fromLoopback, runner); err != nil {
2002 t.Fatalf("%s: runner refused after unknown keys: %v", mode, err)
2003 }
2004 }
2005}
2006```
2007
2008- [ ] **Step 2: Run it**
2009
2010Run: `go vet ./internal/sshd && go test ./internal/sshd -run 'TestAuth' -count=1`
2011Expected: PASS.
2012
2013- [ ] **Step 3: Commit**
2014
2015```bash
2016git add internal/sshd/sshd_test.go
2017git commit -S -m "sshd: test the auth limiter's lockout by registration mode
2018
2019Ref #260"
2020```
2021
2022### Task 3.4: host egress rule for the runner's uid
2023
2024No Go code. The check script is the test: `make deploy-runner` runs it
2025after loading the rule and before restarting the runner, and runbook R3
2026runs the whole path against the scratch repository before merge.
2027
2028**Files:**
2029- Create: `deploy/gitbay-runner-egress.nft`, `deploy/gitbay-runner-egress.service`, `deploy/runner-egress-check.sh`
2030- Modify: `deploy/gitbay-runner.override.conf` (a `[Unit]` section before `[Service]` at line 25)
2031- Modify: `deploy/runner-podman-setup.sh` (after `podman --version`, line 29)
2032- Modify: `Makefile:69-85` (`deploy-runner`)
2033
2034**Interfaces:**
2035- Produces: nftables table `inet gitbay_runner`; unit
2036 `gitbay-runner-egress.service`, required by `gitbay-runner.service`;
2037 rule file at `/etc/gitbay-runner/egress.nft` on the runner host.
2038
2039- [ ] **Step 1: The rule**
2040
2041Create `deploy/gitbay-runner-egress.nft`:
2042
2043```
2044#!/usr/sbin/nft -f
2045# Host egress for CI builds (#260). Loaded by gitbay-runner-egress.service,
2046# which gitbay-runner.service requires, so the runner does not start
2047# without it. `make deploy-runner` installs it as
2048# /etc/gitbay-runner/egress.nft.
2049#
2050# Under rootless podman with pasta, a build's connections are made by
2051# pasta on the host, from sockets owned by the runner's user, ci-runner.
2052# nftables sees them exactly as it sees the runner's own ssh, so this
2053# table cannot tell a build from its runner. It limits what that user
2054# reaches on this host, and the runner needs little: 127.0.0.1:22, to
2055# poll, clone and stream logs.
2056#
2057# Every packet to one of the host's own addresses, loopback or public,
2058# leaves through lo, so the output hook sees host-bound traffic as
2059# oifname "lo". Traffic to other hosts is not matched: builds keep
2060# outbound internet access, trusted or not (go mod download needs it).
2061#
2062# What ci-runner may reach on this host:
2063# 127.0.0.1:22 the forge over loopback, for the runner. Builds do
2064# not reach loopback at all: the runner starts them
2065# with pasta's gateway mapping off (--no-map-gw).
2066# loopback :53 the host's resolver, which pasta forwards a
2067# build's DNS to when the host's nameserver is a
2068# loopback address.
2069# public 22/80/443 the forge, as anyone on the internet reaches it.
2070# Everything else is rejected: the admin sshd on 2222 on every address,
2071# and any service bound to loopback. -isolation none builds run as the
2072# same user and get the same rule.
2073#
2074# The account name is resolved when the file is loaded. A restart of
2075# nftables.service (flush ruleset) removes this table; `systemctl
2076# reload gitbay-runner-egress` puts it back.
2077
2078table inet gitbay_runner
2079delete table inet gitbay_runner
2080
2081table inet gitbay_runner {
2082 chain output {
2083 type filter hook output priority filter; policy accept;
2084 oifname "lo" meta skuid "ci-runner" jump host
2085 }
2086
2087 chain host {
2088 ip daddr 127.0.0.1 tcp dport 22 accept
2089 ip daddr 127.0.0.0/8 meta l4proto { tcp, udp } th dport 53 accept
2090 ip6 daddr ::1 meta l4proto { tcp, udp } th dport 53 accept
2091 ip daddr != 127.0.0.0/8 tcp dport { 22, 80, 443 } accept
2092 ip6 daddr != ::1 tcp dport { 22, 80, 443 } accept
2093 counter reject
2094 }
2095}
2096```
2097
2098The first `table` line creates the table if it is missing so the
2099`delete` never fails; the file then replaces it in one transaction, so
2100a reload never leaves a moment without the rule. The uid match is in
2101the base chain's one rule rather than a `!=` accept, because a packet
2102with no socket (a kernel-sent reset) matches neither `==` nor `!=` on
2103`skuid` and would otherwise fall through to the reject.
2104
2105- [ ] **Step 2: The unit**
2106
2107Create `deploy/gitbay-runner-egress.service`:
2108
2109```
2110# Loads the CI runner's host egress rule (#260,
2111# deploy/gitbay-runner-egress.nft). gitbay-runner.service requires this
2112# unit, so the runner starts only with the rule in force; stopping this
2113# unit removes the table and stops the runner with it.
2114#
2115# Ordered after nftables.service and ufw.service: either may rewrite the
2116# ruleset at boot, and nftables.service's default config starts with
2117# flush ruleset. A missing unit in After= is ignored.
2118#
2119# Reload re-reads the file and replaces the table in one transaction; it
2120# does not restart the runner, which a restart of this unit would
2121# (Requires= propagates restarts). `make deploy-runner` reloads.
2122[Unit]
2123Description=Host egress rule for CI builds
2124After=nftables.service ufw.service
2125Before=gitbay-runner.service
2126
2127[Service]
2128Type=oneshot
2129RemainAfterExit=yes
2130ExecStart=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
2131ExecReload=/usr/sbin/nft -f /etc/gitbay-runner/egress.nft
2132ExecStop=/usr/sbin/nft delete table inet gitbay_runner
2133
2134[Install]
2135WantedBy=multi-user.target
2136```
2137
2138- [ ] **Step 3: The runner requires it**
2139
2140In `deploy/gitbay-runner.override.conf`, insert before `[Service]`
2141(line 25):
2142
2143```
2144[Unit]
2145# The host egress rule (#260, gitbay-runner-egress.nft) limits what this
2146# unit's user reaches on the host: 127.0.0.1:22 for the runner, the
2147# forge's public 22, 80 and 443 for builds, nothing else. Required, so
2148# the runner does not start without it: a table that failed to load must
2149# not mean builds reach the admin sshd.
2150Requires=gitbay-runner-egress.service
2151After=gitbay-runner-egress.service
2152```
2153
2154- [ ] **Step 4: The check**
2155
2156Create `deploy/runner-egress-check.sh`:
2157
2158```sh
2159#!/bin/sh
2160# Check the CI runner's host egress rule (#260) as the runner's user:
2161# the forge over loopback on 22 must answer (the runner polls there),
2162# and the admin sshd on 2222 must not, on loopback or the public
2163# address. `make deploy-runner` runs this after loading the rule and
2164# before restarting the runner, and stops on a failure.
2165#
2166# ssh -p 2222 root@bay1 'sh -s' < deploy/runner-egress-check.sh
2167set -eu
2168
2169RUNNER_USER="${RUNNER_USER:-ci-runner}"
2170public=$(hostname -I | awk '{print $1}')
2171
2172probe() {
2173 su -s /bin/bash "$RUNNER_USER" -c "timeout 5 bash -c 'exec 3<>/dev/tcp/$1/$2'" 2>/dev/null
2174}
2175
2176nft list table inet gitbay_runner >/dev/null
2177
2178for dest in 127.0.0.1:22 "$public:22"; do
2179 if ! probe "${dest%:*}" "${dest##*:}"; then
2180 echo "$RUNNER_USER cannot reach $dest: the egress rule would stop the runner" >&2
2181 exit 1
2182 fi
2183done
2184for dest in 127.0.0.1:2222 "$public:2222"; do
2185 if probe "${dest%:*}" "${dest##*:}"; then
2186 echo "$RUNNER_USER reaches $dest: the egress rule is not in force" >&2
2187 exit 1
2188 fi
2189done
2190echo "egress for $RUNNER_USER: 127.0.0.1:22 and $public:22 open, 2222 refused"
2191```
2192
2193`hostname -I` lists the host's addresses, IPv4 first on bay1; the
2194first is the public one there.
2195
2196- [ ] **Step 5: nftables on the host**
2197
2198In `deploy/runner-podman-setup.sh`, after the podman install block
2199(after `podman --version`, line 29):
2200
2201```sh
2202# nft loads the runner's host egress rule (#260,
2203# deploy/gitbay-runner-egress.nft), which `make deploy-runner` ships and
2204# the runner's unit requires. Without nft the runner does not start.
2205echo "==> installing nftables"
2206if ! command -v nft >/dev/null 2>&1; then
2207 apt-get update
2208 DEBIAN_FRONTEND=noninteractive apt-get install -y nftables
2209fi
2210nft --version
2211```
2212
2213- [ ] **Step 6: `make deploy-runner` ships, loads and checks it**
2214
2215Replace `Makefile:69-85` with:
2216
2217```make
2218deploy-runner: preflight
2219 @echo "==> building $(RUNNER_BIN)"
2220 $(CROSS) go build -trimpath -ldflags='$(LDFLAGS)' -o $(RUNNER_BIN) ./cmd/gitbay-runner
2221 @echo "==> pushing runner to $(HOST)"
2222 ./deploy/copy.sh $(HOST) $(PORT) $(RUNNER_BIN) /usr/local/bin/gitbay-runner.new
2223 ssh -p $(PORT) root@$(HOST) 'mkdir -p /etc/systemd/system/gitbay-runner.service.d /etc/gitbay-runner'
2224 ./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner.override.conf /etc/systemd/system/gitbay-runner.service.d/override.conf
2225 ./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-prune.service /etc/systemd/system/gitbay-runner-prune.service
2226 ./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-prune.timer /etc/systemd/system/gitbay-runner-prune.timer
2227 ./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-egress.nft /etc/gitbay-runner/egress.nft
2228 ./deploy/copy.sh $(HOST) $(PORT) deploy/gitbay-runner-egress.service /etc/systemd/system/gitbay-runner-egress.service
2229 @echo "==> loading the egress rule"
2230 ssh -p $(PORT) root@$(HOST) 'set -eu; \
2231 nft -c -f /etc/gitbay-runner/egress.nft; \
2232 systemctl daemon-reload; \
2233 systemctl enable gitbay-runner-egress.service; \
2234 systemctl reload-or-restart gitbay-runner-egress.service'
2235 ssh -p $(PORT) root@$(HOST) 'sh -s' < deploy/runner-egress-check.sh
2236 ssh -p $(PORT) root@$(HOST) 'set -eu; \
2237 chmod 755 /usr/local/bin/gitbay-runner.new; \
2238 mv /usr/local/bin/gitbay-runner.new /usr/local/bin/gitbay-runner; \
2239 systemctl enable --now gitbay-runner-prune.timer; \
2240 systemctl restart gitbay-runner; \
2241 systemctl --no-pager --lines=3 status gitbay-runner; \
2242 systemctl --no-pager list-timers gitbay-runner-prune.timer'
2243```
2244
2245`nft -c` checks the file without applying it, so a syntax error stops
2246the deploy with the old table and the old runner in place. On the first
2247deploy `reload-or-restart` starts the unit, and starting a required unit
2248does not restart the runner that requires it; later deploys reload it.
2249The check runs before the runner restarts, so a failure stops the
2250deploy with the old binary in place, but the new table is already
2251loaded and applies to the running runner too. If the check says the
2252rule blocks 127.0.0.1:22, remove the table with `ssh -p 2222
2253root@gitbay.org nft delete table inet gitbay_runner` (the unit stays
2254active, so the runner is not stopped with it), fix the rule, and deploy
2255again. Runbook R3 runs this path on the scratch runner before merge.
2256
2257- [ ] **Step 7: Verify locally**
2258
2259Run: `sh -n deploy/runner-egress-check.sh && sh -n deploy/runner-podman-setup.sh && make -n deploy-runner HOST=example.test`
2260Expected: no syntax errors; the dry run lists the two new copies, the
2261`nft -c` / `reload-or-restart` block, the check, then the runner block.
2262
2263- [ ] **Step 8: Commit**
2264
2265```bash
2266git add deploy/gitbay-runner-egress.nft deploy/gitbay-runner-egress.service deploy/runner-egress-check.sh deploy/gitbay-runner.override.conf deploy/runner-podman-setup.sh Makefile
2267git commit -S -m "runner host: builds reach only the forge's public ports on it
2268
2269Ref #260"
2270```
2271
2272### Task 3.5: egress policy in the wiki, and the MR
2273
2274**Files:**
2275- Modify: `.gitbay/wiki/Threat-Model.org` ("The CI runner", after the *Images* bullet at line 169)
2276- Modify: `.gitbay/wiki/CI.org` (new section before "* The table", line 55)
2277- Modify: `.gitbay/wiki/Users.org:551-555` (the `GITBAY_SSH` sentence)
2278- Modify: `.gitbay/wiki/Admin.org` (after "It is idempotent.", line 668)
2279- Modify: `.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org:70` (Network row), `04-Trust-Boundaries.org:27` (TB7), `09-Controls.org:91`
2280
2281- [ ] **Step 1: Threat-Model**
2282
2283Add a bullet after *Images are provisioned…*:
2284
2285```org
2286- *What a build can reach.* Outbound internet, trusted or not: a fork's
2287 merge request to a Go repository has to fetch its modules. On the
2288 runner's host, only the forge's public ports 22, 80 and 443, exactly
2289 as anyone on the internet reaches them. Two layers keep it there. A
2290 runner that polls the daemon over loopback starts its containers with
2291 pasta's gateway mapping off, so a build does not reach the host's
2292 loopback, and =GITBAY_SSH= names the public address; that keeps the
2293 runner's source address, =127.0.0.1=, one no build connects from. And
2294 an nftables table (=deploy/gitbay-runner-egress.nft=) rejects every
2295 connection the runner's user makes to the host's own addresses except
2296 =127.0.0.1:22=, DNS on loopback, and 22, 80 and 443 on the public
2297 address: the operator's sshd on 2222 and anything bound to loopback
2298 are closed to builds. Under rootless podman a build's connections are
2299 made by pasta as the runner's user, so the table cannot tell a build
2300 from its runner and leaves =127.0.0.1:22= open; the gateway mapping
2301 is what closes it to builds. The runner does not start without the
2302 table. The SSH auth limiter counts failures per source address and,
2303 once an address is over the limit, refuses every key from it until
2304 the window passes, the runner's included; with registration open or
2305 by invite an unknown key never counts (krz/gitbay#260). Under
2306 =-isolation none= a build runs on the host and shares its loopback;
2307 the table still applies, since it runs as the same user.
2308```
2309
2310- [ ] **Step 2: CI.org**
2311
2312Add before `* The table`:
2313
2314```org
2315* What a build can reach
2316
2317Builds have outbound internet access, trusted and untrusted alike. On
2318the runner's host they reach only the forge's public ports 22, 80 and
2319443: not the host's loopback, not the operator's sshd. The forge is
2320reached at its public address, the one in =GITBAY_SSH=. See the
2321Threat-Model page, "What a build can reach", for how and why
2322(krz/gitbay#260).
2323```
2324
2325- [ ] **Step 3: Users.org**
2326
2327Replace "(=git@gitbay.org= from a runner elsewhere; inside a container
2328on the server's own runner the host is at a private address the runner
2329fills in)" with "(=git@gitbay.org=, the instance's public address, from
2330a runner elsewhere and from a container on the server's own runner
2331alike; =git@host:port= on an instance whose ssh is not on 22, so use it
2332as =ssh://$GITBAY_SSH/owner/name.git= or =ssh ssh://$GITBAY_SSH …=,
2333which work in both forms)".
2334
2335- [ ] **Step 4: Admin.org**
2336
2337After "…verifies rootless podman actually runs as that user. It is
2338idempotent." add:
2339
2340```org
2341It also installs nftables. =make deploy-runner= ships
2342=deploy/gitbay-runner-egress.nft= to =/etc/gitbay-runner/egress.nft=
2343with =gitbay-runner-egress.service=, which loads it and which the
2344runner's unit requires; it checks the file with =nft -c=, reloads the
2345unit, and runs =deploy/runner-egress-check.sh= as =ci-runner= before
2346restarting the runner: =127.0.0.1:22= and the public 22 must answer,
23472222 must not. The table limits the runner's user to =127.0.0.1:22=,
2348DNS on loopback, and 22, 80 and 443 on the host's public address; the
2349Threat-Model page says why. A restart of =nftables.service= flushes it;
2350=systemctl reload gitbay-runner-egress= restores it.
2351```
2352
2353- [ ] **Step 5: Architecture**
2354
2355`07-CI-and-Supply-Chain.org:70` Network row:
2356
2357```org
2358| Network | pasta; outbound open; a loopback runner's builds run with =--no-map-gw= (=main.go=); on the host only public 22/80/443 (=gitbay-runner-egress.nft=, #260) |
2359```
2360
2361`04-Trust-Boundaries.org:27` TB7: replace "the network is open (#260)"
2362with "outbound is open; on the host only the forge's public ports
2363(#260)". `09-Controls.org:91`:
2364
2365```org
2366| Build network egress restricted | partial | host: loopback closed, public 22/80/443 only (=gitbay-runner-egress.nft=); internet outbound open by decision (#260) |
2367```
2368
2369The Known-Gaps row for #260 and its "What can a build reach…" question
2370stay until the runbook's R3 results are recorded on the CI page.
2371
2372- [ ] **Step 6: Verify, commit, MR**
2373
2374Run: `go build ./... && go vet ./... && go test ./cmd/gitbay-runner ./internal/control ./internal/sshd -count=1`
2375Expected: PASS.
2376
2377```bash
2378git add .gitbay/wiki
2379git commit -S -m "wiki: what a build can reach
2380
2381Ref #260"
2382git push -u origin runner-source-address
2383gitbay mr create --source runner-source-address --target main --title "runner: keep builds off the runner's source address and the host's other ports"
2384```
2385
2386Before merging, run the runbook's R3 on the scratch repository. Merge
2387with `--strategy ff`; delete the branch both places.
2388
2389---
2390
2391# Part 4: failed step and duration (branch `build-failure-report`, #266)
2392
2393### Task 4.1: store the failed step
2394
2395**Files:**
2396- Create: `internal/store/migrations/0065_build_failure.up.sql`, `0065_build_failure.down.sql`
2397- Modify: `internal/store/builds.go:12-33` (`Build`), `:67-78` (`buildSelect`, `scanBuild`), after `FinishBuild` (`:251-264`)
2398- Test: `internal/store/builds_test.go` (append; add `"errors"` to imports)
2399
2400**Interfaces:**
2401- Produces:
2402 - `Build.FailedStep int`, `Build.FailedReason string`
2403 - `func (s *Store) SetBuildFailure(id int64, step int, reason string) error` — only on a running build; `ErrNotFound` otherwise.
2404
2405- [ ] **Step 1: Write the failing test**
2406
2407```go
2408// Where a failed build stopped is recorded while it runs, before the
2409// outcome, and a finished build is not rewritten (#266).
2410func TestSetBuildFailure(t *testing.T) {
2411 s := open(t)
2412 if err := s.MigrateUp(); err != nil {
2413 t.Fatal(err)
2414 }
2415 uid, _ := s.CreateUser("cmc", true)
2416 repoID, _ := s.CreateRepo("user", uid, "app", "public")
2417 if _, err := s.CreateBuild(repoID, "unit", "abc", "main", `["true","false"]`, "", "", true); err != nil {
2418 t.Fatal(err)
2419 }
2420 b, ok, err := s.ClaimBuild(nil, false)
2421 if err != nil || !ok {
2422 t.Fatalf("claim: ok=%v err=%v", ok, err)
2423 }
2424 if err := s.SetBuildFailure(b.ID, 2, "exit 1"); err != nil {
2425 t.Fatal(err)
2426 }
2427 if err := s.FinishBuild(b.ID, "failure"); err != nil {
2428 t.Fatal(err)
2429 }
2430 got, err := s.BuildByID(b.ID)
2431 if err != nil {
2432 t.Fatal(err)
2433 }
2434 if got.FailedStep != 2 || got.FailedReason != "exit 1" {
2435 t.Fatalf("failed step %d reason %q", got.FailedStep, got.FailedReason)
2436 }
2437 if err := s.SetBuildFailure(b.ID, 1, "late"); !errors.Is(err, ErrNotFound) {
2438 t.Fatalf("rewrote a finished build: %v", err)
2439 }
2440}
2441```
2442
2443- [ ] **Step 2: Run it and see it fail**
2444
2445Run: `go test ./internal/store -run TestSetBuildFailure -count=1`
2446Expected: build failure (`s.SetBuildFailure undefined`).
2447
2448- [ ] **Step 3: Migration**
2449
2450`0065_build_failure.up.sql`:
2451
2452```sql
2453-- Where a failed build stopped: the 1-based step, 0 when it stopped
2454-- before any step or did not fail, and the runner's one-line reason.
2455ALTER TABLE builds ADD COLUMN failed_step INTEGER NOT NULL DEFAULT 0;
2456ALTER TABLE builds ADD COLUMN failed_reason TEXT NOT NULL DEFAULT '';
2457```
2458
2459`0065_build_failure.down.sql`:
2460
2461```sql
2462ALTER TABLE builds DROP COLUMN failed_reason;
2463ALTER TABLE builds DROP COLUMN failed_step;
2464```
2465
2466- [ ] **Step 4: Store**
2467
2468In `Build`, after `Trusted`:
2469
2470```go
2471 // FailedStep is the 1-based step a failed build stopped at, 0 when it
2472 // stopped before its first step or did not fail. FailedReason is the
2473 // runner's one line: "exit 1", "build timed out after 45m0s".
2474 FailedStep int
2475 FailedReason string
2476```
2477
2478`buildSelect` and `scanBuild`:
2479
2480```go
2481const buildSelect = `
2482 SELECT id, repo_id, number, job, sha, ref, steps, image, tree, status, created_at, started_at, finished_at, log_closed_at, trusted,
2483 failed_step, failed_reason
2484 FROM builds`
2485
2486func scanBuild(row interface{ Scan(...any) error }) (Build, error) {
2487 var b Build
2488 var trusted int
2489 err := row.Scan(&b.ID, &b.RepoID, &b.Number, &b.Job, &b.SHA, &b.Ref, &b.Steps, &b.Image, &b.Tree,
2490 &b.Status, &b.CreatedAt, &b.StartedAt, &b.FinishedAt, &b.LogClosedAt, &trusted,
2491 &b.FailedStep, &b.FailedReason)
2492 b.Trusted = trusted != 0
2493 return b, err
2494}
2495```
2496
2497After `FinishBuild`:
2498
2499```go
2500// SetBuildFailure records where a running build failed. The runner
2501// reports it with the outcome; it is written first, so a reader woken
2502// by the finish sees both.
2503func (s *Store) SetBuildFailure(id int64, step int, reason string) error {
2504 res, err := s.DB.Exec(`UPDATE builds SET failed_step = ?, failed_reason = ?
2505 WHERE id = ? AND status = 'running'`, step, reason, id)
2506 if err != nil {
2507 return err
2508 }
2509 if n, _ := res.RowsAffected(); n == 0 {
2510 return ErrNotFound
2511 }
2512 return nil
2513}
2514```
2515
2516- [ ] **Step 5: Run the tests and see them pass**
2517
2518Run: `go test ./internal/store -count=1`
2519Expected: PASS, `TestMigrateUpDown` included.
2520
2521- [ ] **Step 6: Commit**
2522
2523```bash
2524git add internal/store
2525git commit -S -m "store: failed step and reason on a build
2526
2527Ref #266"
2528```
2529
2530### Task 4.2: `runner done --step --reason`
2531
2532**Files:**
2533- Modify: `internal/control/build.go:102-106` (registration), `:648-713` (`runRunnerDone`)
2534- Test: `internal/control/runnernext_test.go` (append)
2535
2536**Interfaces:**
2537- Consumes: `Store.SetBuildFailure`.
2538- Produces: `runner done <build-id> success|failure [--step <n>] [--reason <text>]`; `func failureReason(s string) string`.
2539
2540- [ ] **Step 1: Write the failing tests**
2541
2542```go
2543// runner done records the failed step and a one-line reason (#266).
2544func TestRunnerDoneRecordsFailedStep(t *testing.T) {
2545 st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
2546 n, err := st.CreateBuild(repo.ID, "unit", baseSHA, "main", `["go build ./...","go test ./..."]`, "", "", true)
2547 if err != nil {
2548 t.Fatal(err)
2549 }
2550 b, ok, err := st.ClaimBuild([]int64{repo.ID}, false)
2551 if err != nil || !ok {
2552 t.Fatalf("claim: ok=%v err=%v", ok, err)
2553 }
2554 c, out := runnerCtx(st, uid, root)
2555 if code := runRunnerDone(c, []string{fmt.Sprint(b.ID), "failure", "--step", "2", "--reason", "exit 1\n"}); code != protocol.ExitOK {
2556 t.Fatalf("runner done: exit %d\n%s", code, out.String())
2557 }
2558 got, _ := st.BuildByNumber(repo.ID, n)
2559 if got.Status != "failure" || got.FailedStep != 2 || got.FailedReason != "exit 1" {
2560 t.Fatalf("status %s step %d reason %q", got.Status, got.FailedStep, got.FailedReason)
2561 }
2562}
2563
2564// A report with no flags — an older runner — or with a step past the
2565// job's still finishes the build; the step is then recorded as 0.
2566func TestRunnerDoneToleratesMissingOrBadStep(t *testing.T) {
2567 st, repo, uid, root, baseSHA, _ := setupOrphanRepo(t)
2568 for _, extra := range [][]string{nil, {"--step", "9"}} {
2569 n, _ := st.CreateBuild(repo.ID, "unit", baseSHA, "main", `["true"]`, "", "", true)
2570 b, ok, err := st.ClaimBuild([]int64{repo.ID}, false)
2571 if err != nil || !ok {
2572 t.Fatalf("claim: ok=%v err=%v", ok, err)
2573 }
2574 c, out := runnerCtx(st, uid, root)
2575 if code := runRunnerDone(c, append([]string{fmt.Sprint(b.ID), "failure"}, extra...)); code != protocol.ExitOK {
2576 t.Fatalf("runner done %v: exit %d\n%s", extra, code, out.String())
2577 }
2578 if got, _ := st.BuildByNumber(repo.ID, n); got.Status != "failure" || got.FailedStep != 0 {
2579 t.Fatalf("%v: status %s step %d", extra, got.Status, got.FailedStep)
2580 }
2581 }
2582}
2583```
2584
2585- [ ] **Step 2: Run them and see them fail**
2586
2587Run: `go test ./internal/control -run TestRunnerDone -count=1`
2588Expected: FAIL, the first with exit 2 (usage: four arguments where two
2589are accepted).
2590
2591- [ ] **Step 3: Implement**
2592
2593Registration:
2594
2595```go
2596 register(Command{Path: []string{"runner", "done"},
2597 Summary: "finish a build",
2598 Usage: "runner done <build-id> success|failure [--step <n>] [--reason <text>]",
2599 Flags: []Flag{
2600 {"--step", "<n>", "the 1-based step a failed build stopped at", ""},
2601 {"--reason", "<text>", "how it failed, one line", ""},
2602 },
2603 Examples: []string{"runner done 431 success", "runner done 431 failure --step 3 --reason 'exit 1'"},
2604 Run: runRunnerDone})
2605```
2606
2607Replace `runRunnerDone` with:
2608
2609```go
2610func runRunnerDone(c *Ctx, args []string) int {
2611 key, code := runnerSession(c)
2612 if code >= 0 {
2613 return code
2614 }
2615 f, err := parseFlags(args, flagSpec{Values: []string{"--step", "--reason"}, MaxPos: 2,
2616 Usage: "runner done <build-id> success|failure [--step <n>] [--reason <text>]"})
2617 if err != nil {
2618 return c.fail(protocol.ExitUsage, "%v", err)
2619 }
2620 if len(f.Pos) != 2 || (f.Pos[1] != "success" && f.Pos[1] != "failure") {
2621 return c.usage()
2622 }
2623 outcome := f.Pos[1]
2624 id, err := strconv.ParseInt(f.Pos[0], 10, 64)
2625 if err != nil {
2626 return c.fail(protocol.ExitUsage, "bad build id %q", f.Pos[0])
2627 }
2628 b, err := c.Store.BuildByID(id)
2629 if err != nil {
2630 return c.fail(protocol.ExitNotFound, "no build %d", id)
2631 }
2632 if ok, err := runnerMayBuild(c, key, b.RepoID); err != nil {
2633 return c.fail(protocol.ExitFailure, "%v", err)
2634 } else if !ok {
2635 return c.fail(protocol.ExitDenied, "this key is not attached to the build's repository; a repository admin attaches it with repo runner add")
2636 }
2637 // Cancelled underneath the runner: its report is late, not wrong.
2638 // The row, the status and the log were settled by the cancel.
2639 if b.Status == "cancelled" {
2640 c.Store.RunnerDone(key.ID)
2641 return c.emit(map[string]any{"build": b.Number, "status": "cancelled"}, func(w io.Writer) {
2642 fmt.Fprintf(w, "build %d was cancelled\n", b.Number)
2643 })
2644 }
2645 if outcome == "failure" {
2646 // A step the job does not have is recorded as none rather than
2647 // refused: refusing would lose the outcome over a detail (#266).
2648 var steps []string
2649 json.Unmarshal([]byte(b.Steps), &steps)
2650 step, _ := strconv.Atoi(f.Value("--step"))
2651 if step < 0 || step > len(steps) {
2652 step = 0
2653 }
2654 if err := c.Store.SetBuildFailure(id, step, failureReason(f.Value("--reason"))); err != nil && !errors.Is(err, store.ErrNotFound) {
2655 return c.fail(protocol.ExitFailure, "recording build %d's failure: %v", id, err)
2656 }
2657 }
2658 if err := c.Store.FinishBuild(id, outcome); err != nil {
2659 return c.fail(protocol.ExitFailure, "finishing build %d: %v", id, err)
2660 }
2661 c.Store.RunnerDone(key.ID)
2662 repo, err := c.Store.RepoByID(b.RepoID)
2663 if err != nil {
2664 return c.fail(protocol.ExitFailure, "%v", err)
2665 }
2666 url := fmt.Sprintf("%s/%s/builds/%d", c.Cfg.Server.SiteURL, repo.Path(), b.Number)
2667 desc := "build " + outcome
2668 if err := c.Store.SetCommitStatus(repo.ID, b.SHA, "ci/"+b.Job, outcome, desc, url, c.User.ID); err != nil {
2669 return c.fail(protocol.ExitFailure, "%v", err)
2670 }
2671 c.Store.RecordEvent(repo.ID, c.User.ID, "build."+outcome,
2672 fmt.Sprintf(`{"number":%d,"job":%q,"sha":%q}`, b.Number, b.Job, b.SHA))
2673 // A red build mails the repo's notify targets with the log tail — a
2674 // failed scheduled job must not wait to be noticed.
2675 if outcome == "failure" {
2676 if targets, err := c.Store.RepoNotifyTargets(repo); err == nil {
2677 tail := ""
2678 if log, err := c.Store.BuildLog(id); err == nil && len(log) > 0 {
2679 if len(log) > 2000 {
2680 log = log[len(log)-2000:]
2681 }
2682 tail = string(log)
2683 }
2684 notify(c, targets, notice{repo: repo, kind: "build",
2685 subject: fmt.Sprintf("[%s] build %d failed: %s on %s", repo.Path(), b.Number, b.Job, b.Ref),
2686 action: fmt.Sprintf("build %d failed: %s on %s", b.Number, b.Job, b.Ref),
2687 body: fmt.Sprintf("job %s failed at %.10s.\n\n…%s\n\n%s\n", b.Job, b.SHA, tail, url),
2688 path: fmt.Sprintf("%s/builds/%d", repo.Path(), b.Number)})
2689 }
2690 }
2691 return c.emit(map[string]any{"build": b.Number, "status": outcome}, func(w io.Writer) {
2692 fmt.Fprintf(w, "build %d %s\n", b.Number, outcome)
2693 })
2694}
2695
2696// failureReason keeps a runner's reason to one line of at most 200
2697// bytes: it is shown on the build page and by build show.
2698func failureReason(s string) string {
2699 s = strings.Join(strings.Fields(s), " ")
2700 if len(s) > 200 {
2701 s = s[:200]
2702 }
2703 return strings.ToValidUTF8(s, "")
2704}
2705```
2706
2707- [ ] **Step 4: Run the tests and see them pass**
2708
2709Run: `go vet ./... && go test ./internal/control -count=1`
2710Expected: PASS.
2711
2712- [ ] **Step 5: Commit**
2713
2714```bash
2715git add internal/control/build.go internal/control/runnernext_test.go
2716git commit -S -m "runner done: record the failed step and reason
2717
2718Ref #266"
2719```
2720
2721### Task 4.3: the runner names the failed step
2722
2723**Files:**
2724- Modify: `cmd/gitbay-runner/main.go:245-277` (`step`), `:309-435` (`run`), `:363-398` (`runStep`)
2725- Modify: `cmd/gitbay-runner/isolate.go:79-197` (`runSteps`, `runStepsPodman`)
2726- Modify: `cmd/gitbay-runner/report.go:19-23` (`reportDone`), new `doneArgs`
2727- Test: `cmd/gitbay-runner/steps_test.go` (create), `cmd/gitbay-runner/report_test.go` (append)
2728
2729**Interfaces:**
2730- Consumes: `runner done … --step <n> --reason <text>` from Task 4.2.
2731- Produces:
2732 - `type failure struct { Step int; Reason string }`
2733 - `func exitReason(err error) string`
2734 - `run(j job) *failure`, `runSteps(…) *failure`, `runStepsPodman(…) *failure` (nil is success)
2735 - `func (r *runner) reportDone(id int64, status string, f *failure) error`
2736 - `func doneArgs(id int64, status string, f *failure) []string`
2737
2738- [ ] **Step 1: Write the failing tests**
2739
2740Create `cmd/gitbay-runner/steps_test.go`:
2741
2742```go
2743package main
2744
2745import (
2746 "os"
2747 "os/exec"
2748 "strings"
2749 "testing"
2750 "time"
2751)
2752
2753// The failing step is named by number in the log and in the outcome
2754// reported to the server (#266).
2755func TestRunStepsNamesTheFailedStep(t *testing.T) {
2756 r := &runner{isolation: isolationNone}
2757 run := func(cmd *exec.Cmd, _ time.Time) (bool, string) {
2758 if err := cmd.Run(); err != nil {
2759 return false, exitReason(err)
2760 }
2761 return true, ""
2762 }
2763 env := []string{"PATH=" + os.Getenv("PATH")}
2764 var log strings.Builder
2765 f := r.runSteps(job{Steps: []string{"true", "exit 3", "true"}}, t.TempDir(), env, &log, time.Now().Add(time.Minute), run)
2766 if f == nil || f.Step != 2 || f.Reason != "exit 3" {
2767 t.Fatalf("failure %+v, want step 2, exit 3", f)
2768 }
2769 if !strings.Contains(log.String(), "step 2/3 failed: exit 3\n") {
2770 t.Fatalf("log does not name the step:\n%s", log.String())
2771 }
2772 if f := r.runSteps(job{Steps: []string{"true"}}, t.TempDir(), env, &log, time.Now().Add(time.Minute), run); f != nil {
2773 t.Fatalf("a passing job failed: %+v", f)
2774 }
2775}
2776```
2777
2778Append to `cmd/gitbay-runner/report_test.go` (add imports
2779`"strings"` and `"gitbay.org/gitbay/internal/protocol"`):
2780
2781```go
2782// The reason survives the trip: ssh joins arguments with spaces and the
2783// server splits the line again with POSIX rules (#266).
2784func TestDoneArgsNameTheFailedStep(t *testing.T) {
2785 got := doneArgs(7, "failure", &failure{Step: 3, Reason: "can't: exit 1"})
2786 argv, err := protocol.Tokenize(strings.Join(got, " "))
2787 if err != nil {
2788 t.Fatal(err)
2789 }
2790 want := []string{"runner", "done", "7", "failure", "--step", "3", "--reason", "can't: exit 1"}
2791 if strings.Join(argv, "|") != strings.Join(want, "|") {
2792 t.Fatalf("server reads %q, want %q", argv, want)
2793 }
2794 if got := doneArgs(7, "success", nil); strings.Join(got, " ") != "runner done 7 success" {
2795 t.Fatalf("success: %q", got)
2796 }
2797 if got := doneArgs(7, "failure", &failure{Reason: "git clone: exit 128"}); strings.Contains(strings.Join(got, " "), "--step") {
2798 t.Fatalf("a failure before any step sent a step: %q", got)
2799 }
2800}
2801```
2802
2803- [ ] **Step 2: Run them and see them fail**
2804
2805Run: `go test ./cmd/gitbay-runner -count=1`
2806Expected: build failure (`undefined: exitReason`, `undefined: doneArgs`, `undefined: failure`).
2807
2808- [ ] **Step 3: `failure` and `exitReason`**
2809
2810In `main.go`, before `run`:
2811
2812```go
2813// failure says where a build stopped: Step is the 1-based step that
2814// failed, 0 when the build stopped before its first step (the clone, the
2815// container), and Reason is one short line (#266).
2816type failure struct {
2817 Step int
2818 Reason string
2819}
2820
2821// exitReason is how a finished command's failure reads in a build's log
2822// and on the build: "exit 1" for a command that exited, the error
2823// otherwise (a signal, a start failure).
2824func exitReason(err error) string {
2825 var ee *exec.ExitError
2826 if errors.As(err, &ee) && ee.ExitCode() >= 0 {
2827 return fmt.Sprintf("exit %d", ee.ExitCode())
2828 }
2829 return err.Error()
2830}
2831```
2832
2833Add `"errors"` to `main.go`'s imports.
2834
2835- [ ] **Step 4: `run` and `step`**
2836
2837In `run`, the signature becomes `func (r *runner) run(j job) *failure`
2838with its comment "…Returns nil when every step succeeded, else where the
2839build stopped." Each early `return false` returns a failure instead:
2840
2841```go
2842 home, doneHome, err := buildHome(r.workdir, j)
2843 if err != nil {
2844 log.Printf("build %d: build home: %v", j.ID, err)
2845 return &failure{Reason: "preparing the build home failed"}
2846 }
2847 defer doneHome()
2848```
2849
2850```go
2851 pipe, err := logCmd.StdinPipe()
2852 if err != nil {
2853 log.Printf("build %d: log pipe: %v", j.ID, err)
2854 return &failure{Reason: "opening the log stream failed"}
2855 }
2856```
2857
2858```go
2859 if err := logCmd.Start(); err != nil {
2860 log.Printf("build %d: log stream: %v", j.ID, err)
2861 return &failure{Reason: "opening the log stream failed"}
2862 }
2863```
2864
2865In `runStep`, `return false, fmt.Sprintf("step failed: %v", err)`
2866becomes `return false, exitReason(err)`.
2867
2868The clone loop's failure:
2869
2870```go
2871 if ok, why := runStep(cmd, deadline); !ok {
2872 fmt.Fprintf(sink, "git %s: %s\n", args[0], why)
2873 return &failure{Reason: "git " + args[0] + ": " + why}
2874 }
2875```
2876
2877`step()`, from `status := "failure"` to the report:
2878
2879```go
2880 f := r.run(j)
2881 status := "success"
2882 if f != nil {
2883 status = "failure"
2884 }
2885 if err := r.reportDone(j.ID, status, f); err != nil {
2886 return true, err
2887 }
2888```
2889
2890- [ ] **Step 5: `runSteps`, `runStepsPodman`**
2891
2892In `isolate.go`, the `runSteps` comment ends "…Returns nil when every
2893step succeeded." and the function becomes:
2894
2895```go
2896func (r *runner) runSteps(j job, dir string, env []string, sink io.Writer, deadline time.Time, runStep stepRunner) *failure {
2897 if r.isolation == isolationNone {
2898 for i, step := range j.Steps {
2899 fmt.Fprintf(sink, "$ %s\n", step)
2900 cmd := exec.Command(toolpath.Look("sh"), "-c", step)
2901 cmd.Dir, cmd.Env = dir, env
2902 cmd.Stdout, cmd.Stderr = sink, sink
2903 if ok, why := runStep(cmd, deadline); !ok {
2904 fmt.Fprintf(sink, "step %d/%d failed: %s\n", i+1, len(j.Steps), why)
2905 return &failure{Step: i + 1, Reason: why}
2906 }
2907 }
2908 return nil
2909 }
2910 return r.runStepsPodman(j, dir, env, sink, deadline, runStep)
2911}
2912```
2913
2914`runStepsPodman` returns `*failure`. Its setup failures (env file,
2915cgroup, container start) keep their log lines and return
2916`&failure{Reason: "preparing the build environment failed"}`,
2917`&failure{Reason: "preparing the build cgroup failed"}` and
2918`&failure{Reason: "starting the build container failed"}`
2919respectively. The step loop:
2920
2921```go
2922 for i, step := range j.Steps {
2923 fmt.Fprintf(sink, "$ %s\n", step)
2924 cmd := exec.Command(podman, append(r.podmanGlobal(), "exec", "--workdir", "/workspace", name, "sh", "-c", step)...)
2925 cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "HOME=" + r.podmanHome()}
2926 intoCgroup(cmd, cgroupFD)
2927 cmd.Stdout, cmd.Stderr = sink, sink
2928 if ok, why := runStep(cmd, deadline); !ok {
2929 fmt.Fprintf(sink, "step %d/%d failed: %s\n", i+1, len(j.Steps), why)
2930 return &failure{Step: i + 1, Reason: why}
2931 }
2932 }
2933 return nil
2934```
2935
2936`podman exec` exits with the step's own status, so `exit 1` is the
2937step's.
2938
2939- [ ] **Step 6: `reportDone`, `doneArgs`**
2940
2941In `report.go` (add `"strconv"` and `"strings"` to its imports):
2942
2943```go
2944func (r *runner) reportDone(id int64, status string, f *failure) error {
2945 args := doneArgs(id, status, f)
2946 return reportWithRetry(func() (string, error) {
2947 return r.ssh(nil, args...)
2948 }, id, retryDelays)
2949}
2950
2951// doneArgs is the runner done command for a build's outcome. The reason
2952// is single-quoted: ssh joins arguments with spaces, and the server
2953// splits the line again with POSIX rules.
2954func doneArgs(id int64, status string, f *failure) []string {
2955 args := []string{"runner", "done", fmt.Sprint(id), status}
2956 if f == nil {
2957 return args
2958 }
2959 if f.Step > 0 {
2960 args = append(args, "--step", strconv.Itoa(f.Step))
2961 }
2962 if f.Reason != "" {
2963 args = append(args, "--reason", "'"+strings.ReplaceAll(f.Reason, "'", `'\''`)+"'")
2964 }
2965 return args
2966}
2967```
2968
2969(The comment above `reportDone` is unchanged.)
2970
2971- [ ] **Step 7: Run the tests and see them pass**
2972
2973Run: `go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner -count=1`
2974Expected: PASS.
2975
2976- [ ] **Step 8: Commit**
2977
2978```bash
2979git add cmd/gitbay-runner
2980git commit -S -m "runner: name the failed step and report it
2981
2982Ref #266"
2983```
2984
2985### Task 4.4: `build show`, `build log --step/--tail`
2986
2987**Files:**
2988- Create: `internal/control/buildlog.go`, `internal/control/buildlog_test.go`
2989- Modify: `internal/control/build.go:40-47` (registration), `:109-126` (`BuildOut`, `buildToOut`), `:221-238` (`runBuildShow`), `:240-258` (`runBuildLog`)
2990
2991**Interfaces:**
2992- Consumes: `Build.FailedStep`, `Build.FailedReason`, `Build.Elapsed()`.
2993- Produces (used by Task 4.5):
2994 - `type LogSection struct { N int; Step string; Text string }`
2995 - `func SplitBuildLog(log string, steps []string) []LogSection`
2996 - `func FailedSection(sections []LogSection, status string, failedStep int) int`
2997 - `BuildOut.FailedStep int` (`failed_step`), `BuildOut.FailedReason string` (`failed_reason`), `BuildOut.DurationS int64` (`duration_s`), `BuildOut.Steps []string` (`steps`, on `build show` only)
2998
2999- [ ] **Step 1: Write the failing tests**
3000
3001Create `internal/control/buildlog_test.go`:
3002
3003```go
3004package control
3005
3006import (
3007 "bytes"
3008 "fmt"
3009 "reflect"
3010 "regexp"
3011 "strings"
3012 "testing"
3013
3014 "gitbay.org/gitbay/internal/protocol"
3015 "gitbay.org/gitbay/internal/store"
3016)
3017
3018func TestSplitBuildLog(t *testing.T) {
3019 log := "$ git clone ssh://x/a.git (abc)\n" +
3020 "$ go build ./...\n" +
3021 "built\n" +
3022 "$ go test ./...\n" +
3023 "--- FAIL: TestX\n" +
3024 "step 2/2 failed: exit 1\n"
3025 got := SplitBuildLog(log, []string{"go build ./...", "go test ./..."})
3026 want := []LogSection{
3027 {N: 0, Text: "$ git clone ssh://x/a.git (abc)\n"},
3028 {N: 1, Step: "go build ./...", Text: "built\n"},
3029 {N: 2, Step: "go test ./...", Text: "--- FAIL: TestX\nstep 2/2 failed: exit 1\n"},
3030 }
3031 if !reflect.DeepEqual(got, want) {
3032 t.Fatalf("got %+v\nwant %+v", got, want)
3033 }
3034}
3035
3036// A step's line inside other output, not at a line start, does not cut;
3037// a build that stopped before a step has no section for it; an empty
3038// setup is left out.
3039func TestSplitBuildLogStopsAtMissingStep(t *testing.T) {
3040 got := SplitBuildLog("$ make\nrunning: $ make test\nerror\n", []string{"make", "make test"})
3041 want := []LogSection{{N: 1, Step: "make", Text: "running: $ make test\nerror\n"}}
3042 if !reflect.DeepEqual(got, want) {
3043 t.Fatalf("got %+v\nwant %+v", got, want)
3044 }
3045}
3046
3047func TestTailLines(t *testing.T) {
3048 for _, tc := range []struct {
3049 in string
3050 n int
3051 want string
3052 }{
3053 {"a\nb\nc\n", 2, "b\nc\n"},
3054 {"a\nb\nc\n", 5, "a\nb\nc\n"},
3055 {"a\nb", 1, "b"},
3056 } {
3057 if got := string(tailLines([]byte(tc.in), tc.n)); got != tc.want {
3058 t.Errorf("tailLines(%q, %d) = %q, want %q", tc.in, tc.n, got, tc.want)
3059 }
3060 }
3061}
3062
3063// failedBuild is a finished failure whose second of two steps failed,
3064// having run 10m56s.
3065func failedBuild(t *testing.T) (*store.Store, store.Repo, int64, int64) {
3066 t.Helper()
3067 st, repo, uid := newQueueTestRepo(t)
3068 n, err := st.CreateBuild(repo.ID, "unit", "abc", "main", `["go build ./...","go test ./..."]`, "", "", true)
3069 if err != nil {
3070 t.Fatal(err)
3071 }
3072 b, ok, err := st.ClaimBuild([]int64{repo.ID}, false)
3073 if err != nil || !ok {
3074 t.Fatalf("claim: ok=%v err=%v", ok, err)
3075 }
3076 st.AppendBuildLog(b.ID, []byte("$ git clone x (abc)\n$ go build ./...\nok\n$ go test ./...\none\n--- FAIL: TestX\nstep 2/2 failed: exit 1\n"))
3077 if err := st.SetBuildFailure(b.ID, 2, "exit 1"); err != nil {
3078 t.Fatal(err)
3079 }
3080 if err := st.FinishBuild(b.ID, "failure"); err != nil {
3081 t.Fatal(err)
3082 }
3083 if _, err := st.DB.Exec(`UPDATE builds SET started_at = '2026-09-27T10:00:00Z', finished_at = '2026-09-27T10:10:56Z' WHERE id = ?`, b.ID); err != nil {
3084 t.Fatal(err)
3085 }
3086 return st, repo, uid, n
3087}
3088
3089func TestBuildLogStepAndTail(t *testing.T) {
3090 st, repo, uid, n := failedBuild(t)
3091 run := func(args ...string) (string, int) {
3092 t.Helper()
3093 c, errOut := pruneCtx(st, t.TempDir(), store.User{ID: uid})
3094 code := Dispatch(c, append([]string{"build", "log", repo.Path(), fmt.Sprint(n)}, args...))
3095 return c.Stdout.(*bytes.Buffer).String() + errOut.String(), code
3096 }
3097 if out, _ := run("--step", "1"); out != "ok\n" {
3098 t.Errorf("--step 1: %q", out)
3099 }
3100 if out, _ := run("--step", "failed", "--tail", "2"); out != "--- FAIL: TestX\nstep 2/2 failed: exit 1\n" {
3101 t.Errorf("--step failed --tail 2: %q", out)
3102 }
3103 if out, _ := run("--tail", "1"); out != "step 2/2 failed: exit 1\n" {
3104 t.Errorf("--tail 1: %q", out)
3105 }
3106 if _, code := run("--step", "3"); code != protocol.ExitUsage {
3107 t.Errorf("--step past the job: exit %d", code)
3108 }
3109 if _, code := run("--follow", "--tail", "1"); code != protocol.ExitUsage {
3110 t.Errorf("--follow with --tail: exit %d", code)
3111 }
3112}
3113
3114func TestBuildShowNamesFailedStepAndDuration(t *testing.T) {
3115 st, repo, uid, n := failedBuild(t)
3116 c, errOut := pruneCtx(st, t.TempDir(), store.User{ID: uid})
3117 if code := Dispatch(c, []string{"build", "show", repo.Path(), fmt.Sprint(n)}); code != protocol.ExitOK {
3118 t.Fatalf("exit %d: %s", code, errOut)
3119 }
3120 out := c.Stdout.(*bytes.Buffer).String()
3121 for _, re := range []string{`failed step\s+2/2 go test \./\.\.\. \(exit 1\)`, `duration\s+10m56s`} {
3122 if !regexp.MustCompile(re).MatchString(out) {
3123 t.Errorf("build show missing %s:\n%s", re, out)
3124 }
3125 }
3126 c, _ = pruneCtx(st, t.TempDir(), store.User{ID: uid})
3127 c.JSON = true
3128 Dispatch(c, []string{"build", "show", repo.Path(), fmt.Sprint(n)})
3129 for _, want := range []string{`"failed_step":2`, `"failed_reason":"exit 1"`, `"duration_s":656`, `"steps":["go build ./...","go test ./..."]`} {
3130 if !strings.Contains(c.Stdout.(*bytes.Buffer).String(), want) {
3131 t.Errorf("build show --json missing %s", want)
3132 }
3133 }
3134}
3135```
3136
3137- [ ] **Step 2: Run them and see them fail**
3138
3139Run: `go test ./internal/control -run 'TestSplitBuildLog|TestTailLines|TestBuildLogStep|TestBuildShowNames' -count=1`
3140Expected: build failure (`undefined: SplitBuildLog`).
3141
3142- [ ] **Step 3: `buildlog.go`**
3143
3144```go
3145package control
3146
3147import "strings"
3148
3149// LogSection is one part of a build log: the setup before the first
3150// step (N 0), or one step and its output.
3151type LogSection struct {
3152 N int // 0 for the setup, else the 1-based step
3153 Step string // the step's command; "" for the setup
3154 Text string
3155}
3156
3157// SplitBuildLog cuts a log at the "$ <step>" line the runner writes
3158// before each step, matching the build's steps in order and only at a
3159// line start. Output before the first step is the setup section, left
3160// out when empty. A step with no line in the log — the build stopped
3161// before it — has no section, and neither has any step after it.
3162func SplitBuildLog(log string, steps []string) []LogSection {
3163 var out []LogSection
3164 cur := LogSection{}
3165 start := 0
3166 for i, step := range steps {
3167 marker := "$ " + step + "\n"
3168 at := findLine(log, marker, start)
3169 if at < 0 {
3170 break
3171 }
3172 cur.Text = log[start:at]
3173 if cur.N > 0 || cur.Text != "" {
3174 out = append(out, cur)
3175 }
3176 cur = LogSection{N: i + 1, Step: step}
3177 start = at + len(marker)
3178 }
3179 cur.Text = log[start:]
3180 if cur.N > 0 || cur.Text != "" {
3181 out = append(out, cur)
3182 }
3183 return out
3184}
3185
3186// findLine is the index of line in log at or after from where it starts
3187// a line, or -1.
3188func findLine(log, line string, from int) int {
3189 for i := from; i <= len(log)-len(line); {
3190 j := strings.Index(log[i:], line)
3191 if j < 0 {
3192 return -1
3193 }
3194 at := i + j
3195 if at == 0 || log[at-1] == '\n' {
3196 return at
3197 }
3198 i = at + 1
3199 }
3200 return -1
3201}
3202
3203// FailedSection is the index of the section a failed build stopped in:
3204// the step the runner named, or the last section when it named none (an
3205// older runner, or a failure the runner could not tie to a step). -1
3206// when the build did not fail or its log is empty.
3207func FailedSection(sections []LogSection, status string, failedStep int) int {
3208 if status != "failure" || len(sections) == 0 {
3209 return -1
3210 }
3211 for i, s := range sections {
3212 if failedStep > 0 && s.N == failedStep {
3213 return i
3214 }
3215 }
3216 return len(sections) - 1
3217}
3218
3219// tailLines is the last n lines of b; a final newline ends the last line
3220// rather than starting another.
3221func tailLines(b []byte, n int) []byte {
3222 end := len(b)
3223 if end > 0 && b[end-1] == '\n' {
3224 end--
3225 }
3226 for i := end - 1; i >= 0; i-- {
3227 if b[i] == '\n' {
3228 n--
3229 if n == 0 {
3230 return b[i+1:]
3231 }
3232 }
3233 }
3234 return b
3235}
3236```
3237
3238- [ ] **Step 4: `BuildOut`, `build show`**
3239
3240`BuildOut`, after `Subject`:
3241
3242```go
3243 // FailedStep is the 1-based step a failed build stopped at, 0 when
3244 // none; FailedReason says how ("exit 1") (#266).
3245 FailedStep int `json:"failed_step,omitempty"`
3246 FailedReason string `json:"failed_reason,omitempty"`
3247 // DurationS is how long the build ran, once it has a start and a
3248 // finish.
3249 DurationS int64 `json:"duration_s,omitempty"`
3250 // Steps are the job's commands; build show only.
3251 Steps []string `json:"steps,omitempty"`
3252```
3253
3254`buildToOut`:
3255
3256```go
3257func buildToOut(b store.Build) BuildOut {
3258 return BuildOut{Number: b.Number, Job: b.Job, Status: b.Status, SHA: b.SHA,
3259 Ref: b.Ref, CreatedAt: b.CreatedAt, FinishedAt: b.FinishedAt,
3260 FailedStep: b.FailedStep, FailedReason: b.FailedReason,
3261 DurationS: int64(b.Elapsed() / time.Second)}
3262}
3263```
3264
3265`runBuildShow`:
3266
3267```go
3268func runBuildShow(c *Ctx, args []string) int {
3269 repo, b, code := buildRef(c, args)
3270 if code >= 0 {
3271 return code
3272 }
3273 d := buildToOut(b)
3274 json.Unmarshal([]byte(b.Steps), &d.Steps)
3275 return c.emit(d, func(w io.Writer) {
3276 failedStep, failed := "", ""
3277 if d.FailedStep > 0 && d.FailedStep <= len(d.Steps) {
3278 step, _, _ := strings.Cut(d.Steps[d.FailedStep-1], "\n")
3279 failedStep = fmt.Sprintf("%d/%d %s", d.FailedStep, len(d.Steps), step)
3280 if d.FailedReason != "" {
3281 failedStep += " (" + d.FailedReason + ")"
3282 }
3283 } else {
3284 failed = d.FailedReason
3285 }
3286 duration := ""
3287 if d.DurationS > 0 {
3288 duration = (time.Duration(d.DurationS) * time.Second).String()
3289 }
3290 v := c.view(w)
3291 v.title(fmt.Sprintf("#%d", d.Number), d.Job, d.Status)
3292 v.fields(
3293 "sha", fmt.Sprintf("%.10s", d.SHA),
3294 "ref", d.Ref,
3295 "queued", c.when(d.CreatedAt),
3296 "finished", c.when(d.FinishedAt),
3297 "duration", duration,
3298 "failed step", failedStep,
3299 "failed", failed,
3300 "url", c.siteURL(repo.Path(), "builds", strconv.FormatInt(d.Number, 10)),
3301 )
3302 })
3303}
3304```
3305
3306- [ ] **Step 5: `build log`**
3307
3308Registration:
3309
3310```go
3311 register(Command{Path: []string{"build", "log"},
3312 Summary: "print a build's log, or follow it until the build ends",
3313 Usage: "build log <owner/name> <n> [--follow] [--step <step>|failed] [--tail <lines>]",
3314 Flags: []Flag{
3315 {"--follow", "", "stream the log until the build ends", ""},
3316 {"--step", "<step>|failed", "only one step's output: 0 for the setup, a step number, or the one that failed", ""},
3317 {"--tail", "<lines>", "only the last lines", ""},
3318 },
3319 Examples: []string{"build log krz/gitbay 431 --follow", "build log krz/gitbay 431 --step failed --tail 40"},
3320 ReadOnly: true, Run: runBuildLog})
3321```
3322
3323`runBuildLog`:
3324
3325```go
3326func runBuildLog(c *Ctx, args []string) int {
3327 f, err := parseFlags(args, flagSpec{Bools: []string{"--follow"}, Values: []string{"--step", "--tail"}, MaxPos: 2, Usage: c.Cmd.Usage})
3328 if err != nil {
3329 return c.fail(protocol.ExitUsage, "%v", err)
3330 }
3331 repo, b, code := buildRef(c, f.Pos)
3332 if code >= 0 {
3333 return code
3334 }
3335 if f.Has("--follow") {
3336 if f.Has("--step") || f.Has("--tail") {
3337 return c.fail(protocol.ExitUsage, "--step and --tail read the stored log; drop --follow")
3338 }
3339 return followBuildLog(c, repo, b)
3340 }
3341 tail := 0
3342 if f.Has("--tail") {
3343 if tail, err = strconv.Atoi(f.Value("--tail")); err != nil || tail < 1 {
3344 return c.fail(protocol.ExitUsage, "--tail takes a number of lines, 1 or more")
3345 }
3346 }
3347 log, err := c.Store.BuildLog(b.ID)
3348 if err != nil {
3349 return c.fail(protocol.ExitFailure, "%v", err)
3350 }
3351 if f.Has("--step") {
3352 var steps []string
3353 json.Unmarshal([]byte(b.Steps), &steps)
3354 sections := SplitBuildLog(string(log), steps)
3355 at := -1
3356 if want := f.Value("--step"); want == "failed" {
3357 if at = FailedSection(sections, b.Status, b.FailedStep); at < 0 {
3358 return c.fail(protocol.ExitNotFound, "build %d did not fail", b.Number)
3359 }
3360 } else {
3361 n, err := strconv.Atoi(want)
3362 if err != nil || n < 0 || n > len(steps) {
3363 return c.fail(protocol.ExitUsage, "--step takes 0 (the setup) to %d, or failed", len(steps))
3364 }
3365 for i, s := range sections {
3366 if s.N == n {
3367 at = i
3368 }
3369 }
3370 if at < 0 {
3371 return c.fail(protocol.ExitNotFound, "build %d has no output for step %d", b.Number, n)
3372 }
3373 }
3374 log = []byte(sections[at].Text)
3375 }
3376 if tail > 0 {
3377 log = tailLines(log, tail)
3378 }
3379 c.Stdout.Write(log)
3380 return protocol.ExitOK
3381}
3382```
3383
3384- [ ] **Step 6: Run the tests and see them pass**
3385
3386Run: `go vet ./... && go test ./internal/control -count=1 && go test ./cmd/gitbay -run TestSummariesAreCurrent -count=1`
3387Expected: PASS (the `build log` summary is unchanged; no regeneration
3388needed).
3389
3390- [ ] **Step 7: Commit**
3391
3392```bash
3393git add internal/control
3394git commit -S -m "build show: failed step and duration; build log --step, --tail
3395
3396Ref #266"
3397```
3398
3399### Task 4.5: the build page folds by step
3400
3401**Files:**
3402- Modify: `internal/httpd/builds.go:1-18` (imports: add `"time"`), `:287-322` (`build`, `buildView`)
3403- Modify: `internal/web/templates/build.html:14-17`
3404- Modify: `internal/web/static/style.css:1116`
3405- Test: `internal/httpd/buildpages_test.go` (append)
3406
3407**Interfaces:**
3408- Consumes: `control.SplitBuildLog`, `control.FailedSection`, `control.LogSection`, `BuildOut` fields from Task 4.4.
3409- Produces: `buildView.Steps []logStep`, `buildView.Failed bool`, `buildView.Duration string`; `func logSteps(log string, b control.BuildOut) ([]logStep, bool)`.
3410
3411- [ ] **Step 1: Write the failing test**
3412
3413Append to `internal/httpd/buildpages_test.go`:
3414
3415```go
3416// A failed build's page folds its log by step, opens the step that
3417// failed and links to it; no JavaScript (#266).
3418func TestBuildPageFoldsStepsAndOpensFailure(t *testing.T) {
3419 b := control.BuildOut{Number: 61, Job: "test", Status: "failure",
3420 SHA: "ff6271a9d4570cd46f169091637a9d2e40ad5c2b", Ref: "main",
3421 CreatedAt: "2026-08-28T04:42:54Z", FinishedAt: "2026-08-28T04:53:50Z", DurationS: 656,
3422 Steps: []string{"go build ./...", "go test ./..."}, FailedStep: 2, FailedReason: "exit 1"}
3423 log := "$ git clone x (ff6271a9d4)\n$ go build ./...\n$ go test ./...\n--- FAIL: TestCLI\nstep 2/2 failed: exit 1\n"
3424 v := buildView{repoPage: testRepoPage(), Build: b, Log: log, Duration: "10m56s"}
3425 v.Steps, v.Failed = logSteps(log, b)
3426 var sb strings.Builder
3427 if err := web.Render(&sb, "build.html", v); err != nil {
3428 t.Fatalf("render: %v", err)
3429 }
3430 out := sb.String()
3431 for _, want := range []string{
3432 `<details class="difffold buildstep" id="failed" open>`,
3433 "step 2/2", "<code>go test ./...</code>", `href="#failed"`, "Jump to failure",
3434 "ran 10m56s", "--- FAIL: TestCLI",
3435 } {
3436 if !strings.Contains(out, want) {
3437 t.Errorf("build.html missing %q", want)
3438 }
3439 }
3440 if n := strings.Count(out, `class="difffold buildstep"`); n != 3 {
3441 t.Errorf("%d step folds, want 3 (setup and two steps)", n)
3442 }
3443 if n := strings.Count(out, `id="failed"`); n != 1 {
3444 t.Errorf("%d failed anchors, want 1", n)
3445 }
3446}
3447```
3448
3449- [ ] **Step 2: Run it and see it fail**
3450
3451Run: `go test ./internal/httpd -run TestBuildPageFoldsStepsAndOpensFailure -count=1`
3452Expected: build failure (`undefined: logSteps`).
3453
3454- [ ] **Step 3: Handler**
3455
3456Replace `buildView` and add `logStep` and `logSteps`:
3457
3458```go
3459type buildView struct {
3460 repoPage
3461 Build control.BuildOut
3462 Log string
3463 // Steps is the finished log cut at its steps, nil when there is no
3464 // step to cut at; Failed says whether one of them is marked failed.
3465 Steps []logStep
3466 Failed bool
3467 Duration string
3468 Live bool
3469 CanWrite bool
3470 Notice string
3471}
3472
3473type logStep struct {
3474 control.LogSection
3475 Failed bool
3476}
3477
3478// logSteps cuts a finished build's log at its steps and marks the one it
3479// failed at. Nil when no step's line is in the log — a build that
3480// stopped in the clone — which renders as one block.
3481func logSteps(log string, b control.BuildOut) ([]logStep, bool) {
3482 sections := control.SplitBuildLog(log, b.Steps)
3483 stepped := false
3484 for _, s := range sections {
3485 if s.N > 0 {
3486 stepped = true
3487 }
3488 }
3489 if !stepped {
3490 return nil, false
3491 }
3492 failed := control.FailedSection(sections, b.Status, b.FailedStep)
3493 out := make([]logStep, len(sections))
3494 for i, s := range sections {
3495 out[i] = logStep{LogSection: s, Failed: i == failed}
3496 }
3497 return out, failed >= 0
3498}
3499```
3500
3501In `build`, replace the last two lines (`v.Log, _, _ = …` and
3502`s.render(…)`) with:
3503
3504```go
3505 v.Log, _, _ = s.runControl(viewer, []string{"build", "log", p.Repo.Path(), n})
3506 v.Steps, v.Failed = logSteps(v.Log, b)
3507 if b.DurationS > 0 {
3508 v.Duration = (time.Duration(b.DurationS) * time.Second).String()
3509 }
3510 s.render(w, "build.html", v)
3511```
3512
3513Add `"time"` to the imports.
3514
3515- [ ] **Step 4: Template**
3516
3517Replace `build.html:14-17` with:
3518
3519```html
3520<p class="meta">{{.Build.Job}} on {{.Build.Ref}} · <code><a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/commit/{{.Build.SHA}}">{{printf "%.10s" .Build.SHA}}</a></code> · queued {{when .Build.CreatedAt}}{{if .Build.FinishedAt}} · finished {{when .Build.FinishedAt}}{{with .Duration}} · ran {{.}}{{end}}{{end}}{{if .Failed}} · <a href="#failed">Jump to failure</a>{{end}}</p>
3521{{if .Live}}<p class="meta">Live: the log streams here until the build ends. If it stops without a “build finished” line, reload to pick it up again. <a href="?follow=0">Show it without updates</a></p>
3522<pre class="code buildlog" tabindex="0">{{.Log}}</pre>
3523{{else if .Steps}}{{$total := len .Build.Steps}}{{range .Steps}}
3524<details class="difffold buildstep"{{if .Failed}} id="failed" open{{end}}>
3525 <summary>{{if .N}}<span>step {{.N}}/{{$total}}</span> <code>{{.Step}}</code>{{else}}<span>setup</span>{{end}}{{if .Failed}} <span class="chip check-failure">failed</span>{{end}}</summary>
3526 <pre class="code buildlog" tabindex="0">{{.Text}}</pre>
3527</details>{{end}}
3528{{else if .Log}}<pre class="code buildlog" tabindex="0">{{.Log}}</pre>{{else}}<p class="empty-note">no log yet</p>{{end}}
3529```
3530
3531The live branch keeps its single `<pre>` directly after the marker, which
3532`streamBuild` requires (`builds.go:340`).
3533
3534- [ ] **Step 5: CSS**
3535
3536Replace `style.css:1116` with:
3537
3538```css
3539pre.buildlog { max-height: 40rem; overflow: auto; white-space: pre-wrap; overflow-wrap: anywhere; }
3540details.buildstep pre.buildlog { margin: 0; border: 0; border-radius: 0; }
3541details.buildstep summary code { overflow-wrap: anywhere; }
3542```
3543
3544- [ ] **Step 6: Run the tests**
3545
3546Run: `go test ./internal/httpd ./internal/web -count=1`
3547Expected: PASS (`TestPreBlocksAreFocusable` sees the new `<pre>` with
3548`tabindex="0"`; no new template, so `TestMainWidthClass` is unchanged).
3549
3550- [ ] **Step 7: Commit**
3551
3552```bash
3553git add internal/httpd internal/web
3554git commit -S -m "web: build log folded by step, failed step open
3555
3556Ref #266"
3557```
3558
3559### Task 4.6: e2e, wiki, and the MR
3560
3561**Files:**
3562- Modify: `e2e/ci_test.go:145-149` (`TestCI`)
3563- Modify: `.gitbay/wiki/CI.org` (after the `build log --follow` paragraph), `.gitbay/wiki/Users.org:542-547`, `.gitbay/wiki/Parity.org:211-213`, `.gitbay/wiki/Architecture/07-CI-and-Supply-Chain.org` (lifecycle step 5)
3564
3565- [ ] **Step 1: e2e**
3566
3567Replace `e2e/ci_test.go:145-149` with:
3568
3569```go
3570 out, _, _ = inst.ssh(t, aliceKey, "", "build", "log", "alice/app", brokenN)
3571 if !strings.Contains(out, "step 1/1 failed: exit 1") {
3572 t.Fatalf("broken log:\n%s", out)
3573 }
3574 out, _, _ = inst.ssh(t, aliceKey, "", "build", "show", "alice/app", brokenN)
3575 if !strings.Contains(out, "1/1 false (exit 1)") {
3576 t.Fatalf("build show does not name the failed step:\n%s", out)
3577 }
3578 if out, _, _ = inst.ssh(t, aliceKey, "", "build", "log", "alice/app", brokenN, "--step", "failed"); strings.Contains(out, "git clone") || !strings.Contains(out, "exit 1") {
3579 t.Fatalf("build log --step failed:\n%s", out)
3580 }
3581 if _, body := inst.get(t, "/alice/app/builds/"+brokenN); !strings.Contains(body, `id="failed" open`) {
3582 t.Fatalf("build page does not open the failed step:\n%s", body)
3583 }
3584```
3585
3586Before editing, check the `broken` job's step in `TestCI`'s `ci.yml`
3587(line ~100) is `"false"`; the expected `1/1 false` follows from it.
3588
3589Run: `go test ./e2e -run 'TestCI$' -count=1`
3590Expected: PASS.
3591
3592- [ ] **Step 2: Wiki**
3593
3594CI.org, after the `build log --follow` paragraph:
3595
3596```org
3597A failed build names the step it stopped at: the log's last line reads
3598=step 3/3 failed: exit 1=, and =build show= prints =failed step= (=3/3
3599go test ./... (exit 1)=) and =duration=. =build log <owner/name> <n>
3600--step failed= prints only that step's output, =--step 2= another one
3601(=0= is the clone before the first step), and =--tail 40= the last forty
3602lines of whichever was chosen; neither combines with =--follow=. The
3603build page folds the finished log into one section per step, opens the
3604failed one and links to it from the top as "Jump to failure". Builds
3605from before this reported no step; their last section is taken as the
3606failed one.
3607```
3608
3609Users.org, after "…=build list= takes =--ref=, =--status= and =--job=
3610to narrow the listing, combinable;" sentence group, add: "=build show=
3611names a failed build's step and how long it ran, and =build log= takes
3612=--step <n>|failed= and =--tail <lines>=."
3613
3614Parity.org, after `build log follow (until it ends)`:
3615
3616```org
3617| build failed step, duration | yes | yes | no |
3618| build log one step | yes | yes | no |
3619| build log tail | yes | no | no |
3620```
3621
3622`07-CI-and-Supply-Chain.org`, lifecycle step 5, replace
3623"=runner done <id> success|failure= sets the status," with "=runner
3624done <id> success|failure [--step <n>] [--reason <text>]= records where
3625a failed build stopped, sets the status,".
3626
3627- [ ] **Step 3: Verify, commit, MR**
3628
3629Run: `go build ./... && go vet ./... && go test ./cmd/gitbay-runner ./cmd/gitbay ./internal/control ./internal/store ./internal/httpd ./internal/web -count=1`
3630Expected: PASS.
3631
3632```bash
3633git add e2e/ci_test.go .gitbay/wiki
3634git commit -S -m "wiki: failed step, build log --step and --tail
3635
3636Closes #266"
3637git push -u origin build-failure-report
3638gitbay mr create --source build-failure-report --target main --title "builds: name the failed step and duration; jump to failure"
3639```
3640
3641Deploy `gitbayd` (schema 64→65 on the first start, or whatever the
3642number is after renumbering), then validate per runbook R4, then merge
3643with `--strategy ff` and delete the branch both places.
3644
3645---
3646
3647# Open questions
3648
36491. **pasta on bay1.** The plan relies on `--network pasta:--no-map-gw`
3650 removing the host-loopback path and on a pasta container reaching the
3651 host's public address. The code comment at `main.go:465-470` says
3652 pasta exposes the host at `169.254.1.2` as its `--map-host-loopback`
3653 default; newer podman instead passes `--map-guest-addr 169.254.1.2`,
3654 which maps to the host's public address. Which one bay1's podman
3655 does is not in the repository. Runbook R3 step 0 measures it before
3656 Part 3 deploys; if `--no-map-gw` is refused or leaves `127.0.0.1`
3657 reachable, stop and revisit Part 3 before merging. The nftables
3658 table does not cover this path: a build's connection to 127.0.0.1:22
3659 through pasta is ci-runner's, like the runner's own poll.
3660
3661---
3662
3663# Operator runbook (cmc)
3664
3665Everything here runs from the laptop against bay1. One forge write per
3666Bash call; after `make deploy` the CLI's control master is gone, so do
3667not poll with several ssh calls a tick. Operator ssh is
3668`ssh -p 2222 root@gitbay.org`.
3669
3670### R1. Scratch repository and scoped runner (once, before Part 1's runner deploy)
3671
36721. Create the scratch repository and its fork, and attach the bay1
3673 runner key to the scratch repository only:
3674
3675 ```sh
3676 gitbay repo create cmc/ci-scratch --private
3677 gitbay repo fork cmc/ci-scratch --name ci-scratch-fork
3678 ssh -p 2222 root@gitbay.org cat /var/lib/gitbay-runner/.ssh/id_ed25519.pub > /tmp/ci-runner.pub
3679 gitbay repo runner add cmc/ci-scratch < /tmp/ci-runner.pub
3680 ```
3681
36822. Scope the service to it with a second drop-in that sorts after
3683 `override.conf`. Copy the current `ExecStart` from
3684 `deploy/gitbay-runner.override.conf` and add
3685 `-repos cmc/ci-scratch`:
3686
3687 ```sh
3688 ssh -p 2222 root@gitbay.org 'cat > /etc/systemd/system/gitbay-runner.service.d/zz-scratch.conf' <<'EOF'
3689 [Service]
3690 ExecStart=
3691 ExecStart=/usr/local/bin/gitbay-runner -remote git@127.0.0.1 -workdir /var/lib/gitbay-runner/work -poll 5s -timeout 45m -isolation podman -image localhost/gitbay-ci:2 -cpus 3 -memory 6g -untrusted -repos cmc/ci-scratch
3692 EOF
3693 ```
3694
3695 `make deploy-runner` reloads and restarts the unit, so the scoped
3696 `ExecStart` is live for each validation below. While it is in place
3697 builds of real repositories queue and wait.
3698
36993. After each validation: remove the drop-in and restart.
3700
3701 ```sh
3702 ssh -p 2222 root@gitbay.org 'rm /etc/systemd/system/gitbay-runner.service.d/zz-scratch.conf && systemctl daemon-reload && systemctl restart gitbay-runner'
3703 ```
3704
3705### R2. Part 1 (#255): validate, then discard the old homes
3706
37071. `make deploy` from the Part 1 branch; `curl -s https://gitbay.org/healthz`
3708 names its commit. Install the R1 drop-in, then `make deploy-runner`.
37092. On `cmc/ci-scratch` `main`, push a `.gitbay/ci.yml`:
3710
3711 ```yaml
3712 jobs:
3713 home:
3714 steps:
3715 - echo "HOME=$HOME"
3716 - test ! -e "$HOME/poison" || { echo "POISONED"; exit 1; }
3717 - touch "$HOME/trusted-marker"
3718 ```
3719
3720 The build passes; its log shows `HOME=/var/lib/gitbay-runner/work/trusted-home/cmc/ci-scratch`.
37213. In `cmc/ci-scratch-fork`, change the step list to
3722 `echo "HOME=$HOME"`, `ls -a "$HOME"`, `touch "$HOME/poison"`, push a
3723 branch and open an MR into `cmc/ci-scratch`. The untrusted build's
3724 log shows `HOME=/var/lib/gitbay-runner/work/build-<id>-home` and an
3725 empty listing (no `trusted-marker`).
37264. On bay1: `ls /var/lib/gitbay-runner/work` shows no `build-<id>-home`
3727 left behind.
37285. `gitbay build trigger cmc/ci-scratch home`: passes (no `POISONED`).
37296. Remove the R1 drop-in (R1 step 3). Merge Part 1.
37307. Discard the homes that trusted and untrusted builds shared:
3731
3732 ```sh
3733 ssh -p 2222 root@gitbay.org 'chmod -R u+w /var/lib/gitbay-runner/work/home && rm -rf /var/lib/gitbay-runner/work/home'
3734 ```
3735
3736 The laptop runner (`~/Library/Caches/gitbay-runner/home` or its
3737 configured workdir) gets the same once its brew bottle carries Part 1.
37388. Record: the release's CHANGELOG upgrade note says to deploy gitbayd
3739 before runners, and that `<workdir>/home` can be deleted after the
3740 runner upgrade. Nothing further in the wiki; Part 1's MR updated it.
3741
3742### R3. Part 3 (#260): pasta check, egress rule, measurement from a build
3743
3744No throttling test runs on production; Task 3.3's unit test covers the
3745limiter. What is measured here is what a build sees.
3746
37470. Before merging Part 3, on bay1: re-run the host setup (idempotent;
3748 it now installs nftables), then check pasta as the runner user:
3749
3750 ```sh
3751 ssh -p 2222 root@gitbay.org 'sh -s' < deploy/runner-podman-setup.sh
3752 ssh -p 2222 root@gitbay.org "podman --version; pasta --version | head -1; nft --version; hostname -I"
3753 ssh -p 2222 root@gitbay.org "su - ci-runner -s /bin/sh -c 'podman --cgroup-manager=cgroupfs run --rm --pull=never --network pasta:--no-map-gw --entrypoint sh localhost/gitbay-ci:2 -c \"getent hosts proxy.golang.org; timeout 5 bash -c \\\"exec 3<>/dev/tcp/gitbay.org/22\\\" && echo public-ok; cat /proc/net/route\"'"
3754 ```
3755
3756 Expected: `proxy.golang.org` resolves, `public-ok` prints, and
3757 `hostname -I` lists the public IPv4 address first (the check script
3758 takes the first). If podman rejects the option, stop (open question
3759 1).
37601. `make deploy` from the Part 3 branch, the R1 drop-in, then
3761 `make deploy-runner`. Its output shows `==> loading the egress rule`
3762 and then `egress for ci-runner: 127.0.0.1:22 and <public>:22 open,
3763 2222 refused` before the runner restarts. If the check fails, follow
3764 Task 3.4 Step 6 (remove the table, fix, deploy again) before
3765 anything else: the running runner is under the new table.
37662. On `cmc/ci-scratch` `main`, a probe job (keep each step under 4096
3767 bytes):
3768
3769 ```yaml
3770 jobs:
3771 probe:
3772 steps:
3773 - echo "GITBAY_SSH=$GITBAY_SSH"
3774 - |
3775 host=${GITBAY_SSH#*@}; host=${host%:*}
3776 for t in 127.0.0.1:22 127.0.0.1:2222 169.254.1.2:22 $host:22 $host:80 $host:443 $host:2222 proxy.golang.org:443; do
3777 timeout 5 bash -c "exec 3<>/dev/tcp/${t%:*}/${t##*:}" 2>/dev/null && echo "open $t" || echo "closed $t"
3778 done
3779 - bash -c 'exec 3<>/dev/tcp/${GITBAY_SSH#*@}/22; sleep 90'
3780 ```
3781
37823. While the last step holds its connection open, on bay1:
3783
3784 ```sh
3785 ssh -p 2222 root@gitbay.org "ss -tn state established '( sport = :22 )'"
3786 ```
3787
3788 Note the peer address of the build's connection and of the runner's
3789 `runner log` session. Expected: the runner's is `127.0.0.1`, the
3790 build's is the host's public address, never `127.0.0.1`.
37914. Expected build log: `GITBAY_SSH=git@gitbay.org`; `closed
3792 127.0.0.1:22`, `closed 127.0.0.1:2222`; `169.254.1.2:22` as
3793 measured (open only if podman maps that address to the public one);
3794 `open gitbay.org:22`, `:80`, `:443`; `closed gitbay.org:2222`;
3795 `open proxy.golang.org:443`.
37965. The runner kept polling: `gitbay admin runners` shows a recent poll
3797 for the bay1 key, and the probe build finished with its log.
37986. Remove the R1 drop-in. Trigger one real trusted job that talks back
3799 (`gitbay build trigger krz/orgo <its release or pages job>` only if
3800 one is due; otherwise wait for the next hutch/orgo scheduled job) and
3801 check it reached `git@gitbay.org`.
38027. Record the result, one commit on a branch `wiki-260-results` with
3803 `Closes #260`: in CI.org under "What a build can reach", a dated
3804 paragraph with the podman, pasta and nft versions, the source
3805 address the forge saw for a build and for the runner (step 3), and
3806 the reachability list from step 4. In
3807 `Architecture/10-Known-Gaps.org`, remove the `#260` row and answer
3808 "What can a build reach on the host's network?" with the date and a
3809 pointer to the CI page. `09-Controls` stays `partial` (outbound is
3810 open by decision). MR, `--strategy ff`, delete the branch.
3811
3812### R4. Part 4 (#266): validate the failure report
3813
38141. `make deploy` from the Part 4 branch (migration 0065 runs on start),
3815 the R1 drop-in, `make deploy-runner`.
38162. On `cmc/ci-scratch` `main`:
3817
3818 ```yaml
3819 jobs:
3820 fail:
3821 steps:
3822 - echo one
3823 - echo two
3824 - echo about to fail; exit 7
3825 ```
3826
38273. Expected: the log ends `step 3/3 failed: exit 7`; `gitbay build show
3828 cmc/ci-scratch <n>` prints `failed step 3/3 echo about to fail; exit 7
3829 (exit 7)` and a `duration`; `gitbay build log cmc/ci-scratch <n> --step
3830 failed` prints `about to fail` and the failure line; the build page
3831 opens step 3 and "Jump to failure" scrolls to it; at phone width the
3832 log wraps with no sideways scroll.
38334. Remove the R1 drop-in; merge Part 4. Once all four parts are in,
3834 delete `cmc/ci-scratch` and its fork, or keep them as the standing
3835 scratch pair for the next runner change.
3836
3837---
3838
3839# Self-review
3840
3841- **Coverage.** #255: explicit trust (1.1), disposable untrusted home
3842 and trusted-only caches (1.2), the discard (R2.7), wiki (1.3). #258:
3843 `ci/` refused (2.1), reuse by trust and declared image, the default
3844 image's limit documented (2.2, 2.4), required contexts turning the
3845 gate on, shown by `settings show` and the web page, missing as
3846 pending in `MergeGates` (2.3), wiki (2.4). #260: public destination
3847 with the port off 22 (3.1), builds off loopback (3.2), the limiter's
3848 lockout as a unit test instead of a production test (3.3), the host
3849 egress table with its unit, check and deploy wiring (3.4), egress
3850 policy in Threat-Model, CI and Admin (3.5), the source address and
3851 reachability measured from a scratch build and recorded on the CI
3852 page (R3), with the scratch-repository rule (R1). #266: runner names the step (4.3), stored
3853 step and duration (4.1; duration derived), `build show` (4.4),
3854 `build log --step`/`--tail` (4.4), web `<details>` per step with the
3855 failed one open, `id="failed"`, "Jump to failure", duration beside
3856 finished (4.5), `pre.buildlog` wraps (4.5).
3857- **Placeholders.** Every code step carries the code. The one reference
3858 to "the number after renumbering" is the migration rule, not a gap.
3859- **Names across tasks.** `buildHome` (1.2) is used by `run` in 1.2 and
3860 4.3. `job.Trusted` (1.2), `job.SSH` (3.2). `buildSSH(public string)`
3861 replaces `buildSSH()` in 3.2 and the call site changes there.
3862 `failure`/`exitReason`/`doneArgs` (4.3). `SetBuildFailure` (4.1) is
3863 used in 4.2 and 4.4's test fixture. `SplitBuildLog`, `FailedSection`,
3864 `LogSection` (4.4) are used by `logSteps` (4.5). `SuccessBuildForTree`
3865 gains `image` in 2.2 and its one caller changes there.
3866 `GatesOut.ChecksMissing` (2.3) is rendered in `mr.html` (2.3).
3867 `publicSSH` (3.1) fills the claim's `ssh`, read as `job.SSH` (3.2).
3868 The table name `inet gitbay_runner` and the path
3869 `/etc/gitbay-runner/egress.nft` match across the rule, the unit, the
3870 check script and the Makefile (3.4).