Commit a7aac9e9e6

a7aac9e9e651a8cef17bf5a1e3cb5dd7df615af3

parent: 8d58c0f01b

Verified · cmc

cmc <hello@cleberg.net> · 2026-09-09 02:23 UTC

Plan: runners attached to repositories

Ref #184

Layout: unified · split

docs/plans/2026-09-08-user-runners.md added +2378
@@ -0,0 +1,2378 @@
1# Runners attached to repositories: implementation plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** A `gitbay-runner` anyone installs, pairs with their repositories on any instance, and runs as a service; the server hands a runner key only the builds of repositories it is attached to.
6
7**Architecture:** One new table (`runner_repos`) maps an SSH key to repositories; `runner next` claims only from a key's attachments and skips untrusted builds unless asked; `repo runner add|list|remove` manage attachments and render on the settings page. The runner gains `init`, a config file, its own identity, and `-untrusted`.
8
9**Tech Stack:** Go, SQLite via hand-written SQL, `github.com/BurntSushi/toml` (already a dependency), Go templates, e2e tests against real ssh/git.
10
11**Spec:** `docs/specs/2026-09-08-user-runners-design.md`
12
13## Global Constraints
14
15- Commit messages: `<area>, <area>: <what>` on the first line, body with `Ref #184`. No attribution trailers of any kind (top rule of `~/CLAUDE.md`).
16- Never push to `main`. Work on branch `user-runners`; MR at the end.
17- Locally: `go build ./... && go vet ./...`, the unit tests of the touched packages, and at most the one e2e test being written. The full suite runs in CI on bay1.
18- Every control command parses argv through `parseFlags` (`internal/control/flags.go`). A command that reads stdin sets `ReadsStdin: true`. A read command sets `ReadOnly: true`.
19- Every new control command needs a `pass()` entry in `cmd/gitbay/main.go`; a coverage test fails otherwise.
20- Secrets never in argv, never logged. A public key is not a secret.
21- Templates: `str`/`field` are nil-safe helpers; the whole stylesheet is `internal/web/static/style.css`.
22- Migrations: next number is `0050`, both `.up.sql` and `.down.sql`. Foreign keys are on.
23- Comments and docs in plain English, no hype. Wiki is `.gitbay/wiki/*.org`.
24
25---
26
27### Task 1: Store: ClaimBuild skips untrusted builds unless asked
28
29**Files:**
30- Modify: `internal/store/builds.go:80-118` (`ClaimBuild`)
31- Modify: `internal/store/builds_test.go` (every `ClaimBuild(` call gains `, false`; one new test)
32- Modify: `internal/store/queues_test.go:27` (`ClaimBuild(nil, false)`)
33
34**Interfaces:**
35- Produces: `Store.ClaimBuild(repoIDs []int64, untrusted bool) (Build, bool, error)`. With `untrusted` false only rows with `trusted = 1` are candidates.
36
37- [ ] **Step 1: Write the failing test**
38
39Append to `internal/store/builds_test.go`:
40
41```go
42// A merge request head from a fork is untrusted. A claim skips it unless
43// the runner asked for untrusted builds, so a runner on someone's laptop
44// never executes a stranger's branch by default.
45func TestClaimBuildSkipsUntrustedUnlessAsked(t *testing.T) {
46 s := open(t)
47 if err := s.MigrateUp(); err != nil {
48 t.Fatal(err)
49 }
50 uid, err := s.CreateUser("cmc", true)
51 if err != nil {
52 t.Fatal(err)
53 }
54 repo, err := s.CreateRepo("user", uid, "app", "public")
55 if err != nil {
56 t.Fatal(err)
57 }
58 // Queued first, so an unfiltered claim would take it.
59 forkBuild, err := s.CreateBuild(repo, "unit", "abc123", "refs/merge-requests/1/head", `["true"]`, "", "", false)
60 if err != nil {
61 t.Fatal(err)
62 }
63 own, err := s.CreateBuild(repo, "unit", "def456", "main", `["true"]`, "", "", true)
64 if err != nil {
65 t.Fatal(err)
66 }
67 b, ok, err := s.ClaimBuild(nil, false)
68 if err != nil || !ok || b.Number != own {
69 t.Fatalf("trusted-only claim: err=%v ok=%v number=%d, want %d", err, ok, b.Number, own)
70 }
71 if _, ok, _ := s.ClaimBuild(nil, false); ok {
72 t.Fatal("trusted-only claim took the fork build")
73 }
74 b, ok, err = s.ClaimBuild(nil, true)
75 if err != nil || !ok || b.Number != forkBuild {
76 t.Fatalf("untrusted claim: err=%v ok=%v number=%d, want %d", err, ok, b.Number, forkBuild)
77 }
78}
79```
80
81- [ ] **Step 2: Run it to see it fail**
82
83Run: `go test ./internal/store -run TestClaimBuildSkipsUntrusted 2>&1 | head -5`
84Expected: compile error, too many arguments to `ClaimBuild`.
85
86- [ ] **Step 3: Change `ClaimBuild`**
87
88Replace the signature, doc comment and query construction in `internal/store/builds.go`:
89
90```go
91// ClaimBuild atomically hands the oldest pending build to a runner and
92// marks it running. A non-empty repoIDs restricts the claim to those
93// repositories. Untrusted builds — merge request heads from another
94// repository — are skipped unless untrusted is set: they run a stranger's
95// code, which only a runner that isolates should take.
96func (s *Store) ClaimBuild(repoIDs []int64, untrusted bool) (Build, bool, error) {
97 tx, err := s.DB.Begin()
98 if err != nil {
99 return Build{}, false, err
100 }
101 defer tx.Rollback()
102 query := "SELECT id FROM builds WHERE status = 'pending'"
103 args := []any{}
104 if !untrusted {
105 query += " AND trusted = 1"
106 }
107 if len(repoIDs) > 0 {
108 marks := strings.TrimSuffix(strings.Repeat("?,", len(repoIDs)), ",")
109 query += " AND repo_id IN (" + marks + ")"
110 for _, id := range repoIDs {
111 args = append(args, id)
112 }
113 }
114 query += " ORDER BY id LIMIT 1"
115 var id int64
116 err = tx.QueryRow(query, args...).Scan(&id)
117```
118
119The rest of the function is unchanged.
120
121- [ ] **Step 4: Update existing callers in store tests**
122
123In `internal/store/builds_test.go` and `internal/store/queues_test.go`, every `s.ClaimBuild(x)` becomes `s.ClaimBuild(x, false)`:
124
125```bash
126sed -i '' -E 's/ClaimBuild\((nil|\[\]int64\{[a-zA-Z]+\})\)/ClaimBuild(\1, false)/g' internal/store/builds_test.go internal/store/queues_test.go
127grep -n "ClaimBuild(" internal/store/*_test.go
128```
129
130Every hit must now show two arguments.
131
132- [ ] **Step 5: Run the store tests**
133
134Run: `go test ./internal/store 2>&1 | tail -3`
135Expected: PASS.
136
137- [ ] **Step 6: Commit**
138
139```bash
140git add internal/store/builds.go internal/store/builds_test.go internal/store/queues_test.go
141git commit -m "store: ClaimBuild skips untrusted builds unless asked
142
143Ref #184"
144```
145
146---
147
148### Task 2: Store: migration 0050 and runner attachments
149
150**Files:**
151- Create: `internal/store/migrations/0050_runner_repos.up.sql`
152- Create: `internal/store/migrations/0050_runner_repos.down.sql`
153- Modify: `internal/store/runners.go` (whole file)
154- Create: `internal/store/runners_test.go`
155
156**Interfaces:**
157- Consumes: `Store.AddSSHKey(userID int64, fingerprint, algo string, blob []byte, scope string) error`, `Store.SSHKeyByFingerprint(fp) (SSHKey, error)`, `Store.CreateUser(name string, admin bool) (int64, error)`, `Store.CreateRepo(kind string, ownerID int64, name, visibility string) (int64, error)`, `Store.CreateBuild(repoID int64, job, sha, ref, steps, image, tree string, trusted bool) (int64, error)`.
158- Produces:
159 - `type RepoRunner struct { Fingerprint, Algo, Username, AddedAt, LastSeen, BuildRepo string; BuildNumber int64; BuildJob, StartedAt string }`
160 - `Runner` gains `Fingerprint string` and `KeyID int64`.
161 - `Store.AttachRunner(keyID, repoID int64) error` (idempotent)
162 - `Store.DetachRunner(repoID int64, fingerprint string) error` (`ErrNotFound` when not attached)
163 - `Store.RunnerRepoIDs(keyID int64) ([]int64, error)`
164 - `Store.RunnerRepoPaths(keyID int64) ([]string, error)` (owner/name, sorted)
165 - `Store.RunnerAttached(keyID, repoID int64) (bool, error)`
166 - `Store.ListRepoRunners(repoID int64) ([]RepoRunner, error)`
167 - `Store.TouchRunner(keyID, userID int64, scope string, buildID int64) error`
168 - `Store.RunnerDone(keyID int64) error`
169 - `Store.ListRunners() ([]Runner, error)` unchanged signature.
170
171- [ ] **Step 1: Write the migration**
172
173`internal/store/migrations/0050_runner_repos.up.sql`:
174
175```sql
176-- A runner key is attached to the repositories it may claim builds for
177-- (#184). runner_seen is rekeyed by key so two runners on one account
178-- are two rows; what it held were heartbeats, so the rows are dropped.
179CREATE TABLE runner_repos (
180 key_id INTEGER NOT NULL REFERENCES ssh_keys(id) ON DELETE CASCADE,
181 repo_id INTEGER NOT NULL REFERENCES repos(id) ON DELETE CASCADE,
182 added_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%fZ','now')),
183 PRIMARY KEY (key_id, repo_id)
184);
185CREATE INDEX runner_repos_repo ON runner_repos(repo_id);
186
187DROP TABLE runner_seen;
188CREATE TABLE runner_seen (
189 key_id INTEGER PRIMARY KEY REFERENCES ssh_keys(id) ON DELETE CASCADE,
190 user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
191 last_seen TEXT NOT NULL,
192 scope TEXT NOT NULL DEFAULT '',
193 build_id INTEGER REFERENCES builds(id) ON DELETE SET NULL
194);
195```
196
197`internal/store/migrations/0050_runner_repos.down.sql`:
198
199```sql
200DROP TABLE runner_repos;
201DROP TABLE runner_seen;
202CREATE TABLE runner_seen (
203 user_id INTEGER PRIMARY KEY REFERENCES users(id) ON DELETE CASCADE,
204 last_seen TEXT NOT NULL,
205 scope TEXT NOT NULL DEFAULT '',
206 build_id INTEGER REFERENCES builds(id) ON DELETE SET NULL
207);
208```
209
210- [ ] **Step 2: Write the failing store tests**
211
212`internal/store/runners_test.go`:
213
214```go
215package store
216
217import (
218 "errors"
219 "testing"
220)
221
222// runnerFixture is one user with a runner key and two repositories.
223func runnerFixture(t *testing.T) (s *Store, uid, keyID, repoA, repoB int64) {
224 t.Helper()
225 s = open(t)
226 if err := s.MigrateUp(); err != nil {
227 t.Fatal(err)
228 }
229 uid, err := s.CreateUser("alice", false)
230 if err != nil {
231 t.Fatal(err)
232 }
233 if err := s.AddSSHKey(uid, "SHA256:runnerkey", "ssh-ed25519", []byte("blob"), "runner"); err != nil {
234 t.Fatal(err)
235 }
236 k, err := s.SSHKeyByFingerprint("SHA256:runnerkey")
237 if err != nil {
238 t.Fatal(err)
239 }
240 repoA, err = s.CreateRepo("user", uid, "a", "public")
241 if err != nil {
242 t.Fatal(err)
243 }
244 repoB, err = s.CreateRepo("user", uid, "b", "public")
245 if err != nil {
246 t.Fatal(err)
247 }
248 return s, uid, k.ID, repoA, repoB
249}
250
251// Attaching twice is one row; detaching what is not attached is not found.
252func TestAttachRunnerIdempotentAndDetach(t *testing.T) {
253 s, _, keyID, repoA, repoB := runnerFixture(t)
254 for range 2 {
255 if err := s.AttachRunner(keyID, repoA); err != nil {
256 t.Fatal(err)
257 }
258 }
259 ids, err := s.RunnerRepoIDs(keyID)
260 if err != nil || len(ids) != 1 || ids[0] != repoA {
261 t.Fatalf("attached repos %v err=%v, want [%d]", ids, err, repoA)
262 }
263 if ok, _ := s.RunnerAttached(keyID, repoB); ok {
264 t.Fatal("attached to a repo it was never attached to")
265 }
266 if err := s.DetachRunner(repoB, "SHA256:runnerkey"); !errors.Is(err, ErrNotFound) {
267 t.Fatalf("detach of an unattached repo: %v, want ErrNotFound", err)
268 }
269 if err := s.DetachRunner(repoA, "SHA256:runnerkey"); err != nil {
270 t.Fatal(err)
271 }
272 if ok, _ := s.RunnerAttached(keyID, repoA); ok {
273 t.Fatal("still attached after detach")
274 }
275}
276
277// Removing the key or the repository removes the attachment with it.
278func TestRunnerAttachmentCascades(t *testing.T) {
279 s, uid, keyID, repoA, repoB := runnerFixture(t)
280 if err := s.AttachRunner(keyID, repoA); err != nil {
281 t.Fatal(err)
282 }
283 if err := s.AttachRunner(keyID, repoB); err != nil {
284 t.Fatal(err)
285 }
286 if _, err := s.DB.Exec("DELETE FROM repos WHERE id = ?", repoB); err != nil {
287 t.Fatal(err)
288 }
289 if ids, _ := s.RunnerRepoIDs(keyID); len(ids) != 1 {
290 t.Fatalf("after repo delete: %v, want one attachment", ids)
291 }
292 if err := s.RemoveSSHKey(uid, "SHA256:runnerkey"); err != nil {
293 t.Fatal(err)
294 }
295 var n int
296 if err := s.DB.QueryRow("SELECT count(*) FROM runner_repos").Scan(&n); err != nil || n != 0 {
297 t.Fatalf("after key delete: %d rows err=%v, want 0", n, err)
298 }
299}
300
301// The heartbeat is per key: two keys on one account are two rows, and a
302// repository's runner list shows each key's last poll and the build it holds.
303func TestRunnerSeenPerKeyAndRepoList(t *testing.T) {
304 s, uid, keyID, repoA, _ := runnerFixture(t)
305 if err := s.AddSSHKey(uid, "SHA256:second", "ssh-ed25519", []byte("blob2"), "runner"); err != nil {
306 t.Fatal(err)
307 }
308 k2, _ := s.SSHKeyByFingerprint("SHA256:second")
309 for _, id := range []int64{keyID, k2.ID} {
310 if err := s.AttachRunner(id, repoA); err != nil {
311 t.Fatal(err)
312 }
313 }
314 if _, err := s.CreateBuild(repoA, "unit", "abc123", "main", `["true"]`, "", "", true); err != nil {
315 t.Fatal(err)
316 }
317 b, ok, err := s.ClaimBuild(nil, false)
318 if err != nil || !ok {
319 t.Fatalf("claim: %v ok=%v", err, ok)
320 }
321 if err := s.TouchRunner(keyID, uid, "", b.ID); err != nil {
322 t.Fatal(err)
323 }
324 if err := s.TouchRunner(k2.ID, uid, "", 0); err != nil {
325 t.Fatal(err)
326 }
327 runners, err := s.ListRunners()
328 if err != nil || len(runners) != 2 {
329 t.Fatalf("ListRunners: %v err=%v, want two rows", runners, err)
330 }
331 list, err := s.ListRepoRunners(repoA)
332 if err != nil || len(list) != 2 {
333 t.Fatalf("ListRepoRunners: %v err=%v, want two rows", list, err)
334 }
335 var held, idle int
336 for _, r := range list {
337 if r.Username != "alice" || r.LastSeen == "" || r.AddedAt == "" {
338 t.Fatalf("row %+v lacks username, last_seen or added_at", r)
339 }
340 if r.BuildNumber == b.Number && r.BuildJob == "unit" && r.BuildRepo == "alice/a" {
341 held++
342 } else if r.BuildNumber == 0 {
343 idle++
344 }
345 }
346 if held != 1 || idle != 1 {
347 t.Fatalf("held=%d idle=%d, want 1 and 1: %+v", held, idle, list)
348 }
349 if err := s.RunnerDone(keyID); err != nil {
350 t.Fatal(err)
351 }
352 list, _ = s.ListRepoRunners(repoA)
353 for _, r := range list {
354 if r.BuildNumber != 0 {
355 t.Fatalf("build still held after RunnerDone: %+v", r)
356 }
357 }
358 paths, err := s.RunnerRepoPaths(keyID)
359 if err != nil || len(paths) != 1 || paths[0] != "alice/a" {
360 t.Fatalf("RunnerRepoPaths: %v err=%v", paths, err)
361 }
362}
363```
364
365- [ ] **Step 3: Run the tests to see them fail**
366
367Run: `go test ./internal/store -run 'TestAttachRunner|TestRunnerAttachment|TestRunnerSeen' 2>&1 | head -20`
368Expected: compile errors, `s.AttachRunner undefined` and friends.
369
370- [ ] **Step 4: Replace `internal/store/runners.go`**
371
372```go
373package store
374
375import "sort"
376
377// Runner is one runner key as the instance admin sees it.
378type Runner struct {
379 Username string `json:"username"`
380 Fingerprint string `json:"fingerprint"`
381 KeyID int64 `json:"-"`
382 LastSeen string `json:"last_seen"`
383 // Scope is what the runner asked for: comma-joined owner/name, ""
384 // for any. admin runners replaces it with the attachments for a
385 // runner key.
386 Scope string `json:"scope,omitempty"`
387 // The build it holds, if any.
388 BuildRepo string `json:"build_repo,omitempty"`
389 BuildNumber int64 `json:"build_number,omitempty"`
390 BuildJob string `json:"build_job,omitempty"`
391 StartedAt string `json:"started_at,omitempty"`
392}
393
394// RepoRunner is one key attached to a repository, as repo runner list
395// shows it.
396type RepoRunner struct {
397 Fingerprint string `json:"fingerprint"`
398 Algo string `json:"algo"`
399 Username string `json:"username"`
400 AddedAt string `json:"added_at"`
401 LastSeen string `json:"last_seen,omitempty"`
402 BuildRepo string `json:"build_repo,omitempty"`
403 BuildNumber int64 `json:"build_number,omitempty"`
404 BuildJob string `json:"build_job,omitempty"`
405 StartedAt string `json:"started_at,omitempty"`
406}
407
408// AttachRunner lets a key claim a repository's builds. Attaching twice is
409// one row.
410func (s *Store) AttachRunner(keyID, repoID int64) error {
411 _, err := s.DB.Exec("INSERT OR IGNORE INTO runner_repos (key_id, repo_id) VALUES (?, ?)", keyID, repoID)
412 return err
413}
414
415// DetachRunner removes one attachment by fingerprint. The key itself stays.
416func (s *Store) DetachRunner(repoID int64, fingerprint string) error {
417 res, err := s.DB.Exec(`DELETE FROM runner_repos WHERE repo_id = ?
418 AND key_id = (SELECT id FROM ssh_keys WHERE fingerprint = ?)`, repoID, fingerprint)
419 if err != nil {
420 return err
421 }
422 if n, _ := res.RowsAffected(); n == 0 {
423 return ErrNotFound
424 }
425 return nil
426}
427
428// RunnerRepoIDs is every repository a key is attached to.
429func (s *Store) RunnerRepoIDs(keyID int64) ([]int64, error) {
430 rows, err := s.DB.Query("SELECT repo_id FROM runner_repos WHERE key_id = ? ORDER BY repo_id", keyID)
431 if err != nil {
432 return nil, err
433 }
434 defer rows.Close()
435 var ids []int64
436 for rows.Next() {
437 var id int64
438 if err := rows.Scan(&id); err != nil {
439 return nil, err
440 }
441 ids = append(ids, id)
442 }
443 return ids, rows.Err()
444}
445
446// RunnerRepoPaths is RunnerRepoIDs as owner/name, sorted.
447func (s *Store) RunnerRepoPaths(keyID int64) ([]string, error) {
448 rows, err := s.DB.Query(`SELECT COALESCE(u.username, o.name) || '/' || r.name
449 FROM runner_repos rr JOIN repos r ON r.id = rr.repo_id
450 LEFT JOIN users u ON r.owner_kind = 'user' AND u.id = r.owner_id
451 LEFT JOIN orgs o ON r.owner_kind = 'org' AND o.id = r.owner_id
452 WHERE rr.key_id = ?`, keyID)
453 if err != nil {
454 return nil, err
455 }
456 defer rows.Close()
457 var paths []string
458 for rows.Next() {
459 var p string
460 if err := rows.Scan(&p); err != nil {
461 return nil, err
462 }
463 paths = append(paths, p)
464 }
465 sort.Strings(paths)
466 return paths, rows.Err()
467}
468
469// RunnerAttached reports whether a key may claim a repository's builds.
470func (s *Store) RunnerAttached(keyID, repoID int64) (bool, error) {
471 var n int
472 err := s.DB.QueryRow("SELECT count(*) FROM runner_repos WHERE key_id = ? AND repo_id = ?", keyID, repoID).Scan(&n)
473 return n > 0, err
474}
475
476// ListRepoRunners is every key attached to a repository with its last
477// poll and the build it holds, oldest attachment first.
478func (s *Store) ListRepoRunners(repoID int64) ([]RepoRunner, error) {
479 rows, err := s.DB.Query(`SELECT k.fingerprint, k.algo, u.username, rr.added_at,
480 COALESCE(rs.last_seen, ''),
481 COALESCE(COALESCE(bu.username, bo.name) || '/' || br.name, ''),
482 COALESCE(b.number, 0), COALESCE(b.job, ''), COALESCE(b.started_at, '')
483 FROM runner_repos rr
484 JOIN ssh_keys k ON k.id = rr.key_id
485 JOIN users u ON u.id = k.user_id
486 LEFT JOIN runner_seen rs ON rs.key_id = rr.key_id
487 LEFT JOIN builds b ON b.id = rs.build_id AND b.status = 'running'
488 LEFT JOIN repos br ON br.id = b.repo_id
489 LEFT JOIN users bu ON br.owner_kind = 'user' AND bu.id = br.owner_id
490 LEFT JOIN orgs bo ON br.owner_kind = 'org' AND bo.id = br.owner_id
491 WHERE rr.repo_id = ? ORDER BY rr.added_at, k.id`, repoID)
492 if err != nil {
493 return nil, err
494 }
495 defer rows.Close()
496 var out []RepoRunner
497 for rows.Next() {
498 var r RepoRunner
499 if err := rows.Scan(&r.Fingerprint, &r.Algo, &r.Username, &r.AddedAt, &r.LastSeen,
500 &r.BuildRepo, &r.BuildNumber, &r.BuildJob, &r.StartedAt); err != nil {
501 return nil, err
502 }
503 out = append(out, r)
504 }
505 return out, rows.Err()
506}
507
508// TouchRunner records a poll by one key: the time, the scope the runner
509// asked for, and the build it just claimed (0 for none).
510func (s *Store) TouchRunner(keyID, userID int64, scope string, buildID int64) error {
511 _, err := s.DB.Exec(`INSERT INTO runner_seen (key_id, user_id, last_seen, scope, build_id)
512 VALUES (?1, ?2, strftime('%Y-%m-%dT%H:%M:%fZ','now'), ?3, NULLIF(?4, 0))
513 ON CONFLICT (key_id) DO UPDATE SET
514 last_seen = excluded.last_seen, scope = excluded.scope,
515 build_id = COALESCE(excluded.build_id, runner_seen.build_id)`,
516 keyID, userID, scope, buildID)
517 return err
518}
519
520// RunnerDone records that the key reported and holds nothing now.
521func (s *Store) RunnerDone(keyID int64) error {
522 _, err := s.DB.Exec(`UPDATE runner_seen SET last_seen = strftime('%Y-%m-%dT%H:%M:%fZ','now'),
523 build_id = NULL WHERE key_id = ?`, keyID)
524 return err
525}
526
527// ListRunners lists every key that has ever polled as a runner, most
528// recently seen first.
529func (s *Store) ListRunners() ([]Runner, error) {
530 rows, err := s.DB.Query(`SELECT u.username, k.fingerprint, k.id, r.last_seen, r.scope,
531 COALESCE(COALESCE(bu.username, bo.name) || '/' || br.name, ''),
532 COALESCE(b.number, 0), COALESCE(b.job, ''), COALESCE(b.started_at, '')
533 FROM runner_seen r JOIN users u ON u.id = r.user_id
534 JOIN ssh_keys k ON k.id = r.key_id
535 LEFT JOIN builds b ON b.id = r.build_id AND b.status = 'running'
536 LEFT JOIN repos br ON br.id = b.repo_id
537 LEFT JOIN users bu ON br.owner_kind = 'user' AND bu.id = br.owner_id
538 LEFT JOIN orgs bo ON br.owner_kind = 'org' AND bo.id = br.owner_id
539 ORDER BY r.last_seen DESC`)
540 if err != nil {
541 return nil, err
542 }
543 defer rows.Close()
544 var out []Runner
545 for rows.Next() {
546 var r Runner
547 if err := rows.Scan(&r.Username, &r.Fingerprint, &r.KeyID, &r.LastSeen, &r.Scope,
548 &r.BuildRepo, &r.BuildNumber, &r.BuildJob, &r.StartedAt); err != nil {
549 return nil, err
550 }
551 out = append(out, r)
552 }
553 return out, rows.Err()
554}
555```
556
557- [ ] **Step 5: Run the store tests**
558
559Run: `go test ./internal/store 2>&1 | tail -5`
560Expected: PASS. Callers in `internal/control` do not compile yet; that is Task 3. `go build ./internal/store` must pass here.
561
562- [ ] **Step 6: Commit**
563
564```bash
565git add internal/store/migrations/0050_runner_repos.up.sql internal/store/migrations/0050_runner_repos.down.sql internal/store/runners.go internal/store/runners_test.go
566git commit -m "store: runner keys attach to repositories, heartbeat per key
567
568Migration 0050 adds runner_repos and rekeys runner_seen by ssh key.
569
570Ref #184"
571```
572
573---
574
575### Task 3: Control: the claim rule, per-key heartbeat, and admin runners
576
577**Files:**
578- Modify: `internal/control/build.go` (`requireRunner`, `runRunnerNext`, `runRunnerLog`, `runRunnerDone`, the `runner next` registration)
579- Modify: `internal/control/admin.go:444-475` (`runAdminRunners`)
580- Modify: `internal/control/runnernext_test.go:17-30` (`runnerCtx`)
581- Create: `internal/control/runnerattach_test.go`
582- Modify: `e2e/reap_test.go:112-115` (the admin runners row gains a fingerprint column)
583- Modify: `e2e/mrbuilds_test.go:95`, `e2e/build_cancel_test.go` (claims of a fork head pass `--untrusted`)
584
585**Interfaces:**
586- Consumes: the store functions from Tasks 1 and 2; `Ctx.Source` is the SSH key fingerprint for SSH sessions (`internal/sshd/sshd.go:338`).
587- Produces:
588 - `runnerSession(c *Ctx) (store.SSHKey, int)`: the key behind the session, or an exit code.
589 - `runnerMayBuild(c *Ctx, key store.SSHKey, repoID int64) (bool, error)`: admin, or attached.
590 - `runner next [--untrusted] [<owner/name>...]`.
591 - `admin runners` JSON rows carry `fingerprint`; the text row is `username<TAB>fingerprint<TAB>last_seen<TAB>scope<TAB>held`.
592
593- [ ] **Step 1: Write the failing control tests**
594
595`internal/control/runnerattach_test.go`:
596
597```go
598package control
599
600import (
601 "bytes"
602 "strconv"
603 "strings"
604 "testing"
605
606 "gitbay.org/gitbay/internal/config"
607 "gitbay.org/gitbay/internal/protocol"
608 "gitbay.org/gitbay/internal/store"
609)
610
611// attachFixture: alice (not admin) owns alice/app with a build queued;
612// mallory (not admin) owns mallory/evil with an older build queued. Each
613// has a runner-scoped key. The Ctx polls as the given user with the given
614// key, which is what the SSH listener produces.
615type attachFixture struct {
616 st *store.Store
617 alice, mallory int64
618 aliceKey, malloryKey store.SSHKey
619 app, evil store.Repo
620 appBuild, evilBuild int64
621}
622
623func newAttachFixture(t *testing.T) attachFixture {
624 t.Helper()
625 st, err := store.Open(":memory:")
626 if err != nil {
627 t.Fatal(err)
628 }
629 t.Cleanup(func() { st.Close() })
630 if err := st.MigrateUp(); err != nil {
631 t.Fatal(err)
632 }
633 var f attachFixture
634 f.st = st
635 mk := func(name, fp string) (int64, store.SSHKey, store.Repo, string) {
636 uid, err := st.CreateUser(name, false)
637 if err != nil {
638 t.Fatal(err)
639 }
640 if err := st.AddSSHKey(uid, fp, "ssh-ed25519", []byte(fp), "runner"); err != nil {
641 t.Fatal(err)
642 }
643 k, _ := st.SSHKeyByFingerprint(fp)
644 repoName := map[string]string{"alice": "app", "mallory": "evil"}[name]
645 rid, err := st.CreateRepo("user", uid, repoName, "public")
646 if err != nil {
647 t.Fatal(err)
648 }
649 repo, _ := st.RepoByID(rid)
650 return uid, k, repo, repoName
651 }
652 f.mallory, f.malloryKey, f.evil, _ = mk("mallory", "SHA256:mallory")
653 f.alice, f.aliceKey, f.app, _ = mk("alice", "SHA256:alice")
654 // mallory's build is older, so an unrestricted claim would take it.
655 f.evilBuild, err = st.CreateBuild(f.evil.ID, "unit", "aaa111", "main", "[]", "", "", true)
656 if err != nil {
657 t.Fatal(err)
658 }
659 f.appBuild, err = st.CreateBuild(f.app.ID, "unit", "bbb222", "main", "[]", "", "", true)
660 if err != nil {
661 t.Fatal(err)
662 }
663 return f
664}
665
666func (f attachFixture) ctx(uid int64, key store.SSHKey, admin bool) (*Ctx, *bytes.Buffer) {
667 var out bytes.Buffer
668 name := "alice"
669 if uid == f.mallory {
670 name = "mallory"
671 }
672 return &Ctx{
673 User: store.User{ID: uid, Username: name, IsAdmin: admin},
674 Scope: key.Scope,
675 Source: key.Fingerprint,
676 Store: f.st,
677 Cfg: config.Config{Server: config.Server{Root: "/nonexistent", SiteURL: "https://x.test"}},
678 Stdin: strings.NewReader(""),
679 Stdout: &out,
680 Stderr: &out,
681 }, &out
682}
683
684// A runner key with no attachment claims nothing, whatever is queued.
685func TestRunnerNextUnattachedClaimsNothing(t *testing.T) {
686 f := newAttachFixture(t)
687 c, out := f.ctx(f.alice, f.aliceKey, false)
688 if code := runRunnerNext(c, nil); code != protocol.ExitOK || !strings.Contains(out.String(), "no pending builds") {
689 t.Fatalf("exit %d: %s", code, out.String())
690 }
691 b, _ := f.st.BuildByNumber(f.evil.ID, f.evilBuild)
692 if b.Status != "pending" {
693 t.Fatalf("unattached key claimed a build: %s", b.Status)
694 }
695}
696
697// An attached key claims its repository's build and not the older one
698// queued elsewhere; naming a repository outside the attachments is refused.
699func TestRunnerNextAttachedClaimsOwnRepoOnly(t *testing.T) {
700 f := newAttachFixture(t)
701 if err := f.st.AttachRunner(f.aliceKey.ID, f.app.ID); err != nil {
702 t.Fatal(err)
703 }
704 c, out := f.ctx(f.alice, f.aliceKey, false)
705 if code := runRunnerNext(c, nil); code != protocol.ExitOK || !strings.Contains(out.String(), "alice/app") {
706 t.Fatalf("exit %d: %s", code, out.String())
707 }
708 if b, _ := f.st.BuildByNumber(f.evil.ID, f.evilBuild); b.Status != "pending" {
709 t.Fatalf("mallory's build was touched: %s", b.Status)
710 }
711 c, out = f.ctx(f.alice, f.aliceKey, false)
712 if code := runRunnerNext(c, []string{"mallory/evil"}); code != protocol.ExitDenied {
713 t.Fatalf("naming an unattached repo: exit %d, want %d: %s", code, protocol.ExitDenied, out.String())
714 }
715}
716
717// The heartbeat is recorded against the key, and admin runners shows it
718// with its fingerprint and attachments.
719func TestAdminRunnersShowsKeyAndAttachments(t *testing.T) {
720 f := newAttachFixture(t)
721 if err := f.st.AttachRunner(f.aliceKey.ID, f.app.ID); err != nil {
722 t.Fatal(err)
723 }
724 c, _ := f.ctx(f.alice, f.aliceKey, false)
725 runRunnerNext(c, nil)
726 admin, out := f.ctx(f.alice, f.aliceKey, true)
727 admin.Scope = "full"
728 if code := runAdminRunners(admin, nil); code != protocol.ExitOK {
729 t.Fatalf("admin runners: exit %d: %s", code, out.String())
730 }
731 if !strings.Contains(out.String(), "alice\tSHA256:alice\t") || !strings.Contains(out.String(), "\talice/app\t") {
732 t.Fatalf("row lacks fingerprint or attachments:\n%s", out.String())
733 }
734}
735
736// Untrusted builds are skipped unless the runner asks.
737func TestRunnerNextUntrustedFlag(t *testing.T) {
738 f := newAttachFixture(t)
739 if err := f.st.AttachRunner(f.aliceKey.ID, f.app.ID); err != nil {
740 t.Fatal(err)
741 }
742 c, _ := f.ctx(f.alice, f.aliceKey, false)
743 runRunnerNext(c, nil) // takes the trusted build
744 fork, err := f.st.CreateBuild(f.app.ID, "unit", "ccc333", "refs/merge-requests/1/head", "[]", "", "", false)
745 if err != nil {
746 t.Fatal(err)
747 }
748 c, out := f.ctx(f.alice, f.aliceKey, false)
749 runRunnerNext(c, nil)
750 if !strings.Contains(out.String(), "no pending builds") {
751 t.Fatalf("fork head claimed without --untrusted: %s", out.String())
752 }
753 c, out = f.ctx(f.alice, f.aliceKey, false)
754 if code := runRunnerNext(c, []string{"--untrusted"}); code != protocol.ExitOK || !strings.Contains(out.String(), "alice/app") {
755 t.Fatalf("--untrusted did not claim the fork head: exit %d %s", code, out.String())
756 }
757 if b, _ := f.st.BuildByNumber(f.app.ID, fork); b.Status != "running" {
758 t.Fatalf("fork build is %s, want running", b.Status)
759 }
760}
761
762// runner done and runner log on a build whose repository is not attached
763// to the key are refused.
764func TestRunnerDoneRefusedForUnattachedBuild(t *testing.T) {
765 f := newAttachFixture(t)
766 if err := f.st.AttachRunner(f.malloryKey.ID, f.evil.ID); err != nil {
767 t.Fatal(err)
768 }
769 c, _ := f.ctx(f.mallory, f.malloryKey, false)
770 runRunnerNext(c, nil) // mallory holds her own build
771 evil, _ := f.st.BuildByNumber(f.evil.ID, f.evilBuild)
772 c, out := f.ctx(f.alice, f.aliceKey, false)
773 id := strconv.FormatInt(evil.ID, 10)
774 if code := runRunnerDone(c, []string{id, "success"}); code != protocol.ExitDenied {
775 t.Fatalf("done on an unattached build: exit %d, want %d: %s", code, protocol.ExitDenied, out.String())
776 }
777 c, out = f.ctx(f.alice, f.aliceKey, false)
778 if code := runRunnerLog(c, []string{id}); code != protocol.ExitDenied {
779 t.Fatalf("log on an unattached build: exit %d, want %d: %s", code, protocol.ExitDenied, out.String())
780 }
781 if b, _ := f.st.BuildByNumber(f.evil.ID, f.evilBuild); b.Status != "running" {
782 t.Fatalf("build was finished by a foreign key: %s", b.Status)
783 }
784}
785```
786
787- [ ] **Step 2: Run them to see them fail**
788
789Run: `go test ./internal/control -run 'TestRunnerNext(Unattached|Attached|Untrusted)|TestAdminRunnersShows|TestRunnerDoneRefused' 2>&1 | head`
790Expected: compile errors from `ClaimBuild`, `TouchRunner`, `RunnerDone` signature changes in `build.go`.
791
792- [ ] **Step 3: Rewrite the runner protocol in `internal/control/build.go`**
793
794Change the registration:
795
796```go
797 register(Command{Path: []string{"runner", "next"},
798 Summary: "claim the oldest pending build this key may run (runner protocol)",
799 Usage: "runner next [--untrusted] [<owner/name>...]", SSHOnly: true, Run: runRunnerNext})
800```
801
802Replace `requireRunner` with two helpers:
803
804```go
805// runnerSession resolves the key behind a runner-protocol session. The
806// runner commands are SSHOnly, so Source is the key's fingerprint. An
807// admin key is accepted so an operator can rotate at their own pace; a
808// runner host should hold a key added with --scope runner.
809func runnerSession(c *Ctx) (store.SSHKey, int) {
810 if c.Scope != "runner" && !c.User.IsAdmin {
811 return store.SSHKey{}, c.fail(protocol.ExitDenied, "runner commands need a key added with --scope runner")
812 }
813 key, err := c.Store.SSHKeyByFingerprint(c.Source)
814 if err != nil {
815 return store.SSHKey{}, c.fail(protocol.ExitDenied, "runner commands need an SSH key session")
816 }
817 return key, -1
818}
819
820// runnerMayBuild reports whether a runner session may act on a
821// repository's builds: an admin user may on any, a runner key on the
822// repositories it is attached to (#184).
823func runnerMayBuild(c *Ctx, key store.SSHKey, repoID int64) (bool, error) {
824 if c.User.IsAdmin {
825 return true, nil
826 }
827 return c.Store.RunnerAttached(key.ID, repoID)
828}
829```
830
831Rewrite the head of `runRunnerNext` down to the claim loop:
832
833```go
834func runRunnerNext(c *Ctx, args []string) int {
835 key, code := runnerSession(c)
836 if code >= 0 {
837 return code
838 }
839 f, err := parseFlags(args, flagSpec{Bools: []string{"--untrusted"}, MaxPos: -1,
840 Usage: "runner next [--untrusted] [<owner/name>...]"})
841 if err != nil {
842 return c.fail(protocol.ExitUsage, "%v", err)
843 }
844 // The candidate set. An admin key claims from any repository, narrowed
845 // by the names given. A runner key claims from the repositories it is
846 // attached to; a name outside them is refused, not ignored, so a
847 // misconfigured runner says so instead of idling.
848 var repoIDs []int64
849 for _, arg := range f.Pos {
850 repo, code := resolveRepo(c, arg, policy.CanRead)
851 if code >= 0 {
852 return code
853 }
854 ok, err := runnerMayBuild(c, key, repo.ID)
855 if err != nil {
856 return c.fail(protocol.ExitFailure, "%v", err)
857 }
858 if !ok {
859 return c.fail(protocol.ExitDenied, "this key is not attached to %s", repo.Path())
860 }
861 repoIDs = append(repoIDs, repo.ID)
862 }
863 if !c.User.IsAdmin && len(repoIDs) == 0 {
864 repoIDs, err = c.Store.RunnerRepoIDs(key.ID)
865 if err != nil {
866 return c.fail(protocol.ExitFailure, "%v", err)
867 }
868 if len(repoIDs) == 0 {
869 // Nothing attached: nothing to claim. Still a heartbeat, so
870 // admin runners shows the key polling.
871 c.Store.TouchRunner(key.ID, c.User.ID, "", 0)
872 return c.emit(map[string]any{}, func(w io.Writer) { fmt.Fprintln(w, "no pending builds") })
873 }
874 }
875 untrusted := f.Has("--untrusted")
876 var b store.Build
877 var repo store.Repo
878 var ok bool
879 for attempt := 0; attempt < maxOrphanSkip; attempt++ {
880 b, ok, err = c.Store.ClaimBuild(repoIDs, untrusted)
881```
882
883The rest of the loop is unchanged. Delete the old `var err error` line, since `err` now comes from `parseFlags`. Replace the heartbeat line:
884
885```go
886 c.Store.TouchRunner(key.ID, c.User.ID, strings.Join(f.Pos, ","), b.ID)
887```
888
889In `runRunnerLog`, replace `if code := requireRunner(c); code >= 0 { return code }` with:
890
891```go
892 key, code := runnerSession(c)
893 if code >= 0 {
894 return code
895 }
896```
897
898and directly after the `id, err := strconv.ParseInt(args[0], 10, 64)` block, add:
899
900```go
901 if b, err := c.Store.BuildByID(id); err != nil {
902 return c.fail(protocol.ExitNotFound, "no build %d", id)
903 } else if ok, err := runnerMayBuild(c, key, b.RepoID); err != nil {
904 return c.fail(protocol.ExitFailure, "%v", err)
905 } else if !ok {
906 return c.fail(protocol.ExitDenied, "this key is not attached to the build's repository")
907 }
908```
909
910In `runRunnerDone`, the same `runnerSession` replacement; after `b, err := c.Store.BuildByID(id)` succeeds add:
911
912```go
913 if ok, err := runnerMayBuild(c, key, b.RepoID); err != nil {
914 return c.fail(protocol.ExitFailure, "%v", err)
915 } else if !ok {
916 return c.fail(protocol.ExitDenied, "this key is not attached to the build's repository")
917 }
918```
919
920and both `c.Store.RunnerDone(c.User.ID)` become `c.Store.RunnerDone(key.ID)`.
921
922Delete `requireRunner` if nothing else references it (`grep -n requireRunner internal/`).
923
924- [ ] **Step 4: `admin runners` shows the fingerprint and attachments**
925
926In `internal/control/admin.go`, `runAdminRunners`, after `ListRunners`:
927
928```go
929 for i := range runners {
930 if runners[i].Scope != "" {
931 continue
932 }
933 key, err := c.Store.SSHKeyByID(runners[i].KeyID)
934 if err != nil || key.Scope != "runner" {
935 continue // an admin key with no -repos: any
936 }
937 paths, err := c.Store.RunnerRepoPaths(runners[i].KeyID)
938 if err != nil {
939 return c.fail(protocol.ExitFailure, "%v", err)
940 }
941 runners[i].Scope = strings.Join(paths, ",")
942 }
943```
944
945A runner key that asked for `-repos` shows that; one that did not shows its attachments. Both are what the key may claim. Text row:
946
947```go
948 fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", r.Username, r.Fingerprint, r.LastSeen, scope, held)
949```
950
951Add `"strings"` to admin.go imports if missing.
952
953- [ ] **Step 5: Fix the existing `runnerCtx` test helper**
954
955`internal/control/runnernext_test.go`, `runnerCtx`: the session needs a real key. Replace the helper body:
956
957```go
958func runnerCtx(st *store.Store, uid int64, root string) (*Ctx, *bytes.Buffer) {
959 var out bytes.Buffer
960 fp := fmt.Sprintf("SHA256:runner-%d", uid)
961 st.AddSSHKey(uid, fp, "ssh-ed25519", []byte(fp), "full") // ErrDuplicateKey on reuse is fine
962 c := &Ctx{
963 User: store.User{ID: uid, Username: "ci", IsAdmin: true},
964 Scope: "full",
965 Source: fp,
966 Store: st,
967 Cfg: config.Config{Server: config.Server{Root: root, SiteURL: "https://x.test"}},
968 Stdin: strings.NewReader(""),
969 Stdout: &out,
970 Stderr: &out,
971 }
972 return c, &out
973}
974```
975
976Any other control test that builds a `Ctx` for `runRunnerNext`, `runRunnerLog` or `runRunnerDone` (`grep -ln "runRunner" internal/control/*_test.go`) needs the same: an `AddSSHKey` and `Source` set to its fingerprint.
977
978- [ ] **Step 6: Run the control tests**
979
980Run: `go build ./... && go vet ./internal/control && go test ./internal/control 2>&1 | tail -5`
981Expected: PASS, including `TestStdinCommandsReadStdin` and `TestReadOnlyCommandsWriteNothing`.
982
983- [ ] **Step 7: Update the e2e tests that this changes**
984
985`e2e/reap_test.go:112-115`: the row is now `ci<TAB>SHA256:...<TAB>last_seen<TAB>alice/app<TAB>idle`. The existing assertions `strings.Contains(out, "\nci\t")` and `strings.Contains(out, "\talice/app\tidle")` still hold. No change unless the test fails; run it in Step 8.
986
987`e2e/mrbuilds_test.go:95` claims a fork head with an admin key. Add the flag:
988
989```go
990 out, errOut, code := inst.ssh(t, runnerKey, "", "runner", "next", "--untrusted", "alice/app", "--json")
991```
992
993`e2e/build_cancel_test.go`: find every `"runner", "next"` that claims a merge request head from a fork (`grep -n 'runner", "next"' e2e/build_cancel_test.go`, and read the test around each). Add `"--untrusted"` right after `"next"` where the queued build is a fork head. Same-repository branches are trusted and need nothing.
994
995- [ ] **Step 8: Run those e2e tests**
996
997Run: `go test ./e2e -run 'TestStaleBuildReapedWithoutRunner|TestForkMRHeadIsBuilt|TestRunnerNextScopedToRepos|TestRunnerScopedKey' -count=1 2>&1 | tail -5`
998Expected: PASS.
999
1000- [ ] **Step 9: Commit**
1001
1002```bash
1003git add internal/control/build.go internal/control/admin.go internal/control/runnernext_test.go internal/control/runnerattach_test.go e2e/reap_test.go e2e/mrbuilds_test.go e2e/build_cancel_test.go
1004git commit -m "control: a runner key claims only the repositories it is attached to
1005
1006runner next takes --untrusted; without it fork heads are skipped. runner
1007log and runner done refuse a build outside the key's attachments. The
1008heartbeat and admin runners are per key.
1009
1010Ref #184"
1011```
1012
1013---
1014
1015### Task 4: Control and CLI: `repo runner add|list|remove`
1016
1017**Files:**
1018- Create: `internal/control/runnerrepo.go`
1019- Create: `internal/control/runnerrepo_test.go`
1020- Modify: `cmd/gitbay/main.go:437-440` (a `group("runner", ...)` beside `deploy-key`)
1021
1022**Interfaces:**
1023- Consumes: the store functions from Task 2; `resolveRepo(c, path, policy.CanAdmin)`; `c.Store.Audit(userID, action string, fields map[string]any)`.
1024- Produces:
1025 - `repo runner add <owner/name>` (stdin: public key) → `{"fingerprint": ..., "repo": ...}`
1026 - `repo runner list <owner/name>` → `[]store.RepoRunner`
1027 - `repo runner remove <owner/name> <fingerprint>` → `{"removed": fingerprint}`
1028
1029- [ ] **Step 1: Write the failing tests**
1030
1031`internal/control/runnerrepo_test.go`:
1032
1033```go
1034package control
1035
1036import (
1037 "bytes"
1038 "strings"
1039 "testing"
1040
1041 "gitbay.org/gitbay/internal/config"
1042 "gitbay.org/gitbay/internal/protocol"
1043 "gitbay.org/gitbay/internal/store"
1044)
1045
1046// Generated once with ssh-keygen -t ed25519; a valid authorized_keys line.
1047const testRunnerPub = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILAr2r82jFsCJwsEyrEf2wgKy9Dv45xYYici6Ii7NyCS runner@test\n"
1048
1049func repoRunnerCtx(t *testing.T, st *store.Store, uid int64, admin bool, stdin string) (*Ctx, *bytes.Buffer) {
1050 t.Helper()
1051 var out bytes.Buffer
1052 return &Ctx{
1053 User: store.User{ID: uid, Username: "alice", IsAdmin: admin},
1054 Scope: "full",
1055 Source: "SHA256:session",
1056 Store: st,
1057 Cfg: config.Config{Server: config.Server{SiteURL: "https://x.test"}},
1058 Stdin: strings.NewReader(stdin),
1059 Stdout: &out,
1060 Stderr: &out,
1061 JSON: true,
1062 }, &out
1063}
1064
1065// A fresh key is registered on the caller's account with scope runner and
1066// attached; a second add is a no-op; list shows it; remove detaches and
1067// leaves the key on the account.
1068func TestRepoRunnerAddListRemove(t *testing.T) {
1069 st, repo, uid := newQueueTestRepo(t)
1070 c, out := repoRunnerCtx(t, st, uid, false, testRunnerPub)
1071 if code := runRepoRunnerAdd(c, []string{repo.Path()}); code != protocol.ExitOK {
1072 t.Fatalf("add: exit %d %s", code, out.String())
1073 }
1074 if !strings.Contains(out.String(), `"fingerprint":"SHA256:`) {
1075 t.Fatalf("add output: %s", out.String())
1076 }
1077 keys, _ := st.ListSSHKeys(uid)
1078 if len(keys) != 1 || keys[0].Scope != "runner" {
1079 t.Fatalf("key not registered as runner: %+v", keys)
1080 }
1081 fp := keys[0].Fingerprint
1082 c, out = repoRunnerCtx(t, st, uid, false, testRunnerPub)
1083 if code := runRepoRunnerAdd(c, []string{repo.Path()}); code != protocol.ExitOK {
1084 t.Fatalf("second add: exit %d %s", code, out.String())
1085 }
1086 c, out = repoRunnerCtx(t, st, uid, false, "")
1087 if code := runRepoRunnerList(c, []string{repo.Path()}); code != protocol.ExitOK || strings.Count(out.String(), fp) != 1 {
1088 t.Fatalf("list: exit %d %s", code, out.String())
1089 }
1090 c, out = repoRunnerCtx(t, st, uid, false, "")
1091 if code := runRepoRunnerRemove(c, []string{repo.Path(), fp}); code != protocol.ExitOK {
1092 t.Fatalf("remove: exit %d %s", code, out.String())
1093 }
1094 if ok, _ := st.RunnerAttached(keys[0].ID, repo.ID); ok {
1095 t.Fatal("still attached after remove")
1096 }
1097 if keys, _ = st.ListSSHKeys(uid); len(keys) != 1 {
1098 t.Fatal("remove dropped the key from the account")
1099 }
1100 c, out = repoRunnerCtx(t, st, uid, false, "")
1101 if code := runRepoRunnerRemove(c, []string{repo.Path(), fp}); code != protocol.ExitNotFound {
1102 t.Fatalf("remove twice: exit %d, want %d", code, protocol.ExitNotFound)
1103 }
1104}
1105
1106// A key that already exists with another scope is never promoted, and
1107// another account's runner key is refused unless the caller is an admin.
1108func TestRepoRunnerAddRefusesWrongKeys(t *testing.T) {
1109 st, repo, uid := newQueueTestRepo(t)
1110 c, _ := repoRunnerCtx(t, st, uid, false, testRunnerPub)
1111 // Register the same key as a full key first.
1112 if code := runKeysAdd(c, nil); code != protocol.ExitOK {
1113 t.Fatal("keys add failed")
1114 }
1115 c, out := repoRunnerCtx(t, st, uid, false, testRunnerPub)
1116 if code := runRepoRunnerAdd(c, []string{repo.Path()}); code != protocol.ExitDenied {
1117 t.Fatalf("full key accepted as runner: exit %d %s", code, out.String())
1118 }
1119 keys, _ := st.ListSSHKeys(uid)
1120 if keys[0].Scope != "full" {
1121 t.Fatalf("scope changed to %s", keys[0].Scope)
1122 }
1123 // Someone else's runner key.
1124 bob, _ := st.CreateUser("bob", false)
1125 if err := st.AddSSHKey(bob, "SHA256:bobrunner", "ssh-ed25519", []byte("x"), "runner"); err != nil {
1126 t.Fatal(err)
1127 }
1128 st.RemoveSSHKey(uid, keys[0].Fingerprint)
1129 if err := st.AddSSHKey(bob, keys[0].Fingerprint, "ssh-ed25519", keys[0].Blob, "runner"); err != nil {
1130 t.Fatal(err)
1131 }
1132 c, out = repoRunnerCtx(t, st, uid, false, testRunnerPub)
1133 if code := runRepoRunnerAdd(c, []string{repo.Path()}); code != protocol.ExitDenied {
1134 t.Fatalf("another account's key attached by a non-admin: exit %d %s", code, out.String())
1135 }
1136 c, out = repoRunnerCtx(t, st, uid, true, testRunnerPub)
1137 if code := runRepoRunnerAdd(c, []string{repo.Path()}); code != protocol.ExitOK {
1138 t.Fatalf("admin could not attach another account's runner key: exit %d %s", code, out.String())
1139 }
1140}
1141```
1142
1143- [ ] **Step 2: Run them to see them fail**
1144
1145Run: `go test ./internal/control -run TestRepoRunner 2>&1 | head -5`
1146Expected: `undefined: runRepoRunnerAdd`.
1147
1148- [ ] **Step 3: Write `internal/control/runnerrepo.go`**
1149
1150```go
1151package control
1152
1153import (
1154 "errors"
1155 "fmt"
1156 "io"
1157
1158 "golang.org/x/crypto/ssh"
1159
1160 "gitbay.org/gitbay/internal/policy"
1161 "gitbay.org/gitbay/internal/protocol"
1162 "gitbay.org/gitbay/internal/store"
1163)
1164
1165// Runners attached to a repository (#184). A runner key claims builds only
1166// for the repositories it is attached to; a repository admin attaches it
1167// by pasting the runner's public key. The key lands on the admin's own
1168// account with scope runner, which confines it to the runner protocol and
1169// read-only git.
1170func init() {
1171 register(Command{Path: []string{"repo", "runner", "add"},
1172 Summary: "attach a runner's public key to a repository",
1173 Usage: "repo runner add <owner/name> < key.pub",
1174 ReadsStdin: true, Run: runRepoRunnerAdd})
1175 register(Command{Path: []string{"repo", "runner", "list"},
1176 Summary: "list the runners attached to a repository",
1177 Usage: "repo runner list <owner/name>", ReadOnly: true, Run: runRepoRunnerList})
1178 register(Command{Path: []string{"repo", "runner", "remove"},
1179 Summary: "detach a runner from a repository",
1180 Usage: "repo runner remove <owner/name> <fingerprint>", Run: runRepoRunnerRemove})
1181}
1182
1183func runRepoRunnerAdd(c *Ctx, args []string) int {
1184 f, err := parseFlags(args, flagSpec{MaxPos: 1, Usage: "repo runner add <owner/name> < key.pub"})
1185 if err != nil || len(f.Pos) != 1 {
1186 return c.fail(protocol.ExitUsage, "usage: repo runner add <owner/name> < key.pub")
1187 }
1188 repo, code := resolveRepo(c, f.Pos[0], policy.CanAdmin)
1189 if code >= 0 {
1190 return code
1191 }
1192 raw, err := io.ReadAll(io.LimitReader(c.Stdin, 64<<10))
1193 if err != nil {
1194 return c.fail(protocol.ExitFailure, "reading key: %v", err)
1195 }
1196 pub, _, _, _, err := ssh.ParseAuthorizedKey(raw)
1197 if err != nil {
1198 return c.fail(protocol.ExitUsage, "not a valid public key in authorized_keys format: %v", err)
1199 }
1200 fp := ssh.FingerprintSHA256(pub)
1201 key, err := c.Store.SSHKeyByFingerprint(fp)
1202 switch {
1203 case errors.Is(err, store.ErrNotFound):
1204 if err := c.Store.AddSSHKey(c.User.ID, fp, pub.Type(), pub.Marshal(), "runner"); err != nil {
1205 return c.fail(protocol.ExitFailure, "adding key: %v", err)
1206 }
1207 if key, err = c.Store.SSHKeyByFingerprint(fp); err != nil {
1208 return c.fail(protocol.ExitFailure, "%v", err)
1209 }
1210 case err != nil:
1211 return c.fail(protocol.ExitFailure, "%v", err)
1212 case key.Scope != "runner":
1213 // A full key would let a build step administer the account; a
1214 // deploy key is bound elsewhere. A runner gets a key of its own.
1215 return c.fail(protocol.ExitDenied, "%s is a %s key, not a runner key; give the runner a key of its own", fp, key.Scope)
1216 case key.UserID != c.User.ID && !c.User.IsAdmin:
1217 return c.fail(protocol.ExitDenied, "%s belongs to another account", fp)
1218 }
1219 if err := c.Store.AttachRunner(key.ID, repo.ID); err != nil {
1220 return c.fail(protocol.ExitFailure, "%v", err)
1221 }
1222 c.Store.Audit(c.User.ID, "repo.runner.add", map[string]any{"repo": repo.Path(), "fingerprint": fp})
1223 d := map[string]string{"fingerprint": fp, "repo": repo.Path()}
1224 return c.emit(d, func(w io.Writer) {
1225 fmt.Fprintf(w, "runner %s attached to %s\n", fp, repo.Path())
1226 })
1227}
1228
1229func runRepoRunnerList(c *Ctx, args []string) int {
1230 if len(args) != 1 {
1231 return c.fail(protocol.ExitUsage, "usage: repo runner list <owner/name>")
1232 }
1233 repo, code := resolveRepo(c, args[0], policy.CanAdmin)
1234 if code >= 0 {
1235 return code
1236 }
1237 runners, err := c.Store.ListRepoRunners(repo.ID)
1238 if err != nil {
1239 return c.fail(protocol.ExitFailure, "%v", err)
1240 }
1241 if runners == nil {
1242 runners = []store.RepoRunner{}
1243 }
1244 return c.emit(runners, func(w io.Writer) {
1245 for _, r := range runners {
1246 seen := r.LastSeen
1247 if seen == "" {
1248 seen = "never"
1249 }
1250 held := "idle"
1251 if r.BuildNumber != 0 {
1252 held = fmt.Sprintf("%s #%d %s since %s", r.BuildRepo, r.BuildNumber, r.BuildJob, r.StartedAt)
1253 }
1254 fmt.Fprintf(w, "%s\t%s\t%s\t%s\t%s\n", r.Fingerprint, r.Algo, r.Username, seen, held)
1255 }
1256 })
1257}
1258
1259func runRepoRunnerRemove(c *Ctx, args []string) int {
1260 if len(args) != 2 {
1261 return c.fail(protocol.ExitUsage, "usage: repo runner remove <owner/name> <fingerprint>")
1262 }
1263 repo, code := resolveRepo(c, args[0], policy.CanAdmin)
1264 if code >= 0 {
1265 return code
1266 }
1267 if err := c.Store.DetachRunner(repo.ID, args[1]); err != nil {
1268 if errors.Is(err, store.ErrNotFound) {
1269 return c.fail(protocol.ExitNotFound, "no runner %s on %s", args[1], repo.Path())
1270 }
1271 return c.fail(protocol.ExitFailure, "%v", err)
1272 }
1273 c.Store.Audit(c.User.ID, "repo.runner.remove", map[string]any{"repo": repo.Path(), "fingerprint": args[1]})
1274 return c.emit(map[string]string{"removed": args[1]}, func(w io.Writer) {
1275 fmt.Fprintf(w, "runner %s detached from %s\n", args[1], repo.Path())
1276 })
1277}
1278```
1279
1280`Store.Audit(actorID int64, action string, data map[string]any)` returns nothing.
1281
1282- [ ] **Step 4: Add the CLI table entries**
1283
1284In `cmd/gitbay/main.go`, directly after the `group("deploy-key", ...)` block (line 437-440):
1285
1286```go
1287 group("runner", "runners attached to a repository",
1288 pass("add", "attach a runner's public key: < key.pub", passOpts{server: []string{"repo", "runner", "add"}, needsRepo: true, alwaysStdin: true, stdinWhat: "an SSH public key"}),
1289 pass("list", "list attached runners", passOpts{server: []string{"repo", "runner", "list"}, needsRepo: true}),
1290 pass("remove", "detach a runner: <fingerprint>", passOpts{server: []string{"repo", "runner", "remove"}, needsRepo: true}),
1291 ),
1292```
1293
1294- [ ] **Step 5: Run the tests**
1295
1296Run: `go build ./... && go test ./internal/control ./cmd/gitbay 2>&1 | tail -5`
1297Expected: PASS, including the CLI coverage test.
1298
1299- [ ] **Step 6: Commit**
1300
1301```bash
1302git add internal/control/runnerrepo.go internal/control/runnerrepo_test.go cmd/gitbay/main.go
1303git commit -m "control, cli: repo runner add, list, remove
1304
1305Ref #184"
1306```
1307
1308---
1309
1310### Task 5: Web: Runners on the repository settings page
1311
1312**Files:**
1313- Modify: `internal/httpd/settings.go:18-49` (`settingsPage`, `settingsForm`) and the `switch` in `settingsSubmit`
1314- Modify: `internal/web/templates/settings.html` (a section after Dependencies, before Lifecycle)
1315- Create: `e2e/runnerweb_test.go`
1316
1317**Interfaces:**
1318- Consumes: `repo runner list|add|remove` from Task 4; `s.runControlInto`, `s.runControlStdin(u, argv, stdin) (msg string, ok bool)`, `s.runControl`.
1319- Produces: form fields `field=runner-add` with `key`, and `field=runner-remove` with `fingerprint`.
1320
1321- [ ] **Step 1: Write the failing e2e test**
1322
1323`e2e/runnerweb_test.go`:
1324
1325```go
1326package e2e
1327
1328import (
1329 "net/url"
1330 "os"
1331 "strings"
1332 "testing"
1333)
1334
1335// The settings page attaches and detaches runners through the same
1336// commands the CLI uses, and lists what is attached.
1337func TestRunnerSettingsWeb(t *testing.T) {
1338 inst := startInstanceWith(t, "[web]\nmode = \"accounts\"\n")
1339 aliceKey := inst.newKey(t, "alice")
1340 inst.admin(t, "admin", "user", "create", "alice",
1341 "--key", aliceKey+".pub", "--email", "alice@example.test", "--verified")
1342 if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "create", "alice/app"); code != 0 {
1343 t.Fatalf("repo create: %s", errOut)
1344 }
1345 runnerKey := inst.newKey(t, "laptop")
1346 pub, _ := os.ReadFile(runnerKey + ".pub")
1347
1348 alice := inst.login(t, aliceKey)
1349 settings := inst.base() + "/alice/app/settings"
1350 _, body := browserGet(t, alice, settings)
1351 if !strings.Contains(body, "No runners attached") {
1352 t.Fatalf("empty state missing:\n%s", body)
1353 }
1354 if status, _ := browserPost(t, alice, settings, url.Values{"field": {"runner-add"}, "key": {string(pub)}}); status != 200 {
1355 t.Fatalf("runner-add post: %d", status)
1356 }
1357 out, _, _ := inst.ssh(t, aliceKey, "", "repo", "runner", "list", "alice/app", "--json")
1358 if !strings.Contains(out, `"fingerprint":"SHA256:`) {
1359 t.Fatalf("not attached after the form: %s", out)
1360 }
1361 fp := out[strings.Index(out, "SHA256:"):]
1362 fp = fp[:strings.Index(fp, `"`)]
1363 _, body = browserGet(t, alice, settings)
1364 if !strings.Contains(body, fp) || !strings.Contains(body, `value="runner-remove"`) {
1365 t.Fatalf("attached runner not listed:\n%s", body)
1366 }
1367 if status, _ := browserPost(t, alice, settings, url.Values{"field": {"runner-remove"}, "fingerprint": {fp}}); status != 200 {
1368 t.Fatalf("runner-remove post: %d", status)
1369 }
1370 if out, _, _ = inst.ssh(t, aliceKey, "", "repo", "runner", "list", "alice/app", "--json"); strings.Contains(out, fp) {
1371 t.Fatalf("still attached after remove: %s", out)
1372 }
1373}
1374```
1375
1376- [ ] **Step 2: Run it to see it fail**
1377
1378Run: `go test ./e2e -run TestRunnerSettingsWeb -count=1 2>&1 | tail -5`
1379Expected: FAIL at "empty state missing".
1380
1381- [ ] **Step 3: Handler changes in `internal/httpd/settings.go`**
1382
1383`settingsPage` gains:
1384
1385```go
1386 Runners []store.RepoRunner
1387```
1388
1389In `settingsForm`, after the deps read:
1390
1391```go
1392 var runners []store.RepoRunner
1393 s.runControlInto(u, []string{"repo", "runner", "list", repo.Path()}, &runners)
1394```
1395
1396and pass `Runners: runners` to the struct literal.
1397
1398In `settingsSubmit`'s switch, before `default:`:
1399
1400```go
1401 case "runner-add":
1402 body := v("key")
1403 if body == "" {
1404 s.settingsRedirect(w, r, "paste the runner's public key")
1405 return
1406 }
1407 msg, ok := s.runControlStdin(u, []string{"repo", "runner", "add", repo}, body+"\n")
1408 if ok {
1409 msg = ""
1410 }
1411 s.settingsRedirect(w, r, msg)
1412 return
1413 case "runner-remove":
1414 argv = []string{"repo", "runner", "remove", repo, v("fingerprint")}
1415```
1416
1417- [ ] **Step 4: Template section**
1418
1419In `internal/web/templates/settings.html`, before `<h2>Lifecycle</h2>`:
1420
1421```html
1422<h2>Runners</h2>
1423{{if .Runners}}
1424<ul class="protlist">
1425{{range .Runners}}<li><code>{{.Fingerprint}}</code> <span class="meta">{{.Username}}{{if .LastSeen}}, last poll {{.LastSeen}}{{else}}, never polled{{end}}{{if .BuildNumber}}, running {{.BuildRepo}} #{{.BuildNumber}} {{.BuildJob}}{{end}}</span>
1426 <form method="post" action="{{$base}}" class="inline">
1427 <input type="hidden" name="field" value="runner-remove">
1428 <input type="hidden" name="fingerprint" value="{{.Fingerprint}}">
1429 <button type="submit" class="linklike">Detach</button>
1430 </form></li>
1431{{end}}
1432</ul>
1433{{else}}<p class="meta">No runners attached. Builds for this repository run on the runners attached here; a repository with none queues builds nothing claims.</p>{{end}}
1434<form method="post" action="{{$base}}" class="setform">
1435 <input type="hidden" name="field" value="runner-add">
1436 <label for="runner-key">Attach a runner</label>
1437 <textarea id="runner-key" name="key" rows="3" placeholder="ssh-ed25519 AAAA… (from gitbay-runner init)"></textarea>
1438 <button type="submit">Attach</button>
1439</form>
1440<p class="meta">Install <code>gitbay-runner</code>, run <code>gitbay-runner init</code>, and paste the key it prints. The runner builds your commits with the repository's secrets; merge requests from forks wait unless it runs with <code>-untrusted</code>.</p>
1441```
1442
1443- [ ] **Step 5: Run the test**
1444
1445Run: `go test ./e2e -run TestRunnerSettingsWeb -count=1 2>&1 | tail -5`
1446Expected: PASS.
1447
1448- [ ] **Step 6: Commit**
1449
1450```bash
1451git add internal/httpd/settings.go internal/web/templates/settings.html e2e/runnerweb_test.go
1452git commit -m "httpd: attach and detach runners on the settings page
1453
1454Ref #184"
1455```
1456
1457---
1458
1459### Task 6: Runner: config file, `-identity`, `-untrusted`
1460
1461**Files:**
1462- Create: `cmd/gitbay-runner/config.go`
1463- Create: `cmd/gitbay-runner/config_test.go`
1464- Modify: `cmd/gitbay-runner/main.go` (flag block, `runner` struct, `step`, ssh option assembly)
1465
1466**Interfaces:**
1467- Produces:
1468 - `configDir() string`: `$XDG_CONFIG_HOME/gitbay-runner` or `$HOME/.config/gitbay-runner`.
1469 - `defaultConfigPath() string`: `configDir()/config.toml`.
1470 - `configPathFromArgs(args []string, def string) string`: honours `-config X`, `--config X`, `-config=X`.
1471 - `loadConfig(path string) (map[string]string, bool, error)`: flag name to value, false when the file is absent.
1472 - `applyConfig(fs *flag.FlagSet, values map[string]string) error`: `fs.Set` each.
1473 - `identityOpts(path string) []string`: `["-i", path, "-o", "IdentitiesOnly=yes"]` or nil.
1474 - Flags `-config`, `-identity`, `-untrusted`.
1475
1476- [ ] **Step 1: Write the failing tests**
1477
1478`cmd/gitbay-runner/config_test.go`:
1479
1480```go
1481package main
1482
1483import (
1484 "flag"
1485 "os"
1486 "path/filepath"
1487 "testing"
1488)
1489
1490// A config file sets the flags' values; a flag on the command line wins.
1491func TestConfigFileFeedsFlagsAndFlagsOverride(t *testing.T) {
1492 dir := t.TempDir()
1493 path := filepath.Join(dir, "config.toml")
1494 os.WriteFile(path, []byte("remote = \"git@example.test\"\npoll = \"9s\"\nuntrusted = true\nidentity = \"/k\"\njobs = 2\n"), 0o600)
1495
1496 values, found, err := loadConfig(path)
1497 if err != nil || !found {
1498 t.Fatalf("loadConfig: found=%v err=%v", found, err)
1499 }
1500 fs := flag.NewFlagSet("t", flag.ContinueOnError)
1501 remote := fs.String("remote", "git@gitbay.org", "")
1502 poll := fs.Duration("poll", 0, "")
1503 untrusted := fs.Bool("untrusted", false, "")
1504 identity := fs.String("identity", "", "")
1505 jobs := fs.Int("jobs", 1, "")
1506 if err := applyConfig(fs, values); err != nil {
1507 t.Fatal(err)
1508 }
1509 if err := fs.Parse([]string{"-poll", "3s"}); err != nil {
1510 t.Fatal(err)
1511 }
1512 if *remote != "git@example.test" || poll.String() != "3s" || !*untrusted || *identity != "/k" || *jobs != 2 {
1513 t.Fatalf("remote=%s poll=%s untrusted=%v identity=%s jobs=%d", *remote, poll, *untrusted, *identity, *jobs)
1514 }
1515 if _, found, err := loadConfig(filepath.Join(dir, "missing.toml")); found || err != nil {
1516 t.Fatalf("missing file: found=%v err=%v", found, err)
1517 }
1518 if _, _, err := loadConfig(path); err != nil {
1519 t.Fatal(err)
1520 }
1521 os.WriteFile(path, []byte("nonsense = \"x\"\n"), 0o600)
1522 if _, _, err := loadConfig(path); err == nil {
1523 t.Fatal("an unknown key was accepted")
1524 }
1525}
1526
1527func TestConfigPathFromArgs(t *testing.T) {
1528 for _, tc := range []struct {
1529 args []string
1530 want string
1531 }{
1532 {nil, "/def"},
1533 {[]string{"-once"}, "/def"},
1534 {[]string{"-config", "/a"}, "/a"},
1535 {[]string{"--config", "/b", "-once"}, "/b"},
1536 {[]string{"-config=/c"}, "/c"},
1537 } {
1538 if got := configPathFromArgs(tc.args, "/def"); got != tc.want {
1539 t.Errorf("%v: got %s want %s", tc.args, got, tc.want)
1540 }
1541 }
1542}
1543
1544func TestConfigDirHonoursXDG(t *testing.T) {
1545 t.Setenv("XDG_CONFIG_HOME", "/x")
1546 if got := configDir(); got != "/x/gitbay-runner" {
1547 t.Fatalf("got %s", got)
1548 }
1549 t.Setenv("XDG_CONFIG_HOME", "")
1550 t.Setenv("HOME", "/h")
1551 if got := configDir(); got != "/h/.config/gitbay-runner" {
1552 t.Fatalf("got %s", got)
1553 }
1554}
1555
1556func TestIdentityOpts(t *testing.T) {
1557 if got := identityOpts(""); got != nil {
1558 t.Fatalf("empty identity produced %v", got)
1559 }
1560 got := identityOpts("/k")
1561 if len(got) != 4 || got[0] != "-i" || got[1] != "/k" || got[3] != "IdentitiesOnly=yes" {
1562 t.Fatalf("got %v", got)
1563 }
1564}
1565```
1566
1567- [ ] **Step 2: Run them to see them fail**
1568
1569Run: `go test ./cmd/gitbay-runner -run 'TestConfig|TestIdentity' 2>&1 | head -5`
1570Expected: `undefined: loadConfig` and friends.
1571
1572- [ ] **Step 3: Write `cmd/gitbay-runner/config.go`**
1573
1574```go
1575package main
1576
1577import (
1578 "errors"
1579 "flag"
1580 "fmt"
1581 "os"
1582 "path/filepath"
1583 "strings"
1584
1585 "github.com/BurntSushi/toml"
1586)
1587
1588// The runner takes everything as flags, which does not work under a
1589// service manager. config.toml in the config directory carries the same
1590// names; a flag on the command line overrides it (#184).
1591
1592func configDir() string {
1593 if x := os.Getenv("XDG_CONFIG_HOME"); x != "" {
1594 return filepath.Join(x, "gitbay-runner")
1595 }
1596 return filepath.Join(os.Getenv("HOME"), ".config", "gitbay-runner")
1597}
1598
1599func defaultConfigPath() string { return filepath.Join(configDir(), "config.toml") }
1600
1601// configPathFromArgs finds -config before the flag set is parsed, since
1602// the file's values must be set before parsing for flags to override them.
1603func configPathFromArgs(args []string, def string) string {
1604 for i, a := range args {
1605 a = strings.TrimPrefix(a, "-")
1606 if a == "-config" || a == "config" {
1607 if i+1 < len(args) {
1608 return args[i+1]
1609 }
1610 }
1611 if v, ok := strings.CutPrefix(a, "config="); ok {
1612 return v
1613 }
1614 if v, ok := strings.CutPrefix(a, "-config="); ok {
1615 return v
1616 }
1617 }
1618 return def
1619}
1620
1621// configKeys is every key the file may carry: the flag names.
1622var configKeys = map[string]bool{"remote": true, "ssh-opts": true, "clone-base": true, "workdir": true,
1623 "poll": true, "timeout": true, "repos": true, "jobs": true, "image": true, "isolation": true,
1624 "memory": true, "cpus": true, "untrusted": true, "identity": true}
1625
1626// loadConfig reads path into flag name → value. Absent file: found is
1627// false and there is no error. An unknown key is an error, not a typo
1628// the runner silently ignores.
1629func loadConfig(path string) (values map[string]string, found bool, err error) {
1630 var raw map[string]any
1631 if _, err := toml.DecodeFile(path, &raw); errors.Is(err, os.ErrNotExist) {
1632 return nil, false, nil
1633 } else if err != nil {
1634 return nil, true, fmt.Errorf("%s: %w", path, err)
1635 }
1636 values = map[string]string{}
1637 for k, v := range raw {
1638 if !configKeys[k] {
1639 return nil, true, fmt.Errorf("%s: unknown key %s", path, k)
1640 }
1641 values[k] = fmt.Sprint(v)
1642 }
1643 return values, true, nil
1644}
1645
1646// applyConfig sets each value on the flag set, which is what parsing the
1647// command line would do; parse afterwards and the command line wins.
1648func applyConfig(fs *flag.FlagSet, values map[string]string) error {
1649 for k, v := range values {
1650 if fs.Lookup(k) == nil {
1651 return fmt.Errorf("config: unknown key %s", k)
1652 }
1653 if err := fs.Set(k, v); err != nil {
1654 return fmt.Errorf("config: %s: %w", k, err)
1655 }
1656 }
1657 return nil
1658}
1659
1660// identityOpts is what makes ssh and git use the runner's own key and no
1661// other: on a laptop the ambient key is the user's full-scope one, which
1662// the runner protocol refuses.
1663func identityOpts(path string) []string {
1664 if path == "" {
1665 return nil
1666 }
1667 return []string{"-i", path, "-o", "IdentitiesOnly=yes"}
1668}
1669```
1670
1671- [ ] **Step 4: Wire it into `main.go`**
1672
1673In `main()`, replace `flag.Parse()` and the flag block with a flag set fed by the config file. Add three flags and keep the others as they are:
1674
1675```go
1676 var (
1677 configPath = flag.String("config", defaultConfigPath(), "config file; keys are these flag names, flags override it")
1678 identity = flag.String("identity", "", "ssh private key to poll and clone with (default: the key gitbay-runner init generated, if present)")
1679 untrusted = flag.Bool("untrusted", false, "also claim untrusted builds: merge request heads from forks (needs -isolation podman to be safe)")
1680 // ... existing flags unchanged ...
1681 )
1682 path := configPathFromArgs(os.Args[1:], *configPath)
1683 if values, found, err := loadConfig(path); err != nil {
1684 log.Fatal(err)
1685 } else if found {
1686 if err := applyConfig(flag.CommandLine, values); err != nil {
1687 log.Fatal(err)
1688 }
1689 log.Printf("config: %s", path)
1690 }
1691 flag.Parse()
1692```
1693
1694After `if *sshOpts != "" { r.sshOpts = strings.Fields(*sshOpts) }`:
1695
1696```go
1697 if *identity == "" {
1698 if p := filepath.Join(configDir(), "id_ed25519"); fileExists(p) {
1699 *identity = p
1700 }
1701 }
1702 r.sshOpts = append(identityOpts(*identity), r.sshOpts...)
1703 r.untrusted = *untrusted
1704```
1705
1706with
1707
1708```go
1709func fileExists(p string) bool { _, err := os.Stat(p); return err == nil }
1710```
1711
1712`runner` struct gains `untrusted bool`. In `step()`:
1713
1714```go
1715 args := []string{"runner", "next"}
1716 if r.untrusted {
1717 args = append(args, "--untrusted")
1718 }
1719 args = append(append(args, r.repos...), "--json")
1720 out, err := r.ssh(nil, args...)
1721```
1722
1723Both `ssh()` and the `gitSSH` line already use `r.sshOpts`, so the identity reaches both.
1724
1725Move the `init` dispatch hook in now so Task 7 has a place to land, at the top of `main()`:
1726
1727```go
1728 if len(os.Args) > 1 && os.Args[1] == "init" {
1729 os.Exit(runInit(os.Args[2:]))
1730 }
1731```
1732
1733and a stub in `config.go` until Task 7 replaces it:
1734
1735```go
1736func runInit(args []string) int { fmt.Fprintln(os.Stderr, "init: not implemented"); return 2 }
1737```
1738
1739- [ ] **Step 5: Run the tests**
1740
1741Run: `go build ./... && go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner 2>&1 | tail -3`
1742Expected: PASS.
1743
1744- [ ] **Step 6: Commit**
1745
1746```bash
1747git add cmd/gitbay-runner/config.go cmd/gitbay-runner/config_test.go cmd/gitbay-runner/main.go
1748git commit -m "runner: config.toml, -identity, -untrusted
1749
1750Ref #184"
1751```
1752
1753---
1754
1755### Task 7: Runner: `gitbay-runner init`
1756
1757**Files:**
1758- Create: `cmd/gitbay-runner/init.go` (replaces the stub `runInit` in `config.go`; delete the stub)
1759- Create: `cmd/gitbay-runner/init_test.go`
1760
1761**Interfaces:**
1762- Consumes: `configDir()`, `defaultWorkdir()`, `toolpath.Look("ssh-keygen")`.
1763- Produces: `runInit(args []string) int`; files `<configDir>/id_ed25519`, `id_ed25519.pub`, `config.toml`.
1764
1765- [ ] **Step 1: Write the failing test**
1766
1767`cmd/gitbay-runner/init_test.go`:
1768
1769```go
1770package main
1771
1772import (
1773 "bytes"
1774 "os"
1775 "os/exec"
1776 "path/filepath"
1777 "strings"
1778 "testing"
1779)
1780
1781// init creates the key and config once, prints the key and the attach
1782// command, and running it again changes nothing.
1783func TestInitWritesKeyAndConfigOnce(t *testing.T) {
1784 if _, err := exec.LookPath("ssh-keygen"); err != nil {
1785 t.Skip("ssh-keygen not on PATH")
1786 }
1787 dir := t.TempDir()
1788 t.Setenv("XDG_CONFIG_HOME", dir)
1789 var out bytes.Buffer
1790 initOut = &out
1791 defer func() { initOut = os.Stdout }()
1792
1793 if code := runInit([]string{"-remote", "git@example.test"}); code != 0 {
1794 t.Fatalf("init: exit %d\n%s", code, out.String())
1795 }
1796 cdir := filepath.Join(dir, "gitbay-runner")
1797 key := filepath.Join(cdir, "id_ed25519")
1798 pub, err := os.ReadFile(key + ".pub")
1799 if err != nil || !strings.HasPrefix(string(pub), "ssh-ed25519 ") {
1800 t.Fatalf("public key: %v %q", err, pub)
1801 }
1802 if fi, _ := os.Stat(key); fi.Mode().Perm() != 0o600 {
1803 t.Fatalf("private key mode %o", fi.Mode().Perm())
1804 }
1805 if fi, _ := os.Stat(cdir); fi.Mode().Perm() != 0o700 {
1806 t.Fatalf("config dir mode %o", fi.Mode().Perm())
1807 }
1808 cfg, _ := os.ReadFile(filepath.Join(cdir, "config.toml"))
1809 for _, want := range []string{"remote = \"git@example.test\"", "isolation = \"none\"", "untrusted = false", "identity = \"" + key + "\""} {
1810 if !strings.Contains(string(cfg), want) {
1811 t.Fatalf("config lacks %q:\n%s", want, cfg)
1812 }
1813 }
1814 for _, want := range []string{strings.TrimSpace(string(pub)), "gitbay repo runner add owner/name < " + key + ".pub", "https://example.test/owner/name/settings"} {
1815 if !strings.Contains(out.String(), want) {
1816 t.Fatalf("output lacks %q:\n%s", want, out.String())
1817 }
1818 }
1819
1820 out.Reset()
1821 if code := runInit([]string{"-remote", "git@other.test"}); code != 0 {
1822 t.Fatalf("second init: exit %d\n%s", code, out.String())
1823 }
1824 if pub2, _ := os.ReadFile(key + ".pub"); string(pub2) != string(pub) {
1825 t.Fatal("second init replaced the key")
1826 }
1827 if cfg2, _ := os.ReadFile(filepath.Join(cdir, "config.toml")); string(cfg2) != string(cfg) {
1828 t.Fatal("second init rewrote the config")
1829 }
1830}
1831
1832// podman needs an image; init refuses to write a config the runner would
1833// refuse to start with.
1834func TestInitPodmanNeedsImage(t *testing.T) {
1835 t.Setenv("XDG_CONFIG_HOME", t.TempDir())
1836 var out bytes.Buffer
1837 initOut = &out
1838 defer func() { initOut = os.Stdout }()
1839 if code := runInit([]string{"-isolation", "podman"}); code != 2 {
1840 t.Fatalf("exit %d, want 2:\n%s", code, out.String())
1841 }
1842}
1843```
1844
1845- [ ] **Step 2: Run it to see it fail**
1846
1847Run: `go test ./cmd/gitbay-runner -run TestInit 2>&1 | head -5`
1848Expected: `undefined: initOut`.
1849
1850- [ ] **Step 3: Write `cmd/gitbay-runner/init.go`**
1851
1852```go
1853package main
1854
1855import (
1856 "flag"
1857 "fmt"
1858 "io"
1859 "os"
1860 "os/exec"
1861 "path/filepath"
1862 "strings"
1863
1864 "gitbay.org/gitbay/internal/toolpath"
1865)
1866
1867// initOut is where init prints; tests capture it.
1868var initOut io.Writer = os.Stdout
1869
1870// runInit makes a fresh install ready to attach: a key of its own, a
1871// config file the service reads, and the one command to run next. It never
1872// overwrites a key or a config that exists, so running it twice is safe.
1873func runInit(args []string) int {
1874 fs := flag.NewFlagSet("init", flag.ContinueOnError)
1875 fs.SetOutput(initOut)
1876 remote := fs.String("remote", "git@gitbay.org", "ssh destination of the gitbay server")
1877 workdir := fs.String("workdir", defaultWorkdir(), "build workspace root")
1878 isolation := fs.String("isolation", isolationNone, "how steps run: none, or podman with -image")
1879 image := fs.String("image", "", "container image for -isolation podman")
1880 if err := fs.Parse(args); err != nil {
1881 return 2
1882 }
1883 if *isolation == isolationPodman && *image == "" {
1884 fmt.Fprintln(initOut, "-isolation podman needs -image <ref>: the runner refuses to start without one, and there is no image to guess")
1885 return 2
1886 }
1887 if *isolation != isolationPodman && *isolation != isolationNone {
1888 fmt.Fprintf(initOut, "unknown isolation %q\n", *isolation)
1889 return 2
1890 }
1891
1892 dir := configDir()
1893 if err := os.MkdirAll(dir, 0o700); err != nil {
1894 fmt.Fprintln(initOut, err)
1895 return 1
1896 }
1897 os.Chmod(dir, 0o700)
1898 key := filepath.Join(dir, "id_ed25519")
1899 if !fileExists(key) {
1900 cmd := exec.Command(toolpath.Look("ssh-keygen"), "-q", "-t", "ed25519", "-N", "", "-C", "gitbay-runner", "-f", key)
1901 if out, err := cmd.CombinedOutput(); err != nil {
1902 fmt.Fprintf(initOut, "ssh-keygen: %v\n%s", err, out)
1903 return 1
1904 }
1905 }
1906 os.Chmod(key, 0o600)
1907
1908 cfgPath := filepath.Join(dir, "config.toml")
1909 if !fileExists(cfgPath) {
1910 var b strings.Builder
1911 fmt.Fprintf(&b, "remote = %q\n", *remote)
1912 fmt.Fprintf(&b, "workdir = %q\n", *workdir)
1913 fmt.Fprintf(&b, "isolation = %q\n", *isolation)
1914 if *image != "" {
1915 fmt.Fprintf(&b, "image = %q\n", *image)
1916 }
1917 fmt.Fprintf(&b, "untrusted = false\n")
1918 fmt.Fprintf(&b, "identity = %q\n", key)
1919 if err := os.WriteFile(cfgPath, []byte(b.String()), 0o600); err != nil {
1920 fmt.Fprintln(initOut, err)
1921 return 1
1922 }
1923 }
1924
1925 pub, err := os.ReadFile(key + ".pub")
1926 if err != nil {
1927 fmt.Fprintln(initOut, err)
1928 return 1
1929 }
1930 host := *remote
1931 if i := strings.LastIndex(host, "@"); i >= 0 {
1932 host = host[i+1:]
1933 }
1934 fmt.Fprintf(initOut, "config: %s\nkey: %s\n\n", cfgPath, key)
1935 if *isolation == isolationNone {
1936 fmt.Fprintln(initOut, "Steps run on this machine as your user, with no container. Untrusted builds\n(merge requests from forks) are excluded unless the runner is started with\n-untrusted, so that means your own commits.\n")
1937 }
1938 fmt.Fprintf(initOut, "This runner's public key:\n\n %s\nAttach it to each repository it should build, as a repository admin:\n\n gitbay repo runner add owner/name < %s.pub\n\nor paste it under Runners at https://%s/owner/name/settings\n\nThen start it:\n\n brew services start krz/tap/gitbay-runner\n\nor run gitbay-runner with no arguments.\n",
1939 strings.TrimSpace(string(pub)), key, host)
1940 return 0
1941}
1942```
1943
1944`isolationNone` and `isolationPodman` are the constants in `isolate.go:20-21`. Delete the stub `runInit` from `config.go`.
1945
1946- [ ] **Step 4: Run the tests**
1947
1948Run: `go build ./... && go vet ./cmd/gitbay-runner && go test ./cmd/gitbay-runner 2>&1 | tail -3`
1949Expected: PASS.
1950
1951- [ ] **Step 5: Commit**
1952
1953```bash
1954git add cmd/gitbay-runner/init.go cmd/gitbay-runner/init_test.go cmd/gitbay-runner/config.go
1955git commit -m "runner: init generates the key and config and prints the attach step
1956
1957Ref #184"
1958```
1959
1960---
1961
1962### Task 8: e2e: init, attach, build; fork head waits
1963
1964**Files:**
1965- Create: `e2e/runnerattach_test.go`
1966
1967**Interfaces:**
1968- Consumes: `buildRunner(t)`, `inst.newKey`, `inst.admin`, `inst.ssh(t, key, stdin, argv...)`, `inst.gitEnv`, `inst.sshURL`, `mustGit`, `inst.port`, `inst.sshDir` (all in `e2e/ci_test.go` and the instance helpers).
1969
1970- [ ] **Step 1: Write the test**
1971
1972`e2e/runnerattach_test.go`:
1973
1974```go
1975package e2e
1976
1977import (
1978 "fmt"
1979 "os"
1980 "os/exec"
1981 "path/filepath"
1982 "strings"
1983 "testing"
1984)
1985
1986// The whole flow a user goes through: init on their machine, attach the
1987// printed key to their repository, start the runner from the config init
1988// wrote. The runner builds their push and leaves a fork's merge request
1989// head alone until started with -untrusted.
1990func TestAttachedRunnerBuildsOwnRepo(t *testing.T) {
1991 inst := startInstance(t)
1992 inst.runner = buildRunner(t)
1993 aliceKey := inst.newKey(t, "alice")
1994 inst.admin(t, "admin", "user", "create", "alice", "--key", aliceKey+".pub")
1995 bobKey := inst.newKey(t, "bob")
1996 inst.admin(t, "admin", "user", "create", "bob", "--key", bobKey+".pub")
1997 if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "create", "alice/app"); code != 0 {
1998 t.Fatalf("repo create: %s", errOut)
1999 }
2000
2001 // init on "alice's laptop".
2002 xdg := t.TempDir()
2003 initCmd := exec.Command(inst.runner, "init", "-remote", "git@127.0.0.1")
2004 initCmd.Env = append(os.Environ(), "XDG_CONFIG_HOME="+xdg)
2005 initOut, err := initCmd.CombinedOutput()
2006 if err != nil {
2007 t.Fatalf("init: %v\n%s", err, initOut)
2008 }
2009 cdir := filepath.Join(xdg, "gitbay-runner")
2010 pub, err := os.ReadFile(filepath.Join(cdir, "id_ed25519.pub"))
2011 if err != nil {
2012 t.Fatal(err)
2013 }
2014 if !strings.Contains(string(initOut), strings.TrimSpace(string(pub))) {
2015 t.Fatalf("init did not print the key:\n%s", initOut)
2016 }
2017
2018 // The unattached key claims nothing, even with a build queued.
2019 work := t.TempDir()
2020 env := inst.gitEnv(aliceKey)
2021 mustGit(t, work, env, "clone", inst.sshURL("alice/app"), "w")
2022 dir := filepath.Join(work, "w")
2023 os.MkdirAll(filepath.Join(dir, ".gitbay"), 0o755)
2024 os.WriteFile(filepath.Join(dir, ".gitbay", "ci.yml"), []byte("jobs:\n unit:\n steps:\n - echo built\n"), 0o644)
2025 mustGit(t, dir, env, "checkout", "-q", "-b", "main")
2026 mustGit(t, dir, env, "add", ".")
2027 mustGit(t, dir, env, "commit", "-q", "-m", "ci")
2028 mustGit(t, dir, env, "push", "-q", "origin", "main")
2029
2030 run := func(extra ...string) string {
2031 t.Helper()
2032 // No -i in ssh-opts: the identity from the config is what
2033 // authenticates, which is the point.
2034 opts := fmt.Sprintf("-p %d -o StrictHostKeyChecking=no -o UserKnownHostsFile=%s -o BatchMode=yes",
2035 inst.port, filepath.Join(inst.sshDir, "known_hosts"))
2036 args := append([]string{"-config", filepath.Join(cdir, "config.toml"), "-once",
2037 "-ssh-opts", opts,
2038 "-clone-base", fmt.Sprintf("ssh://git@127.0.0.1:%d", inst.port),
2039 "-workdir", t.TempDir()}, extra...)
2040 cmd := exec.Command(inst.runner, args...)
2041 cmd.Env = append(os.Environ(), "XDG_CONFIG_HOME="+xdg, "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null")
2042 out, err := cmd.CombinedOutput()
2043 if err != nil {
2044 t.Fatalf("runner: %v\n%s", err, out)
2045 }
2046 return string(out)
2047 }
2048 if out, _, _ := inst.ssh(t, aliceKey, "", "build", "list", "alice/app"); !strings.Contains(out, "unit\tpending") {
2049 t.Fatalf("build not pending before attach: %s", out)
2050 }
2051
2052 // Attach with the printed key.
2053 if _, errOut, code := inst.ssh(t, aliceKey, string(pub), "repo", "runner", "add", "alice/app"); code != 0 {
2054 t.Fatalf("repo runner add: %s", errOut)
2055 }
2056 out, _, _ := inst.ssh(t, aliceKey, "", "repo", "runner", "list", "alice/app", "--json")
2057 if !strings.Contains(out, `"username":"alice"`) || strings.Contains(out, `"last_seen":"20`) {
2058 t.Fatalf("list after attach: %s", out)
2059 }
2060
2061 // The runner builds it.
2062 run()
2063 if out, _, _ = inst.ssh(t, aliceKey, "", "build", "list", "alice/app"); !strings.Contains(out, "unit\tsuccess") {
2064 t.Fatalf("build not built by the attached runner: %s", out)
2065 }
2066 if out, _, _ = inst.ssh(t, aliceKey, "", "repo", "runner", "list", "alice/app", "--json"); !strings.Contains(out, `"last_seen":"20`) {
2067 t.Fatalf("no heartbeat after a poll: %s", out)
2068 }
2069
2070 // bob forks and opens a merge request: an untrusted build in the target.
2071 if _, errOut, code := inst.ssh(t, bobKey, "", "repo", "fork", "alice/app"); code != 0 {
2072 t.Fatalf("fork: %s", errOut)
2073 }
2074 bwork := t.TempDir()
2075 benv := inst.gitEnv(bobKey)
2076 mustGit(t, bwork, benv, "clone", inst.sshURL("bob/app"), "w")
2077 bdir := filepath.Join(bwork, "w")
2078 mustGit(t, bdir, benv, "checkout", "-q", "-b", "feat")
2079 os.WriteFile(filepath.Join(bdir, "f.txt"), []byte("y\n"), 0o644)
2080 mustGit(t, bdir, benv, "add", ".")
2081 mustGit(t, bdir, benv, "commit", "-q", "-m", "change")
2082 mustGit(t, bdir, benv, "push", "-q", "origin", "feat")
2083 if _, _, code := inst.ssh(t, bobKey, "", "build", "cancel", "bob/app", "1"); code != 0 {
2084 t.Fatal("cancel bob's own build")
2085 }
2086 if _, errOut, code := inst.ssh(t, bobKey, "", "mr", "create", "alice/app",
2087 "--source", "bob/app:feat", "--target", "main", "--title", "change"); code != 0 {
2088 t.Fatalf("mr create: %s", errOut)
2089 }
2090 run()
2091 if out, _, _ = inst.ssh(t, aliceKey, "", "build", "list", "alice/app"); strings.Count(out, "pending") != 1 {
2092 t.Fatalf("fork head was claimed without -untrusted:\n%s", out)
2093 }
2094 run("-untrusted")
2095 if out, _, _ = inst.ssh(t, aliceKey, "", "build", "list", "alice/app"); strings.Contains(out, "pending") {
2096 t.Fatalf("fork head not built with -untrusted:\n%s", out)
2097 }
2098}
2099```
2100
2101- [ ] **Step 2: Run it**
2102
2103Run: `go test ./e2e -run TestAttachedRunnerBuildsOwnRepo -count=1 -v 2>&1 | tail -20`
2104Expected: PASS. If the fork's build numbering differs (two jobs, or the fork's push queues more than one build), adjust the `build cancel bob/app` loop to cancel every pending build listed by `build list bob/app --json`.
2105
2106- [ ] **Step 3: Commit**
2107
2108```bash
2109git add e2e/runnerattach_test.go
2110git commit -m "e2e: init, attach, and an attached runner building its repository
2111
2112Ref #184"
2113```
2114
2115---
2116
2117### Task 9: Docs: wiki pages
2118
2119**Files:**
2120- Modify: `.gitbay/wiki/Users.org` (after the "CI builds" section's last paragraph, before "* Large files (LFS)")
2121- Modify: `.gitbay/wiki/Admin.org:329-340` and `:395-402`
2122- Modify: `.gitbay/wiki/Threat-Model.org:120-126`
2123- Modify: `.gitbay/wiki/Parity.org` (repo table, after the `webhooks` row; and the "SSH only, by design" paragraph is unchanged)
2124- Modify: `.gitbay/wiki/FAQ.org:20-25`
2125- Modify: `.gitbay/wiki/CI.org` (one sentence after the three mechanisms list)
2126
2127- [ ] **Step 1: Users: "Your own runner"**
2128
2129Insert before `* Large files (LFS)`:
2130
2131```org
2132** Your own runner
2133
2134Builds run on runners attached to the repository. An instance need not
2135offer any: install =gitbay-runner= on a machine of yours and attach it.
2136
2137#+begin_src sh
2138brew install krz/tap/gitbay-runner # or a binary from the release
2139gitbay-runner init -remote git@gitbay.org
2140#+end_src
2141
2142=init= generates a key under =~/.config/gitbay-runner/=, writes
2143=config.toml= beside it, and prints the public key with the command to
2144attach it:
2145
2146#+begin_src sh
2147gitbay repo runner add owner/name < ~/.config/gitbay-runner/id_ed25519.pub
2148#+end_src
2149
2150or paste the key under Runners on the repository's settings page. Then
2151=brew services start krz/tap/gitbay-runner=, or run =gitbay-runner= with
2152no arguments; it reads the config file, and any flag overrides it.
2153
2154What it builds: every build for the repositories it is attached to,
2155with the repository's secrets, and nothing else. Merge requests from
2156forks are untrusted and wait unless the runner runs with =-untrusted=,
2157which is only sensible with =-isolation podman -image <ref>= (see
2158[[Admin][Admin]]). Attach one runner to several repositories by repeating
2159=repo runner add=; run several runners on one account by running =init=
2160on each machine. =repo runner list= shows each attached key, when it
2161last polled and the build it holds; =repo runner remove <fingerprint>=
2162detaches one (the key stays on your account; =keys remove= drops it).
2163A runner key reaches only the runner protocol and read-only git, so a
2164build step that reads it off disk cannot administer your account.
2165```
2166
2167- [ ] **Step 2: Admin**
2168
2169Replace lines 329-340's opening paragraph ("=gitbay-runner= executes builds ... then removed:") with:
2170
2171```org
2172=gitbay-runner= executes builds queued by pushes and merge requests. It
2173polls over SSH with a key of scope =runner=, which reaches only the
2174runner protocol and read-only git (a runner executes arbitrary
2175repository code, so the key it holds must not do more). A runner key
2176claims builds only for the repositories it is attached to, by =repo
2177runner add= from a repository admin or an instance admin; an admin key
2178claims any. Users attach their own runners: see the Users page. For an
2179instance runner, run it as a dedicated unprivileged user on a non-admin
2180account. =admin user create --key= registers a full-scope key, so the
2181runner key is added afterwards through a bootstrap key that is then
2182removed, and attached to each repository it should build:
2183```
2184
2185After the existing bootstrap code block, add:
2186
2187```org
2188#+begin_src sh
2189gitbay repo runner add krz/site < /var/lib/gitbay-runner/.ssh/id_ed25519.pub
2190#+end_src
2191```
2192
2193Replace lines 395-402 (from "and the isolation canary, nothing else" to "the boundary is you choosing how to start it.") with:
2194
2195```org
2196and the isolation canary, nothing else, because it shares the host with
2197the forge; any other repository builds on a runner its owner attaches.
2198
2199=-repos= narrows an admin runner; for a runner key the attachments are
2200the boundary, held by the server, and =-repos= may only name
2201repositories among them. =-untrusted= makes a runner claim merge
2202request heads from forks; the bay1 unit sets it because it isolates in
2203podman. A runner without it builds trusted commits only.
2204```
2205
2206Add `-untrusted` to the `gitbay-runner -remote git@gitbay.org -repos krz/site,krz/docs` example only if that runner isolates; leave the example as is and note under it: "Add =-untrusted= only with =-isolation podman=."
2207
2208- [ ] **Step 3: Threat-Model**
2209
2210Replace the "What the runner holds" bullet (lines 120-126) with:
2211
2212```org
2213- *What the runner holds.* A key of scope =runner=, which the dispatcher
2214 confines to =runner next=, =runner log= and =runner done= and to
2215 read-only git, and which claims, logs and finishes builds only for
2216 the repositories it is attached to (=repo runner add=). A step that
2217 reads the key off the disk gets exactly that: it cannot administer
2218 the instance, push, read a repository the runner's account cannot, or
2219 touch another repository's builds. An admin key still works for the
2220 runner protocol so an operator can rotate at their own pace; a runner
2221 host should not hold one. Untrusted builds are skipped unless the
2222 runner asks with =-untrusted=, so a runner on a user's machine never
2223 executes a stranger's branch by default.
2224```
2225
2226- [ ] **Step 4: Parity, FAQ, CI**
2227
2228Parity, repo table, after the `webhooks` row:
2229
2230```org
2231| runners attach, list, detach | yes | yes | no |
2232```
2233
2234The columns are cli, web, ios. iOS is `no`: outstanding, not intended.
2235
2236FAQ, replace the "Does CI run for my repository on gitbay.org?" answer:
2237
2238```org
2239- Does CI run for my repository on gitbay.org? :: On a runner you
2240 attach. The instance's own runner builds the forge's repositories and
2241 its isolation canary, since it shares the host with the forge.
2242 Install =gitbay-runner= on a machine of yours, run =gitbay-runner
2243 init=, and attach the key it prints with =repo runner add= or on the
2244 repository's settings page; see the Users page. A self-hosted
2245 instance can do the same, or run one runner for whichever
2246 repositories its operator attaches it to.
2247```
2248
2249CI.org, after the three-mechanism list:
2250
2251```org
2252Which runner takes a build is the fourth: a build is claimed only by a
2253runner attached to its repository (or an instance admin's runner), and
2254an untrusted build only by one started with =-untrusted=. A repository
2255with no runner attached queues builds nothing claims. See the Users
2256page.
2257```
2258
2259- [ ] **Step 5: Check the wiki tests**
2260
2261Run: `go test ./internal/hookd ./internal/ci -run 'Wiki|Parity' 2>&1 | tail -3`
2262Expected: PASS (nothing here changes the push-shape table).
2263
2264- [ ] **Step 6: Commit**
2265
2266```bash
2267git add .gitbay/wiki/Users.org .gitbay/wiki/Admin.org .gitbay/wiki/Threat-Model.org .gitbay/wiki/Parity.org .gitbay/wiki/FAQ.org .gitbay/wiki/CI.org
2268git commit -m "wiki: runners attached to repositories
2269
2270Ref #184"
2271```
2272
2273---
2274
2275### Task 10: Deploy, release, and the tap
2276
2277**Files:**
2278- Modify: `deploy/gitbay-runner.override.conf` (the `ExecStart` line)
2279- Modify: `deploy/release.sh:22` (the binary list)
2280- Create in `krz/homebrew-tap` (separate clone, after the release is tagged): `Formula/gitbay-runner.rb`; modify `Formula/gitbay.rb`
2281
2282- [ ] **Step 1: The bay1 unit claims fork heads**
2283
2284In `deploy/gitbay-runner.override.conf`, the `ExecStart=` line gains ` -untrusted` at the end, and the comment above `ExecStart` gains:
2285
2286```
2287# -untrusted: this runner isolates in podman, so it takes merge request
2288# heads from forks; a runner without a container must not.
2289```
2290
2291- [ ] **Step 2: Release binaries include the runner**
2292
2293`deploy/release.sh`: `for bin in gitbay gitbayd; do` becomes `for bin in gitbay gitbayd gitbay-runner; do`, and the header comment's "gitbay and gitbayd" becomes "gitbay, gitbayd and gitbay-runner".
2294
2295- [ ] **Step 3: Build, vet, and the touched unit tests**
2296
2297Run: `go build ./... && go vet ./... && go test ./internal/store ./internal/control ./cmd/gitbay ./cmd/gitbay-runner ./internal/httpd 2>&1 | tail -8`
2298Expected: all PASS.
2299
2300- [ ] **Step 4: Commit and open the MR**
2301
2302```bash
2303git add deploy/gitbay-runner.override.conf deploy/release.sh
2304git commit -m "deploy: the bay1 runner claims untrusted builds; release ships gitbay-runner
2305
2306Ref #184"
2307git push -u origin user-runners
2308gitbay mr create --source user-runners --target main --title "Runners attached to repositories" --file - <<'MR'
2309A runner key claims builds only for the repositories it is attached to
2310(`repo runner add|list|remove`, also on the settings page). `runner next`
2311skips untrusted builds unless `--untrusted`. Migration 0050.
2312`gitbay-runner init`, `config.toml`, `-identity`, `-untrusted`.
2313
2314After deploy, attach the bay1 runner to krz/gitbay and cmc/ci-smoke as
2315the admin; until then it claims nothing.
2316
2317Ref #184
2318MR
2319```
2320
2321Wait for CI on bay1 (`gitbay build list` on the MR head) before merging: `gitbay mr merge <n> --strategy ff`.
2322
2323- [ ] **Step 5: Deploy and attach (operator, after merge)**
2324
2325```bash
2326make deploy && make deploy-runner
2327ssh -p 2222 root@46.232.248.67 cat /var/lib/gitbay-runner/.ssh/id_ed25519.pub | gitbay repo runner add krz/gitbay
2328ssh -p 2222 root@46.232.248.67 cat /var/lib/gitbay-runner/.ssh/id_ed25519.pub | gitbay repo runner add cmc/ci-smoke
2329gitbay admin runners
2330```
2331
2332The last line must show the `ci` row with its fingerprint and `krz/gitbay,cmc/ci-smoke`.
2333
2334- [ ] **Step 6: The tap, after the release is tagged**
2335
2336In a clone of `https://gitbay.org/krz/homebrew-tap.git`, `Formula/gitbay-runner.rb`:
2337
2338```ruby
2339class GitbayRunner < Formula
2340 desc "CI runner for gitbay: builds the repositories you attach it to"
2341 homepage "https://gitbay.org/krz/gitbay"
2342 url "https://gitbay.org/krz/gitbay.git",
2343 tag: "v1.17.0",
2344 revision: "<commit of the tag>"
2345 license "0BSD"
2346 head "https://gitbay.org/krz/gitbay.git", branch: "main"
2347
2348 depends_on "go" => :build
2349
2350 def install
2351 system "go", "build", *std_go_args(ldflags: "-s -w"), "./cmd/gitbay-runner"
2352 end
2353
2354 service do
2355 run [opt_bin/"gitbay-runner"]
2356 keep_alive true
2357 log_path var/"log/gitbay-runner.log"
2358 error_log_path var/"log/gitbay-runner.err.log"
2359 end
2360
2361 def caveats
2362 <<~EOS
2363 Generate this machine's key and config, and print the key to attach:
2364 gitbay-runner init -remote git@gitbay.org
2365 Attach it to each repository it should build (as a repository admin):
2366 gitbay repo runner add owner/name < ~/.config/gitbay-runner/id_ed25519.pub
2367 Then:
2368 brew services start krz/tap/gitbay-runner
2369 EOS
2370 end
2371
2372 test do
2373 assert_match "gitbay-runner", shell_output("#{bin}/gitbay-runner -version")
2374 end
2375end
2376```
2377
2378In `Formula/gitbay.rb`, set `tag:` and `revision:` to the same release. Check with `brew install --build-from-source krz/tap/gitbay-runner && brew test krz/tap/gitbay-runner`, then commit on a branch of the tap and merge with an MR there.
docs/specs/2026-09-08-user-runners-design.md +2 −2
@@ -208,8 +208,8 @@ already lists keys with their scope; a runner key shows as `runner`.
208 necessity" and "the scoping is what the runner asks for, not an ACL the 208 necessity" and "the scoping is what the runner asks for, not an ACL the
209 server holds" with the attachment rule. The bay1 example gains 209 server holds" with the attachment rule. The bay1 example gains
210 `-untrusted`. 210 `-untrusted`.
211- `Parity`: three rows, `repo runner add|list|remove`, yes on SSH, CLI, web, 211- `Parity`: one row in the repository table, runners attach/list/detach,
212 API. 212 yes on CLI and web, no on iOS.
213- `CI` and `FAQ`: one line each pointing at the Users section. The FAQ's 213- `CI` and `FAQ`: one line each pointing at the Users section. The FAQ's
214 "CI builds only the repositories the operator names" becomes "and any 214 "CI builds only the repositories the operator names" becomes "and any
215 repository with a runner attached". 215 repository with a runner attached".