CI path filters !262
9 files changed, +522 −13
Layout: unified · split
docs/specs/2026-09-04-wiki-in-repo-design.md added +156
| @@ -0,0 +1,156 @@ | |||
| 1 | # Wikis in the repository | ||
| 2 | |||
| 3 | Ref #169, #170. Milestone v1.14.0. Two merge requests: path filters first, | ||
| 4 | because without them the second one makes the wiki worse to use than it is | ||
| 5 | today. | ||
| 6 | |||
| 7 | ## Problem | ||
| 8 | |||
| 9 | A wiki is a companion bare repo at `<owner>/<name>.wiki.git`, created on first | ||
| 10 | push (`internal/sshd/sshd.go:359-362, 473-480`). It has no row in the store; | ||
| 11 | access derives from the parent. | ||
| 12 | |||
| 13 | That buys one thing — prose edits stay out of the code repository's history, | ||
| 14 | its protected branches and its builds — and costs four: | ||
| 15 | |||
| 16 | - **Invisible to the store.** Backup verification cannot distinguish a wiki | ||
| 17 | from a leaked directory and prints so (`cmd/gitbayd/backup.go:270`). Quotas, | ||
| 18 | `repo list`, search and the activity feed do not see it either. | ||
| 19 | - **Push is the only write path.** There is no `wiki edit` command and the web | ||
| 20 | renders wikis read-only. It is the one capability that does not follow "the | ||
| 21 | capability lands as a control command, then the surfaces render it". | ||
| 22 | - **`.wiki` is a reserved name suffix**, permanently | ||
| 23 | (`internal/policy/names.go:62`). | ||
| 24 | - **A second clone URL** that cannot be discovered without knowing the | ||
| 25 | convention. | ||
| 26 | |||
| 27 | ## Approach | ||
| 28 | |||
| 29 | Pages move to `.gitbay/wiki/` on the default branch, beside `ci.yml` and | ||
| 30 | `CODEOWNERS`, which is already where repository-scoped gitbay metadata lives. | ||
| 31 | The companion path is removed rather than kept alongside: one wiki exists across | ||
| 32 | 70 repositories on this instance, so a compatibility path would be permanent | ||
| 33 | cost for a single migration. | ||
| 34 | |||
| 35 | Rejected: an orphan ref (`refs/wiki/main`) in the main repository. It keeps | ||
| 36 | prose off the code DAG and out of normal clones, but it is invisible to plain | ||
| 37 | git tooling, needs a custom refspec to fetch, and would need its own write | ||
| 38 | commands to be usable at all. The gain over a directory is that prose stays out | ||
| 39 | of `git log`; the cost is a wiki nobody can edit without forge-specific | ||
| 40 | instructions. | ||
| 41 | |||
| 42 | ## What this does and does not buy | ||
| 43 | |||
| 44 | **Does:** one clone, one backup, one permission model, one history. Wiki edits | ||
| 45 | become reviewable through merge requests, approvals and CODEOWNERS for projects | ||
| 46 | that want that. | ||
| 47 | |||
| 48 | **Does not:** web editing on every repository. `repo commit-file` is the command | ||
| 49 | behind the web editor, and it refuses repositories that require verified | ||
| 50 | signatures, because the server authors those commits unsigned and will not write | ||
| 51 | a commit the repository's own policy would reject | ||
| 52 | (`internal/control/commitfile.go:28-35`). `krz/gitbay` requires signed commits, | ||
| 53 | so its wiki stays push-only. That is not a regression — it is push-only today — | ||
| 54 | but the parity gain is conditional and should not be claimed otherwise. | ||
| 55 | |||
| 56 | ## Design | ||
| 57 | |||
| 58 | ### Phase 1 — path filters (#169) | ||
| 59 | |||
| 60 | `ci.Job` gains `Paths` and `PathsIgnore`, each a list of globs matched against | ||
| 61 | the changed-file list from `gitutil.DiffFiles(dir, old, new)`, which already | ||
| 62 | exists (`internal/gitutil/merge.go:276`). | ||
| 63 | |||
| 64 | **Matching.** `path.Match` alone is not enough: Go's `*` does not cross `/`, so | ||
| 65 | `.gitbay/wiki/**` matches `.gitbay/wiki/Home.md` but not | ||
| 66 | `.gitbay/wiki/sub/Page.md` — a filter that appears to work and quietly misses | ||
| 67 | nested files. Define it explicitly: a pattern ending in `/**` matches that | ||
| 68 | directory and everything beneath it at any depth, implemented as a prefix | ||
| 69 | check; every other pattern goes to `path.Match` against the full path. A test | ||
| 70 | must cover the nested case, since that is the one a reader will assume works. | ||
| 71 | |||
| 72 | **Selection.** A job runs when `Paths` is empty or at least one changed file | ||
| 73 | matches one of its patterns. It is then skipped only when every changed file | ||
| 74 | matches at least one `PathsIgnore` pattern. A push touching one ignored file | ||
| 75 | and one other file runs the job. | ||
| 76 | |||
| 77 | **Fail open.** A job runs whenever the filter cannot be evaluated: a new branch | ||
| 78 | with no diff base (`old` is empty or all zeros), a `DiffFiles` error, or a job | ||
| 79 | declaring neither key. A filter that silently skips CI when it cannot tell is | ||
| 80 | worse than no filter, because the failure is invisible. | ||
| 81 | |||
| 82 | `QueueBranchBuilds` is shared with the merge path, which moves a ref without | ||
| 83 | reaching a hook (`internal/hookd/hookd.go:272-278`), so the old sha must reach | ||
| 84 | both callers. `u.Old` is already available at the hook call site | ||
| 85 | (`hookd.go:212`). | ||
| 86 | |||
| 87 | Tag jobs are unaffected: a tag build has no meaningful diff base. | ||
| 88 | |||
| 89 | ### Phase 2 — the move (#170) | ||
| 90 | |||
| 91 | **Storage.** `.gitbay/wiki/*.{md,org,markdown}` on `repo.DefaultBranch`. | ||
| 92 | `wikiExts` is unchanged. | ||
| 93 | |||
| 94 | **Resolution.** `wikiPages` and `wikiHome` keep their logic; they read a tree at | ||
| 95 | `.gitbay/wiki` on the default branch instead of the root of the companion's | ||
| 96 | `main`. `wiki list` and `wiki show` keep their argv, their JSON fields and their | ||
| 97 | exit codes — only resolution moves, so no surface changes shape. | ||
| 98 | |||
| 99 | `HasWiki` (`internal/httpd/web.go:336`) becomes "the default branch holds a | ||
| 100 | non-empty `.gitbay/wiki/` tree". The web route `/{owner}/{repo}/wiki` is | ||
| 101 | externally identical. | ||
| 102 | |||
| 103 | **Writing.** A push, like any other file. `repo commit-file <owner/name> | ||
| 104 | .gitbay/wiki/Page.md --ref <branch> --file -` is the existing command and the | ||
| 105 | existing web editor path; no `wiki edit` is added, because it would duplicate | ||
| 106 | one. | ||
| 107 | |||
| 108 | **Removal.** The `.wiki` suffix branch and `runWikiGit` in | ||
| 109 | `internal/sshd/sshd.go`; `wikiDir` in `internal/control/wiki.go` and | ||
| 110 | `internal/httpd/wiki.go`; the reservation in `internal/policy/names.go:62` and | ||
| 111 | the test asserting it; companion rename and delete in | ||
| 112 | `internal/control/repo.go:396,452`; the special-case wording in | ||
| 113 | `cmd/gitbayd/backup.go:270`. | ||
| 114 | |||
| 115 | **Migration.** One repository, by hand, not a shipped command: | ||
| 116 | |||
| 117 | ``` | ||
| 118 | git bundle create gitbay-wiki-$(date +%F).bundle --all # in a clone of the companion | ||
| 119 | git subtree add --prefix=.gitbay/wiki <wiki-url> main | ||
| 120 | ``` | ||
| 121 | |||
| 122 | `git subtree add` preserves the wiki's history inside the repository's DAG | ||
| 123 | rather than flattening it into one import commit. Verify pages render, keep the | ||
| 124 | bundle, then remove the bare repo from the server. | ||
| 125 | |||
| 126 | Add `paths-ignore: [".gitbay/wiki/**"]` to this repository's own heavy jobs in | ||
| 127 | the same change, so the migration does not immediately demonstrate the problem | ||
| 128 | phase 1 exists to prevent. | ||
| 129 | |||
| 130 | ## Tests | ||
| 131 | |||
| 132 | Phase 1: | ||
| 133 | - A job with `paths` matching a changed file runs; one matching nothing does not. | ||
| 134 | - `paths-ignore` covering every changed file skips the job; covering some of | ||
| 135 | them does not. | ||
| 136 | - A new branch runs every job. | ||
| 137 | - A `DiffFiles` failure runs every job. | ||
| 138 | - A job with neither key runs, unchanged from today. | ||
| 139 | - Tag builds are unaffected. | ||
| 140 | |||
| 141 | Phase 2: | ||
| 142 | - `wiki list` and `wiki show` return the same JSON for a repository whose pages | ||
| 143 | are in `.gitbay/wiki/` as the old commands returned for a companion. | ||
| 144 | - The web wiki tab renders, and reports no wiki when the directory is absent. | ||
| 145 | - A repository with no `.gitbay/wiki/` reports no wiki rather than erroring. | ||
| 146 | - Pushing to `<name>.wiki.git` is refused, since the route is gone. | ||
| 147 | - A repository may now be named `something.wiki`. | ||
| 148 | - `repo commit-file` writes a page on a repository that permits it, and is | ||
| 149 | refused on one requiring verified signatures. | ||
| 150 | |||
| 151 | ## Documentation | ||
| 152 | |||
| 153 | CLAUDE.md's "the repo's own documentation lives in the wiki" stops being true of | ||
| 154 | the storage and needs rewording. The wiki's Parity rows for wiki capabilities | ||
| 155 | change, and the "SSH only, by design" list does not mention wikis, so it needs | ||
| 156 | no edit. | ||
e2e/cipaths_test.go added +64
| @@ -0,0 +1,64 @@ | |||
| 1 | package e2e | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "os" | ||
| 5 | "path/filepath" | ||
| 6 | "strings" | ||
| 7 | "testing" | ||
| 8 | ) | ||
| 9 | |||
| 10 | // A job with paths only builds when the push touched something it names. | ||
| 11 | // A doc-only push queues nothing; a push touching the named path queues | ||
| 12 | // the job, same as before path filters existed. | ||
| 13 | func TestCIPaths(t *testing.T) { | ||
| 14 | inst := startInstance(t) | ||
| 15 | aliceKey := inst.newKey(t, "alice") | ||
| 16 | inst.admin(t, "admin", "user", "create", "alice", "--key", aliceKey+".pub") | ||
| 17 | |||
| 18 | if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "create", "alice/app"); code != 0 { | ||
| 19 | t.Fatalf("repo create: %s", errOut) | ||
| 20 | } | ||
| 21 | work := t.TempDir() | ||
| 22 | env := inst.gitEnv(aliceKey) | ||
| 23 | mustGit(t, work, env, "clone", inst.sshURL("alice/app"), "w") | ||
| 24 | dir := filepath.Join(work, "w") | ||
| 25 | os.MkdirAll(filepath.Join(dir, ".gitbay"), 0o755) | ||
| 26 | os.MkdirAll(filepath.Join(dir, "src"), 0o755) | ||
| 27 | os.MkdirAll(filepath.Join(dir, "docs"), 0o755) | ||
| 28 | os.WriteFile(filepath.Join(dir, ".gitbay", "ci.yml"), []byte( | ||
| 29 | "jobs:\n unit:\n paths:\n - src/**\n steps:\n - echo fine\n"), 0o644) | ||
| 30 | os.WriteFile(filepath.Join(dir, "src", "x.go"), []byte("package x\n"), 0o644) | ||
| 31 | os.WriteFile(filepath.Join(dir, "docs", "x.md"), []byte("# x\n"), 0o644) | ||
| 32 | mustGit(t, dir, env, "checkout", "-q", "-b", "main") | ||
| 33 | mustGit(t, dir, env, "add", ".") | ||
| 34 | mustGit(t, dir, env, "commit", "-q", "-m", "base") | ||
| 35 | mustGit(t, dir, env, "push", "-q", "origin", "main") | ||
| 36 | // A new branch has no diff base, so the filter fails open: the base | ||
| 37 | // push above already queued build 1. | ||
| 38 | before := strings.Count(inst.buildList(t, aliceKey), "\n") | ||
| 39 | if before != 1 { | ||
| 40 | t.Fatalf("base push did not queue exactly one build:\n%s", inst.buildList(t, aliceKey)) | ||
| 41 | } | ||
| 42 | |||
| 43 | // A push touching only docs does not match src/**: no build queued. | ||
| 44 | os.WriteFile(filepath.Join(dir, "docs", "x.md"), []byte("# x changed\n"), 0o644) | ||
| 45 | mustGit(t, dir, env, "add", ".") | ||
| 46 | mustGit(t, dir, env, "commit", "-q", "-m", "docs only") | ||
| 47 | mustGit(t, dir, env, "push", "-q", "origin", "main") | ||
| 48 | if out := inst.buildList(t, aliceKey); strings.Count(out, "\n") != before { | ||
| 49 | t.Fatalf("doc-only push queued a build:\n%s", out) | ||
| 50 | } | ||
| 51 | |||
| 52 | // A push touching src matches: a build is queued. | ||
| 53 | os.WriteFile(filepath.Join(dir, "src", "x.go"), []byte("package x\n\nvar y int\n"), 0o644) | ||
| 54 | mustGit(t, dir, env, "add", ".") | ||
| 55 | mustGit(t, dir, env, "commit", "-q", "-m", "src change") | ||
| 56 | mustGit(t, dir, env, "push", "-q", "origin", "main") | ||
| 57 | out := inst.buildList(t, aliceKey) | ||
| 58 | if strings.Count(out, "\n") != before+1 { | ||
| 59 | t.Fatalf("src push did not queue a build:\n%s", out) | ||
| 60 | } | ||
| 61 | if !strings.Contains(out, "unit\tpending") { | ||
| 62 | t.Fatalf("src push did not queue the unit job:\n%s", out) | ||
| 63 | } | ||
| 64 | } | ||
internal/ci/ci.go +30 −4
| @@ -25,6 +25,7 @@ const ( | |||
| 25 | maxJobs = 10 | 25 | maxJobs = 10 |
| 26 | maxSteps = 50 | 26 | maxSteps = 50 |
| 27 | maxStepSize = 4096 | 27 | maxStepSize = 4096 |
| 28 | maxPaths = 50 | ||
| 28 | ) | 29 | ) |
| 29 | 30 | ||
| 30 | var jobName = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,39}$`) | 31 | var jobName = regexp.MustCompile(`^[a-z0-9][a-z0-9_-]{0,39}$`) |
| @@ -34,6 +35,10 @@ type Job struct { | |||
| 34 | Steps []string | 35 | Steps []string |
| 35 | Schedule string // cron expression; scheduled jobs run on schedule, not on push | 36 | Schedule string // cron expression; scheduled jobs run on schedule, not on push |
| 36 | Tags string // tag glob (e.g. "v*"); tag jobs run on matching tag pushes only | 37 | Tags string // tag glob (e.g. "v*"); tag jobs run on matching tag pushes only |
| 38 | // Paths and PathsIgnore only gate a job queued on push; a scheduled | ||
| 39 | // or tag job ignores them. | ||
| 40 | Paths []string // globs; the job runs only when a changed file matches one | ||
| 41 | PathsIgnore []string // globs; the job is skipped when every changed file matches one | ||
| 37 | } | 42 | } |
| 38 | 43 | ||
| 39 | // Parse returns the jobs in name order, or an error describing the first | 44 | // Parse returns the jobs in name order, or an error describing the first |
| @@ -41,9 +46,11 @@ type Job struct { | |||
| 41 | func Parse(raw []byte) ([]Job, error) { | 46 | func Parse(raw []byte) ([]Job, error) { |
| 42 | var doc struct { | 47 | var doc struct { |
| 43 | Jobs map[string]struct { | 48 | Jobs map[string]struct { |
| 44 | Steps []string `yaml:"steps"` | 49 | Steps []string `yaml:"steps"` |
| 45 | Schedule string `yaml:"schedule"` | 50 | Schedule string `yaml:"schedule"` |
| 46 | Tags string `yaml:"tags"` | 51 | Tags string `yaml:"tags"` |
| 52 | Paths []string `yaml:"paths"` | ||
| 53 | PathsIgnore []string `yaml:"paths-ignore"` | ||
| 47 | } `yaml:"jobs"` | 54 | } `yaml:"jobs"` |
| 48 | } | 55 | } |
| 49 | if err := yaml.Unmarshal(raw, &doc); err != nil { | 56 | if err := yaml.Unmarshal(raw, &doc); err != nil { |
| @@ -84,7 +91,26 @@ func Parse(raw []byte) ([]Job, error) { | |||
| 84 | return nil, fmt.Errorf("job %q: schedule and tags are mutually exclusive", name) | 91 | return nil, fmt.Errorf("job %q: schedule and tags are mutually exclusive", name) |
| 85 | } | 92 | } |
| 86 | } | 93 | } |
| 87 | jobs = append(jobs, Job{Name: name, Steps: j.Steps, Schedule: j.Schedule, Tags: j.Tags}) | 94 | if len(j.Paths) > maxPaths { |
| 95 | return nil, fmt.Errorf("job %q has %d path patterns; max %d", name, len(j.Paths), maxPaths) | ||
| 96 | } | ||
| 97 | for _, p := range j.Paths { | ||
| 98 | if _, err := path.Match(p, "x"); err != nil { | ||
| 99 | return nil, fmt.Errorf("job %q: bad path pattern %q", name, p) | ||
| 100 | } | ||
| 101 | } | ||
| 102 | if len(j.PathsIgnore) > maxPaths { | ||
| 103 | return nil, fmt.Errorf("job %q has %d paths-ignore patterns; max %d", name, len(j.PathsIgnore), maxPaths) | ||
| 104 | } | ||
| 105 | for _, p := range j.PathsIgnore { | ||
| 106 | if _, err := path.Match(p, "x"); err != nil { | ||
| 107 | return nil, fmt.Errorf("job %q: bad paths-ignore pattern %q", name, p) | ||
| 108 | } | ||
| 109 | } | ||
| 110 | jobs = append(jobs, Job{ | ||
| 111 | Name: name, Steps: j.Steps, Schedule: j.Schedule, Tags: j.Tags, | ||
| 112 | Paths: j.Paths, PathsIgnore: j.PathsIgnore, | ||
| 113 | }) | ||
| 88 | } | 114 | } |
| 89 | sort.Slice(jobs, func(i, k int) bool { return jobs[i].Name < jobs[k].Name }) | 115 | sort.Slice(jobs, func(i, k int) bool { return jobs[i].Name < jobs[k].Name }) |
| 90 | return jobs, nil | 116 | return jobs, nil |
internal/ci/paths.go added +67
| @@ -0,0 +1,67 @@ | |||
| 1 | package ci | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "path" | ||
| 5 | "strings" | ||
| 6 | ) | ||
| 7 | |||
| 8 | // zeroSHA is git's null object id: the old side of a ref update that | ||
| 9 | // created the ref, carrying no diff base. | ||
| 10 | const zeroSHA = "0000000000000000000000000000000000000000" | ||
| 11 | |||
| 12 | // Match reports whether a changed file path matches a job's glob. | ||
| 13 | // A trailing /** matches the directory and everything below it at any | ||
| 14 | // depth; anything else goes to path.Match, whose * stops at a separator. | ||
| 15 | func Match(pattern, file string) bool { | ||
| 16 | if prefix, ok := strings.CutSuffix(pattern, "/**"); ok { | ||
| 17 | return file == prefix || strings.HasPrefix(file, prefix+"/") | ||
| 18 | } | ||
| 19 | ok, err := path.Match(pattern, file) | ||
| 20 | return err == nil && ok | ||
| 21 | } | ||
| 22 | |||
| 23 | // Selected reports whether a job's path filters admit a push that | ||
| 24 | // changed the given files. Call it only once the changed-file list is | ||
| 25 | // known; a caller that cannot compute one must run the job instead of | ||
| 26 | // calling this. | ||
| 27 | // | ||
| 28 | // The job runs when Paths is empty or at least one file matches one of | ||
| 29 | // its patterns, and is then held back only if PathsIgnore is non-empty | ||
| 30 | // and every file matches one of its patterns. | ||
| 31 | func Selected(j Job, files []string) bool { | ||
| 32 | if len(j.Paths) > 0 { | ||
| 33 | hit := false | ||
| 34 | for _, f := range files { | ||
| 35 | for _, p := range j.Paths { | ||
| 36 | if Match(p, f) { | ||
| 37 | hit = true | ||
| 38 | } | ||
| 39 | } | ||
| 40 | } | ||
| 41 | if !hit { | ||
| 42 | return false | ||
| 43 | } | ||
| 44 | } | ||
| 45 | if len(j.PathsIgnore) > 0 { | ||
| 46 | for _, f := range files { | ||
| 47 | ignored := false | ||
| 48 | for _, p := range j.PathsIgnore { | ||
| 49 | if Match(p, f) { | ||
| 50 | ignored = true | ||
| 51 | } | ||
| 52 | } | ||
| 53 | if !ignored { | ||
| 54 | return true | ||
| 55 | } | ||
| 56 | } | ||
| 57 | return false | ||
| 58 | } | ||
| 59 | return true | ||
| 60 | } | ||
| 61 | |||
| 62 | // HasDiffBase reports whether old names a commit a diff can start from. | ||
| 63 | // A branch's first push carries an empty or all-zero old sha, so there | ||
| 64 | // is nothing to compare the new commit against. | ||
| 65 | func HasDiffBase(old string) bool { | ||
| 66 | return old != "" && old != zeroSHA | ||
| 67 | } | ||
internal/ci/paths_test.go added +79
| @@ -0,0 +1,79 @@ | |||
| 1 | package ci | ||
| 2 | |||
| 3 | import "testing" | ||
| 4 | |||
| 5 | func TestMatch(t *testing.T) { | ||
| 6 | cases := []struct { | ||
| 7 | pattern, file string | ||
| 8 | want bool | ||
| 9 | }{ | ||
| 10 | {".gitbay/wiki/**", ".gitbay/wiki/Home.md", true}, | ||
| 11 | {".gitbay/wiki/**", ".gitbay/wiki/sub/Page.md", true}, // nested: the case a reader assumes works | ||
| 12 | {".gitbay/wiki/**", ".gitbay/wiki", true}, | ||
| 13 | {".gitbay/wiki/**", "other/Home.md", false}, | ||
| 14 | {"*.md", "README.md", true}, | ||
| 15 | {"*.md", "docs/README.md", false}, | ||
| 16 | {"docs/*.md", "docs/a.md", true}, | ||
| 17 | {"docs/*.md", "docs/sub/a.md", false}, | ||
| 18 | } | ||
| 19 | for _, c := range cases { | ||
| 20 | if got := Match(c.pattern, c.file); got != c.want { | ||
| 21 | t.Errorf("Match(%q, %q) = %v, want %v", c.pattern, c.file, got, c.want) | ||
| 22 | } | ||
| 23 | } | ||
| 24 | } | ||
| 25 | |||
| 26 | func TestSelected(t *testing.T) { | ||
| 27 | cases := []struct { | ||
| 28 | name string | ||
| 29 | job Job | ||
| 30 | files []string | ||
| 31 | want bool | ||
| 32 | }{ | ||
| 33 | {"paths hit", Job{Paths: []string{"src/**"}}, []string{"src/x.go"}, true}, | ||
| 34 | {"paths miss", Job{Paths: []string{"src/**"}}, []string{"docs/x.md"}, false}, | ||
| 35 | {"paths-ignore covers every file", Job{PathsIgnore: []string{"docs/**"}}, | ||
| 36 | []string{"docs/a.md", "docs/b.md"}, false}, | ||
| 37 | {"paths-ignore covers some files", Job{PathsIgnore: []string{"docs/**"}}, | ||
| 38 | []string{"docs/a.md", "src/x.go"}, true}, | ||
| 39 | {"neither key", Job{}, []string{"anything.txt"}, true}, | ||
| 40 | } | ||
| 41 | for _, c := range cases { | ||
| 42 | if got := Selected(c.job, c.files); got != c.want { | ||
| 43 | t.Errorf("%s: Selected() = %v, want %v", c.name, got, c.want) | ||
| 44 | } | ||
| 45 | } | ||
| 46 | } | ||
| 47 | |||
| 48 | func TestHasDiffBase(t *testing.T) { | ||
| 49 | cases := []struct { | ||
| 50 | old string | ||
| 51 | want bool | ||
| 52 | }{ | ||
| 53 | {"", false}, | ||
| 54 | {"0000000000000000000000000000000000000000", false}, | ||
| 55 | {"a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2", true}, | ||
| 56 | } | ||
| 57 | for _, c := range cases { | ||
| 58 | if got := HasDiffBase(c.old); got != c.want { | ||
| 59 | t.Errorf("HasDiffBase(%q) = %v, want %v", c.old, got, c.want) | ||
| 60 | } | ||
| 61 | } | ||
| 62 | } | ||
| 63 | |||
| 64 | func TestParsePaths(t *testing.T) { | ||
| 65 | raw := []byte("jobs:\n unit:\n steps:\n - echo hi\n paths:\n - src/**\n paths-ignore:\n - docs/**\n") | ||
| 66 | jobs, err := Parse(raw) | ||
| 67 | if err != nil { | ||
| 68 | t.Fatalf("Parse: %v", err) | ||
| 69 | } | ||
| 70 | if len(jobs) != 1 || len(jobs[0].Paths) != 1 || jobs[0].Paths[0] != "src/**" || | ||
| 71 | len(jobs[0].PathsIgnore) != 1 || jobs[0].PathsIgnore[0] != "docs/**" { | ||
| 72 | t.Fatalf("Parse did not round-trip paths: %+v", jobs) | ||
| 73 | } | ||
| 74 | |||
| 75 | bad := []byte("jobs:\n unit:\n steps:\n - echo hi\n paths:\n - \"[\"\n") | ||
| 76 | if _, err := Parse(bad); err == nil { | ||
| 77 | t.Fatal("Parse accepted a malformed path pattern") | ||
| 78 | } | ||
| 79 | } | ||
internal/control/build.go +31 −5
| @@ -500,12 +500,15 @@ func runRunnerDone(c *Ctx, args []string) int { | |||
| 500 | // | 500 | // |
| 501 | // Both paths that move a branch call this: post-receive for a push, and | 501 | // Both paths that move a branch call this: post-receive for a push, and |
| 502 | // the merge path for a merge, which updates the ref directly and so never | 502 | // the merge path for a merge, which updates the ref directly and so never |
| 503 | // reaches a hook. | 503 | // reaches a hook. old is the branch's sha before this update, the diff |
| 504 | // base a job's path filters run against; a new branch has no prior | ||
| 505 | // commit and sends old as empty or all zeros, either of which cannot be | ||
| 506 | // diffed and runs every job. | ||
| 504 | func QueueBranchBuilds( | 507 | func QueueBranchBuilds( |
| 505 | st *store.Store, root, siteURL string, | 508 | st *store.Store, root, siteURL string, |
| 506 | repo store.Repo, userID int64, branch, sha string, now time.Time, | 509 | repo store.Repo, userID int64, branch, old, sha string, now time.Time, |
| 507 | ) { | 510 | ) { |
| 508 | queueJobs(st, root, siteURL, repo, userID, branch, sha, now, true, branch == repo.DefaultBranch) | 511 | queueJobs(st, root, siteURL, repo, userID, branch, old, sha, now, true, branch == repo.DefaultBranch) |
| 509 | } | 512 | } |
| 510 | 513 | ||
| 511 | // QueueMRBuilds queues the push jobs for a merge request head fetched | 514 | // QueueMRBuilds queues the push jobs for a merge request head fetched |
| @@ -519,12 +522,14 @@ func QueueMRBuilds( | |||
| 519 | st *store.Store, root, siteURL string, | 522 | st *store.Store, root, siteURL string, |
| 520 | repo store.Repo, userID, n int64, sha string, | 523 | repo store.Repo, userID, n int64, sha string, |
| 521 | ) { | 524 | ) { |
| 522 | queueJobs(st, root, siteURL, repo, userID, mrHeadRef(n), sha, time.Now(), false, false) | 525 | // No old sha: fails open and runs every job, matching today's |
| 526 | // behaviour for a merge request head. | ||
| 527 | queueJobs(st, root, siteURL, repo, userID, mrHeadRef(n), "", sha, time.Now(), false, false) | ||
| 523 | } | 528 | } |
| 524 | 529 | ||
| 525 | func queueJobs( | 530 | func queueJobs( |
| 526 | st *store.Store, root, siteURL string, | 531 | st *store.Store, root, siteURL string, |
| 527 | repo store.Repo, userID int64, ref, sha string, now time.Time, | 532 | repo store.Repo, userID int64, ref, old, sha string, now time.Time, |
| 528 | trusted, syncSchedules bool, | 533 | trusted, syncSchedules bool, |
| 529 | ) { | 534 | ) { |
| 530 | dir := RepoDir(root, repo.OwnerName, repo.Name) | 535 | dir := RepoDir(root, repo.OwnerName, repo.Name) |
| @@ -546,6 +551,24 @@ func queueJobs( | |||
| 546 | if err != nil { | 551 | if err != nil { |
| 547 | built = nil | 552 | built = nil |
| 548 | } | 553 | } |
| 554 | // The changed-file list a job's path filters run against, computed | ||
| 555 | // once and only if some job actually declares one. When the diff | ||
| 556 | // base does not exist or the diff itself fails, filtered stays | ||
| 557 | // false and every job runs: a filter that cannot be evaluated must | ||
| 558 | // not silently skip CI. | ||
| 559 | filtered := false | ||
| 560 | var changed []string | ||
| 561 | for _, j := range jobs { | ||
| 562 | if len(j.Paths) == 0 && len(j.PathsIgnore) == 0 { | ||
| 563 | continue | ||
| 564 | } | ||
| 565 | if ci.HasDiffBase(old) { | ||
| 566 | if files, err := gitutil.DiffFiles(dir, old, sha); err == nil { | ||
| 567 | changed, filtered = files, true | ||
| 568 | } | ||
| 569 | } | ||
| 570 | break | ||
| 571 | } | ||
| 549 | var schedules []store.Schedule | 572 | var schedules []store.Schedule |
| 550 | for _, j := range jobs { | 573 | for _, j := range jobs { |
| 551 | // Tag jobs run on matching tag pushes only. | 574 | // Tag jobs run on matching tag pushes only. |
| @@ -566,6 +589,9 @@ func queueJobs( | |||
| 566 | } | 589 | } |
| 567 | continue | 590 | continue |
| 568 | } | 591 | } |
| 592 | if filtered && !ci.Selected(j, changed) { | ||
| 593 | continue | ||
| 594 | } | ||
| 569 | steps, _ := json.Marshal(j.Steps) | 595 | steps, _ := json.Marshal(j.Steps) |
| 570 | n, err := st.CreateBuild(repo.ID, j.Name, sha, ref, string(steps), trusted) | 596 | n, err := st.CreateBuild(repo.ID, j.Name, sha, ref, string(steps), trusted) |
| 571 | if err != nil { | 597 | if err != nil { |
internal/control/build_test.go added +91
| @@ -0,0 +1,91 @@ | |||
| 1 | package control | ||
| 2 | |||
| 3 | import ( | ||
| 4 | "os" | ||
| 5 | "os/exec" | ||
| 6 | "path/filepath" | ||
| 7 | "strings" | ||
| 8 | "testing" | ||
| 9 | "time" | ||
| 10 | |||
| 11 | "gitbay.org/gitbay/internal/store" | ||
| 12 | ) | ||
| 13 | |||
| 14 | // A DiffFiles failure must not turn into a silent skip: when the old sha | ||
| 15 | // on record cannot be diffed against, every job runs regardless of what | ||
| 16 | // it names in paths, the same as when there is no diff base at all. | ||
| 17 | func TestQueueBranchBuildsFailsOpenOnDiffFailure(t *testing.T) { | ||
| 18 | st, err := store.Open(":memory:") | ||
| 19 | if err != nil { | ||
| 20 | t.Fatal(err) | ||
| 21 | } | ||
| 22 | defer st.Close() | ||
| 23 | if err := st.MigrateUp(); err != nil { | ||
| 24 | t.Fatal(err) | ||
| 25 | } | ||
| 26 | uid, err := st.CreateUser("alice", false) | ||
| 27 | if err != nil { | ||
| 28 | t.Fatal(err) | ||
| 29 | } | ||
| 30 | repoID, err := st.CreateRepo("user", uid, "app", "public") | ||
| 31 | if err != nil { | ||
| 32 | t.Fatal(err) | ||
| 33 | } | ||
| 34 | repo, err := st.RepoByID(repoID) | ||
| 35 | if err != nil { | ||
| 36 | t.Fatal(err) | ||
| 37 | } | ||
| 38 | |||
| 39 | root := t.TempDir() | ||
| 40 | env := append(os.Environ(), | ||
| 41 | "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", | ||
| 42 | "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.test", | ||
| 43 | "GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.test") | ||
| 44 | git := func(dir string, args ...string) string { | ||
| 45 | t.Helper() | ||
| 46 | cmd := exec.Command("git", args...) | ||
| 47 | cmd.Dir = dir | ||
| 48 | cmd.Env = env | ||
| 49 | out, err := cmd.CombinedOutput() | ||
| 50 | if err != nil { | ||
| 51 | t.Fatalf("git %v: %v\n%s", args, err, out) | ||
| 52 | } | ||
| 53 | return string(out) | ||
| 54 | } | ||
| 55 | |||
| 56 | // A job whose paths would exclude a docs-only change, so the test | ||
| 57 | // proves something: without fail-open, the diff failure would leave | ||
| 58 | // the filter unevaluated and this build would never queue. | ||
| 59 | src := filepath.Join(root, "src") | ||
| 60 | os.MkdirAll(filepath.Join(src, ".gitbay"), 0o755) | ||
| 61 | os.MkdirAll(filepath.Join(src, "docs"), 0o755) | ||
| 62 | os.WriteFile(filepath.Join(src, ".gitbay", "ci.yml"), []byte( | ||
| 63 | "jobs:\n unit:\n paths:\n - src/**\n steps:\n - echo hi\n"), 0o644) | ||
| 64 | os.WriteFile(filepath.Join(src, "docs", "x.md"), []byte("# x\n"), 0o644) | ||
| 65 | git(root, "init", "-q", "-b", "main", "src") | ||
| 66 | git(src, "add", ".") | ||
| 67 | git(src, "commit", "-q", "-m", "base") | ||
| 68 | os.WriteFile(filepath.Join(src, "docs", "x.md"), []byte("# x changed\n"), 0o644) | ||
| 69 | git(src, "add", ".") | ||
| 70 | git(src, "commit", "-q", "-m", "docs only") | ||
| 71 | newSHA := strings.TrimSpace(git(src, "rev-parse", "HEAD")) | ||
| 72 | |||
| 73 | dir := RepoDir(root, repo.OwnerName, repo.Name) | ||
| 74 | os.MkdirAll(filepath.Dir(dir), 0o755) | ||
| 75 | git(root, "clone", "-q", "--bare", src, dir) | ||
| 76 | |||
| 77 | // Well-formed but names no object in this repo: git diff itself | ||
| 78 | // fails, rather than the empty/all-zero short-circuit HasDiffBase | ||
| 79 | // already covers. | ||
| 80 | old := strings.Repeat("1", 40) | ||
| 81 | |||
| 82 | QueueBranchBuilds(st, root, "https://x.test", repo, uid, "main", old, newSHA, time.Now()) | ||
| 83 | |||
| 84 | builds, err := st.ListBuilds(repo.ID, 10) | ||
| 85 | if err != nil || len(builds) != 1 { | ||
| 86 | t.Fatalf("builds after queue: %v %v", builds, err) | ||
| 87 | } | ||
| 88 | if builds[0].Job != "unit" || builds[0].Status != "pending" || builds[0].SHA != newSHA { | ||
| 89 | t.Fatalf("queued build wrong: %+v", builds[0]) | ||
| 90 | } | ||
| 91 | } | ||
internal/control/mr.go +1 −1
| @@ -1089,7 +1089,7 @@ func runMRMerge(c *Ctx, args []string) int { | |||
| 1089 | `{"ref":%q,"old":%q,"new":%q,"forced":false,"deleted":false}`, | 1089 | `{"ref":%q,"old":%q,"new":%q,"forced":false,"deleted":false}`, |
| 1090 | targetRef, targetSHA, newSHA)) | 1090 | targetRef, targetSHA, newSHA)) |
| 1091 | QueueBranchBuilds(c.Store, c.Cfg.Server.Root, c.Cfg.Server.SiteURL, | 1091 | QueueBranchBuilds(c.Store, c.Cfg.Server.Root, c.Cfg.Server.SiteURL, |
| 1092 | repo, c.User.ID, mr.TargetRef, newSHA, time.Now()) | 1092 | repo, c.User.ID, mr.TargetRef, targetSHA, newSHA, time.Now()) |
| 1093 | c.Store.MarkMirrorsDirty(repo.ID, "push") | 1093 | c.Store.MarkMirrorsDirty(repo.ID, "push") |
| 1094 | if parts, err := c.Store.MRParticipants(mr.ID); err == nil { | 1094 | if parts, err := c.Store.MRParticipants(mr.ID); err == nil { |
| 1095 | notify(c, parts, notice{repo: repo, kind: "mr", | 1095 | notify(c, parts, notice{repo: repo, kind: "mr", |
internal/hookd/hookd.go +3 −3
| @@ -214,7 +214,7 @@ func (s *Server) postReceive(req Request) { | |||
| 214 | } | 214 | } |
| 215 | // A branch push with a .gitbay/ci.yml queues one build per job. | 215 | // A branch push with a .gitbay/ci.yml queues one build per job. |
| 216 | if pushedRepoErr == nil && !u.IsDelete { | 216 | if pushedRepoErr == nil && !u.IsDelete { |
| 217 | s.queueBuilds(pushedRepo, req.UserID, branch, u.New) | 217 | s.queueBuilds(pushedRepo, req.UserID, branch, u.Old, u.New) |
| 218 | } | 218 | } |
| 219 | if u.IsForce { | 219 | if u.IsForce { |
| 220 | s.st.Audit(req.UserID, "push.forced", map[string]any{ | 220 | s.st.Audit(req.UserID, "push.forced", map[string]any{ |
| @@ -271,10 +271,10 @@ func (s *Server) postReceive(req Request) { | |||
| 271 | 271 | ||
| 272 | // queueBuilds queues the push jobs for a branch update. The work is | 272 | // queueBuilds queues the push jobs for a branch update. The work is |
| 273 | // shared with the merge path, which moves a ref without reaching a hook. | 273 | // shared with the merge path, which moves a ref without reaching a hook. |
| 274 | func (s *Server) queueBuilds(repo store.Repo, userID int64, branch, sha string) { | 274 | func (s *Server) queueBuilds(repo store.Repo, userID int64, branch, old, sha string) { |
| 275 | control.QueueBranchBuilds( | 275 | control.QueueBranchBuilds( |
| 276 | s.st, s.cfg.Server.Root, s.cfg.Server.SiteURL, | 276 | s.st, s.cfg.Server.Root, s.cfg.Server.SiteURL, |
| 277 | repo, userID, branch, sha, time.Now()) | 277 | repo, userID, branch, old, sha, time.Now()) |
| 278 | } | 278 | } |
| 279 | 279 | ||
| 280 | // queueTagBuilds runs the jobs whose tag pattern matches a pushed tag. | 280 | // queueTagBuilds runs the jobs whose tag pattern matches a pushed tag. |