Commit 39dad8f75c
Verified · cmc ci/build: success ci/quality: success ci/test: success
Layout: unified · split
.gitbay/ci.yml +5 −4
| @@ -21,12 +21,13 @@ jobs: | ||
| 21 | 21 | - .gitbay/wiki/** |
| 22 | 22 | # Static analysis through Qlty (.qlty/qlty.toml). Secrets and gofmt fail |
| 23 | 23 | # the build. The full report, including radarlint-go's Sonar rules, is |
| 24 | # printed without failing it. The CLI is pinned, and unpacked with Python | |
| 25 | # because this image has no xz. | |
| 24 | # printed without failing it. The CLI is pinned and checked against the | |
| 25 | # sha256 Qlty publishes beside it, and unpacked with Python because this | |
| 26 | # image has no xz. | |
| 26 | 27 | quality: |
| 27 | 28 | image: localhost/gitbay-ci:3 |
| 28 | 29 | steps: |
| 29 | - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')" | |
| 30 | - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && echo "fa07eb51fbb32a0854113e81ce2d957b73f1c027c58b478d32c233b6aba7a977 /tmp/qlty.tar.xz" | sha256sum -c - && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')" | |
| 30 | 31 | - QLTY_TELEMETRY=off /tmp/qlty-x86_64-unknown-linux-gnu/qlty check --all --no-progress --filter trufflehog,gofmt |
| 31 | 32 | - QLTY_TELEMETRY=off /tmp/qlty-x86_64-unknown-linux-gnu/qlty check --all --no-progress --no-fail |
| 32 | 33 | paths-ignore: |
| @@ -51,5 +52,5 @@ jobs: | ||
| 51 | 52 | - go run golang.org/x/vuln/cmd/govulncheck@latest ./... |
| 52 | 53 | # osv-scanner also reports advisories in dependencies the code does not |
| 53 | 54 | # call, which govulncheck skips. |
| 54 | - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')" | |
| 55 | - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && echo "fa07eb51fbb32a0854113e81ce2d957b73f1c027c58b478d32c233b6aba7a977 /tmp/qlty.tar.xz" | sha256sum -c - && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')" | |
| 55 | 56 | - QLTY_TELEMETRY=off /tmp/qlty-x86_64-unknown-linux-gnu/qlty check --all --no-progress --filter osv-scanner |
.gitbay/wiki/Admin.org +30 −1
| @@ -454,6 +454,32 @@ push=. | ||
| 454 | 454 | the worker logs an error at start and syncs no mirror, recording the |
| 455 | 455 | reason on each. Sync ignores the system and global gitconfig. |
| 456 | 456 | |
| 457 | ** [security] | |
| 458 | The instance's code scan, which a repository admin turns on with | |
| 459 | =repo security enable=. Off until =image= is set. | |
| 460 | ||
| 461 | - =image= ("") — the image the scan runs in, provisioned on every | |
| 462 | runner host that serves a scanned repository. Build it from | |
| 463 | =deploy/Containerfile.security= (Opengrep and its rules, both pinned | |
| 464 | by sha256); the file has the commands. | |
| 465 | - =schedule= ("0 4 * * *") — when each enabled repository's default | |
| 466 | branch is scanned. =repo security run= queues one now. | |
| 467 | - =steps= — shell steps that write SARIF into =$GITBAY_RESULTS=. The | |
| 468 | default runs Opengrep with the rules in the image at | |
| 469 | =/opt/opengrep-rules=. | |
| 470 | ||
| 471 | The scan is a build of the reserved job =gitbay-security= on a runner | |
| 472 | attached to the repository. It gets none of the repository's secrets | |
| 473 | and sets no commit status. When its steps succeed the runner uploads | |
| 474 | the SARIF (=runner results=, gitbay-runner from this release on) and | |
| 475 | the findings replace the repository's last set; =gitbay-bot= keeps a | |
| 476 | "Code scanning findings" issue, opened, rewritten and closed the way | |
| 477 | the dependency issues are. An older runner runs the steps and uploads | |
| 478 | nothing, so the issue stays as it was. | |
| 479 | ||
| 480 | The bundled rules are =opengrep/opengrep-rules=, LGPL-2.1 with the | |
| 481 | Commons Clause: an instance that sells hosting should supply its own. | |
| 482 | ||
| 457 | 483 | ** [go_import] |
| 458 | 484 | Vanity Go module paths, one per line: ="host/module" = "owner/repo"=. |
| 459 | 485 | Requests with =?go-get=1= at or under the module path answer with the |
| @@ -1337,7 +1363,10 @@ trusts and refuses to do. Operational checklist: | ||
| 1337 | 1363 | tagging a release. CI's own =vuln= job runs =govulncheck= nightly |
| 1338 | 1364 | against main rather than per push, because =@latest= scans today's |
| 1339 | 1365 | advisory database and an advisory lands without anyone pushing; |
| 1340 | =build trigger krz/gitbay vuln= runs it on demand. | |
| 1366 | =build trigger krz/gitbay vuln= runs it on demand. Repositories can | |
| 1367 | also opt into the dependency advisory check (=repo deps enable=), | |
| 1368 | the push-time secret scan (on by default as =warn=), and the | |
| 1369 | scheduled code scan (=repo security enable=, see [security]). | |
| 1341 | 1370 | - *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options: |
| 1342 | 1371 | DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no |
| 1343 | 1372 | configuration needed. |
.gitbay/wiki/Architecture/02-Components.org +2
| @@ -38,6 +38,8 @@ the hidden =hook= used by git (=cmd/gitbayd/main.go=, | ||
| 38 | 38 | | =internal/mailin=, =internal/imapc=, =internal/mailreply= | Reply by mail: the IMAP client, the reply token, and the processor that posts a reply through =issue comment= / =mr comment=. | |
| 39 | 39 | | =internal/deps= | Dependency manifest and lockfile parsing; registry and OSV checks (opt-in per repository). | |
| 40 | 40 | | =internal/leaks= | Credential rules and the blob scan pre-receive and =repo secret-scan= run. | |
| 41 | | =internal/codescan= | SARIF from the code scan, and the findings issue's body. | | |
| 42 | | =internal/botissue= | The issues gitbay-bot keeps: open on first report, rewrite on change, close when empty. | | |
| 41 | 43 | | =internal/config= | Configuration load and validation. | |
| 42 | 44 | | =internal/web= | Embedded templates, stylesheet and fonts. | |
| 43 | 45 | | =internal/protocol= | Exit codes, JSON envelope, argv tokenizer. | |
.gitbay/wiki/Architecture/06-Data-and-Cryptography.org +2 −1
| @@ -19,7 +19,8 @@ content (as confidential as the repository), *O* operational. | ||
| 19 | 19 | | Notifications | =notifications= (mail queue), =inbox=, =push_devices= (APNs token), =push_queue=, =mail_replies= (Message-IDs of posted replies) | P | device tokens sealed; looked up by SHA-256 | |
| 20 | 20 | | Signatures | =commit_signatures=, =settings.key_epoch= | O | verification cache | |
| 21 | 21 | | Audit and feed | =audit_log=, =events= | O, P | actor ids, pruned argv, fingerprints and IPs in some audit rows, a hash chain (=prev_hash=, =hash=) | |
| 22 | | Dependencies | =dep_checks=, =dep_reports=, =dep_advisories= | O | | | |
| 22 | | Dependencies | =dep_checks=, =dep_reports=, =dep_advisories= | O | | |
| 23 | | Code scanning | =security_scans=, =security_findings= | O | | | |
| 23 | 24 | |
| 24 | 25 | Outside the database: |
| 25 | 26 | |
.gitbay/wiki/Parity.org +3
| @@ -245,6 +245,9 @@ there shows the app's rendering rather than the one the web page shows. | ||
| 245 | 245 | | job image (ci.yml) | yes | n/a | n/a | |
| 246 | 246 | | secret scan mode, allowlist | yes | yes | yes | |
| 247 | 247 | | secret scan of the history | yes | no | yes | |
| 248 | | code scanning on/off | yes | yes | yes | | |
| 249 | | code scanning status | yes | yes | yes | | |
| 250 | | code scan now | yes | no | yes | | |
| 248 | 251 | | dependency checks on/off | yes | yes | yes | |
| 249 | 252 | | dependency status | yes | yes | yes | |
| 250 | 253 | | delete, transfer | yes | yes | yes | |
.gitbay/wiki/Users.org +2
| @@ -241,6 +241,8 @@ gitbay repo settings require-signed you/project on # every commit must verify | ||
| 241 | 241 | gitbay repo settings secret-scan you/project block # off | warn (default) | block |
| 242 | 242 | gitbay repo settings secret-scan-allow you/project 'testdata/**' # paths whose findings are ignored |
| 243 | 243 | gitbay repo secret-scan you/project # scan the whole history |
| 244 | gitbay repo security enable you/project # scheduled code scan, where the instance offers one | |
| 245 | gitbay repo security status you/project # its findings, also kept in an issue | |
| 244 | 246 | gitbay repo settings git-daemon you/project on # expose over git:// |
| 245 | 247 | gitbay repo topics add you/project cli forge # free-form tags, shown on the web |
| 246 | 248 | gitbay repo search forge # find repos by name/description/topic |
cmd/gitbay-runner/main.go +26 −1
| @@ -49,6 +49,9 @@ type job struct { | ||
| 49 | 49 | // over loopback takes its port for its builds (#260). |
| 50 | 50 | SSH string `json:"ssh"` |
| 51 | 51 | Secrets map[string]string `json:"secrets"` |
| 52 | // Results asks for the SARIF the steps leave in $GITBAY_RESULTS to be | |
| 53 | // uploaded once they succeed. Only the instance's code scan sets it. | |
| 54 | Results bool `json:"results"` | |
| 52 | 55 | } |
| 53 | 56 | |
| 54 | 57 | type runner struct { |
| @@ -450,7 +453,29 @@ func (r *runner) run(j job) *failure { | ||
| 450 | 453 | } |
| 451 | 454 | |
| 452 | 455 | env := stepEnv(j, home, r.buildSSH(j.SSH)) |
| 453 | return r.runSteps(j, dir, env, sink, deadline, runStep) | |
| 456 | if !j.Results { | |
| 457 | return r.runSteps(j, dir, env, sink, deadline, runStep) | |
| 458 | } | |
| 459 | // Cleared first: the checkout is the repository's, and a results | |
| 460 | // directory committed to it would otherwise be uploaded as the scan's. | |
| 461 | results := filepath.Join(dir, resultsDir) | |
| 462 | if err := os.RemoveAll(results); err != nil { | |
| 463 | fmt.Fprintf(sink, "preparing %s: %v\n", resultsDir, err) | |
| 464 | return &failure{Reason: "preparing the results directory failed"} | |
| 465 | } | |
| 466 | inBuild := results | |
| 467 | if r.isolation == isolationPodman { | |
| 468 | inBuild = "/workspace/" + resultsDir | |
| 469 | } | |
| 470 | if f := r.runSteps(j, dir, append(env, "GITBAY_RESULTS="+inBuild), sink, deadline, runStep); f != nil { | |
| 471 | return f | |
| 472 | } | |
| 473 | if err := r.uploadResults(j.ID, results); err != nil { | |
| 474 | fmt.Fprintf(sink, "uploading results: %v\n", err) | |
| 475 | return &failure{Reason: "uploading results failed"} | |
| 476 | } | |
| 477 | fmt.Fprintln(sink, "results uploaded") | |
| 478 | return nil | |
| 454 | 479 | } |
| 455 | 480 | |
| 456 | 481 | // buildHome is a build's HOME and what to do with it when the build ends. |
cmd/gitbay-runner/results.go added +65
| @@ -0,0 +1,65 @@ | ||
| 1 | package main | |
| 2 | ||
| 3 | import ( | |
| 4 | "bytes" | |
| 5 | "encoding/json" | |
| 6 | "errors" | |
| 7 | "fmt" | |
| 8 | "os" | |
| 9 | "path/filepath" | |
| 10 | "sort" | |
| 11 | ) | |
| 12 | ||
| 13 | // resultsDir is where a code scan's steps write SARIF, relative to the | |
| 14 | // workspace. | |
| 15 | const resultsDir = ".gitbay-results" | |
| 16 | ||
| 17 | // maxResults matches the server's upload limit (codescan.MaxUpload). | |
| 18 | const maxResults = 8 << 20 | |
| 19 | ||
| 20 | // mergeResults reads every .sarif file in dir and joins their runs into | |
| 21 | // one document, the shape runner results takes. Steps that wrote nothing | |
| 22 | // is an error: a scan that produced no results did not run. | |
| 23 | func mergeResults(dir string) ([]byte, error) { | |
| 24 | names, err := filepath.Glob(filepath.Join(dir, "*.sarif")) | |
| 25 | if err != nil { | |
| 26 | return nil, err | |
| 27 | } | |
| 28 | if len(names) == 0 { | |
| 29 | return nil, fmt.Errorf("no .sarif files in %s", resultsDir) | |
| 30 | } | |
| 31 | sort.Strings(names) | |
| 32 | var runs []json.RawMessage | |
| 33 | total := 0 | |
| 34 | for _, name := range names { | |
| 35 | raw, err := os.ReadFile(name) | |
| 36 | if err != nil { | |
| 37 | return nil, err | |
| 38 | } | |
| 39 | if total += len(raw); total > maxResults { | |
| 40 | return nil, fmt.Errorf("results are over %d bytes", maxResults) | |
| 41 | } | |
| 42 | var doc struct { | |
| 43 | Runs []json.RawMessage `json:"runs"` | |
| 44 | } | |
| 45 | if err := json.Unmarshal(raw, &doc); err != nil { | |
| 46 | return nil, fmt.Errorf("%s: %w", filepath.Base(name), err) | |
| 47 | } | |
| 48 | if doc.Runs == nil { | |
| 49 | return nil, errors.New(filepath.Base(name) + ": no runs") | |
| 50 | } | |
| 51 | runs = append(runs, doc.Runs...) | |
| 52 | } | |
| 53 | return json.Marshal(map[string]any{"version": "2.1.0", "runs": runs}) | |
| 54 | } | |
| 55 | ||
| 56 | // uploadResults sends the merged SARIF, retrying as reportDone does. | |
| 57 | func (r *runner) uploadResults(id int64, dir string) error { | |
| 58 | doc, err := mergeResults(dir) | |
| 59 | if err != nil { | |
| 60 | return err | |
| 61 | } | |
| 62 | return reportWithRetry(func() (string, error) { | |
| 63 | return r.ssh(bytes.NewReader(doc), "runner", "results", fmt.Sprint(id)) | |
| 64 | }, id, retryDelays) | |
| 65 | } | |
cmd/gitbay-runner/results_test.go added +36
| @@ -0,0 +1,36 @@ | ||
| 1 | package main | |
| 2 | ||
| 3 | import ( | |
| 4 | "encoding/json" | |
| 5 | "os" | |
| 6 | "path/filepath" | |
| 7 | "testing" | |
| 8 | ) | |
| 9 | ||
| 10 | func TestMergeResults(t *testing.T) { | |
| 11 | dir := t.TempDir() | |
| 12 | if _, err := mergeResults(dir); err == nil { | |
| 13 | t.Error("an empty results directory merged") | |
| 14 | } | |
| 15 | os.WriteFile(filepath.Join(dir, "a.sarif"), []byte(`{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"a"}}}]}`), 0o644) | |
| 16 | os.WriteFile(filepath.Join(dir, "b.sarif"), []byte(`{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"b"}}}]}`), 0o644) | |
| 17 | os.WriteFile(filepath.Join(dir, "notes.txt"), []byte(`ignored`), 0o644) | |
| 18 | raw, err := mergeResults(dir) | |
| 19 | if err != nil { | |
| 20 | t.Fatal(err) | |
| 21 | } | |
| 22 | var doc struct { | |
| 23 | Runs []struct { | |
| 24 | Tool struct { | |
| 25 | Driver struct{ Name string } `json:"driver"` | |
| 26 | } `json:"tool"` | |
| 27 | } `json:"runs"` | |
| 28 | } | |
| 29 | if err := json.Unmarshal(raw, &doc); err != nil || len(doc.Runs) != 2 || doc.Runs[1].Tool.Driver.Name != "b" { | |
| 30 | t.Fatalf("merged %s: %v", raw, err) | |
| 31 | } | |
| 32 | os.WriteFile(filepath.Join(dir, "c.sarif"), []byte(`not json`), 0o644) | |
| 33 | if _, err := mergeResults(dir); err == nil { | |
| 34 | t.Error("a broken file merged") | |
| 35 | } | |
| 36 | } | |
cmd/gitbay/coverage_test.go +1
| @@ -16,6 +16,7 @@ var notInCLI = map[string]string{ | ||
| 16 | 16 | "runner next": "the CI runner's wire protocol, not for humans", |
| 17 | 17 | "runner done": "the CI runner's wire protocol, not for humans", |
| 18 | 18 | "runner log": "the CI runner's wire protocol, not for humans", |
| 19 | "runner results": "the CI runner's wire protocol, not for humans", | |
| 19 | 20 | "account import-bundle": "the wire format behind `gitbay migrate`", |
| 20 | 21 | } |
| 21 | 22 | |
cmd/gitbay/main.go +6
| @@ -597,6 +597,12 @@ func repoCmd() *cobra.Command { | ||
| 597 | 597 | pass("sync", passOpts{server: []string{"repo", "mirror", "sync"}, needsRepo: true}), |
| 598 | 598 | ), |
| 599 | 599 | pass("secret-scan", passOpts{server: []string{"repo", "secret-scan"}, needsRepo: true}), |
| 600 | group("security", "the instance's scheduled code scan", | |
| 601 | pass("enable", passOpts{server: []string{"repo", "security", "enable"}, needsRepo: true}), | |
| 602 | pass("disable", passOpts{server: []string{"repo", "security", "disable"}, needsRepo: true}), | |
| 603 | pass("run", passOpts{server: []string{"repo", "security", "run"}, needsRepo: true}), | |
| 604 | pass("status", passOpts{server: []string{"repo", "security", "status"}, needsRepo: true}), | |
| 605 | ), | |
| 600 | 606 | group("deps", "check dependencies against upstream registries and OSV", |
| 601 | 607 | pass("enable", passOpts{server: []string{"repo", "deps", "enable"}, needsRepo: true}), |
| 602 | 608 | pass("disable", passOpts{server: []string{"repo", "deps", "disable"}, needsRepo: true}), |
cmd/gitbay/summaries_gen.go +5
| @@ -197,6 +197,10 @@ var summaries = map[string]string{ | ||
| 197 | 197 | "repo secret remove": "remove a build secret", |
| 198 | 198 | "repo secret set": "set a build secret", |
| 199 | 199 | "repo secret-scan": "scan the whole history for credentials", |
| 200 | "repo security disable": "stop code scanning", | |
| 201 | "repo security enable": "scan the default branch on the instance's code scanning schedule", | |
| 202 | "repo security run": "queue a code scan of the default branch now", | |
| 203 | "repo security status": "show code scanning state and the last scan's findings", | |
| 200 | 204 | "repo settings default-branch": "set the default branch", |
| 201 | 205 | "repo settings description": "set the repository description", |
| 202 | 206 | "repo settings git-daemon": "expose over git://", |
| @@ -231,6 +235,7 @@ var summaries = map[string]string{ | ||
| 231 | 235 | "runner done": "finish a build", |
| 232 | 236 | "runner log": "append a build's log from stdin", |
| 233 | 237 | "runner next": "claim the oldest pending build this key may run (runner protocol)", |
| 238 | "runner results": "upload a code scan's SARIF from stdin", | |
| 234 | 239 | "search": "find repositories, issues and merge requests across the instance", |
| 235 | 240 | "snippet create": "create a snippet from one file on stdin", |
| 236 | 241 | "snippet delete": "delete a snippet and its files", |
cmd/gitbayd/main.go +1 −1
| @@ -244,7 +244,7 @@ func serveCmd() *cobra.Command { | ||
| 244 | 244 | } |
| 245 | 245 | go purgeDeletions(whCtx, st, cfg) |
| 246 | 246 | go sweep(whCtx, st, cfg) |
| 247 | go (&ci.Scheduler{St: st, SiteURL: cfg.Server.SiteURL, | |
| 247 | go (&ci.Scheduler{St: st, SiteURL: cfg.Server.SiteURL, Security: cfg.Security, | |
| 248 | 248 | RepoDir: func(owner, name string) string { |
| 249 | 249 | return control.RepoDir(cfg.Server.Root, owner, name) |
| 250 | 250 | }}).Run(whCtx) |
deploy/Containerfile.security added +56
| @@ -0,0 +1,56 @@ | ||
| 1 | # The image the instance's code scan runs in ([security] image in | |
| 2 | # /etc/gitbay/config.toml, #340). Build it on each runner host that | |
| 3 | # serves a repository with `repo security enable`, the same way as | |
| 4 | # Containerfile.ci: | |
| 5 | # | |
| 6 | # scp -P 2222 deploy/Containerfile.security root@gitbay.org:/var/lib/gitbay-runner/gitbay-security.Containerfile | |
| 7 | # ssh -p 2222 root@gitbay.org 'chown ci-runner /var/lib/gitbay-runner/gitbay-security.Containerfile \ | |
| 8 | # && su - ci-runner -s /bin/sh -c "podman build -t localhost/gitbay-security:1 -f gitbay-security.Containerfile ." \ | |
| 9 | # && rm /var/lib/gitbay-runner/gitbay-security.Containerfile' | |
| 10 | # | |
| 11 | # Everything fetched is pinned to a version and checked against a sha256 | |
| 12 | # taken when it was pinned, so a rebuild cannot pick up a different | |
| 13 | # binary or rule set. Bump both together and the tag with them. | |
| 14 | # | |
| 15 | # Opengrep v1.31.0 (2026-10-15), LGPL-2.1. The rules are | |
| 16 | # opengrep/opengrep-rules at its last commit, a fork of semgrep-rules | |
| 17 | # from before that project's licence change: LGPL-2.1 with the Commons | |
| 18 | # Clause, which forbids selling a service whose value is the rules. An | |
| 19 | # instance that charges for hosting should mount its own rules at | |
| 20 | # /opt/opengrep-rules instead. | |
| 21 | FROM docker.io/library/debian:trixie-slim | |
| 22 | ||
| 23 | LABEL org.gitbay.keep=true | |
| 24 | ||
| 25 | ARG OPENGREP_VERSION=v1.31.0 | |
| 26 | ARG OPENGREP_SHA256=dbbe3c2058dbe600c449a8ba22ea79a1643a6660658400090d0323a8d0d31147 | |
| 27 | ARG RULES_COMMIT=f1d2b562b414783763fd02a6ed2736eaed622efa | |
| 28 | ARG RULES_SHA256=9a5f1cd5c625418cc1c776120123e2d4371df9bb66e099426b17c3488e13619d | |
| 29 | ||
| 30 | RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ | |
| 31 | ca-certificates \ | |
| 32 | curl \ | |
| 33 | git \ | |
| 34 | && rm -rf /var/lib/apt/lists/* | |
| 35 | ||
| 36 | RUN curl -fsSL -o /usr/local/bin/opengrep \ | |
| 37 | "https://github.com/opengrep/opengrep/releases/download/${OPENGREP_VERSION}/opengrep_manylinux_x86" \ | |
| 38 | && echo "${OPENGREP_SHA256} /usr/local/bin/opengrep" | sha256sum -c - \ | |
| 39 | && chmod 0755 /usr/local/bin/opengrep | |
| 40 | ||
| 41 | # Only the language directories, and without the rules' own test inputs | |
| 42 | # (*.test.*, *.fixed.*), which would otherwise be read as rules or match | |
| 43 | # as findings. | |
| 44 | RUN curl -fsSL -o /tmp/rules.tar.gz \ | |
| 45 | "https://codeload.github.com/opengrep/opengrep-rules/tar.gz/${RULES_COMMIT}" \ | |
| 46 | && echo "${RULES_SHA256} /tmp/rules.tar.gz" | sha256sum -c - \ | |
| 47 | && mkdir -p /opt/opengrep-rules /tmp/rules \ | |
| 48 | && tar -xzf /tmp/rules.tar.gz -C /tmp/rules --strip-components=1 \ | |
| 49 | && for d in bash c dockerfile generic go java javascript kotlin php python ruby rust swift terraform typescript yaml; do \ | |
| 50 | cp -r "/tmp/rules/$d" /opt/opengrep-rules/; \ | |
| 51 | done \ | |
| 52 | && find /opt/opengrep-rules -type f \( -name '*.test.*' -o -name '*.fixed.*' \) -delete \ | |
| 53 | && find /opt/opengrep-rules -type f ! -name '*.yaml' ! -name '*.yml' -delete \ | |
| 54 | && rm -rf /tmp/rules /tmp/rules.tar.gz | |
| 55 | ||
| 56 | WORKDIR /workspace | |
e2e/codescan_test.go added +65
| @@ -0,0 +1,65 @@ | ||
| 1 | package e2e | |
| 2 | ||
| 3 | import ( | |
| 4 | "os" | |
| 5 | "path/filepath" | |
| 6 | "strings" | |
| 7 | "testing" | |
| 8 | ) | |
| 9 | ||
| 10 | // The instance's code scan end to end: enable, queue, a real runner runs | |
| 11 | // the configured steps and uploads their SARIF, and the findings land in | |
| 12 | // an issue. A results directory committed to the repository is cleared | |
| 13 | // before the steps run, so it is never uploaded as the scan's (#340). | |
| 14 | func TestSecurityScan(t *testing.T) { | |
| 15 | t.Parallel() | |
| 16 | inst := startInstanceWith(t, `[security] | |
| 17 | image = "localhost/scan:1" | |
| 18 | steps = ['mkdir -p "$GITBAY_RESULTS"', 'cp scan.sarif "$GITBAY_RESULTS/scan.sarif"'] | |
| 19 | `) | |
| 20 | inst.runner = buildRunner(t) | |
| 21 | aliceKey := inst.newKey(t, "alice") | |
| 22 | inst.admin(t, "admin", "user", "create", "alice", "--key", aliceKey+".pub") | |
| 23 | runnerKey := inst.newKey(t, "ci") | |
| 24 | inst.admin(t, "admin", "user", "create", "ci", "--key", runnerKey+".pub", "--admin") | |
| 25 | if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "create", "alice/app"); code != 0 { | |
| 26 | t.Fatalf("repo create: %s", errOut) | |
| 27 | } | |
| 28 | work := t.TempDir() | |
| 29 | env := inst.gitEnv(aliceKey) | |
| 30 | mustGit(t, work, env, "clone", "-q", inst.sshURL("alice/app"), "w") | |
| 31 | dir := filepath.Join(work, "w") | |
| 32 | os.MkdirAll(filepath.Join(dir, ".gitbay-results"), 0o755) | |
| 33 | os.WriteFile(filepath.Join(dir, "scan.sarif"), []byte(`{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"opengrep"}}, | |
| 34 | "results":[{"ruleId":"python.exec-used","level":"error","message":{"text":"exec on input"}, | |
| 35 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"app.py"},"region":{"startLine":7}}}]}]}]}`), 0o644) | |
| 36 | os.WriteFile(filepath.Join(dir, ".gitbay-results", "planted.sarif"), []byte(`{"runs":[{"tool":{"driver":{"name":"planted"}}, | |
| 37 | "results":[{"ruleId":"planted.rule","message":{"text":"planted"}}]}]}`), 0o644) | |
| 38 | mustGit(t, dir, env, "checkout", "-q", "-b", "main") | |
| 39 | mustGit(t, dir, env, "add", ".") | |
| 40 | mustGit(t, dir, env, "commit", "-q", "-m", "base") | |
| 41 | mustGit(t, dir, env, "push", "-q", "origin", "main") | |
| 42 | ||
| 43 | if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "security", "enable", "alice/app"); code != 0 { | |
| 44 | t.Fatalf("security enable: %s", errOut) | |
| 45 | } | |
| 46 | if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "security", "run", "alice/app"); code != 0 { | |
| 47 | t.Fatalf("security run: %s", errOut) | |
| 48 | } | |
| 49 | if out := inst.runnerOnce(t, runnerKey); !strings.Contains(out, "success") { | |
| 50 | t.Fatalf("runner:\n%s", out) | |
| 51 | } | |
| 52 | out, errOut, code := inst.ssh(t, aliceKey, "", "repo", "security", "status", "alice/app", "--json") | |
| 53 | if code != 0 || !strings.Contains(out, `"rule":"python.exec-used"`) || strings.Contains(out, "planted") { | |
| 54 | t.Fatalf("status: %d %s\n%s", code, errOut, out) | |
| 55 | } | |
| 56 | out, _, _ = inst.ssh(t, aliceKey, "", "issue", "list", "alice/app") | |
| 57 | if !strings.Contains(out, "Code scanning findings") { | |
| 58 | t.Fatalf("no findings issue:\n%s", out) | |
| 59 | } | |
| 60 | // The scan sets no commit status. | |
| 61 | sha := strings.TrimSpace(mustGit(t, dir, env, "rev-parse", "HEAD")) | |
| 62 | if out, _, _ := inst.ssh(t, aliceKey, "", "status", "list", "alice/app", sha); strings.Contains(out, "gitbay-security") { | |
| 63 | t.Fatalf("scan set a status:\n%s", out) | |
| 64 | } | |
| 65 | } | |
e2e/readonly_test.go +1
| @@ -143,6 +143,7 @@ func TestReadOnlyCommandsWriteNothing(t *testing.T) { | ||
| 143 | 143 | "repo domain list": {"alice/app"}, |
| 144 | 144 | "repo deps status": {"alice/app"}, |
| 145 | 145 | "repo secret-scan": {"alice/app"}, |
| 146 | "repo security status": {"alice/app"}, | |
| 146 | 147 | "status list": {"alice/app", sha}, |
| 147 | 148 | "issue list": {"alice/app"}, |
| 148 | 149 | "issue show": {"alice/app", "1"}, |
internal/botissue/botissue.go added +121
| @@ -0,0 +1,121 @@ | ||
| 1 | // Package botissue keeps the issues gitbay-bot maintains for a repository: | |
| 2 | // dependency updates, dependency advisories, code scanning findings. Each | |
| 3 | // is one issue per repository, opened when something is first reported, | |
| 4 | // rewritten when the set changes, and closed when nothing is left. | |
| 5 | package botissue | |
| 6 | ||
| 7 | import ( | |
| 8 | "fmt" | |
| 9 | "strings" | |
| 10 | ||
| 11 | "gitbay.org/gitbay/internal/config" | |
| 12 | "gitbay.org/gitbay/internal/store" | |
| 13 | ) | |
| 14 | ||
| 15 | // Tracked is one issue and the set it reports. | |
| 16 | type Tracked struct { | |
| 17 | Title string | |
| 18 | Number int64 // the issue last opened, 0 for none | |
| 19 | Record func(number int64) error | |
| 20 | Empty bool // nothing to report | |
| 21 | Changed bool // the set differs from the last report | |
| 22 | Save func() error | |
| 23 | Body func() string | |
| 24 | } | |
| 25 | ||
| 26 | // Maintain brings the issue in line with the set. | |
| 27 | func Maintain(st *store.Store, cfg config.Config, repo store.Repo, t Tracked) error { | |
| 28 | issue, hasIssue := openIssue(st, repo, t.Number) | |
| 29 | if t.Empty { | |
| 30 | if hasIssue { | |
| 31 | st.SetIssueState(issue.ID, "closed") | |
| 32 | t.Record(0) | |
| 33 | } | |
| 34 | if t.Changed { | |
| 35 | return t.Save() | |
| 36 | } | |
| 37 | return nil | |
| 38 | } | |
| 39 | // Nothing changed since the last report: leave it be, whether the issue | |
| 40 | // is still open or the maintainer has closed it. Reopening on an | |
| 41 | // unchanged set would make closing the issue pointless. | |
| 42 | if !t.Changed { | |
| 43 | return nil | |
| 44 | } | |
| 45 | if err := t.Save(); err != nil { | |
| 46 | return err | |
| 47 | } | |
| 48 | body := t.Body() | |
| 49 | if hasIssue { | |
| 50 | if err := st.UpdateIssueText(issue.ID, nil, &body, nil); err != nil { | |
| 51 | return err | |
| 52 | } | |
| 53 | notify(st, cfg, repo, t.Title, issue.Number, fmt.Sprintf("updated issue #%d", issue.Number), body) | |
| 54 | return nil | |
| 55 | } | |
| 56 | author, err := st.UserByUsername(store.BotUsername) | |
| 57 | if err != nil { | |
| 58 | // Migration 0028 leaves the account uncreated when the name was | |
| 59 | // already taken, which is the one case worth spelling out: the | |
| 60 | // feature is stuck until an operator frees the name. | |
| 61 | return fmt.Errorf("no %s account to author the issue (the name was taken when this instance upgraded): %w", | |
| 62 | store.BotUsername, err) | |
| 63 | } | |
| 64 | number, err := st.CreateIssue(repo.ID, author.ID, t.Title, body, "md") | |
| 65 | if err != nil { | |
| 66 | return err | |
| 67 | } | |
| 68 | if err := t.Record(number); err != nil { | |
| 69 | return err | |
| 70 | } | |
| 71 | st.RecordEvent(repo.ID, author.ID, "issue.created", fmt.Sprintf(`{"number":%d}`, number)) | |
| 72 | notify(st, cfg, repo, t.Title, number, fmt.Sprintf("opened issue #%d", number), body) | |
| 73 | return nil | |
| 74 | } | |
| 75 | ||
| 76 | // openIssue loads the issue being maintained, if there still is one. A | |
| 77 | // closed issue counts as gone: reopening one the maintainer closed would be | |
| 78 | // arguing with them, so the next change opens a fresh issue. | |
| 79 | func openIssue(st *store.Store, repo store.Repo, number int64) (store.Issue, bool) { | |
| 80 | if number == 0 { | |
| 81 | return store.Issue{}, false | |
| 82 | } | |
| 83 | issue, err := st.IssueByNumber(repo.ID, number) | |
| 84 | if err != nil || issue.State != "open" { | |
| 85 | return store.Issue{}, false | |
| 86 | } | |
| 87 | return issue, true | |
| 88 | } | |
| 89 | ||
| 90 | // notify files an inbox row for the repo's owners and watchers and mails | |
| 91 | // them, the same targets and shape as an issue opened over SSH. | |
| 92 | // Best-effort, like every other notification. | |
| 93 | func notify(st *store.Store, cfg config.Config, repo store.Repo, title string, number int64, action, body string) { | |
| 94 | targets, err := st.RepoNotifyTargets(repo) | |
| 95 | if err != nil { | |
| 96 | return | |
| 97 | } | |
| 98 | author, err := st.UserByUsername(store.BotUsername) | |
| 99 | if err != nil { | |
| 100 | return | |
| 101 | } | |
| 102 | recipients, err := st.NotifyRecipients(repo.ID, author.ID, targets, true) | |
| 103 | if err != nil { | |
| 104 | return | |
| 105 | } | |
| 106 | subject := fmt.Sprintf("[%s] #%d: %s", repo.Path(), number, title) | |
| 107 | text := fmt.Sprintf("%s %s\n\n%s\n%s/%s/issues/%d\n", store.BotUsername, action, body, | |
| 108 | strings.TrimSuffix(cfg.Server.SiteURL, "/"), repo.Path(), number) | |
| 109 | path := fmt.Sprintf("%s/issues/%d", repo.Path(), number) | |
| 110 | for _, id := range recipients { | |
| 111 | st.AddNotice(id, repo.ID, "issue", store.BotUsername, action, path) | |
| 112 | if cfg.Mail.SMTPHost == "" { | |
| 113 | continue | |
| 114 | } | |
| 115 | email, err := st.ActivityMailAddress(id) | |
| 116 | if err != nil || email == "" { | |
| 117 | continue | |
| 118 | } | |
| 119 | st.EnqueueMail(email, subject, text) | |
| 120 | } | |
| 121 | } | |
internal/ci/ci.go +8
| @@ -21,6 +21,11 @@ import ( | ||
| 21 | 21 | // ConfigPath is where the build configuration lives in a repository. |
| 22 | 22 | const ConfigPath = ".gitbay/ci.yml" |
| 23 | 23 | |
| 24 | // SecurityJob is the instance's code scanning job, defined by the server's | |
| 25 | // [security] configuration rather than the repository's ci.yml. The name is | |
| 26 | // reserved so a repository's own job cannot be mistaken for it. | |
| 27 | const SecurityJob = "gitbay-security" | |
| 28 | ||
| 24 | 29 | const ( |
| 25 | 30 | maxJobs = 10 |
| 26 | 31 | maxSteps = 50 |
| @@ -79,6 +84,9 @@ func Parse(raw []byte) ([]Job, error) { | ||
| 79 | 84 | if !jobName.MatchString(name) { |
| 80 | 85 | return nil, fmt.Errorf("bad job name %q: lowercase letters, digits, - and _; max 40 chars", name) |
| 81 | 86 | } |
| 87 | if name == SecurityJob { | |
| 88 | return nil, fmt.Errorf("job name %q is reserved for the instance's code scan", name) | |
| 89 | } | |
| 82 | 90 | if len(j.Steps) == 0 { |
| 83 | 91 | return nil, fmt.Errorf("job %q has no steps", name) |
| 84 | 92 | } |
internal/ci/sched.go +49 −3
| @@ -3,11 +3,13 @@ package ci | ||
| 3 | 3 | import ( |
| 4 | 4 | "context" |
| 5 | 5 | "encoding/json" |
| 6 | "errors" | |
| 6 | 7 | "fmt" |
| 7 | 8 | "log/slog" |
| 8 | 9 | "os" |
| 9 | 10 | "time" |
| 10 | 11 | |
| 12 | "gitbay.org/gitbay/internal/config" | |
| 11 | 13 | "gitbay.org/gitbay/internal/gitutil" |
| 12 | 14 | "gitbay.org/gitbay/internal/store" |
| 13 | 15 | ) |
| @@ -30,10 +32,13 @@ func NextRun(expr string, after time.Time) string { | ||
| 30 | 32 | } |
| 31 | 33 | |
| 32 | 34 | // Scheduler fires scheduled builds. repoDir maps a repo to its bare path. |
| 35 | // Security is the instance's code scanning job; with no image it never | |
| 36 | // queues. | |
| 33 | 37 | type Scheduler struct { |
| 34 | St *store.Store | |
| 35 | RepoDir func(owner, name string) string | |
| 36 | SiteURL string | |
| 38 | St *store.Store | |
| 39 | RepoDir func(owner, name string) string | |
| 40 | SiteURL string | |
| 41 | Security config.Security | |
| 37 | 42 | } |
| 38 | 43 | |
| 39 | 44 | // Run ticks until the context ends. GITBAY_SCHED_TICK overrides the |
| @@ -62,6 +67,7 @@ func (s *Scheduler) Run(ctx context.Context) { | ||
| 62 | 67 | // ticker for tests. |
| 63 | 68 | func (s *Scheduler) RunDue(now time.Time) { |
| 64 | 69 | s.reapStale() |
| 70 | s.runSecurityDue(now) | |
| 65 | 71 | due, err := s.St.DueSchedules(isoNow(now)) |
| 66 | 72 | if err != nil { |
| 67 | 73 | slog.Error("scheduler: listing due builds", "err", err) |
| @@ -121,6 +127,46 @@ func (s *Scheduler) RunDue(now time.Time) { | ||
| 121 | 127 | } |
| 122 | 128 | } |
| 123 | 129 | |
| 130 | // runSecurityDue queues the code scan for every repository whose next scan | |
| 131 | // is due, and advances it. | |
| 132 | func (s *Scheduler) runSecurityDue(now time.Time) { | |
| 133 | if s.Security.Image == "" { | |
| 134 | return | |
| 135 | } | |
| 136 | due, err := s.St.DueSecurityScans(isoNow(now)) | |
| 137 | if err != nil { | |
| 138 | slog.Error("scheduler: listing due security scans", "err", err) | |
| 139 | return | |
| 140 | } | |
| 141 | for _, repo := range due { | |
| 142 | s.St.SetSecurityNext(repo.ID, NextRun(s.Security.Schedule, now)) | |
| 143 | if _, err := QueueSecurityScan(s.St, s.Security, s.RepoDir(repo.OwnerName, repo.Name), repo); err != nil { | |
| 144 | slog.Warn("scheduler: queueing security scan", "repo", repo.Path(), "err", err) | |
| 145 | } | |
| 146 | } | |
| 147 | } | |
| 148 | ||
| 149 | // ErrScanPending is QueueSecurityScan finding the last scan of the | |
| 150 | // default branch's tip still pending or running. | |
| 151 | var ErrScanPending = errors.New("a scan of this commit is already pending or running") | |
| 152 | ||
| 153 | // QueueSecurityScan queues the code scan of the default branch's tip. It | |
| 154 | // sets no commit status: the scan reports through its issue, and a status | |
| 155 | // would put it in front of merge gates that never asked for it. | |
| 156 | func QueueSecurityScan(st *store.Store, sec config.Security, dir string, repo store.Repo) (int64, error) { | |
| 157 | sha, err := gitutil.ResolveRef(dir, "refs/heads/"+repo.DefaultBranch) | |
| 158 | if err != nil { | |
| 159 | return 0, fmt.Errorf("%s has no commits on %s", repo.Path(), repo.DefaultBranch) | |
| 160 | } | |
| 161 | if built, err := st.BuildsForCommit(repo.ID, sha); err == nil { | |
| 162 | if b, ok := built[SecurityJob]; ok && (b.Status == "pending" || b.Status == "running") { | |
| 163 | return 0, ErrScanPending | |
| 164 | } | |
| 165 | } | |
| 166 | steps, _ := json.Marshal(sec.Steps) | |
| 167 | return st.CreateBuild(repo.ID, SecurityJob, sha, repo.DefaultBranch, string(steps), sec.Image, "", true) | |
| 168 | } | |
| 169 | ||
| 124 | 170 | // reapStale resolves builds running past the deadline, so a runner that |
| 125 | 171 | // died mid-build leaves neither a build running nor a commit pending |
| 126 | 172 | // forever. It runs on the tick rather than on the next runner claim, so it |
internal/ci/sched_test.go +68
| @@ -4,9 +4,11 @@ import ( | ||
| 4 | 4 | "os" |
| 5 | 5 | "os/exec" |
| 6 | 6 | "path/filepath" |
| 7 | "strings" | |
| 7 | 8 | "testing" |
| 8 | 9 | "time" |
| 9 | 10 | |
| 11 | "gitbay.org/gitbay/internal/config" | |
| 10 | 12 | "gitbay.org/gitbay/internal/store" |
| 11 | 13 | ) |
| 12 | 14 | |
| @@ -136,3 +138,69 @@ func TestSchedulerRunDue(t *testing.T) { | ||
| 136 | 138 | t.Fatalf("tick after the last build finished did not queue: %+v", builds) |
| 137 | 139 | } |
| 138 | 140 | } |
| 141 | ||
| 142 | // The code scan has its own schedule: a due repository gets one build of | |
| 143 | // the instance's job, with no commit status, and none while one is pending | |
| 144 | // or while the instance has no image. | |
| 145 | func TestSchedulerQueuesSecurityScan(t *testing.T) { | |
| 146 | st, err := store.Open(":memory:") | |
| 147 | if err != nil { | |
| 148 | t.Fatal(err) | |
| 149 | } | |
| 150 | defer st.Close() | |
| 151 | if err := st.MigrateUp(); err != nil { | |
| 152 | t.Fatal(err) | |
| 153 | } | |
| 154 | uid, _ := st.CreateUser("alice", false) | |
| 155 | repoID, _ := st.CreateRepo("user", uid, "app", "public") | |
| 156 | work := t.TempDir() | |
| 157 | bare := filepath.Join(work, "bare.git") | |
| 158 | for _, args := range [][]string{ | |
| 159 | {"init", "-q", "-b", "main", "src"}, | |
| 160 | {"-C", "src", "-c", "user.name=t", "-c", "user.email=t@example.test", "commit", "-q", "--allow-empty", "-m", "base"}, | |
| 161 | {"clone", "-q", "--bare", "src", "bare.git"}, | |
| 162 | } { | |
| 163 | cmd := exec.Command("git", args...) | |
| 164 | cmd.Dir = work | |
| 165 | cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null") | |
| 166 | if out, err := cmd.CombinedOutput(); err != nil { | |
| 167 | t.Fatalf("git %v: %v\n%s", args, err, out) | |
| 168 | } | |
| 169 | } | |
| 170 | now := time.Now() | |
| 171 | if err := st.EnableSecurityScan(repoID, now.Add(-time.Minute).UTC().Format("2006-01-02T15:04:05Z")); err != nil { | |
| 172 | t.Fatal(err) | |
| 173 | } | |
| 174 | s := &Scheduler{St: st, SiteURL: "https://x.test", RepoDir: func(owner, name string) string { return bare }} | |
| 175 | s.RunDue(now) | |
| 176 | if builds, _ := st.ListBuilds(repoID, store.BuildFilter{}, 10); len(builds) != 0 { | |
| 177 | t.Fatalf("queued with no image: %+v", builds) | |
| 178 | } | |
| 179 | ||
| 180 | s.Security = config.Security{Image: "localhost/gitbay-security:1", Schedule: "0 4 * * *", Steps: []string{"true"}} | |
| 181 | st.SetSecurityNext(repoID, now.Add(-time.Minute).UTC().Format("2006-01-02T15:04:05Z")) | |
| 182 | s.RunDue(now) | |
| 183 | builds, _ := st.ListBuilds(repoID, store.BuildFilter{}, 10) | |
| 184 | if len(builds) != 1 || builds[0].Job != SecurityJob || builds[0].Image != "localhost/gitbay-security:1" { | |
| 185 | t.Fatalf("builds = %+v", builds) | |
| 186 | } | |
| 187 | if statuses, _ := st.CommitStatusesFor(repoID, []string{builds[0].SHA}); len(statuses[builds[0].SHA]) != 0 { | |
| 188 | t.Errorf("a status was set: %+v", statuses) | |
| 189 | } | |
| 190 | if sc, _ := st.SecurityScanFor(repoID); sc.NextRun <= now.UTC().Format("2006-01-02T15:04:05Z") { | |
| 191 | t.Errorf("next run not advanced: %q", sc.NextRun) | |
| 192 | } | |
| 193 | ||
| 194 | // Due again while the first is pending: nothing more. | |
| 195 | st.SetSecurityNext(repoID, now.Add(-time.Minute).UTC().Format("2006-01-02T15:04:05Z")) | |
| 196 | s.RunDue(now) | |
| 197 | if builds, _ := st.ListBuilds(repoID, store.BuildFilter{}, 10); len(builds) != 1 { | |
| 198 | t.Errorf("queued over a pending scan: %d builds", len(builds)) | |
| 199 | } | |
| 200 | } | |
| 201 | ||
| 202 | func TestSecurityJobNameIsReserved(t *testing.T) { | |
| 203 | if _, err := Parse([]byte("jobs:\n gitbay-security:\n steps: [echo]\n")); err == nil || !strings.Contains(err.Error(), "reserved") { | |
| 204 | t.Fatalf("Parse = %v", err) | |
| 205 | } | |
| 206 | } | |
internal/codescan/codescan.go added +192
| @@ -0,0 +1,192 @@ | ||
| 1 | // Package codescan reads the SARIF a security build uploads and renders | |
| 2 | // the issue that reports it. Only the parts of SARIF 2.1.0 a finding needs | |
| 3 | // are read: the tool's name, and each result's rule, level, message and | |
| 4 | // first location. | |
| 5 | package codescan | |
| 6 | ||
| 7 | import ( | |
| 8 | "crypto/sha256" | |
| 9 | "encoding/hex" | |
| 10 | "encoding/json" | |
| 11 | "errors" | |
| 12 | "fmt" | |
| 13 | "net/url" | |
| 14 | "sort" | |
| 15 | "strings" | |
| 16 | ||
| 17 | "gitbay.org/gitbay/internal/store" | |
| 18 | ) | |
| 19 | ||
| 20 | // IssueTitle is fixed so the issue is recognizable across scans. | |
| 21 | const IssueTitle = "Code scanning findings" | |
| 22 | ||
| 23 | // MaxUpload bounds one upload. MaxFindings bounds what is kept from it. | |
| 24 | const ( | |
| 25 | MaxUpload = 8 << 20 | |
| 26 | MaxFindings = 500 | |
| 27 | ) | |
| 28 | ||
| 29 | type sarif struct { | |
| 30 | Runs []struct { | |
| 31 | Tool struct { | |
| 32 | Driver struct { | |
| 33 | Name string `json:"name"` | |
| 34 | } `json:"driver"` | |
| 35 | } `json:"tool"` | |
| 36 | Results []struct { | |
| 37 | RuleID string `json:"ruleId"` | |
| 38 | Level string `json:"level"` | |
| 39 | Message struct { | |
| 40 | Text string `json:"text"` | |
| 41 | } `json:"message"` | |
| 42 | Locations []struct { | |
| 43 | PhysicalLocation struct { | |
| 44 | ArtifactLocation struct { | |
| 45 | URI string `json:"uri"` | |
| 46 | } `json:"artifactLocation"` | |
| 47 | Region struct { | |
| 48 | StartLine int `json:"startLine"` | |
| 49 | } `json:"region"` | |
| 50 | } `json:"physicalLocation"` | |
| 51 | } `json:"locations"` | |
| 52 | } `json:"results"` | |
| 53 | } `json:"runs"` | |
| 54 | } | |
| 55 | ||
| 56 | // Parse returns the findings in a SARIF document, deduplicated by | |
| 57 | // fingerprint and sorted by path and line. It reports whether results past | |
| 58 | // MaxFindings were dropped. | |
| 59 | func Parse(raw []byte) ([]store.SecurityFinding, bool, error) { | |
| 60 | var doc sarif | |
| 61 | if err := json.Unmarshal(raw, &doc); err != nil { | |
| 62 | return nil, false, fmt.Errorf("not SARIF: %w", err) | |
| 63 | } | |
| 64 | if doc.Runs == nil { | |
| 65 | return nil, false, errors.New("not SARIF: no runs") | |
| 66 | } | |
| 67 | seen := map[string]bool{} | |
| 68 | var out []store.SecurityFinding | |
| 69 | truncated := false | |
| 70 | for _, run := range doc.Runs { | |
| 71 | tool := clean(run.Tool.Driver.Name, 40) | |
| 72 | for _, r := range run.Results { | |
| 73 | if len(out) >= MaxFindings { | |
| 74 | truncated = true | |
| 75 | break | |
| 76 | } | |
| 77 | f := store.SecurityFinding{Tool: tool, Rule: clean(r.RuleID, 200), | |
| 78 | Level: level(r.Level), Message: clean(r.Message.Text, 200)} | |
| 79 | if len(r.Locations) > 0 { | |
| 80 | loc := r.Locations[0].PhysicalLocation | |
| 81 | f.Path = relPath(loc.ArtifactLocation.URI) | |
| 82 | f.Line = max(loc.Region.StartLine, 0) | |
| 83 | } | |
| 84 | f.Fingerprint = fingerprint(f) | |
| 85 | if seen[f.Fingerprint] { | |
| 86 | continue | |
| 87 | } | |
| 88 | seen[f.Fingerprint] = true | |
| 89 | out = append(out, f) | |
| 90 | } | |
| 91 | } | |
| 92 | sort.Slice(out, func(i, j int) bool { | |
| 93 | if out[i].Path != out[j].Path { | |
| 94 | return out[i].Path < out[j].Path | |
| 95 | } | |
| 96 | if out[i].Line != out[j].Line { | |
| 97 | return out[i].Line < out[j].Line | |
| 98 | } | |
| 99 | return out[i].Fingerprint < out[j].Fingerprint | |
| 100 | }) | |
| 101 | return out, truncated, nil | |
| 102 | } | |
| 103 | ||
| 104 | // fingerprint identifies a finding across scans. The line is left out so | |
| 105 | // an edit above it does not make it a new finding. | |
| 106 | func fingerprint(f store.SecurityFinding) string { | |
| 107 | sum := sha256.Sum256([]byte(f.Tool + "\x00" + f.Rule + "\x00" + f.Path + "\x00" + f.Message)) | |
| 108 | return hex.EncodeToString(sum[:12]) | |
| 109 | } | |
| 110 | ||
| 111 | // level is SARIF's result level, which defaults to warning. | |
| 112 | func level(l string) string { | |
| 113 | switch l { | |
| 114 | case "error", "warning", "note", "none": | |
| 115 | return l | |
| 116 | } | |
| 117 | return "warning" | |
| 118 | } | |
| 119 | ||
| 120 | // relPath turns an artifact URI into a repository path: tools write | |
| 121 | // `file:///workspace/x.go`, `./x.go` or `x.go` for the same file. | |
| 122 | func relPath(uri string) string { | |
| 123 | p := uri | |
| 124 | if u, err := url.Parse(uri); err == nil && u.Scheme == "file" { | |
| 125 | p = u.Path | |
| 126 | } | |
| 127 | p = strings.TrimPrefix(p, "/workspace/") | |
| 128 | p = strings.TrimPrefix(p, "./") | |
| 129 | return clean(p, 300) | |
| 130 | } | |
| 131 | ||
| 132 | // clean is one line of at most n bytes of valid UTF-8, since every field | |
| 133 | // lands in an issue body and a mail. | |
| 134 | func clean(s string, n int) string { | |
| 135 | s = strings.ToValidUTF8(strings.Join(strings.Fields(s), " "), "") | |
| 136 | if len(s) > n { | |
| 137 | s = strings.ToValidUTF8(s[:n], "") + "…" | |
| 138 | } | |
| 139 | return s | |
| 140 | } | |
| 141 | ||
| 142 | // Same reports whether two sets name the same findings, which decides | |
| 143 | // between leaving the issue alone and rewriting it. Both arrive sorted by | |
| 144 | // fingerprint. A line moving is not a change. | |
| 145 | func Same(a, b []store.SecurityFinding) bool { | |
| 146 | if len(a) != len(b) { | |
| 147 | return false | |
| 148 | } | |
| 149 | for i := range a { | |
| 150 | if a[i].Fingerprint != b[i].Fingerprint { | |
| 151 | return false | |
| 152 | } | |
| 153 | } | |
| 154 | return true | |
| 155 | } | |
| 156 | ||
| 157 | // ByFingerprint sorts a set the way the store returns it, for Same. | |
| 158 | func ByFingerprint(fs []store.SecurityFinding) []store.SecurityFinding { | |
| 159 | out := append([]store.SecurityFinding(nil), fs...) | |
| 160 | sort.Slice(out, func(i, j int) bool { return out[i].Fingerprint < out[j].Fingerprint }) | |
| 161 | return out | |
| 162 | } | |
| 163 | ||
| 164 | // Body renders the issue: one list per tool, a line per finding. | |
| 165 | func Body(branch string, findings []store.SecurityFinding, truncated bool) string { | |
| 166 | var b strings.Builder | |
| 167 | fmt.Fprintf(&b, "The scheduled code scan of `%s` reported these findings. "+ | |
| 168 | "This issue is kept up to date as that changes, and closed once a scan reports none.\n", branch) | |
| 169 | byTool := map[string][]store.SecurityFinding{} | |
| 170 | var tools []string | |
| 171 | for _, f := range findings { | |
| 172 | if _, ok := byTool[f.Tool]; !ok { | |
| 173 | tools = append(tools, f.Tool) | |
| 174 | } | |
| 175 | byTool[f.Tool] = append(byTool[f.Tool], f) | |
| 176 | } | |
| 177 | sort.Strings(tools) | |
| 178 | for _, tool := range tools { | |
| 179 | name := tool | |
| 180 | if name == "" { | |
| 181 | name = "Unnamed tool" | |
| 182 | } | |
| 183 | fmt.Fprintf(&b, "\n### %s\n\n", name) | |
| 184 | for _, f := range byTool[tool] { | |
| 185 | fmt.Fprintf(&b, "- `%s:%d` %s (%s): %s\n", f.Path, f.Line, f.Rule, f.Level, f.Message) | |
| 186 | } | |
| 187 | } | |
| 188 | if truncated { | |
| 189 | fmt.Fprintf(&b, "\nOnly the first %d results are listed.\n", MaxFindings) | |
| 190 | } | |
| 191 | return b.String() | |
| 192 | } | |
internal/codescan/codescan_test.go added +71
| @@ -0,0 +1,71 @@ | ||
| 1 | package codescan | |
| 2 | ||
| 3 | import ( | |
| 4 | "strconv" | |
| 5 | "strings" | |
| 6 | "testing" | |
| 7 | ) | |
| 8 | ||
| 9 | func TestParse(t *testing.T) { | |
| 10 | raw := []byte(`{"version":"2.1.0","runs":[ | |
| 11 | {"tool":{"driver":{"name":"opengrep"}},"results":[ | |
| 12 | {"ruleId":"r1","message":{"text":"first\nline"}, | |
| 13 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"file:///workspace/a/b.go"},"region":{"startLine":3}}}]}, | |
| 14 | {"ruleId":"r1","message":{"text":"first\nline"}, | |
| 15 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"./a/b.go"},"region":{"startLine":3}}}]}, | |
| 16 | {"ruleId":"r2","level":"note","message":{"text":"second"}, | |
| 17 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"a/a.go"},"region":{"startLine":9}}}]} | |
| 18 | ]}, | |
| 19 | {"tool":{"driver":{"name":"other"}},"results":[{"ruleId":"x","level":"bogus","message":{"text":"no location"}}]} | |
| 20 | ]}`) | |
| 21 | got, truncated, err := Parse(raw) | |
| 22 | if err != nil || truncated { | |
| 23 | t.Fatalf("Parse: %v %v", err, truncated) | |
| 24 | } | |
| 25 | if len(got) != 3 { | |
| 26 | t.Fatalf("findings = %+v", got) | |
| 27 | } | |
| 28 | // Sorted by path; the duplicate under two URI spellings folded; the | |
| 29 | // default level applied; whitespace collapsed. | |
| 30 | if got[0].Path != "" || got[0].Level != "warning" || got[0].Tool != "other" { | |
| 31 | t.Errorf("got[0] = %+v", got[0]) | |
| 32 | } | |
| 33 | if got[1].Path != "a/a.go" || got[1].Level != "note" { | |
| 34 | t.Errorf("got[1] = %+v", got[1]) | |
| 35 | } | |
| 36 | if got[2].Path != "a/b.go" || got[2].Line != 3 || got[2].Message != "first line" || got[2].Level != "warning" { | |
| 37 | t.Errorf("got[2] = %+v", got[2]) | |
| 38 | } | |
| 39 | if _, _, err := Parse([]byte(`{"nope":1}`)); err == nil { | |
| 40 | t.Error("a document without runs parsed") | |
| 41 | } | |
| 42 | } | |
| 43 | ||
| 44 | func TestParseTruncates(t *testing.T) { | |
| 45 | var b strings.Builder | |
| 46 | b.WriteString(`{"runs":[{"tool":{"driver":{"name":"t"}},"results":[`) | |
| 47 | for i := 0; i < MaxFindings+5; i++ { | |
| 48 | if i > 0 { | |
| 49 | b.WriteString(",") | |
| 50 | } | |
| 51 | b.WriteString(`{"ruleId":"r","message":{"text":"m` + strings.Repeat("x", i%7) + itoa(i) + `"}}`) | |
| 52 | } | |
| 53 | b.WriteString(`]}]}`) | |
| 54 | got, truncated, err := Parse([]byte(b.String())) | |
| 55 | if err != nil || !truncated || len(got) != MaxFindings { | |
| 56 | t.Fatalf("len %d truncated %v err %v", len(got), truncated, err) | |
| 57 | } | |
| 58 | } | |
| 59 | ||
| 60 | func itoa(i int) string { return strconv.Itoa(i) } | |
| 61 | ||
| 62 | // A finding that moves is the same finding. | |
| 63 | func TestSameIgnoresLines(t *testing.T) { | |
| 64 | a, _, _ := Parse([]byte(`{"runs":[{"tool":{"driver":{"name":"t"}},"results":[{"ruleId":"r","message":{"text":"m"}, | |
| 65 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"x.go"},"region":{"startLine":1}}}]}]}]}`)) | |
| 66 | b, _, _ := Parse([]byte(`{"runs":[{"tool":{"driver":{"name":"t"}},"results":[{"ruleId":"r","message":{"text":"m"}, | |
| 67 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"x.go"},"region":{"startLine":40}}}]}]}]}`)) | |
| 68 | if !Same(ByFingerprint(a), ByFingerprint(b)) { | |
| 69 | t.Error("a moved finding counted as a change") | |
| 70 | } | |
| 71 | } | |
internal/config/config.go +27
| @@ -65,6 +65,7 @@ type Config struct { | ||
| 65 | 65 | Mail Mail `toml:"mail"` |
| 66 | 66 | Mirrors Mirrors `toml:"mirrors"` |
| 67 | 67 | Deps Deps `toml:"deps"` |
| 68 | Security Security `toml:"security"` | |
| 68 | 69 | Retention Retention `toml:"retention"` |
| 69 | 70 | Push Push `toml:"push"` |
| 70 | 71 | Backup Backup `toml:"backup"` |
| @@ -226,6 +227,23 @@ type Deps struct { | ||
| 226 | 227 | CheckIntervalHours int `toml:"check_interval_hours"` |
| 227 | 228 | } |
| 228 | 229 | |
| 230 | // Security configures the built-in code scanning job that `repo security | |
| 231 | // enable` schedules. It is unavailable until an operator names Image, an | |
| 232 | // image built from deploy/Containerfile.security and provisioned on the | |
| 233 | // runner hosts. Steps write SARIF into $GITBAY_RESULTS; the runner uploads | |
| 234 | // it when they succeed. | |
| 235 | type Security struct { | |
| 236 | Image string `toml:"image"` | |
| 237 | Schedule string `toml:"schedule"` | |
| 238 | Steps []string `toml:"steps"` | |
| 239 | } | |
| 240 | ||
| 241 | // DefaultSecuritySteps run Opengrep with the rules the image carries. | |
| 242 | var DefaultSecuritySteps = []string{ | |
| 243 | `mkdir -p "$GITBAY_RESULTS"`, | |
| 244 | `opengrep scan --quiet --disable-version-check --config /opt/opengrep-rules --sarif-output="$GITBAY_RESULTS/opengrep.sarif" .`, | |
| 245 | } | |
| 246 | ||
| 229 | 247 | type Limits struct { |
| 230 | 248 | MaxPackBytes int64 `toml:"max_pack_bytes"` |
| 231 | 249 | MaxBlobBytes int64 `toml:"max_blob_bytes"` |
| @@ -573,6 +591,7 @@ func Default() Config { | ||
| 573 | 591 | GitDaemon: GitDaemon{Port: 9418}, |
| 574 | 592 | Mirrors: Mirrors{PullIntervalMinutes: 15}, |
| 575 | 593 | Deps: Deps{CheckIntervalHours: 24}, |
| 594 | Security: Security{Schedule: "0 4 * * *", Steps: DefaultSecuritySteps}, | |
| 576 | 595 | Limits: Limits{ |
| 577 | 596 | MaxPackBytes: 2 << 30, // 2 GiB |
| 578 | 597 | MaxBlobBytes: 100 << 20, |
| @@ -656,6 +675,14 @@ func (c Config) Validate() error { | ||
| 656 | 675 | if c.Server.Root == "" { |
| 657 | 676 | errs = append(errs, errors.New("server.root is required")) |
| 658 | 677 | } |
| 678 | if c.Security.Image != "" { | |
| 679 | if len(strings.Fields(c.Security.Schedule)) != 5 { | |
| 680 | errs = append(errs, errors.New("security.schedule must be a five-field cron expression")) | |
| 681 | } | |
| 682 | if len(c.Security.Steps) == 0 { | |
| 683 | errs = append(errs, errors.New("security.steps must name at least one step when security.image is set")) | |
| 684 | } | |
| 685 | } | |
| 659 | 686 | if d := c.Pages.Domain; d != "" { |
| 660 | 687 | if d == c.SiteHost() { |
| 661 | 688 | errs = append(errs, errors.New("pages.domain must differ from the site host: pages serve repo-authored scripts, which must not run on the forge's origin")) |
internal/control/build.go +14 −4
| @@ -743,8 +743,10 @@ func runRunnerNext(c *Ctx, args []string) int { | ||
| 743 | 743 | json.Unmarshal([]byte(b.Steps), &steps) |
| 744 | 744 | // Secrets ride the claim: this channel is admin-only and the values |
| 745 | 745 | // land in the build's environment, nowhere else. |
| 746 | // The code scan is the operator's job, and reads code rather than | |
| 747 | // running it: it gets none of the repository's secrets. | |
| 746 | 748 | var secrets map[string]string |
| 747 | if b.Trusted { | |
| 749 | if b.Trusted && b.Job != ci.SecurityJob { | |
| 748 | 750 | secrets, err = c.Store.BuildSecrets(b.RepoID) |
| 749 | 751 | if err != nil { |
| 750 | 752 | return c.fail(protocol.ExitFailure, "%v", err) |
| @@ -766,8 +768,13 @@ func runRunnerNext(c *Ctx, args []string) int { | ||
| 766 | 768 | // over loopback takes its port for the build's GITBAY_SSH (#260). |
| 767 | 769 | SSH string `json:"ssh,omitempty"` |
| 768 | 770 | Secrets map[string]string `json:"secrets,omitempty"` |
| 771 | // Results asks the runner to upload the SARIF its steps leave in | |
| 772 | // $GITBAY_RESULTS with runner results. An older runner ignores it, | |
| 773 | // and the scan's issue stays as it was. | |
| 774 | Results bool `json:"results,omitempty"` | |
| 769 | 775 | }{ID: b.ID, Repo: repo.Path(), Number: b.Number, Job: b.Job, SHA: b.SHA, Ref: b.Ref, |
| 770 | Steps: steps, Image: b.Image, Trusted: b.Trusted, SSH: publicSSH(c), Secrets: secrets} | |
| 776 | Steps: steps, Image: b.Image, Trusted: b.Trusted, SSH: publicSSH(c), Secrets: secrets, | |
| 777 | Results: b.Job == ci.SecurityJob} | |
| 771 | 778 | return c.emit(d, func(w io.Writer) { |
| 772 | 779 | fmt.Fprintf(w, "build %d: %s %s @ %.10s\n", d.ID, d.Repo, d.Job, d.SHA) |
| 773 | 780 | }) |
| @@ -908,8 +915,11 @@ func runRunnerDone(c *Ctx, args []string) int { | ||
| 908 | 915 | } |
| 909 | 916 | url := fmt.Sprintf("%s/%s/builds/%d", c.Cfg.Server.SiteURL, repo.Path(), b.Number) |
| 910 | 917 | desc := "build " + outcome |
| 911 | if err := c.Store.SetCommitStatus(repo.ID, b.SHA, "ci/"+b.Job, outcome, desc, url, c.User.ID); err != nil { | |
| 912 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 918 | // The code scan reports through its issue; see ci.QueueSecurityScan. | |
| 919 | if b.Job != ci.SecurityJob { | |
| 920 | if err := c.Store.SetCommitStatus(repo.ID, b.SHA, "ci/"+b.Job, outcome, desc, url, c.User.ID); err != nil { | |
| 921 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 922 | } | |
| 913 | 923 | } |
| 914 | 924 | c.Store.RecordEvent(repo.ID, c.User.ID, "build."+outcome, |
| 915 | 925 | fmt.Sprintf(`{"number":%d,"job":%q,"sha":%q}`, b.Number, b.Job, b.SHA)) |
internal/control/security.go added +279
| @@ -0,0 +1,279 @@ | ||
| 1 | package control | |
| 2 | ||
| 3 | import ( | |
| 4 | "errors" | |
| 5 | "fmt" | |
| 6 | "io" | |
| 7 | "strconv" | |
| 8 | "time" | |
| 9 | ||
| 10 | "gitbay.org/gitbay/internal/botissue" | |
| 11 | "gitbay.org/gitbay/internal/ci" | |
| 12 | "gitbay.org/gitbay/internal/codescan" | |
| 13 | "gitbay.org/gitbay/internal/policy" | |
| 14 | "gitbay.org/gitbay/internal/protocol" | |
| 15 | "gitbay.org/gitbay/internal/store" | |
| 16 | ) | |
| 17 | ||
| 18 | func init() { | |
| 19 | // Code scanning: the instance's own job ([security] in the config), | |
| 20 | // scheduled per repository that opts in, reporting through an issue. | |
| 21 | // Opt-in because the build runs on a runner the repository is attached | |
| 22 | // to, which is the owner's arrangement to make. | |
| 23 | register(Command{Path: []string{"repo", "security", "enable"}, | |
| 24 | Summary: "scan the default branch on the instance's code scanning schedule", | |
| 25 | Usage: "repo security enable <owner/name>", | |
| 26 | Examples: []string{"repo security enable krz/gitbay"}, | |
| 27 | Run: runSecurityEnable}) | |
| 28 | register(Command{Path: []string{"repo", "security", "disable"}, | |
| 29 | Summary: "stop code scanning", | |
| 30 | Usage: "repo security disable <owner/name>", | |
| 31 | Examples: []string{"repo security disable krz/gitbay"}, | |
| 32 | Run: runSecurityDisable}) | |
| 33 | register(Command{Path: []string{"repo", "security", "run"}, | |
| 34 | Summary: "queue a code scan of the default branch now", | |
| 35 | Usage: "repo security run <owner/name>", | |
| 36 | Examples: []string{"repo security run krz/gitbay"}, | |
| 37 | Run: runSecurityRun}) | |
| 38 | register(Command{Path: []string{"repo", "security", "status"}, | |
| 39 | Summary: "show code scanning state and the last scan's findings", | |
| 40 | Usage: "repo security status <owner/name>", | |
| 41 | Examples: []string{"repo security status krz/gitbay"}, | |
| 42 | ReadOnly: true, Run: runSecurityStatus}) | |
| 43 | register(Command{Path: []string{"runner", "results"}, | |
| 44 | Summary: "upload a code scan's SARIF from stdin", | |
| 45 | Usage: "runner results <build-id>", | |
| 46 | Examples: []string{"runner results 431 < results.sarif"}, | |
| 47 | ReadsStdin: true, Run: runRunnerResults}) | |
| 48 | } | |
| 49 | ||
| 50 | // SecurityOut is what `repo security status` emits. | |
| 51 | type SecurityOut struct { | |
| 52 | Enabled bool `json:"enabled"` | |
| 53 | NextRun string `json:"next_run,omitempty"` | |
| 54 | LastResults string `json:"last_results,omitempty"` | |
| 55 | IssueNumber int64 `json:"issue_number,omitempty"` | |
| 56 | Findings []SecurityFinding `json:"findings"` | |
| 57 | } | |
| 58 | ||
| 59 | // SecurityFinding is one result of the last scan. | |
| 60 | type SecurityFinding struct { | |
| 61 | Tool string `json:"tool"` | |
| 62 | Rule string `json:"rule"` | |
| 63 | Level string `json:"level"` | |
| 64 | Path string `json:"path"` | |
| 65 | Line int `json:"line"` | |
| 66 | Message string `json:"message"` | |
| 67 | } | |
| 68 | ||
| 69 | func runSecurityEnable(c *Ctx, args []string) int { | |
| 70 | if len(args) != 1 { | |
| 71 | return c.usage() | |
| 72 | } | |
| 73 | repo, code := resolveRepo(c, args[0], policy.CanAdmin) | |
| 74 | if code >= 0 { | |
| 75 | return code | |
| 76 | } | |
| 77 | if c.Cfg.Security.Image == "" { | |
| 78 | return c.fail(protocol.ExitFailure, "this instance has no code scanning image; an operator sets [security] image") | |
| 79 | } | |
| 80 | next := ci.NextRun(c.Cfg.Security.Schedule, time.Now()) | |
| 81 | if err := c.Store.EnableSecurityScan(repo.ID, next); err != nil { | |
| 82 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 83 | } | |
| 84 | return c.emit(map[string]any{"enabled": true, "next_run": next}, func(w io.Writer) { | |
| 85 | fmt.Fprintf(w, "code scanning enabled on %s; first scan %s\n", repo.Path(), next) | |
| 86 | }) | |
| 87 | } | |
| 88 | ||
| 89 | func runSecurityDisable(c *Ctx, args []string) int { | |
| 90 | if len(args) != 1 { | |
| 91 | return c.usage() | |
| 92 | } | |
| 93 | repo, code := resolveRepo(c, args[0], policy.CanAdmin) | |
| 94 | if code >= 0 { | |
| 95 | return code | |
| 96 | } | |
| 97 | if err := c.Store.DisableSecurityScan(repo.ID); err != nil { | |
| 98 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 99 | } | |
| 100 | return c.emit(map[string]any{"enabled": false}, func(w io.Writer) { | |
| 101 | fmt.Fprintf(w, "code scanning disabled on %s\n", repo.Path()) | |
| 102 | }) | |
| 103 | } | |
| 104 | ||
| 105 | func runSecurityRun(c *Ctx, args []string) int { | |
| 106 | if len(args) != 1 { | |
| 107 | return c.usage() | |
| 108 | } | |
| 109 | repo, code := resolveRepo(c, args[0], policy.CanAdmin) | |
| 110 | if code >= 0 { | |
| 111 | return code | |
| 112 | } | |
| 113 | if _, err := c.Store.SecurityScanFor(repo.ID); errors.Is(err, store.ErrNotFound) { | |
| 114 | return c.fail(protocol.ExitFailure, "code scanning is off on %s (repo security enable %s)", repo.Path(), repo.Path()) | |
| 115 | } else if err != nil { | |
| 116 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 117 | } | |
| 118 | if c.Cfg.Security.Image == "" { | |
| 119 | return c.fail(protocol.ExitFailure, "this instance has no code scanning image; an operator sets [security] image") | |
| 120 | } | |
| 121 | n, err := ci.QueueSecurityScan(c.Store, c.Cfg.Security, RepoDir(c.Cfg.Server.Root, repo.OwnerName, repo.Name), repo) | |
| 122 | if err != nil { | |
| 123 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 124 | } | |
| 125 | return c.emit(map[string]any{"build": n}, func(w io.Writer) { | |
| 126 | fmt.Fprintf(w, "queued build %d: %s on %s\n", n, ci.SecurityJob, repo.Path()) | |
| 127 | }) | |
| 128 | } | |
| 129 | ||
| 130 | func runSecurityStatus(c *Ctx, args []string) int { | |
| 131 | if len(args) != 1 { | |
| 132 | return c.usage() | |
| 133 | } | |
| 134 | repo, code := resolveRepo(c, args[0], policy.CanRead) | |
| 135 | if code >= 0 { | |
| 136 | return code | |
| 137 | } | |
| 138 | scan, err := c.Store.SecurityScanFor(repo.ID) | |
| 139 | if errors.Is(err, store.ErrNotFound) { | |
| 140 | return c.emitView(map[string]any{"enabled": false}, func(w io.Writer) { | |
| 141 | c.view(w).fields("scanning", fmt.Sprintf("off (repo security enable %s)", repo.Path())) | |
| 142 | }, func() screen { | |
| 143 | return screen{ | |
| 144 | fields: []field{{"Scanning", []cell{cText("off")}}}, | |
| 145 | actions: []action{{"Security", []string{"repo", "security", "enable", repo.Path()}}}, | |
| 146 | } | |
| 147 | }) | |
| 148 | } | |
| 149 | if err != nil { | |
| 150 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 151 | } | |
| 152 | reported, err := c.Store.ReportedFindings(repo.ID) | |
| 153 | if err != nil { | |
| 154 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 155 | } | |
| 156 | out := SecurityOut{Enabled: true, NextRun: scan.NextRun, LastResults: scan.LastResults, | |
| 157 | IssueNumber: scan.IssueNumber, Findings: []SecurityFinding{}} | |
| 158 | for _, f := range reported { | |
| 159 | out.Findings = append(out.Findings, SecurityFinding{f.Tool, f.Rule, f.Level, f.Path, f.Line, f.Message}) | |
| 160 | } | |
| 161 | return c.emitView(out, func(w io.Writer) { | |
| 162 | tracked := "" | |
| 163 | if scan.IssueNumber != 0 { | |
| 164 | tracked = fmt.Sprintf("#%d", scan.IssueNumber) | |
| 165 | } | |
| 166 | v := c.view(w) | |
| 167 | v.fields( | |
| 168 | "scanning", "on", | |
| 169 | "next scan", scan.NextRun, | |
| 170 | "last results", c.when(scan.LastResults), | |
| 171 | "tracked in", tracked, | |
| 172 | ) | |
| 173 | if len(out.Findings) > 0 { | |
| 174 | v.section("findings") | |
| 175 | tb := c.table(w, "TOOL", "RULE", "LEVEL", "PATH", "LINE") | |
| 176 | for _, f := range out.Findings { | |
| 177 | tb.row(cText(f.Tool), cText(f.Rule), cText(f.Level), cRef(f.Path), cNum(int64(f.Line))) | |
| 178 | } | |
| 179 | tb.flush() | |
| 180 | } | |
| 181 | }, func() screen { | |
| 182 | s := screen{fields: []field{ | |
| 183 | {"Scanning", []cell{cGlyph("ok"), cText("on")}}, | |
| 184 | {"Next scan", []cell{cText(scan.NextRun)}}, | |
| 185 | {"Last results", []cell{cAge(scan.LastResults)}}, | |
| 186 | }} | |
| 187 | if scan.IssueNumber != 0 { | |
| 188 | n := strconv.FormatInt(scan.IssueNumber, 10) | |
| 189 | s.fields = append(s.fields, field{"Tracked in", []cell{cLink("#"+n, c.siteURL(repo.Path(), "issues", n))}}) | |
| 190 | } | |
| 191 | found := section{title: "Findings", n: len(out.Findings)} | |
| 192 | for _, f := range out.Findings { | |
| 193 | found.rows = append(found.rows, rowOf(cRef(f.Path+":"+strconv.Itoa(f.Line)), cText(f.Rule), cMeta(f.Level))) | |
| 194 | } | |
| 195 | s.sections = []section{found} | |
| 196 | s.actions = []action{{"Security", []string{"repo", "security", "run", repo.Path()}}} | |
| 197 | return s | |
| 198 | }) | |
| 199 | } | |
| 200 | ||
| 201 | // runRunnerResults takes a security build's SARIF from the runner that | |
| 202 | // claimed it, replaces the repository's findings with it, and brings the | |
| 203 | // findings issue in line. Only the instance's own job may upload: its | |
| 204 | // steps are the operator's, where any other job's are the repository's. | |
| 205 | func runRunnerResults(c *Ctx, args []string) int { | |
| 206 | key, code := runnerSession(c) | |
| 207 | if code >= 0 { | |
| 208 | return code | |
| 209 | } | |
| 210 | if len(args) != 1 { | |
| 211 | return c.usage() | |
| 212 | } | |
| 213 | id, err := strconv.ParseInt(args[0], 10, 64) | |
| 214 | if err != nil { | |
| 215 | return c.fail(protocol.ExitUsage, "bad build id %q", args[0]) | |
| 216 | } | |
| 217 | b, err := c.Store.BuildByID(id) | |
| 218 | if err != nil { | |
| 219 | return c.fail(protocol.ExitNotFound, "no build %d", id) | |
| 220 | } | |
| 221 | if ok, err := runnerMayBuild(c, key, b.RepoID); err != nil { | |
| 222 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 223 | } else if !ok { | |
| 224 | return c.fail(protocol.ExitDenied, "this key is not attached to the build's repository; a repository admin attaches it with repo runner add") | |
| 225 | } | |
| 226 | if b.Job != ci.SecurityJob { | |
| 227 | return c.fail(protocol.ExitDenied, "results are accepted from the %s job only", ci.SecurityJob) | |
| 228 | } | |
| 229 | if b.Status != "running" { | |
| 230 | return c.fail(protocol.ExitFailure, "build %d is %s", id, b.Status) | |
| 231 | } | |
| 232 | scan, err := c.Store.SecurityScanFor(b.RepoID) | |
| 233 | if errors.Is(err, store.ErrNotFound) { | |
| 234 | // Disabled while the build ran: nothing to report into. | |
| 235 | return c.emit(map[string]any{"findings": 0}, func(w io.Writer) {}) | |
| 236 | } | |
| 237 | if err != nil { | |
| 238 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 239 | } | |
| 240 | raw, err := io.ReadAll(io.LimitReader(c.Stdin, codescan.MaxUpload+1)) | |
| 241 | if err != nil { | |
| 242 | return c.fail(protocol.ExitFailure, "reading results: %v", err) | |
| 243 | } | |
| 244 | if len(raw) > codescan.MaxUpload { | |
| 245 | return c.fail(protocol.ExitFailure, "results are over %d bytes", codescan.MaxUpload) | |
| 246 | } | |
| 247 | findings, truncated, err := codescan.Parse(raw) | |
| 248 | if err != nil { | |
| 249 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 250 | } | |
| 251 | repo, err := c.Store.RepoByID(b.RepoID) | |
| 252 | if err != nil { | |
| 253 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 254 | } | |
| 255 | previous, err := c.Store.ReportedFindings(repo.ID) | |
| 256 | if err != nil { | |
| 257 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 258 | } | |
| 259 | err = botissue.Maintain(c.Store, c.Cfg, repo, botissue.Tracked{ | |
| 260 | Title: codescan.IssueTitle, | |
| 261 | Number: scan.IssueNumber, | |
| 262 | Record: func(n int64) error { return c.Store.SetSecurityIssue(repo.ID, n) }, | |
| 263 | Empty: len(findings) == 0, | |
| 264 | Changed: !codescan.Same(previous, codescan.ByFingerprint(findings)), | |
| 265 | Save: func() error { return c.Store.ReplaceFindings(repo.ID, findings) }, | |
| 266 | Body: func() string { return codescan.Body(repo.DefaultBranch, findings, truncated) }, | |
| 267 | }) | |
| 268 | if err != nil { | |
| 269 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 270 | } | |
| 271 | // Stamped whether or not the set changed: the status says when results | |
| 272 | // last arrived. | |
| 273 | if err := c.Store.StampSecurityResults(repo.ID); err != nil { | |
| 274 | return c.fail(protocol.ExitFailure, "%v", err) | |
| 275 | } | |
| 276 | return c.emit(map[string]any{"findings": len(findings), "truncated": truncated}, func(w io.Writer) { | |
| 277 | fmt.Fprintf(w, "%d findings recorded for build %d\n", len(findings), b.Number) | |
| 278 | }) | |
| 279 | } | |
internal/control/security_test.go added +115
| @@ -0,0 +1,115 @@ | ||
| 1 | package control | |
| 2 | ||
| 3 | import ( | |
| 4 | "encoding/json" | |
| 5 | "strings" | |
| 6 | "testing" | |
| 7 | ||
| 8 | "gitbay.org/gitbay/internal/ci" | |
| 9 | "gitbay.org/gitbay/internal/codescan" | |
| 10 | "gitbay.org/gitbay/internal/config" | |
| 11 | "gitbay.org/gitbay/internal/protocol" | |
| 12 | ) | |
| 13 | ||
| 14 | const testSARIF = `{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"opengrep"}},"results":[ | |
| 15 | {"ruleId":"go.lang.security.audit.sqli","level":"error","message":{"text":"SQL built from input"}, | |
| 16 | "locations":[{"physicalLocation":{"artifactLocation":{"uri":"store/q.go"},"region":{"startLine":12}}}]}]}]}` | |
| 17 | ||
| 18 | // The whole round trip without a runner process: enable, queue, claim, | |
| 19 | // upload, and the issue that results in. | |
| 20 | func TestSecurityScanRoundTrip(t *testing.T) { | |
| 21 | st, repo, u, root := browseFixture(t) | |
| 22 | p := repo.Path() | |
| 23 | sec := config.Security{Image: "localhost/gitbay-security:1", Schedule: "0 4 * * *", Steps: config.DefaultSecuritySteps} | |
| 24 | ||
| 25 | c, out := runnerCtx(st, u.ID, root) | |
| 26 | if code := runSecurityEnable(c, []string{p}); code != protocol.ExitFailure || !strings.Contains(out.String(), "no code scanning image") { | |
| 27 | t.Fatalf("enable without an image: exit %d %s", code, out) | |
| 28 | } | |
| 29 | run := func(f func(*Ctx, []string) int, stdin string, args ...string) (int, string) { | |
| 30 | t.Helper() | |
| 31 | c, out := runnerCtx(st, u.ID, root) | |
| 32 | c.Cfg.Security = sec | |
| 33 | c.Cfg.Mail.SMTPHost = "localhost:587" | |
| 34 | c.Stdin = strings.NewReader(stdin) | |
| 35 | c.JSON = true | |
| 36 | return f(c, args), out.String() | |
| 37 | } | |
| 38 | if code, out := run(runSecurityEnable, "", p); code != protocol.ExitOK { | |
| 39 | t.Fatalf("enable: %d %s", code, out) | |
| 40 | } | |
| 41 | if code, out := run(runSecurityRun, "", p); code != protocol.ExitOK { | |
| 42 | t.Fatalf("run: %d %s", code, out) | |
| 43 | } | |
| 44 | if code, out := run(runSecurityRun, "", p); code == protocol.ExitOK || !strings.Contains(out, "already pending") { | |
| 45 | t.Fatalf("second run: %d %s", code, out) | |
| 46 | } | |
| 47 | ||
| 48 | st.SetBuildSecret(repo.ID, "DEPLOY_TOKEN", "s3cret") | |
| 49 | code, claim := run(runRunnerNext, "") | |
| 50 | if code != protocol.ExitOK || !strings.Contains(claim, `"job":"`+ci.SecurityJob+`"`) || !strings.Contains(claim, `"results":true`) { | |
| 51 | t.Fatalf("claim: %d %s", code, claim) | |
| 52 | } | |
| 53 | if strings.Contains(claim, "s3cret") { | |
| 54 | t.Fatalf("the scan was handed the repository's secrets: %s", claim) | |
| 55 | } | |
| 56 | var claimed struct { | |
| 57 | Data struct { | |
| 58 | ID int64 `json:"id"` | |
| 59 | SHA string `json:"sha"` | |
| 60 | } `json:"data"` | |
| 61 | } | |
| 62 | json.Unmarshal([]byte(claim), &claimed) | |
| 63 | id := claimed.Data.ID | |
| 64 | ||
| 65 | if code, out := run(runRunnerResults, testSARIF, itoa(id)); code != protocol.ExitOK { | |
| 66 | t.Fatalf("results: %d %s", code, out) | |
| 67 | } | |
| 68 | scan, _ := st.SecurityScanFor(repo.ID) | |
| 69 | if scan.IssueNumber == 0 || scan.LastResults == "" { | |
| 70 | t.Fatalf("scan = %+v", scan) | |
| 71 | } | |
| 72 | issue, _ := st.IssueByNumber(repo.ID, scan.IssueNumber) | |
| 73 | if issue.Title != codescan.IssueTitle || !strings.Contains(issue.Body, "`store/q.go:12` go.lang.security.audit.sqli (error)") { | |
| 74 | t.Errorf("issue %q:\n%s", issue.Title, issue.Body) | |
| 75 | } | |
| 76 | if code, out := run(runRunnerDone, "", itoa(id), "success"); code != protocol.ExitOK { | |
| 77 | t.Fatalf("done: %d %s", code, out) | |
| 78 | } | |
| 79 | if statuses, _ := st.CommitStatusesFor(repo.ID, []string{claimed.Data.SHA}); len(statuses[claimed.Data.SHA]) != 0 { | |
| 80 | t.Errorf("the scan set a commit status: %+v", statuses) | |
| 81 | } | |
| 82 | ||
| 83 | var status struct { | |
| 84 | Data SecurityOut `json:"data"` | |
| 85 | } | |
| 86 | json.Unmarshal([]byte(dispatchIn(t, st, u, root, "", "repo", "security", "status", p, "--json")), &status) | |
| 87 | if !status.Data.Enabled || len(status.Data.Findings) != 1 || status.Data.Findings[0].Line != 12 { | |
| 88 | t.Errorf("status = %+v", status.Data) | |
| 89 | } | |
| 90 | screen := atTerminalIn(t, st, u, root, "repo", "security", "status", p) | |
| 91 | if !strings.Contains(screen, "Findings (1)") || !strings.Contains(screen, "store/q.go:12") { | |
| 92 | t.Errorf("screen:\n%s", screen) | |
| 93 | } | |
| 94 | checkLegend(t, screen) | |
| 95 | ||
| 96 | // A clean scan closes the issue. | |
| 97 | run(runSecurityRun, "", p) | |
| 98 | _, claim = run(runRunnerNext, "") | |
| 99 | json.Unmarshal([]byte(claim), &claimed) | |
| 100 | if code, out := run(runRunnerResults, `{"version":"2.1.0","runs":[]}`, itoa(claimed.Data.ID)); code != protocol.ExitOK { | |
| 101 | t.Fatalf("clean results: %d %s", code, out) | |
| 102 | } | |
| 103 | if closed, _ := st.IssueByNumber(repo.ID, scan.IssueNumber); closed.State != "closed" { | |
| 104 | t.Errorf("issue state after a clean scan = %q", closed.State) | |
| 105 | } | |
| 106 | ||
| 107 | // A job of the repository's own may not upload. | |
| 108 | other, err := st.CreateBuild(repo.ID, "unit", claimed.Data.SHA, "main", "[]", "", "", true) | |
| 109 | if err != nil { | |
| 110 | t.Fatal(err) | |
| 111 | } | |
| 112 | if code, out := run(runRunnerResults, testSARIF, itoa(other)); code != protocol.ExitDenied { | |
| 113 | t.Errorf("results from another job: %d %s", code, out) | |
| 114 | } | |
| 115 | } | |
internal/deps/worker.go +17 −127
| @@ -7,10 +7,10 @@ import ( | ||
| 7 | 7 | "log/slog" |
| 8 | 8 | "os" |
| 9 | 9 | "sort" |
| 10 | "strings" | |
| 11 | 10 | "sync" |
| 12 | 11 | "time" |
| 13 | 12 | |
| 13 | "gitbay.org/gitbay/internal/botissue" | |
| 14 | 14 | "gitbay.org/gitbay/internal/config" |
| 15 | 15 | "gitbay.org/gitbay/internal/gitutil" |
| 16 | 16 | "gitbay.org/gitbay/internal/store" |
| @@ -189,14 +189,14 @@ func (w *Worker) reconcile(repo store.Repo, behind []store.DepReport) error { | ||
| 189 | 189 | if err != nil { |
| 190 | 190 | return err |
| 191 | 191 | } |
| 192 | return w.maintain(repo, tracked{ | |
| 193 | title: IssueTitle, | |
| 194 | number: check.IssueNumber, | |
| 195 | record: func(n int64) error { return w.St.SetDepIssue(repo.ID, n) }, | |
| 196 | empty: len(behind) == 0, | |
| 197 | changed: !same(previous, behind), | |
| 198 | save: func() error { return w.St.ReplaceDepReports(repo.ID, behind) }, | |
| 199 | body: func() string { return Body(repo.DefaultBranch, behind) }, | |
| 192 | return botissue.Maintain(w.St, w.Cfg, repo, botissue.Tracked{ | |
| 193 | Title: IssueTitle, | |
| 194 | Number: check.IssueNumber, | |
| 195 | Record: func(n int64) error { return w.St.SetDepIssue(repo.ID, n) }, | |
| 196 | Empty: len(behind) == 0, | |
| 197 | Changed: !same(previous, behind), | |
| 198 | Save: func() error { return w.St.ReplaceDepReports(repo.ID, behind) }, | |
| 199 | Body: func() string { return Body(repo.DefaultBranch, behind) }, | |
| 200 | 200 | }) |
| 201 | 201 | } |
| 202 | 202 | |
| @@ -211,123 +211,13 @@ func (w *Worker) reconcileAdvisories(repo store.Repo, matched []store.DepAdvisor | ||
| 211 | 211 | if err != nil { |
| 212 | 212 | return err |
| 213 | 213 | } |
| 214 | return w.maintain(repo, tracked{ | |
| 215 | title: AdvisoryTitle, | |
| 216 | number: check.AdvisoryIssueNumber, | |
| 217 | record: func(n int64) error { return w.St.SetDepAdvisoryIssue(repo.ID, n) }, | |
| 218 | empty: len(matched) == 0, | |
| 219 | changed: !sameAdvisories(previous, matched), | |
| 220 | save: func() error { return w.St.ReplaceDepAdvisories(repo.ID, matched) }, | |
| 221 | body: func() string { return AdvisoryBody(repo.DefaultBranch, matched) }, | |
| 214 | return botissue.Maintain(w.St, w.Cfg, repo, botissue.Tracked{ | |
| 215 | Title: AdvisoryTitle, | |
| 216 | Number: check.AdvisoryIssueNumber, | |
| 217 | Record: func(n int64) error { return w.St.SetDepAdvisoryIssue(repo.ID, n) }, | |
| 218 | Empty: len(matched) == 0, | |
| 219 | Changed: !sameAdvisories(previous, matched), | |
| 220 | Save: func() error { return w.St.ReplaceDepAdvisories(repo.ID, matched) }, | |
| 221 | Body: func() string { return AdvisoryBody(repo.DefaultBranch, matched) }, | |
| 222 | 222 | }) |
| 223 | 223 | } |
| 224 | ||
| 225 | // tracked is one issue the worker maintains for a repository and the set | |
| 226 | // it reports. | |
| 227 | type tracked struct { | |
| 228 | title string | |
| 229 | number int64 // the issue last opened, 0 for none | |
| 230 | record func(number int64) error | |
| 231 | empty bool // nothing to report | |
| 232 | changed bool // the set differs from the last report | |
| 233 | save func() error | |
| 234 | body func() string | |
| 235 | } | |
| 236 | ||
| 237 | // maintain opens the issue when something is first reported, rewrites it | |
| 238 | // when the set changes, and closes it when nothing is left. | |
| 239 | func (w *Worker) maintain(repo store.Repo, t tracked) error { | |
| 240 | issue, hasIssue := w.openIssue(repo, t.number) | |
| 241 | if t.empty { | |
| 242 | if hasIssue { | |
| 243 | w.St.SetIssueState(issue.ID, "closed") | |
| 244 | t.record(0) | |
| 245 | } | |
| 246 | if t.changed { | |
| 247 | return t.save() | |
| 248 | } | |
| 249 | return nil | |
| 250 | } | |
| 251 | // Nothing changed since the last report: leave it be, whether the issue | |
| 252 | // is still open or the maintainer has closed it. Reopening on an | |
| 253 | // unchanged set would make closing the issue pointless. | |
| 254 | if !t.changed { | |
| 255 | return nil | |
| 256 | } | |
| 257 | if err := t.save(); err != nil { | |
| 258 | return err | |
| 259 | } | |
| 260 | body := t.body() | |
| 261 | if hasIssue { | |
| 262 | if err := w.St.UpdateIssueText(issue.ID, nil, &body, nil); err != nil { | |
| 263 | return err | |
| 264 | } | |
| 265 | w.notify(repo, t.title, issue.Number, fmt.Sprintf("updated issue #%d", issue.Number), body) | |
| 266 | return nil | |
| 267 | } | |
| 268 | author, err := w.St.UserByUsername(store.BotUsername) | |
| 269 | if err != nil { | |
| 270 | // Migration 0028 leaves the account uncreated when the name was | |
| 271 | // already taken, which is the one case worth spelling out: the | |
| 272 | // feature is stuck until an operator frees the name. | |
| 273 | return fmt.Errorf("no %s account to author the issue (the name was taken when this instance upgraded): %w", | |
| 274 | store.BotUsername, err) | |
| 275 | } | |
| 276 | number, err := w.St.CreateIssue(repo.ID, author.ID, t.title, body, "md") | |
| 277 | if err != nil { | |
| 278 | return err | |
| 279 | } | |
| 280 | if err := t.record(number); err != nil { | |
| 281 | return err | |
| 282 | } | |
| 283 | w.St.RecordEvent(repo.ID, author.ID, "issue.created", fmt.Sprintf(`{"number":%d}`, number)) | |
| 284 | w.notify(repo, t.title, number, fmt.Sprintf("opened issue #%d", number), body) | |
| 285 | return nil | |
| 286 | } | |
| 287 | ||
| 288 | // openIssue loads the issue this worker maintains, if there still is one. | |
| 289 | // A closed issue counts as gone: reopening one the maintainer closed would | |
| 290 | // be arguing with them, so the next change opens a fresh issue. | |
| 291 | func (w *Worker) openIssue(repo store.Repo, number int64) (store.Issue, bool) { | |
| 292 | if number == 0 { | |
| 293 | return store.Issue{}, false | |
| 294 | } | |
| 295 | issue, err := w.St.IssueByNumber(repo.ID, number) | |
| 296 | if err != nil || issue.State != "open" { | |
| 297 | return store.Issue{}, false | |
| 298 | } | |
| 299 | return issue, true | |
| 300 | } | |
| 301 | ||
| 302 | // notify files an inbox row for the repo's owners and watchers and mails | |
| 303 | // them, the same targets and shape as an issue opened over SSH. | |
| 304 | // Best-effort, like every other notification. | |
| 305 | func (w *Worker) notify(repo store.Repo, title string, number int64, action, body string) { | |
| 306 | targets, err := w.St.RepoNotifyTargets(repo) | |
| 307 | if err != nil { | |
| 308 | return | |
| 309 | } | |
| 310 | author, err := w.St.UserByUsername(store.BotUsername) | |
| 311 | if err != nil { | |
| 312 | return | |
| 313 | } | |
| 314 | recipients, err := w.St.NotifyRecipients(repo.ID, author.ID, targets, true) | |
| 315 | if err != nil { | |
| 316 | return | |
| 317 | } | |
| 318 | subject := fmt.Sprintf("[%s] #%d: %s", repo.Path(), number, title) | |
| 319 | text := fmt.Sprintf("%s %s\n\n%s\n%s/%s/issues/%d\n", store.BotUsername, action, body, | |
| 320 | strings.TrimSuffix(w.Cfg.Server.SiteURL, "/"), repo.Path(), number) | |
| 321 | path := fmt.Sprintf("%s/issues/%d", repo.Path(), number) | |
| 322 | for _, id := range recipients { | |
| 323 | w.St.AddNotice(id, repo.ID, "issue", store.BotUsername, action, path) | |
| 324 | if w.Cfg.Mail.SMTPHost == "" { | |
| 325 | continue | |
| 326 | } | |
| 327 | email, err := w.St.ActivityMailAddress(id) | |
| 328 | if err != nil || email == "" { | |
| 329 | continue | |
| 330 | } | |
| 331 | w.St.EnqueueMail(email, subject, text) | |
| 332 | } | |
| 333 | } | |
internal/httpd/settings.go +21 −8
| @@ -22,14 +22,18 @@ type settingsPage struct { | ||
| 22 | 22 | Branches []gitutil.Ref |
| 23 | 23 | DepsEnabled bool |
| 24 | 24 | Deps control.DepsOut |
| 25 | Runners []store.RepoRunner | |
| 26 | Access []accessRow | |
| 27 | Hooks []hookRow | |
| 28 | Deliveries []deliveryRow | |
| 29 | Notice string | |
| 30 | Saved bool | |
| 31 | Reauth bool // Notice is the stale-session refusal: link to sign in | |
| 32 | Submitted map[string]string | |
| 25 | Security control.SecurityOut | |
| 26 | // ScanAvailable is whether the instance has a code scanning image; | |
| 27 | // without one the toggle would only ever be refused. | |
| 28 | ScanAvailable bool | |
| 29 | Runners []store.RepoRunner | |
| 30 | Access []accessRow | |
| 31 | Hooks []hookRow | |
| 32 | Deliveries []deliveryRow | |
| 33 | Notice string | |
| 34 | Saved bool | |
| 35 | Reauth bool // Notice is the stale-session refusal: link to sign in | |
| 36 | Submitted map[string]string | |
| 33 | 37 | } |
| 34 | 38 | |
| 35 | 39 | type accessRow struct { |
| @@ -79,6 +83,8 @@ func (s *Server) settingsFormWith(w http.ResponseWriter, r *http.Request, u stor | ||
| 79 | 83 | // prints (#164). |
| 80 | 84 | var deps control.DepsOut |
| 81 | 85 | s.runControlInto(u, []string{"repo", "deps", "status", repo.Path()}, &deps) |
| 86 | var security control.SecurityOut | |
| 87 | s.runControlInto(u, []string{"repo", "security", "status", repo.Path()}, &security) | |
| 82 | 88 | var runners []store.RepoRunner |
| 83 | 89 | s.runControlInto(u, []string{"repo", "runner", "list", repo.Path()}, &runners) |
| 84 | 90 | var access []accessRow |
| @@ -105,6 +111,7 @@ func (s *Server) settingsFormWith(w http.ResponseWriter, r *http.Request, u stor | ||
| 105 | 111 | s.render(w, "settings.html", settingsPage{ |
| 106 | 112 | repoPage: p, Topics: topics, Branches: branches, |
| 107 | 113 | DepsEnabled: deps.Enabled, Deps: deps, |
| 114 | Security: security, ScanAvailable: s.cfg.Security.Image != "", | |
| 108 | 115 | Runners: runners, |
| 109 | 116 | Access: access, Hooks: hooks, Deliveries: deliveries, |
| 110 | 117 | Notice: notice, |
| @@ -155,6 +162,12 @@ func (s *Server) settingsSubmit(w http.ResponseWriter, r *http.Request, u store. | ||
| 155 | 162 | argv = []string{"repo", "settings", "require-mr", repo, onOff(v("require-mr"))} |
| 156 | 163 | case "require-signed": |
| 157 | 164 | argv = []string{"repo", "settings", "require-signed", repo, onOff(v("require-signed"))} |
| 165 | case "security": | |
| 166 | verb := "disable" | |
| 167 | if v("security") == "on" { | |
| 168 | verb = "enable" | |
| 169 | } | |
| 170 | argv = []string{"repo", "security", verb, repo} | |
| 158 | 171 | case "secret-scan": |
| 159 | 172 | argv = []string{"repo", "settings", "secret-scan", repo, v("secret-scan")} |
| 160 | 173 | case "secret-scan-allow": |
internal/store/migrations/0081_security_scans.down.sql added +2
| @@ -0,0 +1,2 @@ | ||
| 1 | DROP TABLE security_findings; | |
| 2 | DROP TABLE security_scans; | |
internal/store/migrations/0081_security_scans.up.sql added +24
| @@ -0,0 +1,24 @@ | ||
| 1 | -- Code scanning: an opt-in, scheduled build of the instance's security | |
| 2 | -- job per repository, whose SARIF results the runner uploads. Its own | |
| 3 | -- schedule rather than a row in build_schedules, which a push replaces | |
| 4 | -- wholesale from .gitbay/ci.yml. | |
| 5 | CREATE TABLE security_scans ( | |
| 6 | repo_id INTEGER PRIMARY KEY REFERENCES repos(id) ON DELETE CASCADE, | |
| 7 | next_run TEXT NOT NULL DEFAULT '', | |
| 8 | last_results TEXT NOT NULL DEFAULT '', | |
| 9 | issue_number INTEGER NOT NULL DEFAULT 0 | |
| 10 | ); | |
| 11 | ||
| 12 | -- What each repo was last told about. The fingerprint leaves the line | |
| 13 | -- out, so a finding that moves is the same finding. | |
| 14 | CREATE TABLE security_findings ( | |
| 15 | repo_id INTEGER NOT NULL REFERENCES repos(id) ON DELETE CASCADE, | |
| 16 | fingerprint TEXT NOT NULL, | |
| 17 | tool TEXT NOT NULL, | |
| 18 | rule TEXT NOT NULL, | |
| 19 | level TEXT NOT NULL DEFAULT '', | |
| 20 | path TEXT NOT NULL, | |
| 21 | line INTEGER NOT NULL DEFAULT 0, | |
| 22 | message TEXT NOT NULL DEFAULT '', | |
| 23 | PRIMARY KEY (repo_id, fingerprint) | |
| 24 | ); | |
internal/store/security.go added +140
| @@ -0,0 +1,140 @@ | ||
| 1 | package store | |
| 2 | ||
| 3 | import ( | |
| 4 | "database/sql" | |
| 5 | "errors" | |
| 6 | ) | |
| 7 | ||
| 8 | // SecurityScan is a repo's code scanning state. A row exists only while | |
| 9 | // scanning is enabled. | |
| 10 | type SecurityScan struct { | |
| 11 | RepoID int64 | |
| 12 | NextRun string | |
| 13 | LastResults string // when results were last uploaded, '' for never | |
| 14 | IssueNumber int64 | |
| 15 | } | |
| 16 | ||
| 17 | // SecurityFinding is one result from a scan, as last reported. | |
| 18 | type SecurityFinding struct { | |
| 19 | Fingerprint string | |
| 20 | Tool string | |
| 21 | Rule string | |
| 22 | Level string | |
| 23 | Path string | |
| 24 | Line int | |
| 25 | Message string | |
| 26 | } | |
| 27 | ||
| 28 | func (s *Store) EnableSecurityScan(repoID int64, nextRun string) error { | |
| 29 | _, err := s.DB.Exec("INSERT OR IGNORE INTO security_scans (repo_id, next_run) VALUES (?, ?)", repoID, nextRun) | |
| 30 | return err | |
| 31 | } | |
| 32 | ||
| 33 | // DisableSecurityScan stops scanning and forgets what was reported. | |
| 34 | func (s *Store) DisableSecurityScan(repoID int64) error { | |
| 35 | tx, err := s.DB.Begin() | |
| 36 | if err != nil { | |
| 37 | return err | |
| 38 | } | |
| 39 | defer tx.Rollback() | |
| 40 | if _, err := tx.Exec("DELETE FROM security_findings WHERE repo_id = ?", repoID); err != nil { | |
| 41 | return err | |
| 42 | } | |
| 43 | if _, err := tx.Exec("DELETE FROM security_scans WHERE repo_id = ?", repoID); err != nil { | |
| 44 | return err | |
| 45 | } | |
| 46 | return tx.Commit() | |
| 47 | } | |
| 48 | ||
| 49 | func (s *Store) SecurityScanFor(repoID int64) (SecurityScan, error) { | |
| 50 | var sc SecurityScan | |
| 51 | err := s.DB.QueryRow( | |
| 52 | "SELECT repo_id, next_run, last_results, issue_number FROM security_scans WHERE repo_id = ?", repoID). | |
| 53 | Scan(&sc.RepoID, &sc.NextRun, &sc.LastResults, &sc.IssueNumber) | |
| 54 | if errors.Is(err, sql.ErrNoRows) { | |
| 55 | return sc, ErrNotFound | |
| 56 | } | |
| 57 | return sc, err | |
| 58 | } | |
| 59 | ||
| 60 | // DueSecurityScans returns the repos whose next scan is at or before now. | |
| 61 | // Archived repos are skipped: nobody is going to act on the findings. | |
| 62 | func (s *Store) DueSecurityScans(nowISO string) ([]Repo, error) { | |
| 63 | rows, err := s.DB.Query(repoSelect+` | |
| 64 | JOIN security_scans sc ON sc.repo_id = r.id | |
| 65 | WHERE sc.next_run <= ? | |
| 66 | ORDER BY sc.next_run, r.id`, nowISO) | |
| 67 | if err != nil { | |
| 68 | return nil, err | |
| 69 | } | |
| 70 | defer rows.Close() | |
| 71 | var out []Repo | |
| 72 | for rows.Next() { | |
| 73 | r, err := scanRepo(rows) | |
| 74 | if err != nil { | |
| 75 | return nil, err | |
| 76 | } | |
| 77 | if r.Settings.Archived { | |
| 78 | continue | |
| 79 | } | |
| 80 | out = append(out, r) | |
| 81 | } | |
| 82 | return out, rows.Err() | |
| 83 | } | |
| 84 | ||
| 85 | func (s *Store) SetSecurityNext(repoID int64, nextRun string) error { | |
| 86 | _, err := s.DB.Exec("UPDATE security_scans SET next_run = ? WHERE repo_id = ?", nextRun, repoID) | |
| 87 | return err | |
| 88 | } | |
| 89 | ||
| 90 | func (s *Store) SetSecurityIssue(repoID, number int64) error { | |
| 91 | _, err := s.DB.Exec("UPDATE security_scans SET issue_number = ? WHERE repo_id = ?", number, repoID) | |
| 92 | return err | |
| 93 | } | |
| 94 | ||
| 95 | func (s *Store) ReportedFindings(repoID int64) ([]SecurityFinding, error) { | |
| 96 | rows, err := s.DB.Query(` | |
| 97 | SELECT fingerprint, tool, rule, level, path, line, message FROM security_findings | |
| 98 | WHERE repo_id = ? ORDER BY fingerprint`, repoID) | |
| 99 | if err != nil { | |
| 100 | return nil, err | |
| 101 | } | |
| 102 | defer rows.Close() | |
| 103 | var out []SecurityFinding | |
| 104 | for rows.Next() { | |
| 105 | var f SecurityFinding | |
| 106 | if err := rows.Scan(&f.Fingerprint, &f.Tool, &f.Rule, &f.Level, &f.Path, &f.Line, &f.Message); err != nil { | |
| 107 | return nil, err | |
| 108 | } | |
| 109 | out = append(out, f) | |
| 110 | } | |
| 111 | return out, rows.Err() | |
| 112 | } | |
| 113 | ||
| 114 | // ReplaceFindings swaps in an upload's findings wholesale. | |
| 115 | func (s *Store) ReplaceFindings(repoID int64, findings []SecurityFinding) error { | |
| 116 | tx, err := s.DB.Begin() | |
| 117 | if err != nil { | |
| 118 | return err | |
| 119 | } | |
| 120 | defer tx.Rollback() | |
| 121 | if _, err := tx.Exec("DELETE FROM security_findings WHERE repo_id = ?", repoID); err != nil { | |
| 122 | return err | |
| 123 | } | |
| 124 | for _, f := range findings { | |
| 125 | if _, err := tx.Exec(` | |
| 126 | INSERT OR IGNORE INTO security_findings (repo_id, fingerprint, tool, rule, level, path, line, message) | |
| 127 | VALUES (?, ?, ?, ?, ?, ?, ?, ?)`, | |
| 128 | repoID, f.Fingerprint, f.Tool, f.Rule, f.Level, f.Path, f.Line, f.Message); err != nil { | |
| 129 | return err | |
| 130 | } | |
| 131 | } | |
| 132 | return tx.Commit() | |
| 133 | } | |
| 134 | ||
| 135 | // StampSecurityResults records that a scan's results arrived. | |
| 136 | func (s *Store) StampSecurityResults(repoID int64) error { | |
| 137 | _, err := s.DB.Exec(`UPDATE security_scans SET last_results = strftime('%Y-%m-%dT%H:%M:%fZ','now') | |
| 138 | WHERE repo_id = ?`, repoID) | |
| 139 | return err | |
| 140 | } | |
internal/web/templates/settings.html +25 −1
| @@ -14,6 +14,8 @@ | ||
| 14 | 14 | <li><a href="#access">Access</a></li> |
| 15 | 15 | <li><a href="#webhooks">Webhooks</a></li> |
| 16 | 16 | <li><a href="#gates">Merge gates</a></li> |
| 17 | {{if or .ScanAvailable .Security.Enabled}}<li><a href="#security">Code scanning</a></li>{{end}} | |
| 18 | <li><a href="#secrets">Secret scan</a></li> | |
| 17 | 19 | <li><a href="#branches">Protected branches</a></li> |
| 18 | 20 | <li><a href="#tags">Protected tags</a></li> |
| 19 | 21 | <li><a href="#deps">Dependencies</a></li> |
| @@ -171,7 +173,29 @@ | ||
| 171 | 173 | </form> |
| 172 | 174 | </section> |
| 173 | 175 | |
| 174 | <section id="secret-scan"><h2>Secret scan</h2> | |
| 176 | {{if or .ScanAvailable .Security.Enabled}}<section id="security"><h2>Code scanning</h2> | |
| 177 | <form method="post" action="{{$base}}" class="setform"> | |
| 178 | <input type="hidden" name="field" value="security"> | |
| 179 | <div><label for="security-enabled">Scheduled scan</label><p class="hint">Runs the instance's code scanner over <code>{{.Repo.DefaultBranch}}</code> on its schedule, on a runner attached to this repository, and keeps an issue of what it finds.</p></div> | |
| 180 | <div class="check"><input type="checkbox" id="security-enabled" name="security" value="on"{{if .Security.Enabled}} checked{{end}}></div> | |
| 181 | <div><button type="submit" class="btn">Save</button></div> | |
| 182 | </form> | |
| 183 | {{if .Security.Enabled}} | |
| 184 | <p class="meta">Next scan {{.Security.NextRun}} · last results {{if .Security.LastResults}}{{when .Security.LastResults}}{{else}}never{{end}}{{if .Security.IssueNumber}} · <a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/issues/{{.Security.IssueNumber}}">#{{.Security.IssueNumber}}</a>{{end}}</p> | |
| 185 | {{if .Security.Findings}}<div class="tablewrap"><table class="keys"> | |
| 186 | <tr class="cols"><th scope="col">location</th><th scope="col">rule</th><th scope="col">level</th></tr> | |
| 187 | {{range .Security.Findings}}<tr> | |
| 188 | <td class="mono">{{.Path}}:{{.Line}}</td> | |
| 189 | <td class="mono">{{.Rule}}</td> | |
| 190 | <td>{{.Level}}</td> | |
| 191 | </tr> | |
| 192 | {{end}}</table></div> | |
| 193 | {{else if .Security.LastResults}}<p class="none">The last scan found nothing.</p>{{end}} | |
| 194 | {{end}} | |
| 195 | </section> | |
| 196 | {{end}} | |
| 197 | ||
| 198 | <section id="secrets"><h2>Secret scan</h2> | |
| 175 | 199 | <form method="post" action="{{$base}}" class="setform"> |
| 176 | 200 | <input type="hidden" name="field" value="secret-scan"> |
| 177 | 201 | <div><label for="secret-scan">Pushes</label><p class="hint">Each push's new files are checked for credentials in known formats. Warn accepts the push and tells the pusher where; block refuses it. Only the path, line and kind of credential are reported, never the value.</p></div> |