Commit 39dad8f75c

39dad8f75c1da3e44ca7c8bd1ac6cc229c7bfaa7

parent: 4158ea427f

Verified · cmc ci/build: success ci/quality: success ci/test: success

cmc <hello@cleberg.net> · 2026-10-31 06:51 UTC

Scheduled code scanning on the runners

repo security enable|disable|run|status schedules the instance's own
job, gitbay-security, per repository from the [security] config (image,
schedule, steps; off until an operator names an image). The scheduler
queues it from its own table, since a push replaces build_schedules
wholesale. The build gets no repository secrets and sets no commit
status. Its steps write SARIF into $GITBAY_RESULTS; the runner clears
that directory before the steps, uploads the merged SARIF with the new
runner results command when they succeed, and the server replaces the
repository's findings and keeps a "Code scanning findings" issue.

The issue lifecycle the deps worker used moves to internal/botissue so
both share it. internal/codescan reads SARIF and fingerprints findings
without the line, so a moved finding is not a new one.
deploy/Containerfile.security builds the image: Opengrep v1.31.0 and
opengrep-rules, both pinned by sha256. ci.yml checks the Qlty tarball
against its published sha256. Migration 0081.

Closes #340

Layout: unified · split

.gitbay/ci.yml +5 −4
@@ -21,12 +21,13 @@ jobs:
2121 - .gitbay/wiki/**
2222 # Static analysis through Qlty (.qlty/qlty.toml). Secrets and gofmt fail
2323 # the build. The full report, including radarlint-go's Sonar rules, is
24 # printed without failing it. The CLI is pinned, and unpacked with Python
25 # because this image has no xz.
24 # printed without failing it. The CLI is pinned and checked against the
25 # sha256 Qlty publishes beside it, and unpacked with Python because this
26 # image has no xz.
2627 quality:
2728 image: localhost/gitbay-ci:3
2829 steps:
29 - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')"
30 - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && echo "fa07eb51fbb32a0854113e81ce2d957b73f1c027c58b478d32c233b6aba7a977 /tmp/qlty.tar.xz" | sha256sum -c - && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')"
3031 - QLTY_TELEMETRY=off /tmp/qlty-x86_64-unknown-linux-gnu/qlty check --all --no-progress --filter trufflehog,gofmt
3132 - QLTY_TELEMETRY=off /tmp/qlty-x86_64-unknown-linux-gnu/qlty check --all --no-progress --no-fail
3233 paths-ignore:
@@ -51,5 +52,5 @@ jobs:
5152 - go run golang.org/x/vuln/cmd/govulncheck@latest ./...
5253 # osv-scanner also reports advisories in dependencies the code does not
5354 # call, which govulncheck skips.
54 - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')"
55 - curl -fsSL -o /tmp/qlty.tar.xz https://qlty-releases.s3.amazonaws.com/qlty/v0.661.0/qlty-x86_64-unknown-linux-gnu.tar.xz && echo "fa07eb51fbb32a0854113e81ce2d957b73f1c027c58b478d32c233b6aba7a977 /tmp/qlty.tar.xz" | sha256sum -c - && python3 -c "import tarfile; tarfile.open('/tmp/qlty.tar.xz').extractall('/tmp', filter='data')"
5556 - QLTY_TELEMETRY=off /tmp/qlty-x86_64-unknown-linux-gnu/qlty check --all --no-progress --filter osv-scanner
.gitbay/wiki/Admin.org +30 −1
@@ -454,6 +454,32 @@ push=.
454454 the worker logs an error at start and syncs no mirror, recording the
455455 reason on each. Sync ignores the system and global gitconfig.
456456
457** [security]
458The instance's code scan, which a repository admin turns on with
459=repo security enable=. Off until =image= is set.
460
461- =image= ("") — the image the scan runs in, provisioned on every
462 runner host that serves a scanned repository. Build it from
463 =deploy/Containerfile.security= (Opengrep and its rules, both pinned
464 by sha256); the file has the commands.
465- =schedule= ("0 4 * * *") — when each enabled repository's default
466 branch is scanned. =repo security run= queues one now.
467- =steps= — shell steps that write SARIF into =$GITBAY_RESULTS=. The
468 default runs Opengrep with the rules in the image at
469 =/opt/opengrep-rules=.
470
471The scan is a build of the reserved job =gitbay-security= on a runner
472attached to the repository. It gets none of the repository's secrets
473and sets no commit status. When its steps succeed the runner uploads
474the SARIF (=runner results=, gitbay-runner from this release on) and
475the findings replace the repository's last set; =gitbay-bot= keeps a
476"Code scanning findings" issue, opened, rewritten and closed the way
477the dependency issues are. An older runner runs the steps and uploads
478nothing, so the issue stays as it was.
479
480The bundled rules are =opengrep/opengrep-rules=, LGPL-2.1 with the
481Commons Clause: an instance that sells hosting should supply its own.
482
457483** [go_import]
458484Vanity Go module paths, one per line: ="host/module" = "owner/repo"=.
459485Requests with =?go-get=1= at or under the module path answer with the
@@ -1337,7 +1363,10 @@ trusts and refuses to do. Operational checklist:
13371363 tagging a release. CI's own =vuln= job runs =govulncheck= nightly
13381364 against main rather than per push, because =@latest= scans today's
13391365 advisory database and an advisory lands without anyone pushing;
1340 =build trigger krz/gitbay vuln= runs it on demand.
1366 =build trigger krz/gitbay vuln= runs it on demand. Repositories can
1367 also opt into the dependency advisory check (=repo deps enable=),
1368 the push-time secret scan (on by default as =warn=), and the
1369 scheduled code scan (=repo security enable=, see [security]).
13411370- *Web responses* carry a scripts-forbidden CSP, =X-Frame-Options:
13421371 DENY=, =nosniff=, =no-referrer=, and HSTS when TLS is on — no
13431372 configuration needed.
.gitbay/wiki/Architecture/02-Components.org +2
@@ -38,6 +38,8 @@ the hidden =hook= used by git (=cmd/gitbayd/main.go=,
3838| =internal/mailin=, =internal/imapc=, =internal/mailreply= | Reply by mail: the IMAP client, the reply token, and the processor that posts a reply through =issue comment= / =mr comment=. |
3939| =internal/deps= | Dependency manifest and lockfile parsing; registry and OSV checks (opt-in per repository). |
4040| =internal/leaks= | Credential rules and the blob scan pre-receive and =repo secret-scan= run. |
41| =internal/codescan= | SARIF from the code scan, and the findings issue's body. |
42| =internal/botissue= | The issues gitbay-bot keeps: open on first report, rewrite on change, close when empty. |
4143| =internal/config= | Configuration load and validation. |
4244| =internal/web= | Embedded templates, stylesheet and fonts. |
4345| =internal/protocol= | Exit codes, JSON envelope, argv tokenizer. |
.gitbay/wiki/Architecture/06-Data-and-Cryptography.org +2 −1
@@ -19,7 +19,8 @@ content (as confidential as the repository), *O* operational.
1919| Notifications | =notifications= (mail queue), =inbox=, =push_devices= (APNs token), =push_queue=, =mail_replies= (Message-IDs of posted replies) | P | device tokens sealed; looked up by SHA-256 |
2020| Signatures | =commit_signatures=, =settings.key_epoch= | O | verification cache |
2121| Audit and feed | =audit_log=, =events= | O, P | actor ids, pruned argv, fingerprints and IPs in some audit rows, a hash chain (=prev_hash=, =hash=) |
22| Dependencies | =dep_checks=, =dep_reports=, =dep_advisories= | O | |
22| Dependencies | =dep_checks=, =dep_reports=, =dep_advisories= | O |
23| Code scanning | =security_scans=, =security_findings= | O | |
2324
2425Outside the database:
2526
.gitbay/wiki/Parity.org +3
@@ -245,6 +245,9 @@ there shows the app's rendering rather than the one the web page shows.
245245| job image (ci.yml) | yes | n/a | n/a |
246246| secret scan mode, allowlist | yes | yes | yes |
247247| secret scan of the history | yes | no | yes |
248| code scanning on/off | yes | yes | yes |
249| code scanning status | yes | yes | yes |
250| code scan now | yes | no | yes |
248251| dependency checks on/off | yes | yes | yes |
249252| dependency status | yes | yes | yes |
250253| delete, transfer | yes | yes | yes |
.gitbay/wiki/Users.org +2
@@ -241,6 +241,8 @@ gitbay repo settings require-signed you/project on # every commit must verify
241241gitbay repo settings secret-scan you/project block # off | warn (default) | block
242242gitbay repo settings secret-scan-allow you/project 'testdata/**' # paths whose findings are ignored
243243gitbay repo secret-scan you/project # scan the whole history
244gitbay repo security enable you/project # scheduled code scan, where the instance offers one
245gitbay repo security status you/project # its findings, also kept in an issue
244246gitbay repo settings git-daemon you/project on # expose over git://
245247gitbay repo topics add you/project cli forge # free-form tags, shown on the web
246248gitbay repo search forge # find repos by name/description/topic
cmd/gitbay-runner/main.go +26 −1
@@ -49,6 +49,9 @@ type job struct {
4949 // over loopback takes its port for its builds (#260).
5050 SSH string `json:"ssh"`
5151 Secrets map[string]string `json:"secrets"`
52 // Results asks for the SARIF the steps leave in $GITBAY_RESULTS to be
53 // uploaded once they succeed. Only the instance's code scan sets it.
54 Results bool `json:"results"`
5255}
5356
5457type runner struct {
@@ -450,7 +453,29 @@ func (r *runner) run(j job) *failure {
450453 }
451454
452455 env := stepEnv(j, home, r.buildSSH(j.SSH))
453 return r.runSteps(j, dir, env, sink, deadline, runStep)
456 if !j.Results {
457 return r.runSteps(j, dir, env, sink, deadline, runStep)
458 }
459 // Cleared first: the checkout is the repository's, and a results
460 // directory committed to it would otherwise be uploaded as the scan's.
461 results := filepath.Join(dir, resultsDir)
462 if err := os.RemoveAll(results); err != nil {
463 fmt.Fprintf(sink, "preparing %s: %v\n", resultsDir, err)
464 return &failure{Reason: "preparing the results directory failed"}
465 }
466 inBuild := results
467 if r.isolation == isolationPodman {
468 inBuild = "/workspace/" + resultsDir
469 }
470 if f := r.runSteps(j, dir, append(env, "GITBAY_RESULTS="+inBuild), sink, deadline, runStep); f != nil {
471 return f
472 }
473 if err := r.uploadResults(j.ID, results); err != nil {
474 fmt.Fprintf(sink, "uploading results: %v\n", err)
475 return &failure{Reason: "uploading results failed"}
476 }
477 fmt.Fprintln(sink, "results uploaded")
478 return nil
454479}
455480
456481// buildHome is a build's HOME and what to do with it when the build ends.
cmd/gitbay-runner/results.go added +65
@@ -0,0 +1,65 @@
1package main
2
3import (
4 "bytes"
5 "encoding/json"
6 "errors"
7 "fmt"
8 "os"
9 "path/filepath"
10 "sort"
11)
12
13// resultsDir is where a code scan's steps write SARIF, relative to the
14// workspace.
15const resultsDir = ".gitbay-results"
16
17// maxResults matches the server's upload limit (codescan.MaxUpload).
18const maxResults = 8 << 20
19
20// mergeResults reads every .sarif file in dir and joins their runs into
21// one document, the shape runner results takes. Steps that wrote nothing
22// is an error: a scan that produced no results did not run.
23func mergeResults(dir string) ([]byte, error) {
24 names, err := filepath.Glob(filepath.Join(dir, "*.sarif"))
25 if err != nil {
26 return nil, err
27 }
28 if len(names) == 0 {
29 return nil, fmt.Errorf("no .sarif files in %s", resultsDir)
30 }
31 sort.Strings(names)
32 var runs []json.RawMessage
33 total := 0
34 for _, name := range names {
35 raw, err := os.ReadFile(name)
36 if err != nil {
37 return nil, err
38 }
39 if total += len(raw); total > maxResults {
40 return nil, fmt.Errorf("results are over %d bytes", maxResults)
41 }
42 var doc struct {
43 Runs []json.RawMessage `json:"runs"`
44 }
45 if err := json.Unmarshal(raw, &doc); err != nil {
46 return nil, fmt.Errorf("%s: %w", filepath.Base(name), err)
47 }
48 if doc.Runs == nil {
49 return nil, errors.New(filepath.Base(name) + ": no runs")
50 }
51 runs = append(runs, doc.Runs...)
52 }
53 return json.Marshal(map[string]any{"version": "2.1.0", "runs": runs})
54}
55
56// uploadResults sends the merged SARIF, retrying as reportDone does.
57func (r *runner) uploadResults(id int64, dir string) error {
58 doc, err := mergeResults(dir)
59 if err != nil {
60 return err
61 }
62 return reportWithRetry(func() (string, error) {
63 return r.ssh(bytes.NewReader(doc), "runner", "results", fmt.Sprint(id))
64 }, id, retryDelays)
65}
cmd/gitbay-runner/results_test.go added +36
@@ -0,0 +1,36 @@
1package main
2
3import (
4 "encoding/json"
5 "os"
6 "path/filepath"
7 "testing"
8)
9
10func TestMergeResults(t *testing.T) {
11 dir := t.TempDir()
12 if _, err := mergeResults(dir); err == nil {
13 t.Error("an empty results directory merged")
14 }
15 os.WriteFile(filepath.Join(dir, "a.sarif"), []byte(`{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"a"}}}]}`), 0o644)
16 os.WriteFile(filepath.Join(dir, "b.sarif"), []byte(`{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"b"}}}]}`), 0o644)
17 os.WriteFile(filepath.Join(dir, "notes.txt"), []byte(`ignored`), 0o644)
18 raw, err := mergeResults(dir)
19 if err != nil {
20 t.Fatal(err)
21 }
22 var doc struct {
23 Runs []struct {
24 Tool struct {
25 Driver struct{ Name string } `json:"driver"`
26 } `json:"tool"`
27 } `json:"runs"`
28 }
29 if err := json.Unmarshal(raw, &doc); err != nil || len(doc.Runs) != 2 || doc.Runs[1].Tool.Driver.Name != "b" {
30 t.Fatalf("merged %s: %v", raw, err)
31 }
32 os.WriteFile(filepath.Join(dir, "c.sarif"), []byte(`not json`), 0o644)
33 if _, err := mergeResults(dir); err == nil {
34 t.Error("a broken file merged")
35 }
36}
cmd/gitbay/coverage_test.go +1
@@ -16,6 +16,7 @@ var notInCLI = map[string]string{
1616 "runner next": "the CI runner's wire protocol, not for humans",
1717 "runner done": "the CI runner's wire protocol, not for humans",
1818 "runner log": "the CI runner's wire protocol, not for humans",
19 "runner results": "the CI runner's wire protocol, not for humans",
1920 "account import-bundle": "the wire format behind `gitbay migrate`",
2021}
2122
cmd/gitbay/main.go +6
@@ -597,6 +597,12 @@ func repoCmd() *cobra.Command {
597597 pass("sync", passOpts{server: []string{"repo", "mirror", "sync"}, needsRepo: true}),
598598 ),
599599 pass("secret-scan", passOpts{server: []string{"repo", "secret-scan"}, needsRepo: true}),
600 group("security", "the instance's scheduled code scan",
601 pass("enable", passOpts{server: []string{"repo", "security", "enable"}, needsRepo: true}),
602 pass("disable", passOpts{server: []string{"repo", "security", "disable"}, needsRepo: true}),
603 pass("run", passOpts{server: []string{"repo", "security", "run"}, needsRepo: true}),
604 pass("status", passOpts{server: []string{"repo", "security", "status"}, needsRepo: true}),
605 ),
600606 group("deps", "check dependencies against upstream registries and OSV",
601607 pass("enable", passOpts{server: []string{"repo", "deps", "enable"}, needsRepo: true}),
602608 pass("disable", passOpts{server: []string{"repo", "deps", "disable"}, needsRepo: true}),
cmd/gitbay/summaries_gen.go +5
@@ -197,6 +197,10 @@ var summaries = map[string]string{
197197 "repo secret remove": "remove a build secret",
198198 "repo secret set": "set a build secret",
199199 "repo secret-scan": "scan the whole history for credentials",
200 "repo security disable": "stop code scanning",
201 "repo security enable": "scan the default branch on the instance's code scanning schedule",
202 "repo security run": "queue a code scan of the default branch now",
203 "repo security status": "show code scanning state and the last scan's findings",
200204 "repo settings default-branch": "set the default branch",
201205 "repo settings description": "set the repository description",
202206 "repo settings git-daemon": "expose over git://",
@@ -231,6 +235,7 @@ var summaries = map[string]string{
231235 "runner done": "finish a build",
232236 "runner log": "append a build's log from stdin",
233237 "runner next": "claim the oldest pending build this key may run (runner protocol)",
238 "runner results": "upload a code scan's SARIF from stdin",
234239 "search": "find repositories, issues and merge requests across the instance",
235240 "snippet create": "create a snippet from one file on stdin",
236241 "snippet delete": "delete a snippet and its files",
cmd/gitbayd/main.go +1 −1
@@ -244,7 +244,7 @@ func serveCmd() *cobra.Command {
244244 }
245245 go purgeDeletions(whCtx, st, cfg)
246246 go sweep(whCtx, st, cfg)
247 go (&ci.Scheduler{St: st, SiteURL: cfg.Server.SiteURL,
247 go (&ci.Scheduler{St: st, SiteURL: cfg.Server.SiteURL, Security: cfg.Security,
248248 RepoDir: func(owner, name string) string {
249249 return control.RepoDir(cfg.Server.Root, owner, name)
250250 }}).Run(whCtx)
deploy/Containerfile.security added +56
@@ -0,0 +1,56 @@
1# The image the instance's code scan runs in ([security] image in
2# /etc/gitbay/config.toml, #340). Build it on each runner host that
3# serves a repository with `repo security enable`, the same way as
4# Containerfile.ci:
5#
6# scp -P 2222 deploy/Containerfile.security root@gitbay.org:/var/lib/gitbay-runner/gitbay-security.Containerfile
7# ssh -p 2222 root@gitbay.org 'chown ci-runner /var/lib/gitbay-runner/gitbay-security.Containerfile \
8# && su - ci-runner -s /bin/sh -c "podman build -t localhost/gitbay-security:1 -f gitbay-security.Containerfile ." \
9# && rm /var/lib/gitbay-runner/gitbay-security.Containerfile'
10#
11# Everything fetched is pinned to a version and checked against a sha256
12# taken when it was pinned, so a rebuild cannot pick up a different
13# binary or rule set. Bump both together and the tag with them.
14#
15# Opengrep v1.31.0 (2026-10-15), LGPL-2.1. The rules are
16# opengrep/opengrep-rules at its last commit, a fork of semgrep-rules
17# from before that project's licence change: LGPL-2.1 with the Commons
18# Clause, which forbids selling a service whose value is the rules. An
19# instance that charges for hosting should mount its own rules at
20# /opt/opengrep-rules instead.
21FROM docker.io/library/debian:trixie-slim
22
23LABEL org.gitbay.keep=true
24
25ARG OPENGREP_VERSION=v1.31.0
26ARG OPENGREP_SHA256=dbbe3c2058dbe600c449a8ba22ea79a1643a6660658400090d0323a8d0d31147
27ARG RULES_COMMIT=f1d2b562b414783763fd02a6ed2736eaed622efa
28ARG RULES_SHA256=9a5f1cd5c625418cc1c776120123e2d4371df9bb66e099426b17c3488e13619d
29
30RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
31 ca-certificates \
32 curl \
33 git \
34 && rm -rf /var/lib/apt/lists/*
35
36RUN curl -fsSL -o /usr/local/bin/opengrep \
37 "https://github.com/opengrep/opengrep/releases/download/${OPENGREP_VERSION}/opengrep_manylinux_x86" \
38 && echo "${OPENGREP_SHA256} /usr/local/bin/opengrep" | sha256sum -c - \
39 && chmod 0755 /usr/local/bin/opengrep
40
41# Only the language directories, and without the rules' own test inputs
42# (*.test.*, *.fixed.*), which would otherwise be read as rules or match
43# as findings.
44RUN curl -fsSL -o /tmp/rules.tar.gz \
45 "https://codeload.github.com/opengrep/opengrep-rules/tar.gz/${RULES_COMMIT}" \
46 && echo "${RULES_SHA256} /tmp/rules.tar.gz" | sha256sum -c - \
47 && mkdir -p /opt/opengrep-rules /tmp/rules \
48 && tar -xzf /tmp/rules.tar.gz -C /tmp/rules --strip-components=1 \
49 && for d in bash c dockerfile generic go java javascript kotlin php python ruby rust swift terraform typescript yaml; do \
50 cp -r "/tmp/rules/$d" /opt/opengrep-rules/; \
51 done \
52 && find /opt/opengrep-rules -type f \( -name '*.test.*' -o -name '*.fixed.*' \) -delete \
53 && find /opt/opengrep-rules -type f ! -name '*.yaml' ! -name '*.yml' -delete \
54 && rm -rf /tmp/rules /tmp/rules.tar.gz
55
56WORKDIR /workspace
e2e/codescan_test.go added +65
@@ -0,0 +1,65 @@
1package e2e
2
3import (
4 "os"
5 "path/filepath"
6 "strings"
7 "testing"
8)
9
10// The instance's code scan end to end: enable, queue, a real runner runs
11// the configured steps and uploads their SARIF, and the findings land in
12// an issue. A results directory committed to the repository is cleared
13// before the steps run, so it is never uploaded as the scan's (#340).
14func TestSecurityScan(t *testing.T) {
15 t.Parallel()
16 inst := startInstanceWith(t, `[security]
17image = "localhost/scan:1"
18steps = ['mkdir -p "$GITBAY_RESULTS"', 'cp scan.sarif "$GITBAY_RESULTS/scan.sarif"']
19`)
20 inst.runner = buildRunner(t)
21 aliceKey := inst.newKey(t, "alice")
22 inst.admin(t, "admin", "user", "create", "alice", "--key", aliceKey+".pub")
23 runnerKey := inst.newKey(t, "ci")
24 inst.admin(t, "admin", "user", "create", "ci", "--key", runnerKey+".pub", "--admin")
25 if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "create", "alice/app"); code != 0 {
26 t.Fatalf("repo create: %s", errOut)
27 }
28 work := t.TempDir()
29 env := inst.gitEnv(aliceKey)
30 mustGit(t, work, env, "clone", "-q", inst.sshURL("alice/app"), "w")
31 dir := filepath.Join(work, "w")
32 os.MkdirAll(filepath.Join(dir, ".gitbay-results"), 0o755)
33 os.WriteFile(filepath.Join(dir, "scan.sarif"), []byte(`{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"opengrep"}},
34 "results":[{"ruleId":"python.exec-used","level":"error","message":{"text":"exec on input"},
35 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"app.py"},"region":{"startLine":7}}}]}]}]}`), 0o644)
36 os.WriteFile(filepath.Join(dir, ".gitbay-results", "planted.sarif"), []byte(`{"runs":[{"tool":{"driver":{"name":"planted"}},
37 "results":[{"ruleId":"planted.rule","message":{"text":"planted"}}]}]}`), 0o644)
38 mustGit(t, dir, env, "checkout", "-q", "-b", "main")
39 mustGit(t, dir, env, "add", ".")
40 mustGit(t, dir, env, "commit", "-q", "-m", "base")
41 mustGit(t, dir, env, "push", "-q", "origin", "main")
42
43 if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "security", "enable", "alice/app"); code != 0 {
44 t.Fatalf("security enable: %s", errOut)
45 }
46 if _, errOut, code := inst.ssh(t, aliceKey, "", "repo", "security", "run", "alice/app"); code != 0 {
47 t.Fatalf("security run: %s", errOut)
48 }
49 if out := inst.runnerOnce(t, runnerKey); !strings.Contains(out, "success") {
50 t.Fatalf("runner:\n%s", out)
51 }
52 out, errOut, code := inst.ssh(t, aliceKey, "", "repo", "security", "status", "alice/app", "--json")
53 if code != 0 || !strings.Contains(out, `"rule":"python.exec-used"`) || strings.Contains(out, "planted") {
54 t.Fatalf("status: %d %s\n%s", code, errOut, out)
55 }
56 out, _, _ = inst.ssh(t, aliceKey, "", "issue", "list", "alice/app")
57 if !strings.Contains(out, "Code scanning findings") {
58 t.Fatalf("no findings issue:\n%s", out)
59 }
60 // The scan sets no commit status.
61 sha := strings.TrimSpace(mustGit(t, dir, env, "rev-parse", "HEAD"))
62 if out, _, _ := inst.ssh(t, aliceKey, "", "status", "list", "alice/app", sha); strings.Contains(out, "gitbay-security") {
63 t.Fatalf("scan set a status:\n%s", out)
64 }
65}
e2e/readonly_test.go +1
@@ -143,6 +143,7 @@ func TestReadOnlyCommandsWriteNothing(t *testing.T) {
143143 "repo domain list": {"alice/app"},
144144 "repo deps status": {"alice/app"},
145145 "repo secret-scan": {"alice/app"},
146 "repo security status": {"alice/app"},
146147 "status list": {"alice/app", sha},
147148 "issue list": {"alice/app"},
148149 "issue show": {"alice/app", "1"},
internal/botissue/botissue.go added +121
@@ -0,0 +1,121 @@
1// Package botissue keeps the issues gitbay-bot maintains for a repository:
2// dependency updates, dependency advisories, code scanning findings. Each
3// is one issue per repository, opened when something is first reported,
4// rewritten when the set changes, and closed when nothing is left.
5package botissue
6
7import (
8 "fmt"
9 "strings"
10
11 "gitbay.org/gitbay/internal/config"
12 "gitbay.org/gitbay/internal/store"
13)
14
15// Tracked is one issue and the set it reports.
16type Tracked struct {
17 Title string
18 Number int64 // the issue last opened, 0 for none
19 Record func(number int64) error
20 Empty bool // nothing to report
21 Changed bool // the set differs from the last report
22 Save func() error
23 Body func() string
24}
25
26// Maintain brings the issue in line with the set.
27func Maintain(st *store.Store, cfg config.Config, repo store.Repo, t Tracked) error {
28 issue, hasIssue := openIssue(st, repo, t.Number)
29 if t.Empty {
30 if hasIssue {
31 st.SetIssueState(issue.ID, "closed")
32 t.Record(0)
33 }
34 if t.Changed {
35 return t.Save()
36 }
37 return nil
38 }
39 // Nothing changed since the last report: leave it be, whether the issue
40 // is still open or the maintainer has closed it. Reopening on an
41 // unchanged set would make closing the issue pointless.
42 if !t.Changed {
43 return nil
44 }
45 if err := t.Save(); err != nil {
46 return err
47 }
48 body := t.Body()
49 if hasIssue {
50 if err := st.UpdateIssueText(issue.ID, nil, &body, nil); err != nil {
51 return err
52 }
53 notify(st, cfg, repo, t.Title, issue.Number, fmt.Sprintf("updated issue #%d", issue.Number), body)
54 return nil
55 }
56 author, err := st.UserByUsername(store.BotUsername)
57 if err != nil {
58 // Migration 0028 leaves the account uncreated when the name was
59 // already taken, which is the one case worth spelling out: the
60 // feature is stuck until an operator frees the name.
61 return fmt.Errorf("no %s account to author the issue (the name was taken when this instance upgraded): %w",
62 store.BotUsername, err)
63 }
64 number, err := st.CreateIssue(repo.ID, author.ID, t.Title, body, "md")
65 if err != nil {
66 return err
67 }
68 if err := t.Record(number); err != nil {
69 return err
70 }
71 st.RecordEvent(repo.ID, author.ID, "issue.created", fmt.Sprintf(`{"number":%d}`, number))
72 notify(st, cfg, repo, t.Title, number, fmt.Sprintf("opened issue #%d", number), body)
73 return nil
74}
75
76// openIssue loads the issue being maintained, if there still is one. A
77// closed issue counts as gone: reopening one the maintainer closed would be
78// arguing with them, so the next change opens a fresh issue.
79func openIssue(st *store.Store, repo store.Repo, number int64) (store.Issue, bool) {
80 if number == 0 {
81 return store.Issue{}, false
82 }
83 issue, err := st.IssueByNumber(repo.ID, number)
84 if err != nil || issue.State != "open" {
85 return store.Issue{}, false
86 }
87 return issue, true
88}
89
90// notify files an inbox row for the repo's owners and watchers and mails
91// them, the same targets and shape as an issue opened over SSH.
92// Best-effort, like every other notification.
93func notify(st *store.Store, cfg config.Config, repo store.Repo, title string, number int64, action, body string) {
94 targets, err := st.RepoNotifyTargets(repo)
95 if err != nil {
96 return
97 }
98 author, err := st.UserByUsername(store.BotUsername)
99 if err != nil {
100 return
101 }
102 recipients, err := st.NotifyRecipients(repo.ID, author.ID, targets, true)
103 if err != nil {
104 return
105 }
106 subject := fmt.Sprintf("[%s] #%d: %s", repo.Path(), number, title)
107 text := fmt.Sprintf("%s %s\n\n%s\n%s/%s/issues/%d\n", store.BotUsername, action, body,
108 strings.TrimSuffix(cfg.Server.SiteURL, "/"), repo.Path(), number)
109 path := fmt.Sprintf("%s/issues/%d", repo.Path(), number)
110 for _, id := range recipients {
111 st.AddNotice(id, repo.ID, "issue", store.BotUsername, action, path)
112 if cfg.Mail.SMTPHost == "" {
113 continue
114 }
115 email, err := st.ActivityMailAddress(id)
116 if err != nil || email == "" {
117 continue
118 }
119 st.EnqueueMail(email, subject, text)
120 }
121}
internal/ci/ci.go +8
@@ -21,6 +21,11 @@ import (
2121// ConfigPath is where the build configuration lives in a repository.
2222const ConfigPath = ".gitbay/ci.yml"
2323
24// SecurityJob is the instance's code scanning job, defined by the server's
25// [security] configuration rather than the repository's ci.yml. The name is
26// reserved so a repository's own job cannot be mistaken for it.
27const SecurityJob = "gitbay-security"
28
2429const (
2530 maxJobs = 10
2631 maxSteps = 50
@@ -79,6 +84,9 @@ func Parse(raw []byte) ([]Job, error) {
7984 if !jobName.MatchString(name) {
8085 return nil, fmt.Errorf("bad job name %q: lowercase letters, digits, - and _; max 40 chars", name)
8186 }
87 if name == SecurityJob {
88 return nil, fmt.Errorf("job name %q is reserved for the instance's code scan", name)
89 }
8290 if len(j.Steps) == 0 {
8391 return nil, fmt.Errorf("job %q has no steps", name)
8492 }
internal/ci/sched.go +49 −3
@@ -3,11 +3,13 @@ package ci
33import (
44 "context"
55 "encoding/json"
6 "errors"
67 "fmt"
78 "log/slog"
89 "os"
910 "time"
1011
12 "gitbay.org/gitbay/internal/config"
1113 "gitbay.org/gitbay/internal/gitutil"
1214 "gitbay.org/gitbay/internal/store"
1315)
@@ -30,10 +32,13 @@ func NextRun(expr string, after time.Time) string {
3032}
3133
3234// Scheduler fires scheduled builds. repoDir maps a repo to its bare path.
35// Security is the instance's code scanning job; with no image it never
36// queues.
3337type Scheduler struct {
34 St *store.Store
35 RepoDir func(owner, name string) string
36 SiteURL string
38 St *store.Store
39 RepoDir func(owner, name string) string
40 SiteURL string
41 Security config.Security
3742}
3843
3944// Run ticks until the context ends. GITBAY_SCHED_TICK overrides the
@@ -62,6 +67,7 @@ func (s *Scheduler) Run(ctx context.Context) {
6267// ticker for tests.
6368func (s *Scheduler) RunDue(now time.Time) {
6469 s.reapStale()
70 s.runSecurityDue(now)
6571 due, err := s.St.DueSchedules(isoNow(now))
6672 if err != nil {
6773 slog.Error("scheduler: listing due builds", "err", err)
@@ -121,6 +127,46 @@ func (s *Scheduler) RunDue(now time.Time) {
121127 }
122128}
123129
130// runSecurityDue queues the code scan for every repository whose next scan
131// is due, and advances it.
132func (s *Scheduler) runSecurityDue(now time.Time) {
133 if s.Security.Image == "" {
134 return
135 }
136 due, err := s.St.DueSecurityScans(isoNow(now))
137 if err != nil {
138 slog.Error("scheduler: listing due security scans", "err", err)
139 return
140 }
141 for _, repo := range due {
142 s.St.SetSecurityNext(repo.ID, NextRun(s.Security.Schedule, now))
143 if _, err := QueueSecurityScan(s.St, s.Security, s.RepoDir(repo.OwnerName, repo.Name), repo); err != nil {
144 slog.Warn("scheduler: queueing security scan", "repo", repo.Path(), "err", err)
145 }
146 }
147}
148
149// ErrScanPending is QueueSecurityScan finding the last scan of the
150// default branch's tip still pending or running.
151var ErrScanPending = errors.New("a scan of this commit is already pending or running")
152
153// QueueSecurityScan queues the code scan of the default branch's tip. It
154// sets no commit status: the scan reports through its issue, and a status
155// would put it in front of merge gates that never asked for it.
156func QueueSecurityScan(st *store.Store, sec config.Security, dir string, repo store.Repo) (int64, error) {
157 sha, err := gitutil.ResolveRef(dir, "refs/heads/"+repo.DefaultBranch)
158 if err != nil {
159 return 0, fmt.Errorf("%s has no commits on %s", repo.Path(), repo.DefaultBranch)
160 }
161 if built, err := st.BuildsForCommit(repo.ID, sha); err == nil {
162 if b, ok := built[SecurityJob]; ok && (b.Status == "pending" || b.Status == "running") {
163 return 0, ErrScanPending
164 }
165 }
166 steps, _ := json.Marshal(sec.Steps)
167 return st.CreateBuild(repo.ID, SecurityJob, sha, repo.DefaultBranch, string(steps), sec.Image, "", true)
168}
169
124170// reapStale resolves builds running past the deadline, so a runner that
125171// died mid-build leaves neither a build running nor a commit pending
126172// forever. It runs on the tick rather than on the next runner claim, so it
internal/ci/sched_test.go +68
@@ -4,9 +4,11 @@ import (
44 "os"
55 "os/exec"
66 "path/filepath"
7 "strings"
78 "testing"
89 "time"
910
11 "gitbay.org/gitbay/internal/config"
1012 "gitbay.org/gitbay/internal/store"
1113)
1214
@@ -136,3 +138,69 @@ func TestSchedulerRunDue(t *testing.T) {
136138 t.Fatalf("tick after the last build finished did not queue: %+v", builds)
137139 }
138140}
141
142// The code scan has its own schedule: a due repository gets one build of
143// the instance's job, with no commit status, and none while one is pending
144// or while the instance has no image.
145func TestSchedulerQueuesSecurityScan(t *testing.T) {
146 st, err := store.Open(":memory:")
147 if err != nil {
148 t.Fatal(err)
149 }
150 defer st.Close()
151 if err := st.MigrateUp(); err != nil {
152 t.Fatal(err)
153 }
154 uid, _ := st.CreateUser("alice", false)
155 repoID, _ := st.CreateRepo("user", uid, "app", "public")
156 work := t.TempDir()
157 bare := filepath.Join(work, "bare.git")
158 for _, args := range [][]string{
159 {"init", "-q", "-b", "main", "src"},
160 {"-C", "src", "-c", "user.name=t", "-c", "user.email=t@example.test", "commit", "-q", "--allow-empty", "-m", "base"},
161 {"clone", "-q", "--bare", "src", "bare.git"},
162 } {
163 cmd := exec.Command("git", args...)
164 cmd.Dir = work
165 cmd.Env = append(os.Environ(), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null")
166 if out, err := cmd.CombinedOutput(); err != nil {
167 t.Fatalf("git %v: %v\n%s", args, err, out)
168 }
169 }
170 now := time.Now()
171 if err := st.EnableSecurityScan(repoID, now.Add(-time.Minute).UTC().Format("2006-01-02T15:04:05Z")); err != nil {
172 t.Fatal(err)
173 }
174 s := &Scheduler{St: st, SiteURL: "https://x.test", RepoDir: func(owner, name string) string { return bare }}
175 s.RunDue(now)
176 if builds, _ := st.ListBuilds(repoID, store.BuildFilter{}, 10); len(builds) != 0 {
177 t.Fatalf("queued with no image: %+v", builds)
178 }
179
180 s.Security = config.Security{Image: "localhost/gitbay-security:1", Schedule: "0 4 * * *", Steps: []string{"true"}}
181 st.SetSecurityNext(repoID, now.Add(-time.Minute).UTC().Format("2006-01-02T15:04:05Z"))
182 s.RunDue(now)
183 builds, _ := st.ListBuilds(repoID, store.BuildFilter{}, 10)
184 if len(builds) != 1 || builds[0].Job != SecurityJob || builds[0].Image != "localhost/gitbay-security:1" {
185 t.Fatalf("builds = %+v", builds)
186 }
187 if statuses, _ := st.CommitStatusesFor(repoID, []string{builds[0].SHA}); len(statuses[builds[0].SHA]) != 0 {
188 t.Errorf("a status was set: %+v", statuses)
189 }
190 if sc, _ := st.SecurityScanFor(repoID); sc.NextRun <= now.UTC().Format("2006-01-02T15:04:05Z") {
191 t.Errorf("next run not advanced: %q", sc.NextRun)
192 }
193
194 // Due again while the first is pending: nothing more.
195 st.SetSecurityNext(repoID, now.Add(-time.Minute).UTC().Format("2006-01-02T15:04:05Z"))
196 s.RunDue(now)
197 if builds, _ := st.ListBuilds(repoID, store.BuildFilter{}, 10); len(builds) != 1 {
198 t.Errorf("queued over a pending scan: %d builds", len(builds))
199 }
200}
201
202func TestSecurityJobNameIsReserved(t *testing.T) {
203 if _, err := Parse([]byte("jobs:\n gitbay-security:\n steps: [echo]\n")); err == nil || !strings.Contains(err.Error(), "reserved") {
204 t.Fatalf("Parse = %v", err)
205 }
206}
internal/codescan/codescan.go added +192
@@ -0,0 +1,192 @@
1// Package codescan reads the SARIF a security build uploads and renders
2// the issue that reports it. Only the parts of SARIF 2.1.0 a finding needs
3// are read: the tool's name, and each result's rule, level, message and
4// first location.
5package codescan
6
7import (
8 "crypto/sha256"
9 "encoding/hex"
10 "encoding/json"
11 "errors"
12 "fmt"
13 "net/url"
14 "sort"
15 "strings"
16
17 "gitbay.org/gitbay/internal/store"
18)
19
20// IssueTitle is fixed so the issue is recognizable across scans.
21const IssueTitle = "Code scanning findings"
22
23// MaxUpload bounds one upload. MaxFindings bounds what is kept from it.
24const (
25 MaxUpload = 8 << 20
26 MaxFindings = 500
27)
28
29type sarif struct {
30 Runs []struct {
31 Tool struct {
32 Driver struct {
33 Name string `json:"name"`
34 } `json:"driver"`
35 } `json:"tool"`
36 Results []struct {
37 RuleID string `json:"ruleId"`
38 Level string `json:"level"`
39 Message struct {
40 Text string `json:"text"`
41 } `json:"message"`
42 Locations []struct {
43 PhysicalLocation struct {
44 ArtifactLocation struct {
45 URI string `json:"uri"`
46 } `json:"artifactLocation"`
47 Region struct {
48 StartLine int `json:"startLine"`
49 } `json:"region"`
50 } `json:"physicalLocation"`
51 } `json:"locations"`
52 } `json:"results"`
53 } `json:"runs"`
54}
55
56// Parse returns the findings in a SARIF document, deduplicated by
57// fingerprint and sorted by path and line. It reports whether results past
58// MaxFindings were dropped.
59func Parse(raw []byte) ([]store.SecurityFinding, bool, error) {
60 var doc sarif
61 if err := json.Unmarshal(raw, &doc); err != nil {
62 return nil, false, fmt.Errorf("not SARIF: %w", err)
63 }
64 if doc.Runs == nil {
65 return nil, false, errors.New("not SARIF: no runs")
66 }
67 seen := map[string]bool{}
68 var out []store.SecurityFinding
69 truncated := false
70 for _, run := range doc.Runs {
71 tool := clean(run.Tool.Driver.Name, 40)
72 for _, r := range run.Results {
73 if len(out) >= MaxFindings {
74 truncated = true
75 break
76 }
77 f := store.SecurityFinding{Tool: tool, Rule: clean(r.RuleID, 200),
78 Level: level(r.Level), Message: clean(r.Message.Text, 200)}
79 if len(r.Locations) > 0 {
80 loc := r.Locations[0].PhysicalLocation
81 f.Path = relPath(loc.ArtifactLocation.URI)
82 f.Line = max(loc.Region.StartLine, 0)
83 }
84 f.Fingerprint = fingerprint(f)
85 if seen[f.Fingerprint] {
86 continue
87 }
88 seen[f.Fingerprint] = true
89 out = append(out, f)
90 }
91 }
92 sort.Slice(out, func(i, j int) bool {
93 if out[i].Path != out[j].Path {
94 return out[i].Path < out[j].Path
95 }
96 if out[i].Line != out[j].Line {
97 return out[i].Line < out[j].Line
98 }
99 return out[i].Fingerprint < out[j].Fingerprint
100 })
101 return out, truncated, nil
102}
103
104// fingerprint identifies a finding across scans. The line is left out so
105// an edit above it does not make it a new finding.
106func fingerprint(f store.SecurityFinding) string {
107 sum := sha256.Sum256([]byte(f.Tool + "\x00" + f.Rule + "\x00" + f.Path + "\x00" + f.Message))
108 return hex.EncodeToString(sum[:12])
109}
110
111// level is SARIF's result level, which defaults to warning.
112func level(l string) string {
113 switch l {
114 case "error", "warning", "note", "none":
115 return l
116 }
117 return "warning"
118}
119
120// relPath turns an artifact URI into a repository path: tools write
121// `file:///workspace/x.go`, `./x.go` or `x.go` for the same file.
122func relPath(uri string) string {
123 p := uri
124 if u, err := url.Parse(uri); err == nil && u.Scheme == "file" {
125 p = u.Path
126 }
127 p = strings.TrimPrefix(p, "/workspace/")
128 p = strings.TrimPrefix(p, "./")
129 return clean(p, 300)
130}
131
132// clean is one line of at most n bytes of valid UTF-8, since every field
133// lands in an issue body and a mail.
134func clean(s string, n int) string {
135 s = strings.ToValidUTF8(strings.Join(strings.Fields(s), " "), "")
136 if len(s) > n {
137 s = strings.ToValidUTF8(s[:n], "") + "…"
138 }
139 return s
140}
141
142// Same reports whether two sets name the same findings, which decides
143// between leaving the issue alone and rewriting it. Both arrive sorted by
144// fingerprint. A line moving is not a change.
145func Same(a, b []store.SecurityFinding) bool {
146 if len(a) != len(b) {
147 return false
148 }
149 for i := range a {
150 if a[i].Fingerprint != b[i].Fingerprint {
151 return false
152 }
153 }
154 return true
155}
156
157// ByFingerprint sorts a set the way the store returns it, for Same.
158func ByFingerprint(fs []store.SecurityFinding) []store.SecurityFinding {
159 out := append([]store.SecurityFinding(nil), fs...)
160 sort.Slice(out, func(i, j int) bool { return out[i].Fingerprint < out[j].Fingerprint })
161 return out
162}
163
164// Body renders the issue: one list per tool, a line per finding.
165func Body(branch string, findings []store.SecurityFinding, truncated bool) string {
166 var b strings.Builder
167 fmt.Fprintf(&b, "The scheduled code scan of `%s` reported these findings. "+
168 "This issue is kept up to date as that changes, and closed once a scan reports none.\n", branch)
169 byTool := map[string][]store.SecurityFinding{}
170 var tools []string
171 for _, f := range findings {
172 if _, ok := byTool[f.Tool]; !ok {
173 tools = append(tools, f.Tool)
174 }
175 byTool[f.Tool] = append(byTool[f.Tool], f)
176 }
177 sort.Strings(tools)
178 for _, tool := range tools {
179 name := tool
180 if name == "" {
181 name = "Unnamed tool"
182 }
183 fmt.Fprintf(&b, "\n### %s\n\n", name)
184 for _, f := range byTool[tool] {
185 fmt.Fprintf(&b, "- `%s:%d` %s (%s): %s\n", f.Path, f.Line, f.Rule, f.Level, f.Message)
186 }
187 }
188 if truncated {
189 fmt.Fprintf(&b, "\nOnly the first %d results are listed.\n", MaxFindings)
190 }
191 return b.String()
192}
internal/codescan/codescan_test.go added +71
@@ -0,0 +1,71 @@
1package codescan
2
3import (
4 "strconv"
5 "strings"
6 "testing"
7)
8
9func TestParse(t *testing.T) {
10 raw := []byte(`{"version":"2.1.0","runs":[
11 {"tool":{"driver":{"name":"opengrep"}},"results":[
12 {"ruleId":"r1","message":{"text":"first\nline"},
13 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"file:///workspace/a/b.go"},"region":{"startLine":3}}}]},
14 {"ruleId":"r1","message":{"text":"first\nline"},
15 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"./a/b.go"},"region":{"startLine":3}}}]},
16 {"ruleId":"r2","level":"note","message":{"text":"second"},
17 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"a/a.go"},"region":{"startLine":9}}}]}
18 ]},
19 {"tool":{"driver":{"name":"other"}},"results":[{"ruleId":"x","level":"bogus","message":{"text":"no location"}}]}
20 ]}`)
21 got, truncated, err := Parse(raw)
22 if err != nil || truncated {
23 t.Fatalf("Parse: %v %v", err, truncated)
24 }
25 if len(got) != 3 {
26 t.Fatalf("findings = %+v", got)
27 }
28 // Sorted by path; the duplicate under two URI spellings folded; the
29 // default level applied; whitespace collapsed.
30 if got[0].Path != "" || got[0].Level != "warning" || got[0].Tool != "other" {
31 t.Errorf("got[0] = %+v", got[0])
32 }
33 if got[1].Path != "a/a.go" || got[1].Level != "note" {
34 t.Errorf("got[1] = %+v", got[1])
35 }
36 if got[2].Path != "a/b.go" || got[2].Line != 3 || got[2].Message != "first line" || got[2].Level != "warning" {
37 t.Errorf("got[2] = %+v", got[2])
38 }
39 if _, _, err := Parse([]byte(`{"nope":1}`)); err == nil {
40 t.Error("a document without runs parsed")
41 }
42}
43
44func TestParseTruncates(t *testing.T) {
45 var b strings.Builder
46 b.WriteString(`{"runs":[{"tool":{"driver":{"name":"t"}},"results":[`)
47 for i := 0; i < MaxFindings+5; i++ {
48 if i > 0 {
49 b.WriteString(",")
50 }
51 b.WriteString(`{"ruleId":"r","message":{"text":"m` + strings.Repeat("x", i%7) + itoa(i) + `"}}`)
52 }
53 b.WriteString(`]}]}`)
54 got, truncated, err := Parse([]byte(b.String()))
55 if err != nil || !truncated || len(got) != MaxFindings {
56 t.Fatalf("len %d truncated %v err %v", len(got), truncated, err)
57 }
58}
59
60func itoa(i int) string { return strconv.Itoa(i) }
61
62// A finding that moves is the same finding.
63func TestSameIgnoresLines(t *testing.T) {
64 a, _, _ := Parse([]byte(`{"runs":[{"tool":{"driver":{"name":"t"}},"results":[{"ruleId":"r","message":{"text":"m"},
65 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"x.go"},"region":{"startLine":1}}}]}]}]}`))
66 b, _, _ := Parse([]byte(`{"runs":[{"tool":{"driver":{"name":"t"}},"results":[{"ruleId":"r","message":{"text":"m"},
67 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"x.go"},"region":{"startLine":40}}}]}]}]}`))
68 if !Same(ByFingerprint(a), ByFingerprint(b)) {
69 t.Error("a moved finding counted as a change")
70 }
71}
internal/config/config.go +27
@@ -65,6 +65,7 @@ type Config struct {
6565 Mail Mail `toml:"mail"`
6666 Mirrors Mirrors `toml:"mirrors"`
6767 Deps Deps `toml:"deps"`
68 Security Security `toml:"security"`
6869 Retention Retention `toml:"retention"`
6970 Push Push `toml:"push"`
7071 Backup Backup `toml:"backup"`
@@ -226,6 +227,23 @@ type Deps struct {
226227 CheckIntervalHours int `toml:"check_interval_hours"`
227228}
228229
230// Security configures the built-in code scanning job that `repo security
231// enable` schedules. It is unavailable until an operator names Image, an
232// image built from deploy/Containerfile.security and provisioned on the
233// runner hosts. Steps write SARIF into $GITBAY_RESULTS; the runner uploads
234// it when they succeed.
235type Security struct {
236 Image string `toml:"image"`
237 Schedule string `toml:"schedule"`
238 Steps []string `toml:"steps"`
239}
240
241// DefaultSecuritySteps run Opengrep with the rules the image carries.
242var DefaultSecuritySteps = []string{
243 `mkdir -p "$GITBAY_RESULTS"`,
244 `opengrep scan --quiet --disable-version-check --config /opt/opengrep-rules --sarif-output="$GITBAY_RESULTS/opengrep.sarif" .`,
245}
246
229247type Limits struct {
230248 MaxPackBytes int64 `toml:"max_pack_bytes"`
231249 MaxBlobBytes int64 `toml:"max_blob_bytes"`
@@ -573,6 +591,7 @@ func Default() Config {
573591 GitDaemon: GitDaemon{Port: 9418},
574592 Mirrors: Mirrors{PullIntervalMinutes: 15},
575593 Deps: Deps{CheckIntervalHours: 24},
594 Security: Security{Schedule: "0 4 * * *", Steps: DefaultSecuritySteps},
576595 Limits: Limits{
577596 MaxPackBytes: 2 << 30, // 2 GiB
578597 MaxBlobBytes: 100 << 20,
@@ -656,6 +675,14 @@ func (c Config) Validate() error {
656675 if c.Server.Root == "" {
657676 errs = append(errs, errors.New("server.root is required"))
658677 }
678 if c.Security.Image != "" {
679 if len(strings.Fields(c.Security.Schedule)) != 5 {
680 errs = append(errs, errors.New("security.schedule must be a five-field cron expression"))
681 }
682 if len(c.Security.Steps) == 0 {
683 errs = append(errs, errors.New("security.steps must name at least one step when security.image is set"))
684 }
685 }
659686 if d := c.Pages.Domain; d != "" {
660687 if d == c.SiteHost() {
661688 errs = append(errs, errors.New("pages.domain must differ from the site host: pages serve repo-authored scripts, which must not run on the forge's origin"))
internal/control/build.go +14 −4
@@ -743,8 +743,10 @@ func runRunnerNext(c *Ctx, args []string) int {
743743 json.Unmarshal([]byte(b.Steps), &steps)
744744 // Secrets ride the claim: this channel is admin-only and the values
745745 // land in the build's environment, nowhere else.
746 // The code scan is the operator's job, and reads code rather than
747 // running it: it gets none of the repository's secrets.
746748 var secrets map[string]string
747 if b.Trusted {
749 if b.Trusted && b.Job != ci.SecurityJob {
748750 secrets, err = c.Store.BuildSecrets(b.RepoID)
749751 if err != nil {
750752 return c.fail(protocol.ExitFailure, "%v", err)
@@ -766,8 +768,13 @@ func runRunnerNext(c *Ctx, args []string) int {
766768 // over loopback takes its port for the build's GITBAY_SSH (#260).
767769 SSH string `json:"ssh,omitempty"`
768770 Secrets map[string]string `json:"secrets,omitempty"`
771 // Results asks the runner to upload the SARIF its steps leave in
772 // $GITBAY_RESULTS with runner results. An older runner ignores it,
773 // and the scan's issue stays as it was.
774 Results bool `json:"results,omitempty"`
769775 }{ID: b.ID, Repo: repo.Path(), Number: b.Number, Job: b.Job, SHA: b.SHA, Ref: b.Ref,
770 Steps: steps, Image: b.Image, Trusted: b.Trusted, SSH: publicSSH(c), Secrets: secrets}
776 Steps: steps, Image: b.Image, Trusted: b.Trusted, SSH: publicSSH(c), Secrets: secrets,
777 Results: b.Job == ci.SecurityJob}
771778 return c.emit(d, func(w io.Writer) {
772779 fmt.Fprintf(w, "build %d: %s %s @ %.10s\n", d.ID, d.Repo, d.Job, d.SHA)
773780 })
@@ -908,8 +915,11 @@ func runRunnerDone(c *Ctx, args []string) int {
908915 }
909916 url := fmt.Sprintf("%s/%s/builds/%d", c.Cfg.Server.SiteURL, repo.Path(), b.Number)
910917 desc := "build " + outcome
911 if err := c.Store.SetCommitStatus(repo.ID, b.SHA, "ci/"+b.Job, outcome, desc, url, c.User.ID); err != nil {
912 return c.fail(protocol.ExitFailure, "%v", err)
918 // The code scan reports through its issue; see ci.QueueSecurityScan.
919 if b.Job != ci.SecurityJob {
920 if err := c.Store.SetCommitStatus(repo.ID, b.SHA, "ci/"+b.Job, outcome, desc, url, c.User.ID); err != nil {
921 return c.fail(protocol.ExitFailure, "%v", err)
922 }
913923 }
914924 c.Store.RecordEvent(repo.ID, c.User.ID, "build."+outcome,
915925 fmt.Sprintf(`{"number":%d,"job":%q,"sha":%q}`, b.Number, b.Job, b.SHA))
internal/control/security.go added +279
@@ -0,0 +1,279 @@
1package control
2
3import (
4 "errors"
5 "fmt"
6 "io"
7 "strconv"
8 "time"
9
10 "gitbay.org/gitbay/internal/botissue"
11 "gitbay.org/gitbay/internal/ci"
12 "gitbay.org/gitbay/internal/codescan"
13 "gitbay.org/gitbay/internal/policy"
14 "gitbay.org/gitbay/internal/protocol"
15 "gitbay.org/gitbay/internal/store"
16)
17
18func init() {
19 // Code scanning: the instance's own job ([security] in the config),
20 // scheduled per repository that opts in, reporting through an issue.
21 // Opt-in because the build runs on a runner the repository is attached
22 // to, which is the owner's arrangement to make.
23 register(Command{Path: []string{"repo", "security", "enable"},
24 Summary: "scan the default branch on the instance's code scanning schedule",
25 Usage: "repo security enable <owner/name>",
26 Examples: []string{"repo security enable krz/gitbay"},
27 Run: runSecurityEnable})
28 register(Command{Path: []string{"repo", "security", "disable"},
29 Summary: "stop code scanning",
30 Usage: "repo security disable <owner/name>",
31 Examples: []string{"repo security disable krz/gitbay"},
32 Run: runSecurityDisable})
33 register(Command{Path: []string{"repo", "security", "run"},
34 Summary: "queue a code scan of the default branch now",
35 Usage: "repo security run <owner/name>",
36 Examples: []string{"repo security run krz/gitbay"},
37 Run: runSecurityRun})
38 register(Command{Path: []string{"repo", "security", "status"},
39 Summary: "show code scanning state and the last scan's findings",
40 Usage: "repo security status <owner/name>",
41 Examples: []string{"repo security status krz/gitbay"},
42 ReadOnly: true, Run: runSecurityStatus})
43 register(Command{Path: []string{"runner", "results"},
44 Summary: "upload a code scan's SARIF from stdin",
45 Usage: "runner results <build-id>",
46 Examples: []string{"runner results 431 < results.sarif"},
47 ReadsStdin: true, Run: runRunnerResults})
48}
49
50// SecurityOut is what `repo security status` emits.
51type SecurityOut struct {
52 Enabled bool `json:"enabled"`
53 NextRun string `json:"next_run,omitempty"`
54 LastResults string `json:"last_results,omitempty"`
55 IssueNumber int64 `json:"issue_number,omitempty"`
56 Findings []SecurityFinding `json:"findings"`
57}
58
59// SecurityFinding is one result of the last scan.
60type SecurityFinding struct {
61 Tool string `json:"tool"`
62 Rule string `json:"rule"`
63 Level string `json:"level"`
64 Path string `json:"path"`
65 Line int `json:"line"`
66 Message string `json:"message"`
67}
68
69func runSecurityEnable(c *Ctx, args []string) int {
70 if len(args) != 1 {
71 return c.usage()
72 }
73 repo, code := resolveRepo(c, args[0], policy.CanAdmin)
74 if code >= 0 {
75 return code
76 }
77 if c.Cfg.Security.Image == "" {
78 return c.fail(protocol.ExitFailure, "this instance has no code scanning image; an operator sets [security] image")
79 }
80 next := ci.NextRun(c.Cfg.Security.Schedule, time.Now())
81 if err := c.Store.EnableSecurityScan(repo.ID, next); err != nil {
82 return c.fail(protocol.ExitFailure, "%v", err)
83 }
84 return c.emit(map[string]any{"enabled": true, "next_run": next}, func(w io.Writer) {
85 fmt.Fprintf(w, "code scanning enabled on %s; first scan %s\n", repo.Path(), next)
86 })
87}
88
89func runSecurityDisable(c *Ctx, args []string) int {
90 if len(args) != 1 {
91 return c.usage()
92 }
93 repo, code := resolveRepo(c, args[0], policy.CanAdmin)
94 if code >= 0 {
95 return code
96 }
97 if err := c.Store.DisableSecurityScan(repo.ID); err != nil {
98 return c.fail(protocol.ExitFailure, "%v", err)
99 }
100 return c.emit(map[string]any{"enabled": false}, func(w io.Writer) {
101 fmt.Fprintf(w, "code scanning disabled on %s\n", repo.Path())
102 })
103}
104
105func runSecurityRun(c *Ctx, args []string) int {
106 if len(args) != 1 {
107 return c.usage()
108 }
109 repo, code := resolveRepo(c, args[0], policy.CanAdmin)
110 if code >= 0 {
111 return code
112 }
113 if _, err := c.Store.SecurityScanFor(repo.ID); errors.Is(err, store.ErrNotFound) {
114 return c.fail(protocol.ExitFailure, "code scanning is off on %s (repo security enable %s)", repo.Path(), repo.Path())
115 } else if err != nil {
116 return c.fail(protocol.ExitFailure, "%v", err)
117 }
118 if c.Cfg.Security.Image == "" {
119 return c.fail(protocol.ExitFailure, "this instance has no code scanning image; an operator sets [security] image")
120 }
121 n, err := ci.QueueSecurityScan(c.Store, c.Cfg.Security, RepoDir(c.Cfg.Server.Root, repo.OwnerName, repo.Name), repo)
122 if err != nil {
123 return c.fail(protocol.ExitFailure, "%v", err)
124 }
125 return c.emit(map[string]any{"build": n}, func(w io.Writer) {
126 fmt.Fprintf(w, "queued build %d: %s on %s\n", n, ci.SecurityJob, repo.Path())
127 })
128}
129
130func runSecurityStatus(c *Ctx, args []string) int {
131 if len(args) != 1 {
132 return c.usage()
133 }
134 repo, code := resolveRepo(c, args[0], policy.CanRead)
135 if code >= 0 {
136 return code
137 }
138 scan, err := c.Store.SecurityScanFor(repo.ID)
139 if errors.Is(err, store.ErrNotFound) {
140 return c.emitView(map[string]any{"enabled": false}, func(w io.Writer) {
141 c.view(w).fields("scanning", fmt.Sprintf("off (repo security enable %s)", repo.Path()))
142 }, func() screen {
143 return screen{
144 fields: []field{{"Scanning", []cell{cText("off")}}},
145 actions: []action{{"Security", []string{"repo", "security", "enable", repo.Path()}}},
146 }
147 })
148 }
149 if err != nil {
150 return c.fail(protocol.ExitFailure, "%v", err)
151 }
152 reported, err := c.Store.ReportedFindings(repo.ID)
153 if err != nil {
154 return c.fail(protocol.ExitFailure, "%v", err)
155 }
156 out := SecurityOut{Enabled: true, NextRun: scan.NextRun, LastResults: scan.LastResults,
157 IssueNumber: scan.IssueNumber, Findings: []SecurityFinding{}}
158 for _, f := range reported {
159 out.Findings = append(out.Findings, SecurityFinding{f.Tool, f.Rule, f.Level, f.Path, f.Line, f.Message})
160 }
161 return c.emitView(out, func(w io.Writer) {
162 tracked := ""
163 if scan.IssueNumber != 0 {
164 tracked = fmt.Sprintf("#%d", scan.IssueNumber)
165 }
166 v := c.view(w)
167 v.fields(
168 "scanning", "on",
169 "next scan", scan.NextRun,
170 "last results", c.when(scan.LastResults),
171 "tracked in", tracked,
172 )
173 if len(out.Findings) > 0 {
174 v.section("findings")
175 tb := c.table(w, "TOOL", "RULE", "LEVEL", "PATH", "LINE")
176 for _, f := range out.Findings {
177 tb.row(cText(f.Tool), cText(f.Rule), cText(f.Level), cRef(f.Path), cNum(int64(f.Line)))
178 }
179 tb.flush()
180 }
181 }, func() screen {
182 s := screen{fields: []field{
183 {"Scanning", []cell{cGlyph("ok"), cText("on")}},
184 {"Next scan", []cell{cText(scan.NextRun)}},
185 {"Last results", []cell{cAge(scan.LastResults)}},
186 }}
187 if scan.IssueNumber != 0 {
188 n := strconv.FormatInt(scan.IssueNumber, 10)
189 s.fields = append(s.fields, field{"Tracked in", []cell{cLink("#"+n, c.siteURL(repo.Path(), "issues", n))}})
190 }
191 found := section{title: "Findings", n: len(out.Findings)}
192 for _, f := range out.Findings {
193 found.rows = append(found.rows, rowOf(cRef(f.Path+":"+strconv.Itoa(f.Line)), cText(f.Rule), cMeta(f.Level)))
194 }
195 s.sections = []section{found}
196 s.actions = []action{{"Security", []string{"repo", "security", "run", repo.Path()}}}
197 return s
198 })
199}
200
201// runRunnerResults takes a security build's SARIF from the runner that
202// claimed it, replaces the repository's findings with it, and brings the
203// findings issue in line. Only the instance's own job may upload: its
204// steps are the operator's, where any other job's are the repository's.
205func runRunnerResults(c *Ctx, args []string) int {
206 key, code := runnerSession(c)
207 if code >= 0 {
208 return code
209 }
210 if len(args) != 1 {
211 return c.usage()
212 }
213 id, err := strconv.ParseInt(args[0], 10, 64)
214 if err != nil {
215 return c.fail(protocol.ExitUsage, "bad build id %q", args[0])
216 }
217 b, err := c.Store.BuildByID(id)
218 if err != nil {
219 return c.fail(protocol.ExitNotFound, "no build %d", id)
220 }
221 if ok, err := runnerMayBuild(c, key, b.RepoID); err != nil {
222 return c.fail(protocol.ExitFailure, "%v", err)
223 } else if !ok {
224 return c.fail(protocol.ExitDenied, "this key is not attached to the build's repository; a repository admin attaches it with repo runner add")
225 }
226 if b.Job != ci.SecurityJob {
227 return c.fail(protocol.ExitDenied, "results are accepted from the %s job only", ci.SecurityJob)
228 }
229 if b.Status != "running" {
230 return c.fail(protocol.ExitFailure, "build %d is %s", id, b.Status)
231 }
232 scan, err := c.Store.SecurityScanFor(b.RepoID)
233 if errors.Is(err, store.ErrNotFound) {
234 // Disabled while the build ran: nothing to report into.
235 return c.emit(map[string]any{"findings": 0}, func(w io.Writer) {})
236 }
237 if err != nil {
238 return c.fail(protocol.ExitFailure, "%v", err)
239 }
240 raw, err := io.ReadAll(io.LimitReader(c.Stdin, codescan.MaxUpload+1))
241 if err != nil {
242 return c.fail(protocol.ExitFailure, "reading results: %v", err)
243 }
244 if len(raw) > codescan.MaxUpload {
245 return c.fail(protocol.ExitFailure, "results are over %d bytes", codescan.MaxUpload)
246 }
247 findings, truncated, err := codescan.Parse(raw)
248 if err != nil {
249 return c.fail(protocol.ExitFailure, "%v", err)
250 }
251 repo, err := c.Store.RepoByID(b.RepoID)
252 if err != nil {
253 return c.fail(protocol.ExitFailure, "%v", err)
254 }
255 previous, err := c.Store.ReportedFindings(repo.ID)
256 if err != nil {
257 return c.fail(protocol.ExitFailure, "%v", err)
258 }
259 err = botissue.Maintain(c.Store, c.Cfg, repo, botissue.Tracked{
260 Title: codescan.IssueTitle,
261 Number: scan.IssueNumber,
262 Record: func(n int64) error { return c.Store.SetSecurityIssue(repo.ID, n) },
263 Empty: len(findings) == 0,
264 Changed: !codescan.Same(previous, codescan.ByFingerprint(findings)),
265 Save: func() error { return c.Store.ReplaceFindings(repo.ID, findings) },
266 Body: func() string { return codescan.Body(repo.DefaultBranch, findings, truncated) },
267 })
268 if err != nil {
269 return c.fail(protocol.ExitFailure, "%v", err)
270 }
271 // Stamped whether or not the set changed: the status says when results
272 // last arrived.
273 if err := c.Store.StampSecurityResults(repo.ID); err != nil {
274 return c.fail(protocol.ExitFailure, "%v", err)
275 }
276 return c.emit(map[string]any{"findings": len(findings), "truncated": truncated}, func(w io.Writer) {
277 fmt.Fprintf(w, "%d findings recorded for build %d\n", len(findings), b.Number)
278 })
279}
internal/control/security_test.go added +115
@@ -0,0 +1,115 @@
1package control
2
3import (
4 "encoding/json"
5 "strings"
6 "testing"
7
8 "gitbay.org/gitbay/internal/ci"
9 "gitbay.org/gitbay/internal/codescan"
10 "gitbay.org/gitbay/internal/config"
11 "gitbay.org/gitbay/internal/protocol"
12)
13
14const testSARIF = `{"version":"2.1.0","runs":[{"tool":{"driver":{"name":"opengrep"}},"results":[
15 {"ruleId":"go.lang.security.audit.sqli","level":"error","message":{"text":"SQL built from input"},
16 "locations":[{"physicalLocation":{"artifactLocation":{"uri":"store/q.go"},"region":{"startLine":12}}}]}]}]}`
17
18// The whole round trip without a runner process: enable, queue, claim,
19// upload, and the issue that results in.
20func TestSecurityScanRoundTrip(t *testing.T) {
21 st, repo, u, root := browseFixture(t)
22 p := repo.Path()
23 sec := config.Security{Image: "localhost/gitbay-security:1", Schedule: "0 4 * * *", Steps: config.DefaultSecuritySteps}
24
25 c, out := runnerCtx(st, u.ID, root)
26 if code := runSecurityEnable(c, []string{p}); code != protocol.ExitFailure || !strings.Contains(out.String(), "no code scanning image") {
27 t.Fatalf("enable without an image: exit %d %s", code, out)
28 }
29 run := func(f func(*Ctx, []string) int, stdin string, args ...string) (int, string) {
30 t.Helper()
31 c, out := runnerCtx(st, u.ID, root)
32 c.Cfg.Security = sec
33 c.Cfg.Mail.SMTPHost = "localhost:587"
34 c.Stdin = strings.NewReader(stdin)
35 c.JSON = true
36 return f(c, args), out.String()
37 }
38 if code, out := run(runSecurityEnable, "", p); code != protocol.ExitOK {
39 t.Fatalf("enable: %d %s", code, out)
40 }
41 if code, out := run(runSecurityRun, "", p); code != protocol.ExitOK {
42 t.Fatalf("run: %d %s", code, out)
43 }
44 if code, out := run(runSecurityRun, "", p); code == protocol.ExitOK || !strings.Contains(out, "already pending") {
45 t.Fatalf("second run: %d %s", code, out)
46 }
47
48 st.SetBuildSecret(repo.ID, "DEPLOY_TOKEN", "s3cret")
49 code, claim := run(runRunnerNext, "")
50 if code != protocol.ExitOK || !strings.Contains(claim, `"job":"`+ci.SecurityJob+`"`) || !strings.Contains(claim, `"results":true`) {
51 t.Fatalf("claim: %d %s", code, claim)
52 }
53 if strings.Contains(claim, "s3cret") {
54 t.Fatalf("the scan was handed the repository's secrets: %s", claim)
55 }
56 var claimed struct {
57 Data struct {
58 ID int64 `json:"id"`
59 SHA string `json:"sha"`
60 } `json:"data"`
61 }
62 json.Unmarshal([]byte(claim), &claimed)
63 id := claimed.Data.ID
64
65 if code, out := run(runRunnerResults, testSARIF, itoa(id)); code != protocol.ExitOK {
66 t.Fatalf("results: %d %s", code, out)
67 }
68 scan, _ := st.SecurityScanFor(repo.ID)
69 if scan.IssueNumber == 0 || scan.LastResults == "" {
70 t.Fatalf("scan = %+v", scan)
71 }
72 issue, _ := st.IssueByNumber(repo.ID, scan.IssueNumber)
73 if issue.Title != codescan.IssueTitle || !strings.Contains(issue.Body, "`store/q.go:12` go.lang.security.audit.sqli (error)") {
74 t.Errorf("issue %q:\n%s", issue.Title, issue.Body)
75 }
76 if code, out := run(runRunnerDone, "", itoa(id), "success"); code != protocol.ExitOK {
77 t.Fatalf("done: %d %s", code, out)
78 }
79 if statuses, _ := st.CommitStatusesFor(repo.ID, []string{claimed.Data.SHA}); len(statuses[claimed.Data.SHA]) != 0 {
80 t.Errorf("the scan set a commit status: %+v", statuses)
81 }
82
83 var status struct {
84 Data SecurityOut `json:"data"`
85 }
86 json.Unmarshal([]byte(dispatchIn(t, st, u, root, "", "repo", "security", "status", p, "--json")), &status)
87 if !status.Data.Enabled || len(status.Data.Findings) != 1 || status.Data.Findings[0].Line != 12 {
88 t.Errorf("status = %+v", status.Data)
89 }
90 screen := atTerminalIn(t, st, u, root, "repo", "security", "status", p)
91 if !strings.Contains(screen, "Findings (1)") || !strings.Contains(screen, "store/q.go:12") {
92 t.Errorf("screen:\n%s", screen)
93 }
94 checkLegend(t, screen)
95
96 // A clean scan closes the issue.
97 run(runSecurityRun, "", p)
98 _, claim = run(runRunnerNext, "")
99 json.Unmarshal([]byte(claim), &claimed)
100 if code, out := run(runRunnerResults, `{"version":"2.1.0","runs":[]}`, itoa(claimed.Data.ID)); code != protocol.ExitOK {
101 t.Fatalf("clean results: %d %s", code, out)
102 }
103 if closed, _ := st.IssueByNumber(repo.ID, scan.IssueNumber); closed.State != "closed" {
104 t.Errorf("issue state after a clean scan = %q", closed.State)
105 }
106
107 // A job of the repository's own may not upload.
108 other, err := st.CreateBuild(repo.ID, "unit", claimed.Data.SHA, "main", "[]", "", "", true)
109 if err != nil {
110 t.Fatal(err)
111 }
112 if code, out := run(runRunnerResults, testSARIF, itoa(other)); code != protocol.ExitDenied {
113 t.Errorf("results from another job: %d %s", code, out)
114 }
115}
internal/deps/worker.go +17 −127
@@ -7,10 +7,10 @@ import (
77 "log/slog"
88 "os"
99 "sort"
10 "strings"
1110 "sync"
1211 "time"
1312
13 "gitbay.org/gitbay/internal/botissue"
1414 "gitbay.org/gitbay/internal/config"
1515 "gitbay.org/gitbay/internal/gitutil"
1616 "gitbay.org/gitbay/internal/store"
@@ -189,14 +189,14 @@ func (w *Worker) reconcile(repo store.Repo, behind []store.DepReport) error {
189189 if err != nil {
190190 return err
191191 }
192 return w.maintain(repo, tracked{
193 title: IssueTitle,
194 number: check.IssueNumber,
195 record: func(n int64) error { return w.St.SetDepIssue(repo.ID, n) },
196 empty: len(behind) == 0,
197 changed: !same(previous, behind),
198 save: func() error { return w.St.ReplaceDepReports(repo.ID, behind) },
199 body: func() string { return Body(repo.DefaultBranch, behind) },
192 return botissue.Maintain(w.St, w.Cfg, repo, botissue.Tracked{
193 Title: IssueTitle,
194 Number: check.IssueNumber,
195 Record: func(n int64) error { return w.St.SetDepIssue(repo.ID, n) },
196 Empty: len(behind) == 0,
197 Changed: !same(previous, behind),
198 Save: func() error { return w.St.ReplaceDepReports(repo.ID, behind) },
199 Body: func() string { return Body(repo.DefaultBranch, behind) },
200200 })
201201}
202202
@@ -211,123 +211,13 @@ func (w *Worker) reconcileAdvisories(repo store.Repo, matched []store.DepAdvisor
211211 if err != nil {
212212 return err
213213 }
214 return w.maintain(repo, tracked{
215 title: AdvisoryTitle,
216 number: check.AdvisoryIssueNumber,
217 record: func(n int64) error { return w.St.SetDepAdvisoryIssue(repo.ID, n) },
218 empty: len(matched) == 0,
219 changed: !sameAdvisories(previous, matched),
220 save: func() error { return w.St.ReplaceDepAdvisories(repo.ID, matched) },
221 body: func() string { return AdvisoryBody(repo.DefaultBranch, matched) },
214 return botissue.Maintain(w.St, w.Cfg, repo, botissue.Tracked{
215 Title: AdvisoryTitle,
216 Number: check.AdvisoryIssueNumber,
217 Record: func(n int64) error { return w.St.SetDepAdvisoryIssue(repo.ID, n) },
218 Empty: len(matched) == 0,
219 Changed: !sameAdvisories(previous, matched),
220 Save: func() error { return w.St.ReplaceDepAdvisories(repo.ID, matched) },
221 Body: func() string { return AdvisoryBody(repo.DefaultBranch, matched) },
222222 })
223223}
224
225// tracked is one issue the worker maintains for a repository and the set
226// it reports.
227type tracked struct {
228 title string
229 number int64 // the issue last opened, 0 for none
230 record func(number int64) error
231 empty bool // nothing to report
232 changed bool // the set differs from the last report
233 save func() error
234 body func() string
235}
236
237// maintain opens the issue when something is first reported, rewrites it
238// when the set changes, and closes it when nothing is left.
239func (w *Worker) maintain(repo store.Repo, t tracked) error {
240 issue, hasIssue := w.openIssue(repo, t.number)
241 if t.empty {
242 if hasIssue {
243 w.St.SetIssueState(issue.ID, "closed")
244 t.record(0)
245 }
246 if t.changed {
247 return t.save()
248 }
249 return nil
250 }
251 // Nothing changed since the last report: leave it be, whether the issue
252 // is still open or the maintainer has closed it. Reopening on an
253 // unchanged set would make closing the issue pointless.
254 if !t.changed {
255 return nil
256 }
257 if err := t.save(); err != nil {
258 return err
259 }
260 body := t.body()
261 if hasIssue {
262 if err := w.St.UpdateIssueText(issue.ID, nil, &body, nil); err != nil {
263 return err
264 }
265 w.notify(repo, t.title, issue.Number, fmt.Sprintf("updated issue #%d", issue.Number), body)
266 return nil
267 }
268 author, err := w.St.UserByUsername(store.BotUsername)
269 if err != nil {
270 // Migration 0028 leaves the account uncreated when the name was
271 // already taken, which is the one case worth spelling out: the
272 // feature is stuck until an operator frees the name.
273 return fmt.Errorf("no %s account to author the issue (the name was taken when this instance upgraded): %w",
274 store.BotUsername, err)
275 }
276 number, err := w.St.CreateIssue(repo.ID, author.ID, t.title, body, "md")
277 if err != nil {
278 return err
279 }
280 if err := t.record(number); err != nil {
281 return err
282 }
283 w.St.RecordEvent(repo.ID, author.ID, "issue.created", fmt.Sprintf(`{"number":%d}`, number))
284 w.notify(repo, t.title, number, fmt.Sprintf("opened issue #%d", number), body)
285 return nil
286}
287
288// openIssue loads the issue this worker maintains, if there still is one.
289// A closed issue counts as gone: reopening one the maintainer closed would
290// be arguing with them, so the next change opens a fresh issue.
291func (w *Worker) openIssue(repo store.Repo, number int64) (store.Issue, bool) {
292 if number == 0 {
293 return store.Issue{}, false
294 }
295 issue, err := w.St.IssueByNumber(repo.ID, number)
296 if err != nil || issue.State != "open" {
297 return store.Issue{}, false
298 }
299 return issue, true
300}
301
302// notify files an inbox row for the repo's owners and watchers and mails
303// them, the same targets and shape as an issue opened over SSH.
304// Best-effort, like every other notification.
305func (w *Worker) notify(repo store.Repo, title string, number int64, action, body string) {
306 targets, err := w.St.RepoNotifyTargets(repo)
307 if err != nil {
308 return
309 }
310 author, err := w.St.UserByUsername(store.BotUsername)
311 if err != nil {
312 return
313 }
314 recipients, err := w.St.NotifyRecipients(repo.ID, author.ID, targets, true)
315 if err != nil {
316 return
317 }
318 subject := fmt.Sprintf("[%s] #%d: %s", repo.Path(), number, title)
319 text := fmt.Sprintf("%s %s\n\n%s\n%s/%s/issues/%d\n", store.BotUsername, action, body,
320 strings.TrimSuffix(w.Cfg.Server.SiteURL, "/"), repo.Path(), number)
321 path := fmt.Sprintf("%s/issues/%d", repo.Path(), number)
322 for _, id := range recipients {
323 w.St.AddNotice(id, repo.ID, "issue", store.BotUsername, action, path)
324 if w.Cfg.Mail.SMTPHost == "" {
325 continue
326 }
327 email, err := w.St.ActivityMailAddress(id)
328 if err != nil || email == "" {
329 continue
330 }
331 w.St.EnqueueMail(email, subject, text)
332 }
333}
internal/httpd/settings.go +21 −8
@@ -22,14 +22,18 @@ type settingsPage struct {
2222 Branches []gitutil.Ref
2323 DepsEnabled bool
2424 Deps control.DepsOut
25 Runners []store.RepoRunner
26 Access []accessRow
27 Hooks []hookRow
28 Deliveries []deliveryRow
29 Notice string
30 Saved bool
31 Reauth bool // Notice is the stale-session refusal: link to sign in
32 Submitted map[string]string
25 Security control.SecurityOut
26 // ScanAvailable is whether the instance has a code scanning image;
27 // without one the toggle would only ever be refused.
28 ScanAvailable bool
29 Runners []store.RepoRunner
30 Access []accessRow
31 Hooks []hookRow
32 Deliveries []deliveryRow
33 Notice string
34 Saved bool
35 Reauth bool // Notice is the stale-session refusal: link to sign in
36 Submitted map[string]string
3337}
3438
3539type accessRow struct {
@@ -79,6 +83,8 @@ func (s *Server) settingsFormWith(w http.ResponseWriter, r *http.Request, u stor
7983 // prints (#164).
8084 var deps control.DepsOut
8185 s.runControlInto(u, []string{"repo", "deps", "status", repo.Path()}, &deps)
86 var security control.SecurityOut
87 s.runControlInto(u, []string{"repo", "security", "status", repo.Path()}, &security)
8288 var runners []store.RepoRunner
8389 s.runControlInto(u, []string{"repo", "runner", "list", repo.Path()}, &runners)
8490 var access []accessRow
@@ -105,6 +111,7 @@ func (s *Server) settingsFormWith(w http.ResponseWriter, r *http.Request, u stor
105111 s.render(w, "settings.html", settingsPage{
106112 repoPage: p, Topics: topics, Branches: branches,
107113 DepsEnabled: deps.Enabled, Deps: deps,
114 Security: security, ScanAvailable: s.cfg.Security.Image != "",
108115 Runners: runners,
109116 Access: access, Hooks: hooks, Deliveries: deliveries,
110117 Notice: notice,
@@ -155,6 +162,12 @@ func (s *Server) settingsSubmit(w http.ResponseWriter, r *http.Request, u store.
155162 argv = []string{"repo", "settings", "require-mr", repo, onOff(v("require-mr"))}
156163 case "require-signed":
157164 argv = []string{"repo", "settings", "require-signed", repo, onOff(v("require-signed"))}
165 case "security":
166 verb := "disable"
167 if v("security") == "on" {
168 verb = "enable"
169 }
170 argv = []string{"repo", "security", verb, repo}
158171 case "secret-scan":
159172 argv = []string{"repo", "settings", "secret-scan", repo, v("secret-scan")}
160173 case "secret-scan-allow":
internal/store/migrations/0081_security_scans.down.sql added +2
@@ -0,0 +1,2 @@
1DROP TABLE security_findings;
2DROP TABLE security_scans;
internal/store/migrations/0081_security_scans.up.sql added +24
@@ -0,0 +1,24 @@
1-- Code scanning: an opt-in, scheduled build of the instance's security
2-- job per repository, whose SARIF results the runner uploads. Its own
3-- schedule rather than a row in build_schedules, which a push replaces
4-- wholesale from .gitbay/ci.yml.
5CREATE TABLE security_scans (
6 repo_id INTEGER PRIMARY KEY REFERENCES repos(id) ON DELETE CASCADE,
7 next_run TEXT NOT NULL DEFAULT '',
8 last_results TEXT NOT NULL DEFAULT '',
9 issue_number INTEGER NOT NULL DEFAULT 0
10);
11
12-- What each repo was last told about. The fingerprint leaves the line
13-- out, so a finding that moves is the same finding.
14CREATE TABLE security_findings (
15 repo_id INTEGER NOT NULL REFERENCES repos(id) ON DELETE CASCADE,
16 fingerprint TEXT NOT NULL,
17 tool TEXT NOT NULL,
18 rule TEXT NOT NULL,
19 level TEXT NOT NULL DEFAULT '',
20 path TEXT NOT NULL,
21 line INTEGER NOT NULL DEFAULT 0,
22 message TEXT NOT NULL DEFAULT '',
23 PRIMARY KEY (repo_id, fingerprint)
24);
internal/store/security.go added +140
@@ -0,0 +1,140 @@
1package store
2
3import (
4 "database/sql"
5 "errors"
6)
7
8// SecurityScan is a repo's code scanning state. A row exists only while
9// scanning is enabled.
10type SecurityScan struct {
11 RepoID int64
12 NextRun string
13 LastResults string // when results were last uploaded, '' for never
14 IssueNumber int64
15}
16
17// SecurityFinding is one result from a scan, as last reported.
18type SecurityFinding struct {
19 Fingerprint string
20 Tool string
21 Rule string
22 Level string
23 Path string
24 Line int
25 Message string
26}
27
28func (s *Store) EnableSecurityScan(repoID int64, nextRun string) error {
29 _, err := s.DB.Exec("INSERT OR IGNORE INTO security_scans (repo_id, next_run) VALUES (?, ?)", repoID, nextRun)
30 return err
31}
32
33// DisableSecurityScan stops scanning and forgets what was reported.
34func (s *Store) DisableSecurityScan(repoID int64) error {
35 tx, err := s.DB.Begin()
36 if err != nil {
37 return err
38 }
39 defer tx.Rollback()
40 if _, err := tx.Exec("DELETE FROM security_findings WHERE repo_id = ?", repoID); err != nil {
41 return err
42 }
43 if _, err := tx.Exec("DELETE FROM security_scans WHERE repo_id = ?", repoID); err != nil {
44 return err
45 }
46 return tx.Commit()
47}
48
49func (s *Store) SecurityScanFor(repoID int64) (SecurityScan, error) {
50 var sc SecurityScan
51 err := s.DB.QueryRow(
52 "SELECT repo_id, next_run, last_results, issue_number FROM security_scans WHERE repo_id = ?", repoID).
53 Scan(&sc.RepoID, &sc.NextRun, &sc.LastResults, &sc.IssueNumber)
54 if errors.Is(err, sql.ErrNoRows) {
55 return sc, ErrNotFound
56 }
57 return sc, err
58}
59
60// DueSecurityScans returns the repos whose next scan is at or before now.
61// Archived repos are skipped: nobody is going to act on the findings.
62func (s *Store) DueSecurityScans(nowISO string) ([]Repo, error) {
63 rows, err := s.DB.Query(repoSelect+`
64 JOIN security_scans sc ON sc.repo_id = r.id
65 WHERE sc.next_run <= ?
66 ORDER BY sc.next_run, r.id`, nowISO)
67 if err != nil {
68 return nil, err
69 }
70 defer rows.Close()
71 var out []Repo
72 for rows.Next() {
73 r, err := scanRepo(rows)
74 if err != nil {
75 return nil, err
76 }
77 if r.Settings.Archived {
78 continue
79 }
80 out = append(out, r)
81 }
82 return out, rows.Err()
83}
84
85func (s *Store) SetSecurityNext(repoID int64, nextRun string) error {
86 _, err := s.DB.Exec("UPDATE security_scans SET next_run = ? WHERE repo_id = ?", nextRun, repoID)
87 return err
88}
89
90func (s *Store) SetSecurityIssue(repoID, number int64) error {
91 _, err := s.DB.Exec("UPDATE security_scans SET issue_number = ? WHERE repo_id = ?", number, repoID)
92 return err
93}
94
95func (s *Store) ReportedFindings(repoID int64) ([]SecurityFinding, error) {
96 rows, err := s.DB.Query(`
97 SELECT fingerprint, tool, rule, level, path, line, message FROM security_findings
98 WHERE repo_id = ? ORDER BY fingerprint`, repoID)
99 if err != nil {
100 return nil, err
101 }
102 defer rows.Close()
103 var out []SecurityFinding
104 for rows.Next() {
105 var f SecurityFinding
106 if err := rows.Scan(&f.Fingerprint, &f.Tool, &f.Rule, &f.Level, &f.Path, &f.Line, &f.Message); err != nil {
107 return nil, err
108 }
109 out = append(out, f)
110 }
111 return out, rows.Err()
112}
113
114// ReplaceFindings swaps in an upload's findings wholesale.
115func (s *Store) ReplaceFindings(repoID int64, findings []SecurityFinding) error {
116 tx, err := s.DB.Begin()
117 if err != nil {
118 return err
119 }
120 defer tx.Rollback()
121 if _, err := tx.Exec("DELETE FROM security_findings WHERE repo_id = ?", repoID); err != nil {
122 return err
123 }
124 for _, f := range findings {
125 if _, err := tx.Exec(`
126 INSERT OR IGNORE INTO security_findings (repo_id, fingerprint, tool, rule, level, path, line, message)
127 VALUES (?, ?, ?, ?, ?, ?, ?, ?)`,
128 repoID, f.Fingerprint, f.Tool, f.Rule, f.Level, f.Path, f.Line, f.Message); err != nil {
129 return err
130 }
131 }
132 return tx.Commit()
133}
134
135// StampSecurityResults records that a scan's results arrived.
136func (s *Store) StampSecurityResults(repoID int64) error {
137 _, err := s.DB.Exec(`UPDATE security_scans SET last_results = strftime('%Y-%m-%dT%H:%M:%fZ','now')
138 WHERE repo_id = ?`, repoID)
139 return err
140}
internal/web/templates/settings.html +25 −1
@@ -14,6 +14,8 @@
1414 <li><a href="#access">Access</a></li>
1515 <li><a href="#webhooks">Webhooks</a></li>
1616 <li><a href="#gates">Merge gates</a></li>
17 {{if or .ScanAvailable .Security.Enabled}}<li><a href="#security">Code scanning</a></li>{{end}}
18 <li><a href="#secrets">Secret scan</a></li>
1719 <li><a href="#branches">Protected branches</a></li>
1820 <li><a href="#tags">Protected tags</a></li>
1921 <li><a href="#deps">Dependencies</a></li>
@@ -171,7 +173,29 @@
171173</form>
172174</section>
173175
174<section id="secret-scan"><h2>Secret scan</h2>
176{{if or .ScanAvailable .Security.Enabled}}<section id="security"><h2>Code scanning</h2>
177<form method="post" action="{{$base}}" class="setform">
178 <input type="hidden" name="field" value="security">
179 <div><label for="security-enabled">Scheduled scan</label><p class="hint">Runs the instance's code scanner over <code>{{.Repo.DefaultBranch}}</code> on its schedule, on a runner attached to this repository, and keeps an issue of what it finds.</p></div>
180 <div class="check"><input type="checkbox" id="security-enabled" name="security" value="on"{{if .Security.Enabled}} checked{{end}}></div>
181 <div><button type="submit" class="btn">Save</button></div>
182</form>
183{{if .Security.Enabled}}
184<p class="meta">Next scan {{.Security.NextRun}} · last results {{if .Security.LastResults}}{{when .Security.LastResults}}{{else}}never{{end}}{{if .Security.IssueNumber}} · <a href="/{{.Repo.OwnerName}}/{{.Repo.Name}}/issues/{{.Security.IssueNumber}}">#{{.Security.IssueNumber}}</a>{{end}}</p>
185{{if .Security.Findings}}<div class="tablewrap"><table class="keys">
186<tr class="cols"><th scope="col">location</th><th scope="col">rule</th><th scope="col">level</th></tr>
187{{range .Security.Findings}}<tr>
188 <td class="mono">{{.Path}}:{{.Line}}</td>
189 <td class="mono">{{.Rule}}</td>
190 <td>{{.Level}}</td>
191</tr>
192{{end}}</table></div>
193{{else if .Security.LastResults}}<p class="none">The last scan found nothing.</p>{{end}}
194{{end}}
195</section>
196{{end}}
197
198<section id="secrets"><h2>Secret scan</h2>
175199<form method="post" action="{{$base}}" class="setform">
176200 <input type="hidden" name="field" value="secret-scan">
177201 <div><label for="secret-scan">Pushes</label><p class="hint">Each push's new files are checked for credentials in known formats. Warn accepts the push and tells the pusher where; block refuses it. Only the path, line and kind of credential are reported, never the value.</p></div>