audit-labs/audit-report

 1# Running audit-report in CI
 2
 3The pattern is always the same three steps:
 4
 51. **Collect** an evidence package with `audit-tools` (per-platform CLI).
 62. **Report** on it with `audit-report`, writing `md`/`html`/`json` artifacts.
 73. **Gate** the pipeline with `--fail-on` so a control regression can block a
 8   merge or page a scheduled run.
 9
10Ready-to-copy starting points:
11
12- [`examples/github-actions-audit.yml`](../examples/github-actions-audit.yml)
13- [`examples/gitlab-ci-audit.yml`](../examples/gitlab-ci-audit.yml)
14
15> `audit-tools` is not on PyPI — it is a set of per-platform scripts. Check the
16> repo out (or clone it) and run `applications/<platform>/audit.py` directly;
17> Python puts the script's own directory on `sys.path`, so no `cd` is needed.
18> Pass an absolute `--out` so every platform's package lands in one folder.
19> Everything downstream only needs the
20> `<out>/<platform>_audit_<subject>_<date>/` directory it writes.
21
22## Exact collection commands
23
24Install `audit-tools`' dependencies once (`pip install -r audit-tools/requirements.txt`),
25export the platform's credentials, then:
26
27```bash
28# AWS — credentials come from the standard AWS chain (env vars, profile, OIDC role)
29python audit-tools/applications/aws/audit.py --out "$PWD/output"
30
31# GitHub — needs GITHUB_TOKEN (read-only) in the environment
32python audit-tools/applications/github/audit.py --org "$ORG" --out "$PWD/output"
33
34# GitLab — needs GITLAB_TOKEN (read_api) in the environment
35python audit-tools/applications/gitlab/audit.py \
36  --group "$GROUP" --url "$API_V4_URL" --out "$PWD/output"
37```
38
39## Gating strategies
40
41**Snapshot gate — the current state must be clean.**
42
43```bash
44audit-report "$PKG" --fail-on high
45```
46
47Exits non-zero if any high-severity control is unsupported in the newest
48package. Simple and strict; good for a scheduled run that should stay green.
49
50**Regression gate — this change must not make things worse.**
51
52Keep the previous package in the repo (or restore it from an artifact) and diff
53against it. The build fails only when a control that used to pass now fails,
54which avoids blocking on pre-existing debt.
55
56```bash
57audit-report "$PKG" --baseline ./baseline/"$LAST_PKG" --fail-on high
58```
59
60**Trend artifact — show direction over time.**
61
62Point trend mode at a folder of retained packages to publish a heatmap of every
63control across dates. Its `--fail-on` reflects the latest package, so it can
64double as a snapshot gate while producing the timeline artifact.
65
66```bash
67audit-report ./history --trend --format html,json --out ./trend
68```
69
70## Exit codes
71
72| Code | Meaning |
73| --- | --- |
74| `0` | Ran successfully; no `--fail-on` threshold was breached. |
75| `1` | A finding (or, in diff mode, a regression) met the `--fail-on` severity. |
76| `2` | Usage or input error (missing package, unknown format, mismatched platforms). |
77
78Distinguishing `1` from `2` lets a workflow tell "the audit found a problem"
79(expected, actionable) from "the job is misconfigured" (fix the pipeline).
80
81## Retaining history
82
83`audit-report` never writes back to the package — it only reads. To build a
84trend or a regression baseline, archive each run's package directory (a CI
85artifact, a committed `history/` folder, or object storage) and feed the
86collection back in on the next run.