audit-labs/control-coverage

  1# Using control-coverage
  2
  3A practical, task-oriented guide. For the conceptual overview see the
  4[README](README.md); this walks through actually running the tool.
  5
  6## The one thing to understand first
  7
  8Every other Audit Labs tool is **evidence-first**: it starts from what you collected
  9and tells you what it maps to. `control-coverage` is **control-first**: it starts
 10from the *complete* list of a framework's controls and tells you how much of it your
 11evidence addresses — and, more usefully, what it *doesn't*.
 12
 13So the input is your evidence, and the output is measured against a fixed yardstick
 14(the framework catalog) you didn't have to write.
 15
 16## 1. Get the input: audit-report JSON
 17
 18The corpus is one or more JSON reports from `audit-report`. Produce them with its
 19`--format json` flag, one per platform:
 20
 21```bash
 22audit-report ./output/aws_audit_prod_2026-02-01     --format json --out reports/
 23audit-report ./output/github_audit_prod_2026-02-01  --format json --out reports/
 24```
 25
 26You now have `reports/*.json`. That directory *is* a corpus — coverage aggregates
 27every report in it into one per-framework picture, so AWS, GitHub, and GitLab
 28evidence all count toward the same SOC 2 number.
 29
 30> No audit-report packages yet? Any JSON with the same shape works — a list of
 31> `findings`, each with `controls: ["SOC2:CC6.1", ...]` and a `status` of `pass`,
 32> `fail`, or `not_applicable`.
 33
 34## 2. Install
 35
 36```bash
 37git clone https://github.com/audit-labs/control-coverage
 38cd control-coverage
 39python -m venv .venv && source .venv/bin/activate
 40pip install -e .
 41```
 42
 43## 3. The four things you'll actually do
 44
 45### A. "How covered am I, and what am I missing?"
 46
 47```bash
 48control-coverage reports/
 49```
 50
 51Prints a Markdown report: a per-framework summary table, then the **blind spots**
 52(in-scope controls no finding touches), then the full matrix. Frameworks are inferred
 53from the codes your corpus cites.
 54
 55Want just the gap list, nothing else?
 56
 57```bash
 58control-coverage reports/ --framework SOC2 --blind-spots
 59```
 60
 61Want files to hand off? Write all formats to a directory:
 62
 63```bash
 64control-coverage reports/ --format md,html,json --out coverage-out/
 65```
 66
 67- **md** — human-readable, good for a PR comment or a wiki paste.
 68- **html** — self-contained, printable, has coverage bars. Attach to a workpaper.
 69- **json** — for dashboards or further scripting.
 70
 71### B. "Produce a Statement of Applicability"
 72
 73First write a scope file. It selects frameworks and records exclusions — each with a
 74mandatory reason (an unjustified exclusion is rejected):
 75
 76```yaml
 77# soa.yaml
 78subject: Acme Production
 79frameworks: [SOC2, ISO]
 80exclusions:
 81  - control: ISO:A.7.1
 82    reason: "Fully cloud-hosted; no physical premises are in scope for the ISMS."
 83  - control: ISO:A.5.7
 84    reason: "No formal threat-intelligence program; risk accepted by the CISO for 2026."
 85exclude_families:
 86  - {framework: SOC2, family: Privacy, reason: "Privacy category not in the SOC 2 audit scope."}
 87owners:
 88  SOC2:CC6.1: platform-team
 89```
 90
 91`exclusions` drops one control; `exclude_families` drops a whole category, ISO theme,
 92or NIST family at once (how audit scope is really decided). Both require a reason.
 93
 94Then generate coverage over the in-scope controls, plus the SoA itself:
 95
 96```bash
 97control-coverage reports/ --scope soa.yaml --format md,soa --out coverage-out/
 98```
 99
100`coverage-out/soa.md` lists every control, whether it applies, its implementation
101status (derived from your evidence, not asserted by hand), and the justification.
102Excluded controls are recorded, not counted as gaps.
103
104### C. "What changed since last time?"
105
106Keep last month's reports around. Point `--baseline` at them:
107
108```bash
109control-coverage reports/2026-02/ --baseline reports/2026-01/ --framework SOC2
110```
111
112You get a movement report:
113
114- **improved** — a control got more assurance (e.g. failing → supported).
115- **regressed** — a control lost assurance (e.g. supported → failing).
116- **gained** — a blind spot became addressed (coverage went up).
117- **lost** — an addressed control became a blind spot (coverage went down).
118
119`--baseline` accepts a single file or a directory.
120
121### D. "Which evidence is doing the most work?"
122
123```bash
124control-coverage reports/ --framework SOC2,ISO,NIST --crosswalk
125```
126
127Two things come out:
128
129- **Evidence leverage** — each check and the controls it supports, across all three
130  frameworks. You'll see that one 2FA check earns SOC 2 CC6.1 + ISO A.5.17 + NIST IA-2.
131- **Minimal evidence set** — the fewest checks that still cover every addressed
132  control. This is your walkthrough/sampling short-list: pull these and you've touched
133  everything the full corpus touches.
134
135## 4. Reading the numbers
136
137Every in-scope control is in exactly one state:
138
139| State | What it means | Counts toward… |
140| --- | --- | --- |
141| **supported** | Something passes here, nothing fails | coverage **and** assured |
142| **failing** | Something fails here (worst wins) | coverage |
143| **asserted** | Mapped, but the data was absent | coverage |
144| **unaddressed** | Nothing maps here — a blind spot | neither |
145| **out of scope** | Excluded in the scope file, with a reason | neither (removed from the denominator) |
146
147- **Coverage %** = supported + failing + asserted, over in-scope controls. *"How much
148  of the framework am I even looking at?"*
149- **Assured %** = supported only, over in-scope controls. *"How much do I have good
150  evidence for?"*
151
152A low coverage number on a fresh corpus is expected — the framework is large and your
153automated checks touch a slice of it. The value is knowing *exactly which* slice, and
154watching coverage climb (via `--baseline`) as you add evidence.
155
156## 5. Wire it into CI
157
158Two independent gates, both exit non-zero to fail a build:
159
160```bash
161# Fail if any framework's coverage drops below a floor
162control-coverage reports/ --scope soa.yaml --fail-under 60
163
164# Fail if anything regressed or lost coverage versus the last run
165control-coverage reports/ --baseline last-run/ --fail-on-regression
166```
167
168See [`examples/github-actions-coverage.yml`](examples/github-actions-coverage.yml) for
169a scheduled workflow that runs both and uploads the reports as artifacts.
170
171## 6. Frameworks and codes
172
173| Framework | Pass as | Catalog |
174| --- | --- | --- |
175| SOC 2 Trust Services Criteria | `SOC2` | All five categories, 61 controls |
176| ISO/IEC 27001:2022 Annex A | `ISO` (or `iso27001`) | All 93 Annex A controls |
177| NIST SP 800-53 Rev. 5 | `NIST` (or `800-53`) | Moderate baseline, 177 base controls |
178
179Control codes are `FRAMEWORK:ID` — `SOC2:CC6.1`, `ISO:A.5.17`, `NIST:IA-2` — the same
180codes `audit-report` rulesets already emit, so the two tools line up with no
181translation. If your corpus cites a code whose framework is loaded but the catalog
182doesn't define it (a typo or a renamed control), it's reported under **Unmatched
183control codes** rather than silently dropped.
184
185## Gotchas
186
187- **`--crosswalk` and `--baseline` can't be combined** — they're different analyses.
188- **Trend and crosswalk emit `md`, `html`, and `json`** (not `soa`). Coverage emits all four.
189- **SOC 2 defaults to all five categories (61 controls).** Most reports scope only some.
190  Drop the ones you're not audited on with `exclude_families` (see §2) so coverage
191  reflects your real perimeter — e.g. exclude `Privacy` and `Processing Integrity`.
192- **NIST coverage looks low** because the moderate baseline is large (177 controls) and
193  most are organizational/physical/personnel controls no automated scanner evidences.
194  That's the point — those are your blind spots. Use a scope file to exclude the ones
195  handled by policy rather than tooling, so the number reflects your real perimeter.
196- **An unaddressed control is a gap in *evidence*, not proof of a gap in *controls*.**
197  It may just mean the signal isn't collected yet. The tool produces evidence, never a
198  verdict.