audit-labs/control-coverage
clone: git clone https://gitbay.org/audit-labs/control-coverage.git
main: README.md · raw
1# control-coverage
2
3[](LICENSE)
4[]()
5
6Control-first coverage and blind-spot analysis over an evidence corpus.
7
8The rest of the Audit Labs toolchain is **evidence-first**: [audit-tools](https://github.com/audit-labs/audit-tools)
9collects raw signals, [audit-report](https://github.com/audit-labs/audit-report)
10maps each finding onto the controls it touches, and [evidence-seal](https://github.com/audit-labs/evidence-seal)
11proves the package is authentic. That answers *"what did I collect, and what does it
12map to?"* — but it can never tell you what you are **not** looking at, because it has
13no list of everything a framework requires.
14
15`control-coverage` supplies that missing list — the **denominator**. It starts from
16the *complete* catalog of a framework's controls and scores your evidence against it,
17so it can report two numbers nothing else in the pipeline can:
18
19- **Coverage %** — of everything the framework requires, how much the evidence corpus
20 addresses at all.
21- **Blind spots** — the in-scope controls that *no* finding touches. These are the
22 gaps an auditor finds for you if you don't find them first.
23
24It also produces a **Statement of Applicability** — the ISO 27001 artifact that lists
25every Annex A control, whether it applies, and why — derived from your evidence
26instead of hand-maintained.
27
28> Like every Audit Labs tool, this produces *evidence*, not a verdict. An unaddressed
29> control is a gap in *evidence*, which may reflect a real gap in *controls* or simply
30> a signal not yet collected. The final judgment belongs to the organization and its
31> auditor.
32
33## Install
34
35```bash
36pip install control-coverage
37```
38
39Pure standard library plus PyYAML — no other dependencies. To hack on it from
40a clone instead, see [Development](#development).
41
42## Usage
43
44The input is one or more JSON reports from `audit-report` (its `--format json`
45output). A corpus is typically one report per platform and date — AWS, GitHub,
46GitLab — which `control-coverage` folds into a single per-framework picture.
47
48```bash
49# Coverage across every framework the corpus cites, Markdown to stdout
50control-coverage aws.json github.json
51
52# Just the blind spots — the controls nothing evidences yet
53control-coverage aws.json github.json --framework SOC2 --blind-spots
54
55# A whole directory of reports, all formats into ./out/
56control-coverage ./reports/ --format md,html,json,soa --out out/
57
58# Gate CI: exit non-zero if any framework's coverage is under 60%
59control-coverage ./reports/ --fail-under 60
60```
61
62### Trend — how coverage moved
63
64Point `--baseline` at an earlier corpus (a file or a directory) to see what changed:
65controls that improved, regressed, and — the two that move the coverage number —
66were *gained* (a blind spot became addressed) or *lost* (an addressed control became
67a blind spot).
68
69```bash
70control-coverage ./reports/2026-02/ --baseline ./reports/2026-01/ --framework SOC2
71
72# Gate CI: fail the build if any control regressed or lost coverage
73control-coverage ./reports/2026-02/ --baseline ./reports/2026-01/ --fail-on-regression
74```
75
76Trend mode outputs Markdown, HTML, or JSON (`--format md,html,json`).
77
78### Crosswalk — evidence leverage and the minimal set
79
80One check is rarely worth one control: enforced 2FA is evidence for SOC 2 CC6.1,
81ISO A.5.17, and NIST IA-2 at once. `--crosswalk` shows that leverage per check and
82computes the **minimal evidence set** — the fewest checks that still touch every
83addressed control, which is what you want when scoping a walkthrough or a sample.
84
85```bash
86control-coverage ./reports/ --framework SOC2,ISO,NIST --crosswalk
87```
88
89Crosswalk mode outputs Markdown, HTML, or JSON (`--format md,html,json`).
90
91### Scope and the Statement of Applicability
92
93Not every control applies to every organization. A **scope file** records which
94controls are excluded and — required, never optional — *why*:
95
96```yaml
97# soa.yaml
98subject: Acme Production
99frameworks: [SOC2, ISO]
100exclusions:
101 - control: ISO:A.7.1
102 reason: "Fully cloud-hosted; no physical premises are in scope for the ISMS."
103 - control: ISO:A.5.7
104 reason: "No formal threat-intelligence program; risk accepted by the CISO for 2026."
105owners:
106 SOC2:CC6.1: platform-team
107```
108
109```bash
110# Coverage over in-scope controls, plus a ready-to-file SoA
111control-coverage ./reports/ --scope soa.yaml --format md,soa --out out/
112```
113
114Excluded controls are recorded with their justification rather than counted as gaps.
115An exclusion with no reason is rejected — an unjustified exclusion is the single most
116common SoA audit finding.
117
118## Assurance states
119
120Every in-scope control lands in exactly one state:
121
122| State | Meaning |
123| --- | --- |
124| **supported** | At least one mapped finding passes, and none fail. |
125| **failing** | At least one mapped finding fails. The worst observation wins. |
126| **asserted** | Findings map here, but their data was absent — evidence attempted, not obtained. |
127| **unaddressed** | No finding maps here at all. **The blind spot.** |
128| **out of scope** | Excluded by the scope file, with a recorded justification. |
129
130`coverage %` is the share of in-scope controls in any of the first three states;
131`assured %` is the share that are `supported`.
132
133## Bundled catalogs
134
135| Framework | Code | Catalog |
136| --- | --- | --- |
137| SOC 2 (Trust Services Criteria) | `SOC2` | Complete — all five categories (Common Criteria + Availability, Confidentiality, Processing Integrity, Privacy), 61 controls |
138| ISO/IEC 27001:2022 Annex A | `ISO` | Complete — all 93 controls |
139| NIST SP 800-53 Rev. 5 | `NIST` | Moderate baseline — 177 base controls across 18 families |
140
141Most SOC 2 reports scope only some categories (Security is near-universal; Privacy and
142Processing Integrity often are not). Use `exclude_families` in the scope file to drop a
143whole category — or an ISO theme, or a NIST family — from the denominator in one line:
144
145```yaml
146exclude_families:
147 - {framework: SOC2, family: Privacy, reason: "Privacy category not in the SOC 2 audit scope."}
148 - {framework: SOC2, family: Processing Integrity, reason: "PI category not in the SOC 2 audit scope."}
149```
150
151Control codes are written `FRAMEWORK:ID` (`SOC2:CC6.1`, `ISO:A.5.17`), matching the
152codes `audit-report` rulesets already cite. A partial catalog is reported honestly as
153coverage of the shipped subset, never as the whole standard.
154
155If the corpus cites a code whose framework is loaded but the catalog does not define
156it — a typo or a renamed control — it is surfaced as an **unmatched control code**
157rather than silently ignored.
158
159## How it fits the pipeline
160
161```
162audit-tools ──► CSV package ──► evidence-seal (seal + verify)
163 │
164 ▼
165 audit-report ──► per-package report (--format json)
166 │
167 ▼ one or more reports = a corpus
168 control-coverage ──► coverage %, blind spots, SoA,
169 trend over time, evidence crosswalk
170```
171
172## Stability
173
174`control-coverage` is stable as of **v1.0.0** and follows [semantic versioning](https://semver.org).
175It reads [audit-report](https://github.com/audit-labs/audit-report)'s v1 JSON
176contract; the bundled catalogs, the coverage / Statement of Applicability / JSON
177output schemas, and the `--fail-under` gate are committed within the 1.x line.
178
179## Development
180
181```bash
182pip install -e ".[dev]"
183pytest
184ruff check .
185```
186
187## License
188
189GPL-3.0-or-later. See [LICENSE](LICENSE).