audit-labs/control-coverage

  1# control-coverage
  2
  3[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](LICENSE)
  4[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)]()
  5
  6Control-first coverage and blind-spot analysis over an evidence corpus.
  7
  8The rest of the Audit Labs toolchain is **evidence-first**: [audit-tools](https://github.com/audit-labs/audit-tools)
  9collects raw signals, [audit-report](https://github.com/audit-labs/audit-report)
 10maps each finding onto the controls it touches, and [evidence-seal](https://github.com/audit-labs/evidence-seal)
 11proves the package is authentic. That answers *"what did I collect, and what does it
 12map to?"* — but it can never tell you what you are **not** looking at, because it has
 13no list of everything a framework requires.
 14
 15`control-coverage` supplies that missing list — the **denominator**. It starts from
 16the *complete* catalog of a framework's controls and scores your evidence against it,
 17so it can report two numbers nothing else in the pipeline can:
 18
 19- **Coverage %** — of everything the framework requires, how much the evidence corpus
 20  addresses at all.
 21- **Blind spots** — the in-scope controls that *no* finding touches. These are the
 22  gaps an auditor finds for you if you don't find them first.
 23
 24It also produces a **Statement of Applicability** — the ISO 27001 artifact that lists
 25every Annex A control, whether it applies, and why — derived from your evidence
 26instead of hand-maintained.
 27
 28> Like every Audit Labs tool, this produces *evidence*, not a verdict. An unaddressed
 29> control is a gap in *evidence*, which may reflect a real gap in *controls* or simply
 30> a signal not yet collected. The final judgment belongs to the organization and its
 31> auditor.
 32
 33## Install
 34
 35```bash
 36git clone https://github.com/audit-labs/control-coverage
 37cd control-coverage
 38python -m venv .venv && source .venv/bin/activate
 39pip install -e .
 40```
 41
 42Pure standard library plus PyYAML — no other dependencies.
 43
 44## Usage
 45
 46The input is one or more JSON reports from `audit-report` (its `--format json`
 47output). A corpus is typically one report per platform and date — AWS, GitHub,
 48GitLab — which `control-coverage` folds into a single per-framework picture.
 49
 50```bash
 51# Coverage across every framework the corpus cites, Markdown to stdout
 52control-coverage aws.json github.json
 53
 54# Just the blind spots — the controls nothing evidences yet
 55control-coverage aws.json github.json --framework SOC2 --blind-spots
 56
 57# A whole directory of reports, all formats into ./out/
 58control-coverage ./reports/ --format md,html,json,soa --out out/
 59
 60# Gate CI: exit non-zero if any framework's coverage is under 60%
 61control-coverage ./reports/ --fail-under 60
 62```
 63
 64### Trend — how coverage moved
 65
 66Point `--baseline` at an earlier corpus (a file or a directory) to see what changed:
 67controls that improved, regressed, and — the two that move the coverage number —
 68were *gained* (a blind spot became addressed) or *lost* (an addressed control became
 69a blind spot).
 70
 71```bash
 72control-coverage ./reports/2026-02/ --baseline ./reports/2026-01/ --framework SOC2
 73
 74# Gate CI: fail the build if any control regressed or lost coverage
 75control-coverage ./reports/2026-02/ --baseline ./reports/2026-01/ --fail-on-regression
 76```
 77
 78Trend mode outputs Markdown, HTML, or JSON (`--format md,html,json`).
 79
 80### Crosswalk — evidence leverage and the minimal set
 81
 82One check is rarely worth one control: enforced 2FA is evidence for SOC 2 CC6.1,
 83ISO A.5.17, and NIST IA-2 at once. `--crosswalk` shows that leverage per check and
 84computes the **minimal evidence set** — the fewest checks that still touch every
 85addressed control, which is what you want when scoping a walkthrough or a sample.
 86
 87```bash
 88control-coverage ./reports/ --framework SOC2,ISO,NIST --crosswalk
 89```
 90
 91Crosswalk mode outputs Markdown, HTML, or JSON (`--format md,html,json`).
 92
 93### Scope and the Statement of Applicability
 94
 95Not every control applies to every organization. A **scope file** records which
 96controls are excluded and — required, never optional — *why*:
 97
 98```yaml
 99# soa.yaml
100subject: Acme Production
101frameworks: [SOC2, ISO]
102exclusions:
103  - control: ISO:A.7.1
104    reason: "Fully cloud-hosted; no physical premises are in scope for the ISMS."
105  - control: ISO:A.5.7
106    reason: "No formal threat-intelligence program; risk accepted by the CISO for 2026."
107owners:
108  SOC2:CC6.1: platform-team
109```
110
111```bash
112# Coverage over in-scope controls, plus a ready-to-file SoA
113control-coverage ./reports/ --scope soa.yaml --format md,soa --out out/
114```
115
116Excluded controls are recorded with their justification rather than counted as gaps.
117An exclusion with no reason is rejected — an unjustified exclusion is the single most
118common SoA audit finding.
119
120## Assurance states
121
122Every in-scope control lands in exactly one state:
123
124| State | Meaning |
125| --- | --- |
126| **supported** | At least one mapped finding passes, and none fail. |
127| **failing** | At least one mapped finding fails. The worst observation wins. |
128| **asserted** | Findings map here, but their data was absent — evidence attempted, not obtained. |
129| **unaddressed** | No finding maps here at all. **The blind spot.** |
130| **out of scope** | Excluded by the scope file, with a recorded justification. |
131
132`coverage %` is the share of in-scope controls in any of the first three states;
133`assured %` is the share that are `supported`.
134
135## Bundled catalogs
136
137| Framework | Code | Catalog |
138| --- | --- | --- |
139| SOC 2 (Trust Services Criteria) | `SOC2` | Complete — all five categories (Common Criteria + Availability, Confidentiality, Processing Integrity, Privacy), 61 controls |
140| ISO/IEC 27001:2022 Annex A | `ISO` | Complete — all 93 controls |
141| NIST SP 800-53 Rev. 5 | `NIST` | Moderate baseline — 177 base controls across 18 families |
142
143Most SOC 2 reports scope only some categories (Security is near-universal; Privacy and
144Processing Integrity often are not). Use `exclude_families` in the scope file to drop a
145whole category — or an ISO theme, or a NIST family — from the denominator in one line:
146
147```yaml
148exclude_families:
149  - {framework: SOC2, family: Privacy, reason: "Privacy category not in the SOC 2 audit scope."}
150  - {framework: SOC2, family: Processing Integrity, reason: "PI category not in the SOC 2 audit scope."}
151```
152
153Control codes are written `FRAMEWORK:ID` (`SOC2:CC6.1`, `ISO:A.5.17`), matching the
154codes `audit-report` rulesets already cite. A partial catalog is reported honestly as
155coverage of the shipped subset, never as the whole standard.
156
157If the corpus cites a code whose framework is loaded but the catalog does not define
158it — a typo or a renamed control — it is surfaced as an **unmatched control code**
159rather than silently ignored.
160
161## How it fits the pipeline
162
163```
164audit-tools ──► CSV package ──► evidence-seal (seal + verify)
165                     │
166                     ▼
167               audit-report ──► per-package report (--format json)
168                                        │
169                                        ▼  one or more reports = a corpus
170                               control-coverage ──► coverage %, blind spots, SoA,
171                                                    trend over time, evidence crosswalk
172```
173
174## Development
175
176```bash
177pip install -e ".[dev]"
178pytest
179ruff check .
180```
181
182## License
183
184GPL-3.0-or-later. See [LICENSE](LICENSE).