audit-labs/audit-tools

A collection of scripts, queries, and other goodies you can use in an audit. audit automation compliance evidence scripts

Commit 4221777e89

4221777e89a2d291f229db7a866406db3e879b8b

parent: 41e4b4b5db

Unsigned

cmc <hello@cleberg.net> · 2026-07-07 22:18 UTC 

remove stray files

Layout: unified · split

COMMIT_MESSAGE.txt deleted −50
@@ -1,50 +0,0 @@
1Add production-ready audit sampling CLI
2
3Implement a reusable audit sampling tool for CSV and Excel populations.
4
5This change adds sampling/audit_sample.py and a modular sampling_tool package
6that supports random sampling, stratified sampling, validate-only runs, exact
7match filters, source row tracking, duplicate-ID handling, blank-ID handling,
8reconciliation outputs, methodology documentation, manifests, and run logs.
9
10Key behavior:
11- Supports .csv, .xlsx, .xls, and .xlsm population files using pandas.
12- Adds _source_row_number based on source data row numbers, with first data row
13 recorded as row 2 to align with spreadsheet conventions.
14- Supports CLI options and YAML config files, with CLI arguments overriding
15 config values.
16- Supports simple random samples without replacement using deterministic seeds.
17- Supports stratified sampling by explicit counts or proportional allocation
18 using largest remainder rounding.
19- Generates documented output packages containing applicable sample,
20 population, reconciliation, exclusion, duplicate-ID, strata summary,
21 methodology, manifest, and log files.
22- Fails by default on duplicate IDs while still writing duplicate and
23 reconciliation evidence.
24- Allows blank IDs to be retained with warnings or excluded with evidence.
25- Keeps existing sampling/stratified_sample.py in place for compatibility.
26
27Documentation and examples:
28- Adds safe example populations and a stratified YAML config under
29 sampling/examples.
30- Updates sampling/README.md with installation, examples, output descriptions,
31 reproducibility notes, and limitations.
32- Fixes the top-level README clone URL typo.
33- Updates .gitignore so output and __pycache__ directories are ignored at any
34 folder depth.
35
36Tests:
37- Adds pytest coverage for random sampling, reproducibility, stratified counts,
38 stratified proportions, validation behavior, duplicate-ID evidence,
39 blank-ID exclusion, filters, validate-only behavior, and reconciliation math.
40
41Validation:
42- Ran .venv/bin/python -m pytest sampling/tests.
43- Result: 10 passed.
44- Ran manual CLI validation for random, stratified-counts,
45 stratified-proportions, validate-only, config-file, duplicate-ID failure, and
46 blank-ID exclusion cases.
47- Programmatically validated 7 generated output packages for required files,
48 manifest completeness, reconciliation tie-outs, sample schema, strata math,
49 excluded-row evidence, duplicate-ID evidence, methodology content, and log
50 status.
TESTING_PROOF_LOG.md deleted −307
@@ -1,307 +0,0 @@
1# Testing Proof Log
2
3Date: 2026-07-07
4Repository: `/Users/cmc/git/audit-labs/audit-tools`
5
6## Unit Tests
7
8Command:
9
10```bash
11.venv/bin/python -m pytest sampling/tests
12```
13
14Observed output:
15
16```text
17============================= test session starts ==============================
18platform darwin -- Python 3.14.6, pytest-9.1.1, pluggy-1.6.0
19rootdir: /Users/cmc/git/audit-labs/audit-tools
20plugins: dash-4.4.0
21collected 10 items
22
23sampling/tests/test_random_sample.py ... [ 30%]
24sampling/tests/test_reconciliation.py . [ 40%]
25sampling/tests/test_stratified_sample.py .. [ 60%]
26sampling/tests/test_validation.py .... [100%]
27
28============================== 10 passed in 0.42s ==============================
29```
30
31Result: PASS
32
33## Manual CLI Runs
34
35The following output packages were generated under `output/validation_suite`.
36The `output` directory is ignored by git.
37
38### Random Sample
39
40Command:
41
42```bash
43.venv/bin/python sampling/audit_sample.py \
44 --input sampling/examples/users_population.csv \
45 --id-column "User ID" \
46 --method random \
47 --sample-size 5 \
48 --seed 20260707 \
49 --out ./output/validation_suite
50```
51
52Output package:
53
54```text
55output/validation_suite/sample_2026-07-07_220414
56```
57
58Observed files:
59
60```text
61manifest.json
62methodology.txt
63population_reconciliation.csv
64population_validated.csv
65run.log
66sample.csv
67```
68
69Result: PASS
70
71### Stratified Sample By Counts
72
73Command:
74
75```bash
76.venv/bin/python sampling/audit_sample.py \
77 --input sampling/examples/changes_population.csv \
78 --id-column "Change ID" \
79 --method stratified \
80 --stratify-column "Change Type" \
81 --strata-counts "Normal=3,Emergency=2,Standard=2" \
82 --seed 20260707 \
83 --out ./output/validation_suite
84```
85
86Output package:
87
88```text
89output/validation_suite/sample_2026-07-07_220418
90```
91
92Observed files:
93
94```text
95manifest.json
96methodology.txt
97population_reconciliation.csv
98population_validated.csv
99run.log
100sample.csv
101strata_summary.csv
102```
103
104Result: PASS
105
106### Stratified Sample By Proportions
107
108Command:
109
110```bash
111.venv/bin/python sampling/audit_sample.py \
112 --input sampling/examples/changes_population.csv \
113 --id-column "Change ID" \
114 --method stratified \
115 --stratify-column "Change Type" \
116 --strata-proportions "Normal=0.50,Emergency=0.25,Standard=0.25" \
117 --sample-size 8 \
118 --seed 20260707 \
119 --out ./output/validation_suite
120```
121
122Output package:
123
124```text
125output/validation_suite/sample_2026-07-07_220425
126```
127
128Observed files:
129
130```text
131manifest.json
132methodology.txt
133population_reconciliation.csv
134population_validated.csv
135run.log
136sample.csv
137strata_summary.csv
138```
139
140Result: PASS
141
142### Validate-Only With Filter
143
144Command:
145
146```bash
147.venv/bin/python sampling/audit_sample.py \
148 --input sampling/examples/changes_population.csv \
149 --id-column "Change ID" \
150 --method validate-only \
151 --filter "Status=Closed" \
152 --out ./output/validation_suite
153```
154
155Output package:
156
157```text
158output/validation_suite/sample_2026-07-07_220430
159```
160
161Observed files:
162
163```text
164excluded_rows.csv
165manifest.json
166methodology.txt
167population_reconciliation.csv
168population_validated.csv
169run.log
170```
171
172Result: PASS. No `sample.csv` was produced, as expected for validate-only.
173
174### YAML Config Run
175
176Command:
177
178```bash
179.venv/bin/python sampling/audit_sample.py \
180 --config sampling/examples/stratified_config.yml \
181 --out ./output/validation_suite
182```
183
184Output package:
185
186```text
187output/validation_suite/sample_2026-07-07_220436
188```
189
190Observed files:
191
192```text
193manifest.json
194methodology.txt
195population_reconciliation.csv
196population_validated.csv
197run.log
198sample.csv
199strata_summary.csv
200```
201
202Result: PASS
203
204### Duplicate-ID Failure Evidence
205
206Command:
207
208```bash
209.venv/bin/python sampling/audit_sample.py \
210 --input /private/tmp/audit_sample_validation_inputs/duplicate_ids.csv \
211 --id-column ID \
212 --method validate-only \
213 --out ./output/validation_suite
214```
215
216Output package:
217
218```text
219output/validation_suite/sample_2026-07-07_220453
220```
221
222Observed files:
223
224```text
225duplicate_ids.csv
226manifest.json
227methodology.txt
228population_reconciliation.csv
229population_validated.csv
230run.log
231```
232
233Result: PASS. The command failed as intended because duplicate IDs are rejected
234by default, and `duplicate_ids.csv` was still written as evidence.
235
236### Blank-ID Exclusion Evidence
237
238Command:
239
240```bash
241.venv/bin/python sampling/audit_sample.py \
242 --input /private/tmp/audit_sample_validation_inputs/blank_ids.csv \
243 --id-column ID \
244 --method validate-only \
245 --exclude-blank-id \
246 --out ./output/validation_suite
247```
248
249Output package:
250
251```text
252output/validation_suite/sample_2026-07-07_220500
253```
254
255Observed files:
256
257```text
258excluded_rows.csv
259manifest.json
260methodology.txt
261population_reconciliation.csv
262population_validated.csv
263run.log
264```
265
266Result: PASS. Blank-ID rows were excluded and written to `excluded_rows.csv`.
267
268## Output Integrity Validation
269
270Command:
271
272```bash
273.venv/bin/python - <<'PY'
274# Programmatic validation over output/validation_suite:
275# - required files by method
276# - manifest required keys
277# - manifest output_files matches files on disk
278# - reconciliation tie-outs
279# - population_validated row counts and _source_row_number
280# - sample.csv metadata columns and counts
281# - strata_summary schema and counts
282# - excluded_rows.csv evidence
283# - duplicate_ids.csv evidence
284# - run.log final status
285PY
286```
287
288Observed output:
289
290```text
291Validated 7 output package(s).
292sample_2026-07-07_220414: method=random, status=success, file_count=6
293sample_2026-07-07_220418: method=stratified, status=success, file_count=7
294sample_2026-07-07_220425: method=stratified, status=success, file_count=7
295sample_2026-07-07_220430: method=validate-only, status=success, file_count=6
296sample_2026-07-07_220436: method=stratified, status=success, file_count=7
297sample_2026-07-07_220453: method=validate-only, status=failed, file_count=6
298sample_2026-07-07_220500: method=validate-only, status=success, file_count=6
299All output package integrity checks passed.
300```
301
302Result: PASS
303
304## Overall Result
305
306All automated tests passed, all manual CLI paths completed with expected
307behavior, and all generated output packages passed integrity validation.