audit-labs/gh-attest

GitHub Audit Evidence Extractor attestation audit evidence github https://audit-labs.dev/gh-attest/

Commit 8149e7fc18

8149e7fc18e96307607e6ee4867f896ada91623c

parent: 2d878671dd

Unsigned

cmc <hello@cleberg.net> · 2026-07-26 18:12 UTC 

docs: add architecture overview with mermaid diagrams

Add docs/architecture.md: system context, webhook ingestion, hourly
poll + retention, and export-pipeline diagrams, plus an ER diagram of the
data model and the evidence-query flow. Includes route/auth tables, the
boundaries & isolation notes, and a source-file map. Cross-links
framework-mapping.md.

Layout: unified · split

docs/architecture.md added +266
@@ -0,0 +1,266 @@
1# Architecture
2
3gh-attest is a single [Cloudflare Worker](../src/index.ts) with three entry
4points — an HTTP router, an hourly cron, and a queue consumer — backed by D1,
5R2, and a Queue, all provisioned in Cloudflare's `eu` jurisdiction. It ingests
6GitHub security posture as immutable timestamped snapshots and renders them into
7auditor-ready evidence packages mapped to SOC 2 / ISO 27001 controls.
8
9This document is the shape of the system. For *why* a given GitHub signal counts
10as evidence for a given control, see [framework-mapping.md](framework-mapping.md).
11
12## System context
13
14```mermaid
15flowchart LR
16 subgraph ext[External]
17 GH["GitHub App<br/>webhooks · OAuth · REST API"]
18 USER["User / Auditor<br/>browser"]
19 end
20
21 subgraph cf["Cloudflare Worker — src/index.ts"]
22 FETCH["fetch()<br/>HTTP router"]
23 CRON["scheduled()<br/>hourly cron"]
24 QUEUE["queue()<br/>export consumer"]
25 end
26
27 subgraph store["Storage — EU jurisdiction"]
28 D1[("D1<br/>gh-attest-db-eu")]
29 R2[("R2<br/>gh-attest-exports-eu")]
30 Q[["Queue<br/>generate-export"]]
31 end
32
33 GH -- "POST /webhooks/*" --> FETCH
34 USER -- "login · dashboard · exports" --> FETCH
35 CRON -- "poll (installation token)" --> GH
36
37 FETCH -- "snapshots · export rows" --> D1
38 FETCH -- "enqueue job" --> Q
39 FETCH -- "stream download" --> R2
40
41 CRON -- "snapshots · retention" --> D1
42 CRON -- "retention delete" --> R2
43
44 Q --> QUEUE
45 QUEUE -- "read evidence" --> D1
46 QUEUE -- "write file" --> R2
47```
48
49## Entry points
50
51| Handler | Trigger | Responsibility | Code |
52| --- | --- | --- | --- |
53| `fetch` | HTTP request | Webhooks, OAuth/session dashboard, export requests + downloads, admin ops | [index.ts](../src/index.ts) |
54| `scheduled` | Cron `0 * * * *` (hourly) | Poll GitHub for state webhooks never announce; enforce retention | [index.ts](../src/index.ts) |
55| `queue` | `generate-export` message | Render CSV/PDF off the request path into R2 | [index.ts](../src/index.ts) |
56
57### Routes and their auth
58
59| Route | Auth | Purpose |
60| --- | --- | --- |
61| `POST /webhooks/github`, `/webhooks/marketplace` | HMAC (`GITHUB_WEBHOOK_SECRET`) | Ingest App / marketplace events |
62| `GET /login`, `/callback`, `/logout` | OAuth state cookie | Dashboard sign-in |
63| `GET /`, `/access-review` | Session cookie | Posture dashboard, membership diff |
64| `POST /exports`, `/resync`, `/switch` | Session cookie | Dashboard actions (installation-scoped) |
65| `GET /exports/:id[/download]` | Session cookie | Export status / file (scoped to installation) |
66| `POST /admin/{poll,export,cleanup,purge}`, `GET /admin/export/:id` | Bearer (`ADMIN_TOKEN`) | Operations |
67
68## Ingestion — two paths into one table
69
70Both paths write to the append-only `snapshots` table; nothing is ever updated
71in place, which is what makes the table a point-in-time audit trail.
72
73### Webhooks (change events)
74
75```mermaid
76sequenceDiagram
77 autonumber
78 participant GH as GitHub
79 participant W as Worker (fetch)
80 participant D1 as D1
81 participant R2 as R2
82
83 GH->>W: POST /webhooks/github (event + X-Hub-Signature-256)
84 W->>W: verifySignature(body, GITHUB_WEBHOOK_SECRET)
85 alt invalid signature
86 W-->>GH: 401
87 else valid
88 W->>W: extractFact(event) → {resource, status}
89 W->>D1: upsert installations row
90 alt installation.deleted
91 W->>D1: DELETE all rows for installation
92 W->>R2: delete export objects
93 W-->>GH: 200 (purged)
94 else suspend / unsuspend
95 W->>D1: toggle suspended_at
96 W-->>GH: 200
97 else normal event
98 W->>D1: INSERT snapshot (append-only)
99 W-->>GH: 200
100 end
101 end
102```
103
104### Hourly poll + retention (baseline / drift)
105
106Webhooks only fire on change, so protection that existed *before* install, and
107current membership, would never appear. The cron closes that gap and enforces
108the retention windows in the same invocation.
109
110```mermaid
111sequenceDiagram
112 autonumber
113 participant Cron as scheduled (hourly)
114 participant W as Worker
115 participant GH as GitHub REST
116 participant D1 as D1
117 participant R2 as R2
118
119 Cron->>W: fire
120 par Poll every active installation
121 W->>GH: app JWT → installation token
122 W->>GH: repos · branch protection · rulesets · org/team members
123 GH-->>W: current state
124 W->>D1: INSERT snapshots (one captured_at per batch)
125 and Retention cleanup
126 W->>D1: SELECT expired export r2_keys
127 W->>R2: delete expired objects
128 W->>D1: DELETE exports > 90d, snapshots > 396d
129 end
130```
131
132## Export pipeline
133
134Rendering runs off the request path via the Queue because PDF / large CSV can
135exceed request CPU limits. The `exports` row is the job's state machine
136(`queued → processing → done | error`).
137
138```mermaid
139sequenceDiagram
140 autonumber
141 participant U as User (session)
142 participant W as Worker (fetch)
143 participant D1 as D1
144 participant Q as Queue
145 participant C as Worker (queue)
146 participant R2 as R2
147
148 U->>W: POST /exports (framework, format)
149 W->>D1: INSERT exports (status=queued)
150 W->>Q: send job
151 W-->>U: 303 redirect to dashboard
152
153 Q->>C: deliver job
154 C->>D1: status=processing
155 C->>D1: buildEvidenceRows — snapshots ⋈ control_mappings
156 C->>C: renderCsv / renderPdf
157 C->>R2: put file at exports/{installation}/{jobId}.{fmt}
158 C->>D1: status=done, r2_key
159 Note over C,D1: render failure → status=error (deterministic, not retried)
160
161 U->>W: GET /exports/:id/download
162 W->>D1: lookup scoped to session installation
163 W->>R2: get object
164 R2-->>W: file body
165 W-->>U: stream (Content-Disposition: attachment)
166```
167
168## Data model
169
170`control_mappings` has **no foreign key** to `snapshots`. Mapping is a join on
171`(resource, status)` performed at query/export time — so a mapping can be
172corrected without re-ingesting webhook history. A mapping row with `status =
173NULL` matches any status for that resource.
174
175```mermaid
176erDiagram
177 installations ||--o{ snapshots : has
178 installations ||--o{ exports : has
179 snapshots }o..o{ control_mappings : "query-time join on (resource, status)"
180
181 installations {
182 integer installation_id PK
183 text org_login
184 text installed_at
185 text suspended_at "null unless suspended"
186 }
187 snapshots {
188 integer id PK
189 integer installation_id FK
190 text repo "null for org-level facts"
191 text subject "member/team for access facts"
192 text resource "e.g. branch_protection"
193 text status "e.g. enabled | open | added"
194 text raw_payload "original JSON, audit trail"
195 text captured_at
196 }
197 control_mappings {
198 integer id PK
199 text resource
200 text status "null matches any"
201 text framework "soc2 | iso27001"
202 text control_id "e.g. CC8.1 | A.8.32"
203 text posture "positive | negative | informational"
204 text rationale
205 }
206 exports {
207 text id PK "uuid"
208 integer installation_id FK
209 text framework
210 text format "csv | pdf"
211 text status "queued|processing|done|error"
212 text r2_key "null until rendered"
213 text error
214 text created_at
215 text completed_at
216 }
217```
218
219### Evidence query
220
221`buildEvidenceRows` ([exporter.ts](../src/exporter.ts)) reduces the append-only
222table to current posture, then attaches controls:
223
224```mermaid
225flowchart TB
226 S[("snapshots<br/>append-only")] --> L["latest row per<br/>(repo, subject, resource)"]
227 L --> J{{"JOIN control_mappings<br/>on resource + status"}}
228 CM[("control_mappings")] --> J
229 J --> F["filter by framework<br/>(soc2 | iso27001 | all)"]
230 F --> E["evidence rows<br/>control · posture · rationale"]
231 E --> CSV["renderCsv"]
232 E --> PDF["renderPdf"]
233```
234
235Unmapped `(resource, status)` pairs (e.g. `unavailable`, raw `push`) simply
236produce no rows — no evidence in either direction.
237
238## Boundaries & isolation
239
240- **Multi-tenant scoping.** Every session route is scoped to the session's
241 `installationId`; the allowed set lives in the signed session, so a tampered
242 id can't widen access. Export downloads are looked up with an installation
243 filter, so one org can't read another's file by guessing its UUID.
244- **Three auth schemes.** HMAC for webhooks, signed session cookies
245 (`SESSION_SECRET`) for the dashboard, timing-safe bearer (`ADMIN_TOKEN`) for
246 `/admin/*`.
247- **Data residency.** D1, R2, and the Queue are all in the `eu` jurisdiction;
248 see [wrangler.jsonc](../wrangler.jsonc). No source code or access tokens are
249 stored — only posture facts and their raw event JSON.
250- **Deletion.** Uninstall (`installation.deleted`) purges all D1 rows and R2
251 objects for the installation; `/admin/purge` does the same on request.
252
253## Where things live
254
255| Concern | File |
256| --- | --- |
257| Router, handlers, retention, purge | [src/index.ts](../src/index.ts) |
258| Webhook verification + fact extraction | [src/webhook.ts](../src/webhook.ts) |
259| GitHub polling (protection, rulesets, access) | [src/poller.ts](../src/poller.ts) |
260| App JWT + installation tokens | [src/github-app.ts](../src/github-app.ts) |
261| OAuth + session sign/verify | [src/auth.ts](../src/auth.ts) |
262| Evidence query + CSV/PDF rendering | [src/exporter.ts](../src/exporter.ts) |
263| Access-review diff | [src/access-review.ts](../src/access-review.ts) |
264| Dashboard HTML | [src/dashboard.ts](../src/dashboard.ts) |
265| Schema + control mappings | [migrations/](../migrations) |
266```