Commit `8149e7fc18`

8149e7fc18e96307607e6ee4867f896ada91623c

parent: 2d878671dd

Unsigned

cmc <hello@cleberg.net> · 2026-07-26 18:12 UTC

docs: add architecture overview with mermaid diagrams

Add docs/architecture.md: system context, webhook ingestion, hourly
poll + retention, and export-pipeline diagrams, plus an ER diagram of the
data model and the evidence-query flow. Includes route/auth tables, the
boundaries & isolation notes, and a source-file map. Cross-links
framework-mapping.md.

Layout: unified · split

docs/architecture.md added +266

		@@ -0,0 +1,266 @@
	1	# Architecture
	2
	3	gh-attest is a single [Cloudflare Worker](../src/index.ts) with three entry
	4	points — an HTTP router, an hourly cron, and a queue consumer — backed by D1,
	5	R2, and a Queue, all provisioned in Cloudflare's `eu` jurisdiction. It ingests
	6	GitHub security posture as immutable timestamped snapshots and renders them into
	7	auditor-ready evidence packages mapped to SOC 2 / ISO 27001 controls.
	8
	9	This document is the shape of the system. For why a given GitHub signal counts
	10	as evidence for a given control, see [framework-mapping.md](framework-mapping.md).
	11
	12	## System context
	13
	14	```mermaid
	15	flowchart LR
	16	subgraph ext[External]
	17	GH["GitHub App<br/>webhooks · OAuth · REST API"]
	18	USER["User / Auditor<br/>browser"]
	19	end
	20
	21	subgraph cf["Cloudflare Worker — src/index.ts"]
	22	FETCH["fetch()<br/>HTTP router"]
	23	CRON["scheduled()<br/>hourly cron"]
	24	QUEUE["queue()<br/>export consumer"]
	25	end
	26
	27	subgraph store["Storage — EU jurisdiction"]
	28	D1[("D1<br/>gh-attest-db-eu")]
	29	R2[("R2<br/>gh-attest-exports-eu")]
	30	Q[["Queue<br/>generate-export"]]
	31	end
	32
	33	GH -- "POST /webhooks/*" --> FETCH
	34	USER -- "login · dashboard · exports" --> FETCH
	35	CRON -- "poll (installation token)" --> GH
	36
	37	FETCH -- "snapshots · export rows" --> D1
	38	FETCH -- "enqueue job" --> Q
	39	FETCH -- "stream download" --> R2
	40
	41	CRON -- "snapshots · retention" --> D1
	42	CRON -- "retention delete" --> R2
	43
	44	Q --> QUEUE
	45	QUEUE -- "read evidence" --> D1
	46	QUEUE -- "write file" --> R2
	47	```
	48
	49	## Entry points
	50
	51	\| Handler \| Trigger \| Responsibility \| Code \|
	52	\| --- \| --- \| --- \| --- \|
	53	\| `fetch` \| HTTP request \| Webhooks, OAuth/session dashboard, export requests + downloads, admin ops \| [index.ts](../src/index.ts) \|
	54	\| `scheduled` \| Cron `0 * * * *` (hourly) \| Poll GitHub for state webhooks never announce; enforce retention \| [index.ts](../src/index.ts) \|
	55	\| `queue` \| `generate-export` message \| Render CSV/PDF off the request path into R2 \| [index.ts](../src/index.ts) \|
	56
	57	### Routes and their auth
	58
	59	\| Route \| Auth \| Purpose \|
	60	\| --- \| --- \| --- \|
	61	\| `POST /webhooks/github`, `/webhooks/marketplace` \| HMAC (`GITHUB_WEBHOOK_SECRET`) \| Ingest App / marketplace events \|
	62	\| `GET /login`, `/callback`, `/logout` \| OAuth state cookie \| Dashboard sign-in \|
	63	\| `GET /`, `/access-review` \| Session cookie \| Posture dashboard, membership diff \|
	64	\| `POST /exports`, `/resync`, `/switch` \| Session cookie \| Dashboard actions (installation-scoped) \|
	65	\| `GET /exports/:id[/download]` \| Session cookie \| Export status / file (scoped to installation) \|
	66	\| `POST /admin/{poll,export,cleanup,purge}`, `GET /admin/export/:id` \| Bearer (`ADMIN_TOKEN`) \| Operations \|
	67
	68	## Ingestion — two paths into one table
	69
	70	Both paths write to the append-only `snapshots` table; nothing is ever updated
	71	in place, which is what makes the table a point-in-time audit trail.
	72
	73	### Webhooks (change events)
	74
	75	```mermaid
	76	sequenceDiagram
	77	autonumber
	78	participant GH as GitHub
	79	participant W as Worker (fetch)
	80	participant D1 as D1
	81	participant R2 as R2
	82
	83	GH->>W: POST /webhooks/github (event + X-Hub-Signature-256)
	84	W->>W: verifySignature(body, GITHUB_WEBHOOK_SECRET)
	85	alt invalid signature
	86	W-->>GH: 401
	87	else valid
	88	W->>W: extractFact(event) → {resource, status}
	89	W->>D1: upsert installations row
	90	alt installation.deleted
	91	W->>D1: DELETE all rows for installation
	92	W->>R2: delete export objects
	93	W-->>GH: 200 (purged)
	94	else suspend / unsuspend
	95	W->>D1: toggle suspended_at
	96	W-->>GH: 200
	97	else normal event
	98	W->>D1: INSERT snapshot (append-only)
	99	W-->>GH: 200
	100	end
	101	end
	102	```
	103
	104	### Hourly poll + retention (baseline / drift)
	105
	106	Webhooks only fire on change, so protection that existed before install, and
	107	current membership, would never appear. The cron closes that gap and enforces
	108	the retention windows in the same invocation.
	109
	110	```mermaid
	111	sequenceDiagram
	112	autonumber
	113	participant Cron as scheduled (hourly)
	114	participant W as Worker
	115	participant GH as GitHub REST
	116	participant D1 as D1
	117	participant R2 as R2
	118
	119	Cron->>W: fire
	120	par Poll every active installation
	121	W->>GH: app JWT → installation token
	122	W->>GH: repos · branch protection · rulesets · org/team members
	123	GH-->>W: current state
	124	W->>D1: INSERT snapshots (one captured_at per batch)
	125	and Retention cleanup
	126	W->>D1: SELECT expired export r2_keys
	127	W->>R2: delete expired objects
	128	W->>D1: DELETE exports > 90d, snapshots > 396d
	129	end
	130	```
	131
	132	## Export pipeline
	133
	134	Rendering runs off the request path via the Queue because PDF / large CSV can
	135	exceed request CPU limits. The `exports` row is the job's state machine
	136	(`queued → processing → done \| error`).
	137
	138	```mermaid
	139	sequenceDiagram
	140	autonumber
	141	participant U as User (session)
	142	participant W as Worker (fetch)
	143	participant D1 as D1
	144	participant Q as Queue
	145	participant C as Worker (queue)
	146	participant R2 as R2
	147
	148	U->>W: POST /exports (framework, format)
	149	W->>D1: INSERT exports (status=queued)
	150	W->>Q: send job
	151	W-->>U: 303 redirect to dashboard
	152
	153	Q->>C: deliver job
	154	C->>D1: status=processing
	155	C->>D1: buildEvidenceRows — snapshots ⋈ control_mappings
	156	C->>C: renderCsv / renderPdf
	157	C->>R2: put file at exports/{installation}/{jobId}.{fmt}
	158	C->>D1: status=done, r2_key
	159	Note over C,D1: render failure → status=error (deterministic, not retried)
	160
	161	U->>W: GET /exports/:id/download
	162	W->>D1: lookup scoped to session installation
	163	W->>R2: get object
	164	R2-->>W: file body
	165	W-->>U: stream (Content-Disposition: attachment)
	166	```
	167
	168	## Data model
	169
	170	`control_mappings` has no foreign key to `snapshots`. Mapping is a join on
	171	`(resource, status)` performed at query/export time — so a mapping can be
	172	corrected without re-ingesting webhook history. A mapping row with `status =
	173	NULL` matches any status for that resource.
	174
	175	```mermaid
	176	erDiagram
	177	installations \|\|--o{ snapshots : has
	178	installations \|\|--o{ exports : has
	179	snapshots }o..o{ control_mappings : "query-time join on (resource, status)"
	180
	181	installations {
	182	integer installation_id PK
	183	text org_login
	184	text installed_at
	185	text suspended_at "null unless suspended"
	186	}
	187	snapshots {
	188	integer id PK
	189	integer installation_id FK
	190	text repo "null for org-level facts"
	191	text subject "member/team for access facts"
	192	text resource "e.g. branch_protection"
	193	text status "e.g. enabled \| open \| added"
	194	text raw_payload "original JSON, audit trail"
	195	text captured_at
	196	}
	197	control_mappings {
	198	integer id PK
	199	text resource
	200	text status "null matches any"
	201	text framework "soc2 \| iso27001"
	202	text control_id "e.g. CC8.1 \| A.8.32"
	203	text posture "positive \| negative \| informational"
	204	text rationale
	205	}
	206	exports {
	207	text id PK "uuid"
	208	integer installation_id FK
	209	text framework
	210	text format "csv \| pdf"
	211	text status "queued\|processing\|done\|error"
	212	text r2_key "null until rendered"
	213	text error
	214	text created_at
	215	text completed_at
	216	}
	217	```
	218
	219	### Evidence query
	220
	221	`buildEvidenceRows` ([exporter.ts](../src/exporter.ts)) reduces the append-only
	222	table to current posture, then attaches controls:
	223
	224	```mermaid
	225	flowchart TB
	226	S[("snapshots<br/>append-only")] --> L["latest row per<br/>(repo, subject, resource)"]
	227	L --> J{{"JOIN control_mappings<br/>on resource + status"}}
	228	CM[("control_mappings")] --> J
	229	J --> F["filter by framework<br/>(soc2 \| iso27001 \| all)"]
	230	F --> E["evidence rows<br/>control · posture · rationale"]
	231	E --> CSV["renderCsv"]
	232	E --> PDF["renderPdf"]
	233	```
	234
	235	Unmapped `(resource, status)` pairs (e.g. `unavailable`, raw `push`) simply
	236	produce no rows — no evidence in either direction.
	237
	238	## Boundaries & isolation
	239
	240	- Multi-tenant scoping. Every session route is scoped to the session's
	241	`installationId`; the allowed set lives in the signed session, so a tampered
	242	id can't widen access. Export downloads are looked up with an installation
	243	filter, so one org can't read another's file by guessing its UUID.
	244	- Three auth schemes. HMAC for webhooks, signed session cookies
	245	(`SESSION_SECRET`) for the dashboard, timing-safe bearer (`ADMIN_TOKEN`) for
	246	`/admin/*`.
	247	- Data residency. D1, R2, and the Queue are all in the `eu` jurisdiction;
	248	see [wrangler.jsonc](../wrangler.jsonc). No source code or access tokens are
	249	stored — only posture facts and their raw event JSON.
	250	- Deletion. Uninstall (`installation.deleted`) purges all D1 rows and R2
	251	objects for the installation; `/admin/purge` does the same on request.
	252
	253	## Where things live
	254
	255	\| Concern \| File \|
	256	\| --- \| --- \|
	257	\| Router, handlers, retention, purge \| [src/index.ts](../src/index.ts) \|
	258	\| Webhook verification + fact extraction \| [src/webhook.ts](../src/webhook.ts) \|
	259	\| GitHub polling (protection, rulesets, access) \| [src/poller.ts](../src/poller.ts) \|
	260	\| App JWT + installation tokens \| [src/github-app.ts](../src/github-app.ts) \|
	261	\| OAuth + session sign/verify \| [src/auth.ts](../src/auth.ts) \|
	262	\| Evidence query + CSV/PDF rendering \| [src/exporter.ts](../src/exporter.ts) \|
	263	\| Access-review diff \| [src/access-review.ts](../src/access-review.ts) \|
	264	\| Dashboard HTML \| [src/dashboard.ts](../src/dashboard.ts) \|
	265	\| Schema + control mappings \| [migrations/](../migrations) \|
	266	```

Commit 8149e7fc18

Commit `8149e7fc18`