audit-labs/gh-attest

  1# Architecture
  2
  3gh-attest is a single [Cloudflare Worker](../src/index.ts) with three entry
  4points — an HTTP router, an hourly cron, and a queue consumer — backed by D1,
  5R2, and a Queue, all provisioned in Cloudflare's `eu` jurisdiction. It ingests
  6GitHub security posture as immutable timestamped snapshots and renders them into
  7auditor-ready evidence packages mapped to SOC 2 / ISO 27001 controls.
  8
  9This document is the shape of the system. For *why* a given GitHub signal counts
 10as evidence for a given control, see [framework-mapping.md](framework-mapping.md).
 11
 12## System context
 13
 14```mermaid
 15flowchart LR
 16  subgraph ext[External]
 17    GH["GitHub App<br/>webhooks · OAuth · REST API"]
 18    USER["User / Auditor<br/>browser"]
 19  end
 20
 21  subgraph cf["Cloudflare Worker — src/index.ts"]
 22    FETCH["fetch()<br/>HTTP router"]
 23    CRON["scheduled()<br/>hourly cron"]
 24    QUEUE["queue()<br/>export consumer"]
 25  end
 26
 27  subgraph store["Storage — EU jurisdiction"]
 28    D1[("D1<br/>gh-attest-db-eu")]
 29    R2[("R2<br/>gh-attest-exports-eu")]
 30    Q[["Queue<br/>generate-export"]]
 31  end
 32
 33  GH -- "POST /webhooks/*" --> FETCH
 34  USER -- "login · dashboard · exports" --> FETCH
 35  CRON -- "poll (installation token)" --> GH
 36
 37  FETCH -- "snapshots · export rows" --> D1
 38  FETCH -- "enqueue job" --> Q
 39  FETCH -- "stream download" --> R2
 40
 41  CRON -- "snapshots · retention" --> D1
 42  CRON -- "retention delete" --> R2
 43
 44  Q --> QUEUE
 45  QUEUE -- "read evidence" --> D1
 46  QUEUE -- "write file" --> R2
 47```
 48
 49## Entry points
 50
 51| Handler | Trigger | Responsibility | Code |
 52| --- | --- | --- | --- |
 53| `fetch` | HTTP request | Webhooks, OAuth/session dashboard, export requests + downloads, admin ops | [index.ts](../src/index.ts) |
 54| `scheduled` | Cron `0 * * * *` (hourly) | Poll GitHub for state webhooks never announce; enforce retention | [index.ts](../src/index.ts) |
 55| `queue` | `generate-export` message | Render CSV/PDF off the request path into R2 | [index.ts](../src/index.ts) |
 56
 57### Routes and their auth
 58
 59| Route | Auth | Purpose |
 60| --- | --- | --- |
 61| `POST /webhooks/github`, `/webhooks/marketplace` | HMAC (`GITHUB_WEBHOOK_SECRET`) | Ingest App / marketplace events |
 62| `GET /login`, `/callback`, `/logout` | OAuth state cookie | Dashboard sign-in |
 63| `GET /`, `/access-review` | Session cookie | Posture dashboard, membership diff |
 64| `POST /exports`, `/resync`, `/switch` | Session cookie | Dashboard actions (installation-scoped) |
 65| `GET /exports/:id[/download]` | Session cookie | Export status / file (scoped to installation) |
 66| `POST /admin/{poll,export,cleanup,purge}`, `GET /admin/export/:id` | Bearer (`ADMIN_TOKEN`) | Operations |
 67
 68## Ingestion — two paths into one table
 69
 70Both paths write to the append-only `snapshots` table; nothing is ever updated
 71in place, which is what makes the table a point-in-time audit trail.
 72
 73### Webhooks (change events)
 74
 75```mermaid
 76sequenceDiagram
 77  autonumber
 78  participant GH as GitHub
 79  participant W as Worker (fetch)
 80  participant D1 as D1
 81  participant R2 as R2
 82
 83  GH->>W: POST /webhooks/github (event + X-Hub-Signature-256)
 84  W->>W: verifySignature(body, GITHUB_WEBHOOK_SECRET)
 85  alt invalid signature
 86    W-->>GH: 401
 87  else valid
 88    W->>W: extractFact(event) → {resource, status}
 89    W->>D1: upsert installations row
 90    alt installation.deleted
 91      W->>D1: DELETE all rows for installation
 92      W->>R2: delete export objects
 93      W-->>GH: 200 (purged)
 94    else suspend / unsuspend
 95      W->>D1: toggle suspended_at
 96      W-->>GH: 200
 97    else normal event
 98      W->>D1: INSERT snapshot (append-only)
 99      W-->>GH: 200
100    end
101  end
102```
103
104### Hourly poll + retention (baseline / drift)
105
106Webhooks only fire on change, so protection that existed *before* install, and
107current membership, would never appear. The cron closes that gap and enforces
108the retention windows in the same invocation.
109
110```mermaid
111sequenceDiagram
112  autonumber
113  participant Cron as scheduled (hourly)
114  participant W as Worker
115  participant GH as GitHub REST
116  participant D1 as D1
117  participant R2 as R2
118
119  Cron->>W: fire
120  par Poll every active installation
121    W->>GH: app JWT → installation token
122    W->>GH: repos · branch protection · rulesets · org/team members
123    GH-->>W: current state
124    W->>D1: INSERT snapshots (one captured_at per batch)
125  and Retention cleanup
126    W->>D1: SELECT expired export r2_keys
127    W->>R2: delete expired objects
128    W->>D1: DELETE exports > 90d, snapshots > 396d
129  end
130```
131
132## Export pipeline
133
134Rendering runs off the request path via the Queue because PDF / large CSV can
135exceed request CPU limits. The `exports` row is the job's state machine
136(`queued → processing → done | error`).
137
138```mermaid
139sequenceDiagram
140  autonumber
141  participant U as User (session)
142  participant W as Worker (fetch)
143  participant D1 as D1
144  participant Q as Queue
145  participant C as Worker (queue)
146  participant R2 as R2
147
148  U->>W: POST /exports (framework, format)
149  W->>D1: INSERT exports (status=queued)
150  W->>Q: send job
151  W-->>U: 303 redirect to dashboard
152
153  Q->>C: deliver job
154  C->>D1: status=processing
155  C->>D1: buildEvidenceRows — snapshots ⋈ control_mappings
156  C->>C: renderCsv / renderPdf
157  C->>R2: put file at exports/{installation}/{jobId}.{fmt}
158  C->>D1: status=done, r2_key
159  Note over C,D1: render failure → status=error (deterministic, not retried)
160
161  U->>W: GET /exports/:id/download
162  W->>D1: lookup scoped to session installation
163  W->>R2: get object
164  R2-->>W: file body
165  W-->>U: stream (Content-Disposition: attachment)
166```
167
168## Data model
169
170`control_mappings` has **no foreign key** to `snapshots`. Mapping is a join on
171`(resource, status)` performed at query/export time — so a mapping can be
172corrected without re-ingesting webhook history. A mapping row with `status =
173NULL` matches any status for that resource.
174
175```mermaid
176erDiagram
177  installations ||--o{ snapshots : has
178  installations ||--o{ exports : has
179  snapshots }o..o{ control_mappings : "query-time join on (resource, status)"
180
181  installations {
182    integer installation_id PK
183    text    org_login
184    text    installed_at
185    text    suspended_at "null unless suspended"
186  }
187  snapshots {
188    integer id PK
189    integer installation_id FK
190    text    repo "null for org-level facts"
191    text    subject "member/team for access facts"
192    text    resource "e.g. branch_protection"
193    text    status "e.g. enabled | open | added"
194    text    raw_payload "original JSON, audit trail"
195    text    captured_at
196  }
197  control_mappings {
198    integer id PK
199    text    resource
200    text    status "null matches any"
201    text    framework "soc2 | iso27001"
202    text    control_id "e.g. CC8.1 | A.8.32"
203    text    posture "positive | negative | informational"
204    text    rationale
205  }
206  exports {
207    text    id PK "uuid"
208    integer installation_id FK
209    text    framework
210    text    format "csv | pdf"
211    text    status "queued|processing|done|error"
212    text    r2_key "null until rendered"
213    text    error
214    text    created_at
215    text    completed_at
216  }
217```
218
219### Evidence query
220
221`buildEvidenceRows` ([exporter.ts](../src/exporter.ts)) reduces the append-only
222table to current posture, then attaches controls:
223
224```mermaid
225flowchart TB
226  S[("snapshots<br/>append-only")] --> L["latest row per<br/>(repo, subject, resource)"]
227  L --> J{{"JOIN control_mappings<br/>on resource + status"}}
228  CM[("control_mappings")] --> J
229  J --> F["filter by framework<br/>(soc2 | iso27001 | all)"]
230  F --> E["evidence rows<br/>control · posture · rationale"]
231  E --> CSV["renderCsv"]
232  E --> PDF["renderPdf"]
233```
234
235Unmapped `(resource, status)` pairs (e.g. `unavailable`, raw `push`) simply
236produce no rows — no evidence in either direction.
237
238## Boundaries & isolation
239
240- **Multi-tenant scoping.** Every session route is scoped to the session's
241  `installationId`; the allowed set lives in the signed session, so a tampered
242  id can't widen access. Export downloads are looked up with an installation
243  filter, so one org can't read another's file by guessing its UUID.
244- **Three auth schemes.** HMAC for webhooks, signed session cookies
245  (`SESSION_SECRET`) for the dashboard, timing-safe bearer (`ADMIN_TOKEN`) for
246  `/admin/*`.
247- **Data residency.** D1, R2, and the Queue are all in the `eu` jurisdiction;
248  see [wrangler.jsonc](../wrangler.jsonc). No source code or access tokens are
249  stored — only posture facts and their raw event JSON.
250- **Deletion.** Uninstall (`installation.deleted`) purges all D1 rows and R2
251  objects for the installation; `/admin/purge` does the same on request.
252
253## Where things live
254
255| Concern | File |
256| --- | --- |
257| Router, handlers, retention, purge | [src/index.ts](../src/index.ts) |
258| Webhook verification + fact extraction | [src/webhook.ts](../src/webhook.ts) |
259| GitHub polling (protection, rulesets, access) | [src/poller.ts](../src/poller.ts) |
260| App JWT + installation tokens | [src/github-app.ts](../src/github-app.ts) |
261| OAuth + session sign/verify | [src/auth.ts](../src/auth.ts) |
262| Evidence query + CSV/PDF rendering | [src/exporter.ts](../src/exporter.ts) |
263| Access-review diff | [src/access-review.ts](../src/access-review.ts) |
264| Dashboard HTML | [src/dashboard.ts](../src/dashboard.ts) |
265| Schema + control mappings | [migrations/](../migrations) |
266```