audit-labs/gh-attest

  1# Architecture
  2
  3gh-attest is a single [Cloudflare Worker](../src/index.ts) with three entry
  4points — an HTTP router, an hourly cron, and a queue consumer — backed by D1,
  5R2, and a Queue, all provisioned in Cloudflare's `eu` jurisdiction. It ingests
  6GitHub security posture as immutable timestamped snapshots and renders them into
  7auditor-ready evidence packages mapped to SOC 2 / ISO 27001 controls.
  8
  9This document is the shape of the system. For *why* a given GitHub signal counts
 10as evidence for a given control, see [framework-mapping.md](framework-mapping.md).
 11
 12## System context
 13
 14```mermaid
 15flowchart LR
 16  subgraph ext[External]
 17    GH["GitHub App<br/>webhooks · OAuth · REST API"]
 18    USER["User / Auditor<br/>browser"]
 19  end
 20
 21  subgraph cf["Cloudflare Worker — src/index.ts"]
 22    FETCH["fetch()<br/>HTTP router"]
 23    CRON["scheduled()<br/>hourly cron"]
 24    QUEUE["queue()<br/>export consumer"]
 25  end
 26
 27  subgraph store["Storage — EU jurisdiction"]
 28    D1[("D1<br/>gh-attest-db-eu")]
 29    R2[("R2<br/>gh-attest-exports-eu")]
 30    Q[["Queue<br/>generate-export"]]
 31  end
 32
 33  GH -- "POST /webhooks/*" --> FETCH
 34  USER -- "login · dashboard · exports" --> FETCH
 35  CRON -- "poll (installation token)" --> GH
 36
 37  FETCH -- "snapshots · export rows" --> D1
 38  FETCH -- "enqueue job" --> Q
 39  FETCH -- "stream download" --> R2
 40
 41  CRON -- "snapshots · retention" --> D1
 42  CRON -- "retention delete" --> R2
 43
 44  Q --> QUEUE
 45  QUEUE -- "read evidence" --> D1
 46  QUEUE -- "write file" --> R2
 47```
 48
 49## Entry points
 50
 51| Handler | Trigger | Responsibility | Code |
 52| --- | --- | --- | --- |
 53| `fetch` | HTTP request | Webhooks, OAuth/session dashboard, export requests + downloads, admin ops | [index.ts](../src/index.ts) |
 54| `scheduled` | Cron `0 * * * *` (hourly) | Poll GitHub for state webhooks never announce; enforce retention | [index.ts](../src/index.ts) |
 55| `queue` | `generate-export` message | Render CSV/PDF off the request path into R2 | [index.ts](../src/index.ts) |
 56
 57### Routes and their auth
 58
 59| Route | Auth | Purpose |
 60| --- | --- | --- |
 61| `POST /webhooks/github`, `/webhooks/marketplace` | HMAC (`GITHUB_WEBHOOK_SECRET`) | Ingest App / marketplace events |
 62| `GET /login`, `/callback`, `/logout` | OAuth state cookie | Dashboard sign-in |
 63| `GET /`, `/access-review` | Session cookie | Posture dashboard, membership diff |
 64| `POST /exports`, `/resync`, `/switch`, `/exclusions` | Session cookie | Dashboard actions (installation-scoped) |
 65| `GET /exports/:id[/download]` | Session cookie | Export status / file (scoped to installation) |
 66| `POST /admin/{poll,export,cleanup,purge}`, `GET /admin/export/:id` | Bearer (`ADMIN_TOKEN`) | Operations |
 67
 68## Ingestion — two paths into one table
 69
 70Both paths write to the append-only `snapshots` table; nothing is ever updated
 71in place, which is what makes the table a point-in-time audit trail.
 72
 73### Webhooks (change events)
 74
 75```mermaid
 76sequenceDiagram
 77  autonumber
 78  participant GH as GitHub
 79  participant W as Worker (fetch)
 80  participant D1 as D1
 81  participant R2 as R2
 82
 83  GH->>W: POST /webhooks/github (event + X-Hub-Signature-256)
 84  W->>W: verifySignature(body, GITHUB_WEBHOOK_SECRET)
 85  alt invalid signature
 86    W-->>GH: 401
 87  else valid
 88    W->>W: extractFact(event) → {resource, status}
 89    W->>D1: upsert installations row
 90    alt installation.deleted
 91      W->>D1: DELETE all rows for installation
 92      W->>R2: delete export objects
 93      W-->>GH: 200 (purged)
 94    else suspend / unsuspend
 95      W->>D1: toggle suspended_at
 96      W-->>GH: 200
 97    else normal event
 98      W->>D1: INSERT snapshot (append-only)
 99      W-->>GH: 200
100    end
101  end
102```
103
104### Hourly poll + retention (baseline / drift)
105
106Webhooks only fire on change, so protection that existed *before* install, and
107current membership, would never appear. The cron closes that gap and enforces
108the retention windows in the same invocation.
109
110```mermaid
111sequenceDiagram
112  autonumber
113  participant Cron as scheduled (hourly)
114  participant W as Worker
115  participant GH as GitHub REST
116  participant D1 as D1
117  participant R2 as R2
118
119  Cron->>W: fire
120  par Poll every active installation
121    W->>GH: app JWT → installation token
122    W->>GH: repos (minus exclusions) · branch protection · rulesets · org/team members
123    GH-->>W: current state
124    W->>D1: INSERT snapshots (one captured_at per batch)
125  and Retention cleanup
126    W->>D1: SELECT expired export r2_keys
127    W->>R2: delete expired objects
128    W->>D1: DELETE exports > 90d, snapshots > 396d
129  end
130```
131
132## Export pipeline
133
134Rendering runs off the request path via the Queue because PDF / large CSV can
135exceed request CPU limits. The `exports` row is the job's state machine
136(`queued → processing → done | error`).
137
138```mermaid
139sequenceDiagram
140  autonumber
141  participant U as User (session)
142  participant W as Worker (fetch)
143  participant D1 as D1
144  participant Q as Queue
145  participant C as Worker (queue)
146  participant R2 as R2
147
148  U->>W: POST /exports (framework, format)
149  W->>D1: INSERT exports (status=queued)
150  W->>Q: send job
151  W-->>U: 303 redirect to dashboard
152
153  Q->>C: deliver job
154  C->>D1: status=processing
155  C->>D1: buildEvidenceRows — snapshots ⋈ control_mappings
156  C->>C: renderCsv / renderPdf
157  C->>R2: put file at exports/{installation}/{jobId}.{fmt}
158  C->>D1: status=done, r2_key
159  Note over C,D1: render failure → status=error (deterministic, not retried)
160
161  U->>W: GET /exports/:id/download
162  W->>D1: lookup scoped to session installation
163  W->>R2: get object
164  R2-->>W: file body
165  W-->>U: stream (Content-Disposition: attachment)
166```
167
168## Data model
169
170`control_mappings` has **no foreign key** to `snapshots`. Mapping is a join on
171`(resource, status)` performed at query/export time — so a mapping can be
172corrected without re-ingesting webhook history. A mapping row with `status =
173NULL` matches any status for that resource.
174
175```mermaid
176erDiagram
177  installations ||--o{ snapshots : has
178  installations ||--o{ exports : has
179  installations ||--o{ repo_exclusions : has
180  snapshots }o..o{ control_mappings : "query-time join on (resource, status)"
181
182  installations {
183    integer installation_id PK
184    text    org_login
185    text    installed_at
186    text    suspended_at "null unless suspended"
187  }
188  snapshots {
189    integer id PK
190    integer installation_id FK
191    text    repo "null for org-level facts"
192    text    subject "member/team for access facts"
193    text    resource "e.g. branch_protection"
194    text    status "e.g. enabled | open | added"
195    text    raw_payload "original JSON, audit trail"
196    text    captured_at
197  }
198  control_mappings {
199    integer id PK
200    text    resource
201    text    status "null matches any"
202    text    framework "soc2 | iso27001"
203    text    control_id "e.g. CC8.1 | A.8.32"
204    text    posture "positive | negative | informational"
205    text    rationale
206  }
207  repo_exclusions {
208    integer installation_id PK,FK
209    text    repo PK "full name, out of scope"
210    text    excluded_at
211  }
212  exports {
213    text    id PK "uuid"
214    integer installation_id FK
215    text    framework
216    text    format "csv | pdf"
217    text    status "queued|processing|done|error"
218    text    r2_key "null until rendered"
219    text    error
220    text    created_at
221    text    completed_at
222  }
223```
224
225### Evidence query
226
227`buildEvidenceRows` ([exporter.ts](../src/exporter.ts)) reduces the append-only
228table to current posture, then attaches controls:
229
230```mermaid
231flowchart TB
232  S[("snapshots<br/>append-only")] --> L["latest row per<br/>(repo, subject, resource)"]
233  L --> J{{"JOIN control_mappings<br/>on resource + status"}}
234  CM[("control_mappings")] --> J
235  J --> F["filter by framework<br/>(soc2 | iso27001 | all)"]
236  F --> C["collapse branch_protection<br/>+ repository_ruleset to one row"]
237  C --> E["evidence rows<br/>control · posture · rationale"]
238  E --> CSV["renderCsv"]
239  E --> PDF["renderPdf"]
240```
241
242Unmapped `(resource, status)` pairs (e.g. `unavailable`, raw `push`) simply
243produce no rows — no evidence in either direction.
244
245Repos listed in `repo_exclusions` are filtered out of the result and skipped by
246the poll, so an excluded repo costs no subrequests and reports no gaps; its
247snapshots stay in the table, so removing the exclusion restores its history.
248
249Classic branch protection and repository rulesets both attest the same control,
250so the two are collapsed to one row per (framework, control, repo) — enabled
251wins over disabled — rather than letting an unused mechanism report a gap the
252other one covers.
253
254## Boundaries & isolation
255
256- **Multi-tenant scoping.** Every session route is scoped to the session's
257  `installationId`; the allowed set lives in the signed session, so a tampered
258  id can't widen access. Export downloads are looked up with an installation
259  filter, so one org can't read another's file by guessing its UUID.
260- **Three auth schemes.** HMAC for webhooks, signed session cookies
261  (`SESSION_SECRET`) for the dashboard, timing-safe bearer (`ADMIN_TOKEN`) for
262  `/admin/*`.
263- **Data residency.** D1, R2, and the Queue are all in the `eu` jurisdiction;
264  see [wrangler.jsonc](../wrangler.jsonc). No source code or access tokens are
265  stored — only posture facts and their raw event JSON.
266- **Deletion.** Uninstall (`installation.deleted`) purges all D1 rows and R2
267  objects for the installation; `/admin/purge` does the same on request.
268
269## Where things live
270
271| Concern | File |
272| --- | --- |
273| Router, handlers, retention, purge | [src/index.ts](../src/index.ts) |
274| Webhook verification + fact extraction | [src/webhook.ts](../src/webhook.ts) |
275| GitHub polling (protection, rulesets, access) | [src/poller.ts](../src/poller.ts) |
276| App JWT + installation tokens | [src/github-app.ts](../src/github-app.ts) |
277| OAuth + session sign/verify | [src/auth.ts](../src/auth.ts) |
278| Evidence query + CSV/PDF rendering | [src/exporter.ts](../src/exporter.ts) |
279| Access-review diff | [src/access-review.ts](../src/access-review.ts) |
280| Dashboard HTML | [src/dashboard.ts](../src/dashboard.ts) |
281| Schema + control mappings | [migrations/](../migrations) |
282```