krz/orgo

  1# org-ssg
  2
  3An org-mode static site generator, in Rust. Org is treated as the *source language*,
  4not an inconvenient input to be normalized into markdown. The org element tree —
  5headings, drawers, blocks, links with their org-specific semantics — **is** the
  6document model, and we render that tree straight to HTML. We never round-trip through
  7a markdown-shaped intermediate representation, because the point is to preserve what
  8markdown cannot express: property drawers, TODO/priority/tag metadata on headings,
  9`#+` directives, ID links, named/captioned blocks, footnote semantics.
 10
 11The one non-obvious early commitment is **incremental builds keyed on content
 12hashing**, treated as a first-class architectural concern from day one. The discipline
 13it imposes on the data model — pure, hashable, dependency-tracked units — is the real
 14deliverable, even while the corpus is small enough that a full rebuild is instant.
 15
 16## Pipeline
 17
 18```
 19DISCOVER → PARSE → INDEX → RESOLVE → RENDER → TEMPLATE → EMIT
 20```
 21
 22PARSE and RENDER are pure functions of their inputs (cacheable, hashable). INDEX/RESOLVE
 23is the only inherently global stage — it is where the link dependency graph is born.
 24
 25| Stage | Module | Notes |
 26|---|---|---|
 27| PARSE | `src/parser.rs` | Hand-written recursive descent: line lexer → element builder → inline tokenizer. |
 28| model | `src/model.rs` | The org element tree — Elements (block) vs Objects (inline). |
 29| INDEX | `src/index.rs` | Collect link targets into a symbol table. |
 30| RESOLVE | `src/resolve.rs` | Rewrite links to URLs; return the used-target list (dependency edges). |
 31| RENDER | `src/render.rs` | Tree → HTML fragment; syntect highlighting; footnote two-pass. |
 32| TEMPLATE | `src/template.rs` | minijinja: fragment + metadata → full page. |
 33| incremental | `src/incremental.rs` | Content/config/template hashing, dep graph, cache manifest, invalidation. |
 34
 35## v1 scope (recommended; must be reconciled against a corpus audit first)
 36
 37**IN — v1 must handle:** headings with nesting; TODO keywords; priorities `[#A]`; tags;
 38property drawers; plain lists (unordered/ordered/description, checkboxes, nesting);
 39tables (with rule rows, no `#+TBLFM:`); source blocks with syntax highlighting;
 40example/quote/center blocks; links (external, internal `[[*Heading]]`/`[[#custom-id]]`,
 41`id:`); footnotes (inline and referenced); `#+` keywords/directives; inline markup
 42(bold/italic/underline/verbatim/code/strike); timestamps (active/inactive, ranges);
 43paragraphs and horizontal rules; images with `#+CAPTION`/`#+ATTR_HTML`.
 44
 45**OUT — explicitly not v1 (parse-and-ignore or reject loudly):** Babel execution /
 46`:results`; `#+TBLFM:` formulas; LaTeX / MathJax; `#+INCLUDE:`; radio targets and
 47macros; drawers other than PROPERTIES/LOGBOOK; column view / clocking / agenda
 48semantics; non-HTML export blocks; the full Unicode entity set.
 49
 50**Scope guardrail:** every IN item gets a golden-file fixture from a real document;
 51every OUT item gets a test asserting it degrades predictably (ignored, no crash). The
 52IN/OUT line is enforced by tests, defending against the project's #1 risk: scope creep
 53back toward all-of-org.
 54
 55## Phase plan
 56
 57| Phase | Scope | Status |
 58|---|---|---|
 59| **M0** | **Buildable skeleton: crate layout, module stubs, deps, test harness, fixtures** | **done** |
 60| **v0.1** | **End-to-end core parse → render: `build` a single `.org` file to HTML** | **done** |
 61| **v0.2** | **Multi-file SITE build: INDEX + RESOLVE internal links, minijinja templates, `build <src-dir> <out-dir>`, tables + footnotes** | **done** |
 62| **v0.3** | **Incremental build layer: content/config/template hashing, dependency graph, per-page render keys, persisted cache manifest, invalidation** | **done** |
 63| 0 | Corpus audit + `emacs --batch` ground-truth oracle | todo |
 64| 1 | Line lexer + heading/section skeleton | done |
 65| 2 | Block elements — lists, source blocks, tables, footnote defs done; generic drawers | partial |
 66| 3 | Inline objects — emphasis, links, bare URLs, footnote refs done; timestamps | partial |
 67| 4 | Rendering to HTML — tree walk, tables, footnote two-pass, minijinja templating done; real syntect highlighting | partial |
 68| 5 | Link resolution + symbol table (INDEX + RESOLVE, used-target list, broken-link reporting) | done |
 69| 6 | Incremental build layer (hashing, dep graph, invalidation) done; `watch` is a simple poll loop | done |
 70| 7 | Hardening: rayon parallelism, CLI polish, error locations | todo |
 71
 72### v0.2 in / out
 73
 74**Added in v0.2:** the INDEX stage (`SymbolTable` of `:ID:`/`:CUSTOM_ID:`/heading/`file:`
 75targets across a directory); the RESOLVE stage — rewrites `[[#custom-id]]`, `[[id:...]]`,
 76`[[*Heading]]` and `[[file:other.org]]` links to real relative output URLs, returns the
 77`used_targets` list (the `uses` edges, spec §4.3/R2) and reports unresolved links as
 78warnings rather than crashing; a minijinja base layout (title, nav, body) applied to every
 79page; a `build <src-dir> <out-dir>` path that walks the tree, parses + resolves + renders +
 80templates every `.org` into a linked static site and copies non-`.org` assets through;
 81plus two new constructs — pipe **tables** (with header band from the rule row) and
 82**footnotes** (block `[fn:1]` definitions, referenced `[fn:1]`, and inline `[fn:1:text]`,
 83rendered as a numbered, back-linked notes section).
 84
 85**Still stubbed (`todo!`):** timestamps; TODO keywords and priorities; generic
 86(non-PROPERTIES) drawers. Source-block syntax highlighting remains a `<pre><code>`
 87passthrough behind the `Highlighter` trait; real syntect tokenizing is deferred.
 88
 89### v0.3 in / out
 90
 91**Added in v0.3 — the incremental build layer (spec §4, the flagship, non-retrofittable
 92feature):**
 93
 94- **Three hash classes (spec §4.1)** in `src/incremental.rs`: a **content hash** (blake3
 95  of a file's bytes), a **config hash** (blake3 of the resolved `BuildConfig`), and a
 96  **template hash** (blake3 of the template sources). A change in any one invalidates the
 97  pages it affects.
 98- **Dependency graph (spec §4.3)** built from RESOLVE's `defines`/`uses` edges: a page
 99  depends on the targets it links to, so editing (or renaming a heading in) a file
100  invalidates the pages that *link into* it, not just the file itself — the load-bearing
101  R2 invariant. On rebuild the graph is merged with the previous build's `defines` so a
102  *removed* target still pulls in its linkers.
103- **Per-page `render_key`** = `H(content ⊕ resolved-links ⊕ config ⊕ template)`. If a
104  page's render key is unchanged, its on-disk output is already correct and it is skipped.
105  The config component folds in a **site-structure hash** (every page's `(path, title)`),
106  because the shared nav bar is global chrome — a title change or a page add/remove alters
107  the nav on every page and so must re-render them all (otherwise byte-equivalence breaks).
108- **Persisted cache manifest** (`<out>/.org-ssg-cache.json`, JSON), carrying per-page
109  records, the config/template hashes, and the serialized dependency graph, tagged with
110  `CACHE_FORMAT_VERSION`. A version mismatch, a missing file, or a corrupt file all fall
111  back to a clean full rebuild — the cache is an optimization, never a correctness
112  dependency.
113- **Wired into `build_site`**: only pages whose render key changed (or that link into a
114  changed file's targets) are re-rendered; unchanged outputs are left in place. `--no-cache`
115  forces a full rebuild; `clean <out-dir>` removes the output directory (and its cache).
116  `SiteReport` now reports `rendered` vs `skipped` counts.
117
118The hard gates are enforced by `tests/incremental.rs`: full-vs-incremental **byte
119equivalence** (and a second unchanged build re-rendering **zero** pages); **edit-one-file**
120re-renders exactly the changed page plus its linkers; **renamed-heading** invalidates the
121linking page and updates its emitted anchor; and cache **version-bump / missing / corrupt**
122all fall back to a full rebuild.
123
124**Out of scope in v0.3 (unchanged from v0.2):** real syntect highlighting; timestamps and
125TODO keywords. `watch` is a minimal mtime poll loop (`watch <src-dir> -o <out-dir>`), not an
126OS file-watcher — the fs-notify integration is deferred. The parse-tree cache (spec §4.5,
127"optionally") is not persisted: PARSE/INDEX/RESOLVE run for every file each build (cheap and
128pure); the incremental win is on RENDER + EMIT.
129
130**From v0.1 (core subset):** headings with nesting and anchors (every heading is now
131anchored — `:CUSTOM_ID:`/`:ID:` else a slug of its text) and trailing tags; paragraphs;
132plain lists (unordered + ordered) with checkboxes; source blocks; inline markup (`*bold*`,
133`/italic/`, `_underline_`, `+strike+`, `=verbatim=`, `~code~`); links and bare URLs.
134
135## Dependencies
136
137Parser is hand-written recursive descent (not `nom`/`chumsky`/`pest` — org is
138line-oriented and context-sensitive, not clean CFG). Key crates: `syntect` (syntax
139highlighting, behind a `Highlighter` trait so tree-sitter can be swapped in later),
140`minijinja` (runtime templates), `blake3` (content/cache hashing), `chrono`,
141`camino`, `walkdir`, `clap`, `anyhow`/`thiserror`. `insta` for snapshot tests.
142
143## Build & test
144
145```
146cargo build
147cargo test
148cargo run -- build fixtures/minimal.org -o minimal.html   # single file
149cargo run -- build fixtures/site -o _site                 # whole site (incremental)
150cargo run -- build fixtures/site -o _site --no-cache      # force a full rebuild
151cargo run -- watch fixtures/site -o _site                 # poll + rebuild on change
152cargo run -- clean _site                                  # remove output + cache
153```
154
155A second `build` of an unchanged site re-renders nothing; editing a page re-renders only
156that page and the pages that link into it (watch the `rendered`/`cached` counts).
157
158`fixtures/` holds tiny `.org` samples: single-file ones (`minimal.org`, `core.org`,
159`elements.org`, `table.org`, `footnote.org`) and a linked multi-file site under
160`fixtures/site/` (`index.org`, `guide.org`, `about.org` + a `style.css` asset). The
161real corpus (golden files derived from actual documents) lands in Phase 0. `cargo test`
162includes `insta` snapshots of the element tree and rendered HTML for the single-file
163fixtures, the two templated site pages (proving cross-file link resolution), and the
164table and footnote constructs.