krz/orgo

  1# orgo
  2
  3An org-mode static site generator, in Rust. Org is treated as the *source language*,
  4not an inconvenient input to be normalized into markdown. The org element tree —
  5headings, drawers, blocks, links with their org-specific semantics — **is** the
  6document model, and we render that tree straight to HTML. We never round-trip through
  7a markdown-shaped intermediate representation, because the point is to preserve what
  8markdown cannot express: property drawers, TODO/priority/tag metadata on headings,
  9`#+` directives, ID links, named/captioned blocks, footnote semantics.
 10
 11The one non-obvious early commitment is **incremental builds keyed on content
 12hashing**, treated as a first-class architectural concern from day one. The discipline
 13it imposes on the data model — pure, hashable, dependency-tracked units — is the real
 14deliverable, even while the corpus is small enough that a full rebuild is instant.
 15
 16**Full documentation is in [`docs/`](docs/)** — a site written in org and built by
 17orgo itself. Build and read it with:
 18
 19```bash
 20cargo run -- serve docs -o docs/_site
 21```
 22
 23## Quick start
 24
 25```bash
 26cargo run -- init my-site      # config + an editable copy of the layout + a page
 27cargo run -- build my-site -o _site
 28```
 29
 30Or skip the scaffolding entirely — point it at any directory of `.org` files:
 31
 32```bash
 33cargo run -- build ~/notes -o _site
 34```
 35
 36**Zero configuration is a supported path, not a demo.** With no `orgo.toml`, no
 37templates and no orgo-specific markup in your files, you get a complete site: pages,
 38navigation, syntax-highlighted code and the stylesheet to colour it. Configuration
 39changes what you get; it is never what makes it work.
 40
 41Discovery skips what should not be published — dot-directories such as `.git`, the config
 42file, the templates directory, and the output directory when it sits inside the source, so
 43`orgo build . -o _site` does the obvious thing.
 44
 45## Configuration
 46
 47Everything is optional. `orgo init` writes a fully commented `orgo.toml`; every
 48value below is the default.
 49
 50```toml
 51[site]
 52title = "orgo site"
 53base_url = ""          # absolute URL, no trailing slash; needed for feeds/canonical links
 54description = ""
 55language = "en"
 56
 57[nav]
 58mode = "top-level"     # top-level | all | explicit | none
 59# pages = ["index.org", "about.org"]   # for mode = "explicit"; order is preserved
 60
 61[templates]
 62dir = "templates"      # base.html replaces the built-in layout
 63expose_page_list = false
 64
 65# [[pages]]            # which layout a section renders through; base.html by default
 66# match = "blog"       # a source directory or one .org file; most specific rule wins
 67# template = "post.html"
 68
 69[highlight]
 70theme = "InspiredGitHub"
 71
 72[build]
 73drafts = false
 74assets = []            # extra directories copied to the site root, e.g. ["../theme/static"]
 75
 76[html]
 77heading_offset = 1     # a level-1 org heading becomes <h2>, beneath the layout's <h1>
 78```
 79
 80### Templates
 81
 82Drop a `base.html` into the templates directory and it replaces the built-in layout
 83entirely. Any other `.html` file there is available to `{% include %}` and
 84`{% extends %}`. Templates are [minijinja](https://docs.rs/minijinja) (Jinja2 syntax) and
 85receive:
 86
 87| Variable | What it is |
 88|---|---|
 89| `body` | the rendered page HTML — use `{{ body \| safe }}` |
 90| `page` | `.title`, `.url`, `.source`, `.date`, `.date_iso`, `.year`, `.tags`, `.content`, `.excerpt`, `.word_count`, `.reading_time`, `.toc`, `.keywords` |
 91| `site` | `.title`, `.base_url`, `.description`, `.language` |
 92| `nav` | list of `{title, url}`, relative to this page |
 93| `root` | `../`-prefix back to the site root from this page |
 94| `stylesheet` | URL of the generated `syntax.css` |
 95| `pages` | every page's metadata — only when `expose_page_list = true` |
 96
 97`page.keywords` carries **every** `#+KEYWORD:` in the file under its lowercased name, so
 98your own metadata works without this crate knowing about it: `#+CUSTOM_THING: x` is
 99`{{ page.keywords.custom_thing }}`.
100
101`base.html` is the default layout, not the only one. A `[[pages]]` rule gives a section
102its own — `match = "blog"`, `template = "post.html"` — and `#+TEMPLATE: wide.html` gives
103one page its own, which wins over any rule. A second layout usually starts with
104`{% extends "base.html" %}`.
105
106Editing a template re-renders the pages that use it — template sources are a hash input,
107so a design change never leaves a site half-updated.
108
109### Generated listing pages
110
111A blog index, an archive, a feed — output files with no source `.org` behind them.
112Repeat the block for each one:
113
114```toml
115[[collections]]
116source = "blog"             # directory to list; empty means every page
117output = "blog/index.html"  # where to write it
118template = "list.html"
119title = "Blog"
120sort = "date"               # date | title | path
121order = "desc"              # desc | asc
122nav = true                  # put this listing page in the nav
123```
124
125The template gets the collection's entries as `pages`, already sorted, plus the usual
126`site`/`nav`/`root`. It can `{% extends "base.html" %}` to inherit the site chrome:
127
128```jinja
129{% extends "base.html" %}
130{% block main %}
131<ul>{% for p in pages %}
132  <li><time datetime="{{ p.date_iso }}">{{ p.date_iso }}</time>
133      <a href="{{ root }}{{ p.url }}">{{ p.title }}</a></li>
134{% endfor %}</ul>
135{% endblock %}
136```
137
138`p.date_iso` is the `YYYY-MM-DD` extracted from `#+DATE:`, whatever org syntax it was
139written in — `[2025-09-05 Fri 10:21:00]`, `<2024-05-01 Wed>` or bare `2024-05-01`. It is
140also the sort key; pages without a parseable date sort last, so an undated draft never
141leads a dated archive.
142
143#### Pagination
144
145Set `paginate` to split a long listing across numbered pages:
146
147```toml
148[[collections]]
149source = "blog"
150output = "blog/index.html"
151paginate = 10
152paginate_output = "blog/page/{n}.html"   # {n} is the 1-based page number
153```
154
155Page 1 stays at `output`, so a section's canonical URL never moves as its page count
156changes; only pages 2..N are named by `paginate_output`. The template gets a `paginator`:
157
158```jinja
159{% if paginator and paginator.total > 1 %}
160<nav>
161  {% if paginator.prev_url %}<a href="{{ paginator.prev_url }}">Newer</a>{% endif %}
162  {% for pg in paginator.pages %}
163    <a href="{{ pg.url }}"{% if pg.current %} aria-current="page"{% endif %}>{{ pg.number }}</a>
164  {% endfor %}
165  {% if paginator.next_url %}<a href="{{ paginator.next_url }}">Older</a>{% endif %}
166</nav>
167{% endif %}
168```
169
170`paginator` carries `current`, `total`, `per_page`, `total_entries`, `prev_url`,
171`next_url`, `first_url`, `last_url`, and `pages`. Every URL is relative to the page
172carrying it, so links work from page 1 (`page/2.html`) and from page 5 (`../index.html`,
173`6.html`) without the template knowing where it sits. An unpaginated collection has no
174`paginator` at all, so `{% if paginator %}` is a reliable test in a shared template.
175
176Grouping and pagination compose: each group paginates independently, which is why
177`paginate_output` needs `{tag}` as well as `{n}` on a grouped collection. An empty
178collection still emits page 1 — a section that exists but has nothing in it should say so
179rather than 404. When the entry count shrinks, pages that no longer exist are deleted
180instead of being left serving stale posts.
181
182#### Tag pages
183
184Add `group_by` and the collection emits one page *per group* instead of one page total,
185plus an optional index of the groups:
186
187```toml
188[[collections]]
189source = "blog"
190group_by = "tags"             # "tags", or any #+KEYWORD: name to group by its value
191output = "tags/{tag}.html"    # {tag} is replaced by each group's slug
192template = "tag.html"
193title = "Tagged: {tag}"
194index_output = "tags/index.html"   # the tag index
195index_template = "tags.html"
196index_title = "Tags"
197nav = true                    # adds the *index*, not every tag
198```
199
200A group page receives its own posts as `pages` and itself as `group`
201(`.name`, `.slug`, `.url`, `.count`). The index receives `groups` — every group, sorted
202by name:
203
204```jinja
205<ul>{% for tag in groups %}
206  <li><a href="{{ root }}{{ tag.url }}">{{ tag.name }}</a> ({{ tag.count }})</li>
207{% endfor %}</ul>
208```
209
210`group_by = "tags"` is multi-valued: a post appears under every tag it carries. Any other
211value names a single-valued `#+KEYWORD:`, so `group_by = "category"` buckets by
212`#+CATEGORY:`.
213
214Two tags that would produce the same URL (`web_dev` and `web@dev` both slugify to
215`web-dev`) are a build error rather than one page silently overwriting the other.
216
217A tag page depends on its own posts and nothing else, so adding a post tagged `rust`
218re-renders that post, its section index, `tags/rust.html`, and the tag index whose counts
219changed — four pages, not one per tag. That precision is why `groups` is given to the
220index and not to every group page: a page that can see every group depends on every
221group.
222
223#### Feeds and absolute URLs
224
225**A feed is a listing page with an XML template**, not a separate feature — templates are
226loaded by full filename and any extension, so `output = "feed.xml"` with
227`template = "feed.xml"` is all it takes. `orgo init` writes a working RSS template.
228
229A feed is read away from the site that served it, so relative links in one are simply
230broken. Set `site.base_url` and use the `absolute` filter:
231
232```jinja
233<link>{{ post.url | absolute }}</link>
234<pubDate>{{ post.date_iso | rfc822 }}</pubDate>
235```
236
237| Filter | Does |
238|---|---|
239| `absolute` | site-root-relative path → absolute URL; already-absolute URLs pass through |
240| `rfc822` | any org or ISO date → the format RSS `pubDate` requires |
241| `truncate(n)` | shorten to at most `n` characters on a word boundary, with an ellipsis |
242
243Apply `absolute` to the site-root-relative values — `page.url`, `pages[].url`,
244`group.url` — and not to `nav[].url`, `paginator.*_url`, `stylesheet` or `root`, which
245are relative to the page carrying them and already correct there.
246
247With no `base_url`, `absolute` is an **error** naming the setting, rather than quietly
248emitting a relative URL that would make the feed invalid everywhere while looking fine.
249The default layout also emits `<link rel="canonical">` when a base URL is set.
250
251Listing pages are cached on the entries they list, so adding a post re-renders that
252section's index and nothing else.
253
254### Table of contents and `#+OPTIONS:`
255
256`page.toc` is the page's headings as a **tree** — `{title, anchor, level, children}` —
257because a table of contents is one, and rebuilding a tree from a flat list of levels
258inside a template is what Jinja is worst at. Its anchors come from the same function the
259renderer uses to emit heading `id`s, so a TOC link cannot drift from the heading it
260points at.
261
262```jinja
263{% macro toc_list(entries) %}
264<ul>{% for e in entries %}
265  <li><a href="#{{ e.anchor }}">{{ e.title }}</a>
266  {%- if e.children %}{{ toc_list(e.children) }}{% endif %}</li>
267{% endfor %}</ul>
268{% endmacro %}
269{% if page.toc %}{{ toc_list(page.toc) }}{% endif %}
270```
271
272Org's own per-file export switches are honoured, so a document can turn a feature off for
273itself the way its author already knows:
274
275| Switch | Effect | Site default |
276|---|---|---|
277| `#+OPTIONS: toc:nil` | empties `page.toc` for this page | `[html] toc = true` |
278| `#+OPTIONS: num:t` | numbers headings `1.`, `1.1.`, … | `[html] section_numbers = false` |
279
280**Section numbers default to off, which differs from Emacs on purpose.**
281`org-export-with-section-numbers` is on there, so an org-published site inherits numbered
282headings whether or not anyone chose them. Most sites do not want them; `num:t` or
283`section_numbers = true` gets Emacs' behaviour back, with Emacs' own
284`section-number-N` classes so the output stays diffable against the oracle.
285
286### Excerpts and drafts
287
288`page.excerpt` is a page's `#+DESCRIPTION:` when it sets one and its first paragraph
289otherwise, so a listing has something to show whether or not the author thought about
290summaries. `page.word_count` and `page.reading_time` (minutes at 200 wpm) count prose
291only — a post that is mostly a shell transcript should not read as an hour's work.
292`truncate` exists because an excerpt is usually a whole paragraph and minijinja has no
293such filter.
294
295`#+DRAFT:` keeps a page out of the build entirely — no page, and absent from listings and
296the nav rather than merely unlinked. `--drafts` includes them, which is what you want
297under `watch` while writing one. A draft is out of the symbol table too, so a link *to*
298one is reported as the dead link it would be once published.
299
300The keyword is read forgivingly: `t`, `yes`, `1` and a bare `#+DRAFT:` all mean draft,
301because writing the keyword at all is the signal. Only an explicit `nil`, `false`, `no`,
302`0` or `off` means published.
303
304### `#+SLUG:`
305
306A page's output filename comes from its `#+SLUG:` when it has one, so
307`2018-11-28-aes-encryption.org` can publish as `aes-encryption.html`. Without one the
308source filename is used. Slugs are sanitized to a single safe path component, and two
309pages claiming one URL is a build error rather than a silently dropped page.
310
311## Pipeline
312
313```
314DISCOVER → PARSE → INDEX → RESOLVE → RENDER → TEMPLATE → EMIT
315```
316
317PARSE and RENDER are pure functions of their inputs (cacheable, hashable). INDEX/RESOLVE
318is the only inherently global stage — it is where the link dependency graph is born.
319
320| Stage | Module | Notes |
321|---|---|---|
322| config | `src/config.rs` | `orgo.toml`: site metadata, nav mode, templates, theme. A hash input. |
323| PARSE | `src/parser.rs` | Hand-written recursive descent: line lexer → element builder → inline tokenizer. |
324| audit | `src/audit.rs` | Phase 0 corpus audit: construct frequencies against the IN/OUT line. |
325| model | `src/model.rs` | The org element tree — Elements (block) vs Objects (inline). |
326| INDEX | `src/index.rs` | Collect link targets into a symbol table. |
327| RESOLVE | `src/resolve.rs` | Rewrite links to URLs; return the used-target list (dependency edges). |
328| RENDER | `src/render.rs` | Tree → HTML fragment; syntect highlighting; footnote two-pass. |
329| TEMPLATE | `src/template.rs` | minijinja: fragment + metadata → full page. |
330| incremental | `src/incremental.rs` | Content/config/template hashing, dep graph, cache manifest, invalidation. |
331
332## v1 scope (delivered as of v0.4; still to be reconciled against a corpus audit)
333
334**IN — v1 must handle:** headings with nesting, at levels relative to the document's
335shallowest; TODO keywords; priorities `[#A]`; tags; property drawers; plain lists
336(unordered/ordered/description, checkboxes, `[@N]` counters, nesting); tables (with rule
337rows and org's special marker column, no `#+TBLFM:`); source blocks with syntax
338highlighting; example/quote/center/verse blocks and named special blocks; links (external,
339internal `[[*Heading]]`/`[[#custom-id]]`, `id:`); footnotes (inline and referenced); `#+`
340keywords/directives; inline markup (bold/italic/underline/verbatim/code/strike); org's
341export-time text conversions (`--`/`---`/`...`, `x^2`, `a_{b}`, `\alpha`); timestamps
342(active/inactive, ranges); paragraphs and horizontal rules; images with
343`#+CAPTION`/`#+ATTR_HTML`, numbered `Figure N:`.
344
345**OUT — explicitly not v1 (parse-and-ignore or reject loudly):** Babel execution /
346`:results`; `#+TBLFM:` formulas; LaTeX / MathJax (passed through untouched, including past
347the text conversions); `#+INCLUDE:` (never expanded — reported as a diagnostic, so a page
348is never quietly short of content); citations; radio targets and macros; drawers other
349than PROPERTIES/LOGBOOK; column view / clocking / agenda semantics; non-HTML export
350blocks.
351
352**Scope guardrail:** every IN item gets a golden-file fixture; every OUT item gets a test
353asserting it degrades predictably (ignored, no crash). The IN/OUT line is enforced by
354`tests/constructs.rs`, defending against the project's #1 risk: scope creep back toward
355all-of-org. Phase 0 checked this line against a real 179-file corpus and found it sound
356(99.9% of construct uses in scope) — but also found one thing missing from it entirely:
357`#+SLUG:`. See [Phase 0](#phase-0-the-corpus-audit-and-the-emacs-oracle).
358
359## Phase plan
360
361| Phase | Scope | Status |
362|---|---|---|
363| **M0** | **Buildable skeleton: crate layout, module stubs, deps, test harness, fixtures** | **done** |
364| **v0.1** | **End-to-end core parse → render: `build` a single `.org` file to HTML** | **done** |
365| **v0.2** | **Multi-file SITE build: INDEX + RESOLVE internal links, minijinja templates, `build <src-dir> <out-dir>`, tables + footnotes** | **done** |
366| **v0.3** | **Incremental build layer: content/config/template hashing, dependency graph, per-page render keys, persisted cache manifest, invalidation** | **done** |
367| **v0.4** | **MVP: the full v1 construct scope — heading metadata, nested/description lists, block types, timestamps, images, syntect highlighting — with the IN/OUT line under test** | **done** |
368| **0** | **Corpus audit + `emacs --batch` ground-truth oracle** | **done** |
369| 1 | Line lexer + heading/section skeleton | done |
370| 2 | Block elements — lists, source blocks, tables, footnote defs, blocks by type, drawers | done |
371| 3 | Inline objects — emphasis, links, bare URLs, footnote refs, timestamps | done |
372| 4 | Rendering to HTML — tree walk, tables, footnote two-pass, minijinja templating, syntect highlighting | done |
373| 5 | Link resolution + symbol table (INDEX + RESOLVE, used-target list, broken-link reporting) | done |
374| 6 | Incremental build layer (hashing, dep graph, invalidation); `watch` on OS filesystem events | done |
375| **7** | **Hardening: rayon parallelism, error locations in parse diagnostics** | **done** |
376| **8** | **General use: config file, user templates, nav modes, `init` scaffold, safe discovery** | **done** |
377| **9** | **Generated listing pages: `[[collections]]`, sorted indexes, feeds via XML templates** | **done** |
378| **10** | **Grouped collections: one page per tag plus a tag index — full parity with the incumbent** | **done** |
379| **11** | **Pagination: numbered pages with a `paginator` context, composing with grouping** | **done** |
380| **12** | **`base_url`: `absolute`/`rfc822` filters, a valid RSS feed in the scaffold, canonical links** | **done** |
381| **13** | **`watch` on OS filesystem events, debounced, with the feedback loop closed** | **done** |
382| **14** | **Authoring: excerpts, word count, reading time, `truncate`, and draft pages** | **done** |
383| **15** | **Table of contents, section numbers, and org's `#+OPTIONS:` per-file switches** | **done** |
384| **16** | **`serve`: development server with long-poll live reload, loopback-bound** | **done** |
385| **17** | **Bundled TOML and Org syntaxes, a user syntax directory, and org's comma escape** | **done** |
386| **18** | **Per-page layouts: `[[pages]]` rules and `#+TEMPLATE:`** | **done** |
387| **19** | **Export parity: relative heading levels, special strings, sub/superscript, caption numbering, checkbox and counter markup, table marker columns, special blocks** | **done** |
388| **20** | **Correctness debt: org's entity table, table captions, a reported `#+INCLUDE:`, and an oracle that separates deliberate divergence from defects** | **done** |
389| **21** | **Extra asset roots; per-template hashing so one layout edit does not re-render the site** | **done** |
390| **22** | **Release engineering: CI on both platforms, a checked MSRV, release binaries, a changelog, and a written compatibility promise** | **done** |
391
392### v0.2 in / out
393
394**Added in v0.2:** the INDEX stage (`SymbolTable` of `:ID:`/`:CUSTOM_ID:`/heading/`file:`
395targets across a directory); the RESOLVE stage — rewrites `[[#custom-id]]`, `[[id:...]]`,
396`[[*Heading]]` and `[[file:other.org]]` links to real relative output URLs, returns the
397`used_targets` list (the `uses` edges, spec §4.3/R2) and reports unresolved links as
398warnings rather than crashing; a minijinja base layout (title, nav, body) applied to every
399page; a `build <src-dir> <out-dir>` path that walks the tree, parses + resolves + renders +
400templates every `.org` into a linked static site and copies non-`.org` assets through;
401plus two new constructs — pipe **tables** (with header band from the rule row) and
402**footnotes** (block `[fn:1]` definitions, referenced `[fn:1]`, and inline `[fn:1:text]`,
403rendered as a numbered, back-linked notes section).
404
405**Left stubbed at v0.2, all closed in v0.4:** timestamps; TODO keywords and priorities;
406generic (non-PROPERTIES) drawers; real syntect tokenizing behind the `Highlighter` trait.
407
408### v0.3 in / out
409
410**Added in v0.3 — the incremental build layer (spec §4, the flagship, non-retrofittable
411feature):**
412
413- **Three hash classes (spec §4.1)** in `src/incremental.rs`: a **content hash** (blake3
414  of a file's bytes), a **config hash** (blake3 of the resolved `BuildConfig`), and a
415  **template hash** (blake3 of the template sources). A change in any one invalidates the
416  pages it affects.
417- **Dependency graph (spec §4.3)** built from RESOLVE's `defines`/`uses` edges: a page
418  depends on the targets it links to, so editing (or renaming a heading in) a file
419  invalidates the pages that *link into* it, not just the file itself — the load-bearing
420  R2 invariant. On rebuild the graph is merged with the previous build's `defines` so a
421  *removed* target still pulls in its linkers.
422- **Per-page `render_key`** = `H(content ⊕ resolved-links ⊕ config ⊕ template)`. If a
423  page's render key is unchanged, its on-disk output is already correct and it is skipped.
424  The config component folds in a **site-structure hash** (every page's `(path, title)`),
425  because the shared nav bar is global chrome — a title change or a page add/remove alters
426  the nav on every page and so must re-render them all (otherwise byte-equivalence breaks).
427- **Persisted cache manifest** (`<out>/.orgo-cache.json`, JSON), carrying per-page
428  records, the config/template hashes, and the serialized dependency graph, tagged with
429  `CACHE_FORMAT_VERSION`. A version mismatch, a missing file, or a corrupt file all fall
430  back to a clean full rebuild — the cache is an optimization, never a correctness
431  dependency.
432- **Wired into `build_site`**: only pages whose render key changed (or that link into a
433  changed file's targets) are re-rendered; unchanged outputs are left in place. `--no-cache`
434  forces a full rebuild; `clean <out-dir>` removes the output directory (and its cache).
435  `SiteReport` now reports `rendered` vs `skipped` counts.
436
437The hard gates are enforced by `tests/incremental.rs`: full-vs-incremental **byte
438equivalence** (and a second unchanged build re-rendering **zero** pages); **edit-one-file**
439re-renders exactly the changed page plus its linkers; **renamed-heading** invalidates the
440linking page and updates its emitted anchor; and cache **version-bump / missing / corrupt**
441all fall back to a full rebuild.
442
443**Out of scope in v0.3:** real syntect highlighting; timestamps and TODO keywords (all
444landed in v0.4). `watch` is a minimal mtime poll loop (`watch <src-dir> -o <out-dir>`), not
445an OS file-watcher — the fs-notify integration is deferred. The parse-tree cache (spec §4.5,
446"optionally") is not persisted: PARSE/INDEX/RESOLVE run for every file each build (cheap and
447pure); the incremental win is on RENDER + EMIT.
448
449### v0.4 in / out — the MVP
450
451v0.4 closes the gap between the v1 scope above and what the code actually did, so every
452construct the IN list claims is now parsed, rendered, and pinned by a golden file:
453
454- **Heading metadata** — TODO keywords (the Emacs default `TODO`/`DONE` set, matched on a
455  word boundary so `TODOs` is not one) and `[#A]` priority cookies, rendered with Emacs'
456  own export classes so the output stays diffable against an `emacs --batch` oracle.
457- **Lists** — indentation-based nesting (a sub-list renders *inside* its parent `<li>`),
458  multi-paragraph item bodies, and `term :: definition` description lists as `<dl>`.
459- **Blocks by type** — `QUOTE`, `CENTER`, `EXAMPLE`, `EXPORT` and `SRC` are now distinct
460  elements rather than all collapsing to a verbatim example block. Block matching is on the
461  specific kind, so a source block can nest inside a quote. An `html` export block passes
462  through; every other backend drops.
463- **Timestamps** — active `<...>` and inactive `[...]`, optional times, same-day time
464  ranges and `--`-joined date ranges, rendered as `<time>` with a machine-readable
465  `datetime`. Repeater/warning cookies are recognized and discarded.
466- **Images** — a description-less link to an image file renders as `<img>`; with an
467  affiliated `#+CAPTION:`/`#+ATTR_HTML:` it is promoted to a `<figure>` with the caption as
468  both `<figcaption>` and alt text. Links to non-`.org` files are now understood as asset
469  links: neither resolved nor reported as broken.
470- **Syntax highlighting** — real syntect tokenizing to CSS classes (never inline styles, so
471  themes live in the stylesheet). Every build emits the matching `syntax.css` and each page
472  links it relative to its own depth. An unknown language degrades to escaped `<pre><code>`.
473- **Diagnostics** — broken links are reported as the org syntax the author wrote
474  (`warning: b.org: unresolved link [[#setup]]`) rather than a Debug-printed enum.
475
476**The OUT line is now enforced, not just asserted.** `tests/constructs.rs` pins each
477excluded construct to a specific degradation: babel is never executed *and* a checked-in
478`#+RESULTS:` block is dropped rather than published as if it were verified output;
479`#+TBLFM:` is inert; `#+INCLUDE:` is never expanded and says so; LaTeX, macros and radio targets survive
480as literal text; drawers other than PROPERTIES are captured and dropped; unmodelled block
481types keep their content verbatim.
482
483**Still out:** `#+TODO:` per-file keyword sequences; planning lines
484(`SCHEDULED:`/`DEADLINE:`), which render as ordinary paragraphs; and fixed-width `: `
485lines.
486
487## Serving
488
489```bash
490cargo run -- serve my-site -o _site        # http://127.0.0.1:3000
491```
492
493Builds, watches, serves, and reloads the browser when a rebuild lands — the loop `watch`
494leaves half-open.
495
496- **Loopback by default.** A dev server serves unreviewed drafts off your laptop, so
497  reaching the local network is something you ask for with `--host 0.0.0.0`, never
498  something you get.
499- **The reload script is injected on the way out**, never written to disk. What you
500  deploy is the built site, and it must not carry a dev server's JavaScript.
501- **Long-polling, not WebSockets or SSE.** The browser asks "anything since generation
502  N?" and the server holds the request until there is. Instant like a push, no protocol
503  beyond ordinary HTTP, and no dependency. A streamed response would have been more
504  elegant and does not work: tiny_http buffers a response until its body ends, so a body
505  that never ends never reaches the client.
506- A reload only follows a **successful** rebuild. Reloading onto a stale page because the
507  build just failed tells you nothing; the error is already on your terminal.
508
509URL resolution is the server's security boundary and is written as a pure function with
510its own tests: `..`, percent-encoded `..`, backslashes, absolute paths and embedded NULs
511all resolve to nothing rather than to somewhere outside the output directory.
512
513## Watching
514
515```bash
516cargo run -- watch my-site -o _site
517```
518
519Rebuilds on OS filesystem events rather than polling, so it costs nothing while nothing
520happens. Write bursts are debounced — an editor saving a file writes a temp file, renames
521it over the original and touches the directory, which is one edit and several events.
522
523Two rules decide what counts as a change, and they are not the same rules the build uses
524to find content:
525
526- **A build input is a change.** Editing `orgo.toml` or a template rebuilds, even
527  though discovery skips both as non-content. The question is "would this change the
528  site?", not "is this a page?".
529- **Our own output is not.** `watch . -o _site` puts the output inside the source, so a
530  rebuild's writes raise events that would trigger a rebuild, forever. Dot-directories go
531  the same way — `.git` churns on every command — as do editor scratch files, including
532  Emacs' `file.org~` backups, which do not start with a dot.
533
534Where native watching is unavailable (some network and container filesystems), it falls
535back to polling and says so, rather than failing.
536
537## Phase 0: the corpus audit and the Emacs oracle
538
539The v1 scope was, by its own admission, *recommended* — a guess about which slice of org
540matters. Phase 0 replaces both halves of that guess with a measurement: an audit that asks
541what a real corpus actually uses, and an oracle that asks whether we render it the way
542Emacs does.
543
544The audit runs against any corpus — point it at your own notes before trusting this tool
545with them. The numbers below come from a 179-file site published today by weblorg, a
546wrapper around org's own HTML exporter, which makes it both a realistic workload and a
547directly comparable incumbent. With collections configured, orgo now reproduces
548**all 182 of that site's URLs**.
549
550```
551cargo run -- audit <src-dir>   # what does this corpus use, and is it in scope?
552cargo test --test oracle       # how does our HTML differ from Emacs' own export?
553```
554
555### What the audit found
556
557**The scope guess was sound.** 99.9% of construct uses in the corpus are in scope. The
558whole out-of-scope tail is 8 uses: four `#+TBLFM:` in a post *about* org-mode, three
559`\name` entities, and one `#+BEGIN_NOTE`.
560
561**`#+SLUG:` was a hole big enough to sink the project.** 178 of 179 files set it, and the
562published URL comes from it, not from the filename: `2018-11-28-aes-encryption.org` is
563served at `blog/aes-encryption.html`. orgo derived output paths from source filenames,
564so **169 of 179 pages would have been published at the wrong URL** — every inbound link and
565every search result, broken, by a tool that reported a clean build. Output paths now come
566from `#+SLUG:` when present ([`util::output_path`](src/util.rs)); slugs are sanitized so an
567author-supplied `../../etc/x` cannot escape the output directory, and two pages claiming one
568URL is a build error rather than a silently dropped page. Building the real corpus now
569reproduces all 179 of the live site's URLs exactly.
570
571**Some machinery is speculative.** The corpus contains no `id:`, `#custom-id` or `*Heading`
572links at all — its cross-page links are hand-written relative URLs. The INDEX/RESOLVE
573symbol table that v0.2 was built around is, against this corpus, unexercised.
574
575**An audit can lie too.** The first run reported 23 uses of a custom TODO keyword sequence.
576All 23 were false: the detector read the leading word of `* CSS Variables` as the keyword
577`CSS`. The corpus defines no `#+TODO:` sequences at all, so the true count was zero. The
578detector now matches conventional keyword names only — a tool that overstates a gap argues
579for work nobody needs.
580
581### What the oracle found
582
583`tests/oracle.rs` exports each fixture with org's own exporter via `emacs --batch`, reduces
584both sides to a semantic skeleton (element opens, closes and text, with layout `div`s,
585inline `span`s and all attributes but `href`/`src` dropped), and **snapshots the
586disagreement**. Snapshotting rather than asserting is deliberate: a checked-in divergence
587report gets reviewed and shows up as a diff, where a permanently red test gets ignored.
588Three invariants are asserted outright, and all three hold — heading structure, list
589nesting, and source-block text match Emacs exactly.
590
591**No bugs in orgo.** Every remaining divergence is a deliberate choice to emit better
592HTML than org does:
593
594| | orgo | Emacs | why |
595|---|---|---|---|
596| emphasis | `<em>`/`<strong>` | `<i>`/`<b>` | semantic, not presentational |
597| captioned image | `<figure>`/`<figcaption>` | `<p>` + `"Figure 1: …"` | real figure semantics |
598| timestamp | `<time datetime="…">` | literal `<2024-01-15 Mon>` | machine-readable |
599| footnotes | `<section><ol>` | `<h2>Footnotes:</h2>` | a list of notes is a list |
600| heading anchor | slug of the text | `org1a2b3c4` | stable, and what the live site serves |
601| code | `<pre><code>` | `<pre>` | the HTML5 idiom |
602
603One genuine semantic difference: org treats a single blank line between a `1.` list and a
604`-` list as *one* list and keeps the first item's bullet type, while we start a second list.
605We keep ours, on measurement rather than taste — the pattern occurs **zero** times in the
606corpus, so matching an org quirk would buy nothing and cost the more obvious reading.
607
608**The oracle's best catch was three bugs in itself.** Naive normalization reported code as
609corrupted (it trimmed each of syntect's per-token text runs, turning `def greet` into
610`defgreet`) and reported blocks at 36% agreement (syntect's spans flooded the diff). Both
611were measurement artifacts. A differential harness is a piece of software like any other,
612and the first divergences it reports are usually its own.
613
614## Phase 7: hardening
615
616### Parse diagnostics (`file:line: message`)
617
618The parser's contract is that it always returns a document — out-of-scope and malformed
619constructs degrade rather than crash. The gap was that they degraded *silently*, and in the
620worst cases the degradation is severe: an unterminated `#+BEGIN_SRC` reads the rest of the
621file as block content, and an unterminated drawer does the same but renders to nothing, so
622one missing line deletes most of a page from a build that reports success.
623
624`parse` now returns `Document::diagnostics`, each carrying a 1-based source line, and the
625build prints them as `file:line: message`. `--strict` turns them (and unresolved links) into
626a non-zero exit. Line numbers are threaded as an absolute offset through every nested parse,
627so a block inside a list item inside a section still reports its real file line — there is a
628test for exactly that, because reconstructed and re-indented nested slices are precisely
629where an off-by-N hides. The 179-file corpus produces zero diagnostics.
630
631### Parallelism
632
633PARSE, RESOLVE and RENDER/EMIT run under rayon. PARSE is a pure function of one file's bytes
634and RESOLVE only reads the shared symbol table, which is what makes both safe to parallelize
635at all; INDEX stays sequential.
636
637| corpus | before | after | speedup |
638|---|---|---|---|
639| 179 files (real) | 0.23s | 0.07s | 3.3× |
640| 1,790 files (10× copy) | 3.98s | 0.82s | 4.9× |
641
642Measured on 12 cores. `RAYON_NUM_THREADS=1` reproduces the old 3.98s exactly, so the gain is
643parallelism rather than incidental change, and the output is byte-identical to the sequential
644build across the whole corpus.
645
646**Parallelism must not be observable in the result.** `par_iter().collect()` preserves input
647order, so the emitted bytes are unaffected — but the build *report* is the fragile half:
648pushing to `rendered`/`skipped` from inside the parallel pass would order them by thread
649scheduling, producing a non-deterministic report over a deterministic site. The parallel pass
650therefore returns only what was written, and the report is assembled sequentially afterwards.
651`parallel_builds_are_deterministic_in_output_and_report_order` holds that line, and it was
652verified by reintroducing the bug and watching it fail.
653
654### The real scaling limit was not the CPU
655
656Going 10× on corpus size cost 17× in time, which parallelism improves without fixing: the
657cause was the nav bar listing **every** page, so an *n*-page site emitted *n*² nav links. At
6581,790 pages each page carried 1,799 links and the output was 284 MB, against 5.5 MB for the
659179-page corpus — 52× the bytes for 10× the input.
660
661The nav is now built from **top-level pages only** ([`is_top_level`](src/site.rs)): a nav is a
662map of the site's top level, not an index of its contents, and section pages reach their
663siblings through that section's landing page. Nav size becomes a function of the top level
664rather than of the corpus, and the quadratic disappears.
665
666| 1,790-page corpus (6 top-level pages) | before | after |
667|---|---|---|
668| full build | 0.82s | 0.39s |
669| total output | 284 MB | 34 MB |
670| nav links per page | 1,799 | 6 |
671
672Scaling is now linear: 179 pages in 0.07s and 1,796 in 0.39s, where the small case is mostly
673the fixed cost of loading syntect's syntax definitions.
674
675The same rule sharpened the incremental build, which is the larger win. The site-structure
676hash — the thing that forces a global re-render — now covers only the pages that appear in
677the nav, because those are the only ones whose title or URL affects another page. **Adding a
678blog post used to re-render the entire site; now it renders one page.** A top-level page's
679title still invalidates everything, correctly, since every page displays it.
680
681**Trade-off worth knowing:** on a site whose sections live in subdirectories, only genuinely
682root-level pages appear — a site keeping its landing pages at `salary/index.org` and friends
683gets a one-entry nav. That is what `nav.mode = "explicit"` is for: list the pages you want,
684in the order you want them.
685
686**From v0.1 (core subset):** headings with nesting and anchors (every heading is now
687anchored — `:CUSTOM_ID:`/`:ID:` else a slug of its text) and trailing tags; paragraphs;
688plain lists (unordered + ordered) with checkboxes; source blocks; inline markup (`*bold*`,
689`/italic/`, `_underline_`, `+strike+`, `=verbatim=`, `~code~`); links and bare URLs.
690
691## Compatibility
692
693Versions mean something as of 1.0. The **stable surface** — changing incompatibly requires
694a major version — is what you actually build a site against:
695
696| Stable | Detail |
697|---|---|
698| `orgo.toml` keys | Names, types and meaning. New keys are minor releases; removing one is major. |
699| Template context | `page`, `site`, `nav`, `root`, `pages`, `group`, `groups`, `paginator`, `stylesheet`, and the `absolute` / `rfc822` / `truncate` filters. |
700| CLI | Command names, flags, and exit codes. |
701| URLs | How a source path becomes an output path, including `#+SLUG:`. A generator that moves your URLs breaks every link anyone has to you. |
702
703Explicitly **not stable**, so that the above can be:
704
705- **The incremental cache.** Versioned, discarded on mismatch, never a correctness
706  dependency. It changes whenever it needs to, in any release.
707- **Rendered HTML details.** orgo tracks what Emacs exports from the same file, and
708  closing a gap changes markup. Changes that affect output are called out in
709  [CHANGELOG.md](CHANGELOG.md) — the class names the documentation names (`post-list`,
710  `figure-number`, `section-number-N`, `footnote-ref`) are the ones to write CSS against.
711- **The Rust API.** The crate is published so the binary can be installed with
712  `cargo install`; the library exists to serve it, and its types move as the tool does.
713
714The **MSRV is 1.88**, checked in CI on every change. orgo's own code compiles on
7151.82; the floor comes from dependencies. Raising it is a minor version, never a patch.
716
717## Dependencies
718
719Parser is hand-written recursive descent (not `nom`/`chumsky`/`pest` — org is
720line-oriented and context-sensitive, not clean CFG). Key crates: `syntect` (syntax
721highlighting, behind a `Highlighter` trait so tree-sitter can be swapped in later),
722`minijinja` (runtime templates), `blake3` (content/cache hashing), `rayon` (parallel
723PARSE/RESOLVE/RENDER), `notify` (filesystem events for `watch`), `tiny_http` (the `serve`
724development server), `toml` (config), `chrono`, `camino`, `walkdir`, `clap`, `anyhow`/`thiserror`.
725`insta` for snapshot tests, and `emacs --batch` — optional, and only for the oracle.
726
727## Build & test
728
729```
730cargo build
731cargo test                                                # 191 tests
732cargo run -- init my-site                                 # scaffold a new site
733cargo run -- build fixtures/minimal.org -o minimal.html   # single file
734cargo run -- build fixtures/site -o _site                 # whole site (incremental)
735cargo run -- audit fixtures/site                          # corpus audit (Phase 0)
736cargo run -- build fixtures/site -o _site --no-cache      # force a full rebuild
737cargo run -- watch fixtures/site -o _site                 # rebuild on filesystem events
738cargo run -- serve fixtures/site -o _site                 # ... and serve with live reload
739cargo run -- clean _site                                  # remove output + cache
740```
741
742A second `build` of an unchanged site re-renders nothing; editing a page re-renders only
743that page and the pages that link into it (watch the `rendered`/`cached` counts).
744
745A build emits `syntax.css` next to its output (the highlighter emits CSS classes, so the
746stylesheet has to come with them) and every page links it.
747
748`fixtures/` holds tiny `.org` samples: the core ones (`minimal.org`, `core.org`,
749`elements.org`, `table.org`, `footnote.org`), one per v1 construct group (`headings.org`,
750`lists.org`, `blocks.org`, `timestamps.org`, `images.org`), the scope guardrail
751(`outofscope.org`), and a linked multi-file site under `fixtures/site/` (`index.org`,
752`guide.org`, `about.org` + a `style.css` asset). The real corpus (golden files derived from
753actual documents) lands in Phase 0. `cargo test` runs `insta` snapshots of the element tree
754and rendered HTML for each fixture, the two templated site pages (proving cross-file link
755resolution), and the incremental gates.