krz/orgo

  1#+TITLE: Incremental builds
  2#+DESCRIPTION: How the cache decides what to re-render, and why that shape is the architecture.
  3#+LEDE: Editing one post rebuilds four pages, whatever the size of the site.
  4
  5Incremental rebuilding is not an optimisation bolted on to orgo; it is the constraint
  6the data model was built around. Parsing is a pure function of one file's bytes, link
  7resolution reports the edges it used, and rendering is a pure function of a resolved
  8document. Those properties are what make caching sound — and they are also what make the
  9build parallel.
 10
 11You do not have to configure any of this. It is described here because knowing what
 12invalidates what explains the behaviour you will see.
 13
 14* What you observe
 15
 16#+BEGIN_EXAMPLE
 17$ orgo build content -o _site
 18built 182 page(s) (182 rendered, 0 cached) ...
 19
 20$ orgo build content -o _site
 21built 182 page(s) (0 rendered, 182 cached) ...
 22
 23$ vim content/blog/post.org && orgo build content -o _site
 24built 182 page(s) (4 rendered, 178 cached) ...
 25#+END_EXAMPLE
 26
 27The four are the post itself, its section index, its tag page, and the tag index whose
 28counts changed.
 29
 30* The render key
 31
 32Every page has a key composed from four hashes:
 33
 34| Component | Changes when |
 35|-----------+--------------|
 36| content | The source file's bytes change. |
 37| resolved links | A link's target moves, is renamed, or disappears. |
 38| config | =orgo.toml= changes, or the shared chrome does. |
 39| templates | *This page's* layout changes, or something that layout extends or includes. |
 40
 41If a page's key matches the cached one and its output file still exists, the file on disk
 42is already correct and is left untouched.
 43
 44** Template scope
 45
 46The template component covers the layout a page actually renders through, plus everything
 47that layout pulls in — followed through ={% extends %}=, ={% include %}=, ={% import %}=
 48and ={% from %}=. Editing =feed.xml= on a 196-page site re-renders one page; editing a
 49=post.html= that only blog posts use re-renders the posts. =base.html= is extended by
 50almost everything, so editing it still re-renders almost everything — which is correct,
 51and is why the win shows up on the *other* edits.
 52
 53A template whose include is computed at render time — ={% include chooser %}= — cannot be
 54followed, so it is treated as depending on every template. Over-invalidating costs time;
 55under-invalidating publishes a stale page.
 56
 57The cache lives in =<output>/.orgo-cache.json= and is tagged with a format version. A
 58version mismatch, a missing file or a corrupt file all fall back to a full rebuild — the
 59cache is an optimisation, never a correctness dependency. There is a test for each of
 60those three fallbacks.
 61
 62* Link dependencies
 63
 64Resolution records which targets each page consumed, which gives the build a dependency
 65graph. That is what makes renaming a heading work:
 66
 67#+BEGIN_EXAMPLE
 68a.org:  * Target Heading
 69b.org:  Jump to [[*Target Heading][there]].
 70#+END_EXAMPLE
 71
 72Rename the heading in =a.org= and *both* pages re-render — =b.org= because the URL it
 73emits has changed. Without the graph, =b.html= would keep a link to an anchor that no
 74longer exists. On a rebuild the graph is merged with the previous build's, so a target
 75that was *removed* still pulls in the pages that linked to it.
 76
 77* Global chrome
 78
 79The navigation appears on every page, so a change to it must re-render every page. The
 80site-structure hash covers exactly the pages that can appear in the nav — which is why
 81the default =nav.mode = "top-level"= matters for more than aesthetics:
 82
 83- Retitling a *top-level* page changes the nav everywhere, and re-renders the site.
 84- Adding a *nested* page cannot change anyone's nav, and re-renders one page.
 85
 86Turning on =[templates] expose_page_list= widens that hash to every page, because then
 87any template can read any page's metadata. That is the documented cost of building an
 88index by hand instead of with a collection.
 89
 90* Generated pages
 91
 92A listing page has no source file, so it is cached on the thing it actually depends on:
 93the entries it lists — their URLs, titles, dates and tags.
 94
 95- Adding a post re-renders the indexes that list it.
 96- Editing a post's *body* changes no listing metadata, so no index is touched.
 97- Retitling a post re-renders the listings that display the title.
 98
 99A tag page depends on its own posts and not on the other groups. That is why the group
100list is given to the tag *index* and not to every tag page: a page that could see every
101group would depend on every group, and one new post would re-render every tag page.
102
103* Byte equivalence
104
105A full build (=--no-cache=) and an incremental rebuild produce *byte-identical* output.
106This is the property everything else rests on, and it is a test rather than an intention:
107the suite builds a site both ways and compares every emitted file.
108
109* Parallelism
110
111Parsing, resolution and rendering run across cores. Measured on a 1,790-page corpus, a
112full build went from 3.98s to 0.82s on 12 cores; the 179-page reference corpus builds in
1130.07s.
114
115Parallelism is not observable in the result. Emitted bytes are unaffected, and the build
116*report* — the order of =rendered= and =skipped= — is assembled sequentially afterwards,
117so a build is reproducible run to run. There is a test for that ordering, because a
118non-deterministic report over a deterministic site would be a confusing thing to debug.
119
120* When to reach for --no-cache
121
122Almost never. Config changes, template edits and cache-format upgrades all invalidate
123correctly on their own. It exists to answer "is the cache lying to me?" — and if it ever
124is, that is a bug worth reporting, with the two builds' output to compare.
125
126#+BEGIN_SRC sh
127orgo build content -o _site --no-cache
128#+END_SRC