krz/orgo
Lightning fast org-mode static site generator.
clone: git clone https://gitbay.org/krz/orgo.git
d6911e2a61dca2635a9a23f5db41be4239ead50e
verified · cmc
author: Christian Cleberg <hello@cleberg.net> · 2026-08-11T04:37:10Z
README.md | 44 ++++++++++++++++++-------- src/site.rs | 34 ++++++++++++++++---- tests/incremental.rs | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 147 insertions(+), 18 deletions(-) @@ -281,18 +281,38 @@ therefore returns only what was written, and the report is assembled sequentiall `parallel_builds_are_deterministic_in_output_and_report_order` holds that line, and it was verified by reintroducing the bug and watching it fail. -### The real scaling limit is not the CPU - -Going 10× on corpus size cost 17× in time before parallelism, which is superlinear — and -parallelism moves that constant without fixing it. The cause is the nav bar: it lists **every** -page, so an *n*-page site emits *n*² nav links. At 1,790 pages each page carries 1,799 links -and the output is 284 MB, against 5.5 MB for the 179-page corpus — 52× the bytes for 10× the -input. Even at the real corpus size this is already visible: 18 KB pages whose nav dwarfs the -prose, where the live site's nav has about six links. - -This is a template and configuration question rather than a bug — *which* pages belong in a -nav is a decision this project has not made yet — so it is recorded here rather than guessed -at. Until it is made, a build's cost is dominated by chrome nobody asked for. +### The real scaling limit was not the CPU + +Going 10× on corpus size cost 17× in time, which parallelism improves without fixing: the +cause was the nav bar listing **every** page, so an *n*-page site emitted *n*² nav links. At +1,790 pages each page carried 1,799 links and the output was 284 MB, against 5.5 MB for the +179-page corpus — 52× the bytes for 10× the input. + +The nav is now built from **top-level pages only** ([`is_top_level`](src/site.rs)): a nav is a +map of the site's top level, not an index of its contents, and section pages reach their +siblings through that section's landing page. Nav size becomes a function of the top level +rather than of the corpus, and the quadratic disappears. + +| 1,790-page corpus (6 top-level pages) | before | after | +|---|---|---| +| full build | 0.82s | 0.39s | +| total output | 284 MB | 34 MB | +| nav links per page | 1,799 | 6 | + +Scaling is now linear: 179 pages in 0.07s and 1,796 in 0.39s, where the small case is mostly +the fixed cost of loading syntect's syntax definitions. + +The same rule sharpened the incremental build, which is the larger win. The site-structure +hash — the thing that forces a global re-render — now covers only the pages that appear in +the nav, because those are the only ones whose title or URL affects another page. **Adding a +blog post used to re-render the entire site; now it renders one page.** A top-level page's +title still invalidates everything, correctly, since every page displays it. + +**Trade-off worth knowing:** on a site whose sections live in subdirectories, only genuinely +root-level pages appear. cleberg.net keeps its landing pages at `content/salary/index.org` +and friends, so its nav comes out as a single `index.org` entry where the live site shows +four. Treating a directory's `index.org` as top-level too is a one-line change to +`is_top_level` if that is the behaviour you want. **From v0.1 (core subset):** headings with nesting and anchors (every heading is now anchored — `:CUSTOM_ID:`/`:ID:` else a slug of its text) and trailing tags; paragraphs; @@ -127,18 +127,23 @@ fn prepare_pages(src: &Utf8Path) -> Result<(Vec<PagePrep>, SymbolTable)> { symbols.index_document(doc); } - // Nav is global; titles come from #+TITLE (falling back to the file stem) and URLs - // from each page's output path, which `#+SLUG:` can rename. - let entries: Vec<(Utf8PathBuf, String)> = docs + // Nav is global chrome; titles come from #+TITLE (falling back to the file stem) and + // URLs from each page's output path, which `#+SLUG:` can rename. + let all_pages: Vec<(Utf8PathBuf, String)> = docs .iter() .map(|d| (output_path(&d.source_path, &d.keywords), page_title(d))) .collect(); + let entries: Vec<(Utf8PathBuf, String)> = all_pages + .iter() + .filter(|(out, _)| is_top_level(out)) + .cloned() + .collect(); // Two sources emitting one page would silently drop a page — and with slugs, a // collision is a typo away and invisible in the source filenames. let mut claimed: std::collections::HashMap<&Utf8PathBuf, &Utf8PathBuf> = std::collections::HashMap::new(); - for (doc, (out, _)) in docs.iter().zip(&entries) { + for (doc, (out, _)) in docs.iter().zip(&all_pages) { if let Some(other) = claimed.insert(out, &doc.source_path) { anyhow::bail!( "output collision: {} and {} both build to {out} (check their #+SLUG:)", @@ -239,10 +244,16 @@ pub fn build_site(src: &Utf8Path, out: &Utf8Path, opts: &BuildOptions) -> Result // chrome on every page — is built from every page's (path, title), so a title/path // change or a page add/remove must re-render every page (else stale nav on disk). let cfg = BuildConfig::default(); - // Keyed on the *output* path: a `#+SLUG:` change moves a page's URL, which changes - // the nav on every other page even though no source filename moved. + // Only the pages that actually appear in the nav belong in the site-structure hash, + // because the nav is the only global chrome a page carries. Hashing *every* page + // here would mean adding one blog post re-rendered the entire site — correct, but + // needlessly: a nested page cannot change any other page's nav. + // + // Keyed on the *output* path, since a `#+SLUG:` change moves a page's URL — and so + // its nav link — even though no source filename moved. let nav_entries: Vec<(String, String)> = preps .iter() + .filter(|p| is_top_level(&p.output)) .map(|p| (p.output.to_string(), p.title.clone())) .collect(); let cfg_hash = combine(config_hash(&cfg), site_structure_hash(&nav_entries)); @@ -489,6 +500,17 @@ fn discover(src: &Utf8Path) -> Result<(Vec<Utf8PathBuf>, Vec<Utf8PathBuf>)> { Ok((org, assets)) } +/// Does this output path sit at the site root? +/// +/// The nav is the site's global chrome, and listing *every* page in it makes an `n`-page +/// site emit `n²` nav links — 1,790 pages produced 284 MB of output, most of it nav. A +/// nav is a map of the site's top level, not an index of its contents, so it is built +/// from root-level pages only. Section pages reach their siblings through that section's +/// own landing page. +fn is_top_level(output: &Utf8Path) -> bool { + output.parent().is_none_or(|p| p.as_str().is_empty()) +} + fn page_title(doc: &Document) -> String { doc.keywords .entries @@ -361,3 +361,90 @@ fn parallel_builds_are_deterministic_in_output_and_report_order() { ); } } + +/// A site with pages in subdirectories. +fn write_nested_site(src: &Utf8PathBuf) { + std::fs::create_dir_all(src.join("blog")).unwrap(); + write(src, "index.org", "#+TITLE: Home\n\nWelcome.\n"); + write(src, "about.org", "#+TITLE: About\n\nAbout me.\n"); + write(&src.join("blog"), "first.org", "#+TITLE: First Post\n\nPost body.\n"); + write(&src.join("blog"), "second.org", "#+TITLE: Second Post\n\nPost body.\n"); +} + +/// The nav is a map of the site's top level, not an index of its contents. Listing every +/// page made an n-page site emit n² nav links: 1,790 pages produced 284 MB of output, +/// nearly all of it nav. +#[test] +fn nav_lists_only_top_level_pages() { + let root = tmpdir("navtop"); + let src = root.join("src"); + std::fs::create_dir_all(&src).unwrap(); + write_nested_site(&src); + let out_dir = root.join("out"); + + build_site(&src, &out_dir, &BuildOptions::default()).unwrap(); + let home = std::fs::read_to_string(out_dir.join("index.html")).unwrap(); + let nav = home + .split("<nav>") + .nth(1) + .and_then(|s| s.split("</nav>").next()) + .expect("a nav element"); + + assert!(nav.contains("About"), "a root-level page belongs in the nav:\n{nav}"); + assert!(nav.contains("Home"), "the index page belongs in the nav:\n{nav}"); + assert!( + !nav.contains("First Post") && !nav.contains("Second Post"), + "pages in subdirectories must not appear in the nav:\n{nav}" + ); + + // Nested pages still get the nav — they just are not *in* it. + let post = std::fs::read_to_string(out_dir.join("blog/first.html")).unwrap(); + assert!( + post.contains("href=\"../about.html\"") && post.contains("href=\"../index.html\""), + "a nested page links up to the top-level nav:\n{post}" + ); +} + +/// The payoff for narrowing the site-structure hash to nav entries. Adding a blog post +/// cannot change any other page's nav, so it must not re-render the site — which is what +/// hashing *every* page's (path, title) used to force. +#[test] +fn adding_a_nested_page_does_not_rebuild_the_site() { + let root = tmpdir("navadd"); + let src = root.join("src"); + std::fs::create_dir_all(&src).unwrap(); + write_nested_site(&src); + let out_dir = root.join("out"); + + build_site(&src, &out_dir, &BuildOptions::default()).unwrap(); + + write(&src.join("blog"), "third.org", "#+TITLE: Third Post\n\nBody.\n"); + let r = build_site(&src, &out_dir, &BuildOptions::default()).unwrap(); + + assert_eq!( + r.rendered, + vec![out("blog/third.html")], + "only the new nested page renders, got: {:?}", + r.rendered + ); + assert_eq!(r.skipped.len(), 4, "every pre-existing page is reused"); +} + +/// The other half of the same rule: a page that IS in the nav still invalidates +/// everything when its title changes, because every page renders that title. +#[test] +fn retitling_a_top_level_page_still_rebuilds_the_site() { + let root = tmpdir("navretitle"); + let src = root.join("src"); + std::fs::create_dir_all(&src).unwrap(); + write_nested_site(&src); + let out_dir = root.join("out"); + + build_site(&src, &out_dir, &BuildOptions::default()).unwrap(); + write(&src, "about.org", "#+TITLE: Colophon\n\nAbout me.\n"); + let r = build_site(&src, &out_dir, &BuildOptions::default()).unwrap(); + + assert_eq!(r.rendered.len(), 4, "a nav title change re-renders every page"); + let post = std::fs::read_to_string(out_dir.join("blog/first.html")).unwrap(); + assert!(post.contains("Colophon"), "nested pages show the updated nav title"); +}