Commit `bb6ef333ca`

bb6ef333ca9e4b42280a2b860e722e76ff9c222b

parent: 94205548ab

Unsigned ci/pages: success ci/test: success

cmc <hello@cleberg.net> · 2026-10-18 20:22 UTC

Track org 9.8 in the Emacs oracle (!11)

`cargo test --locked --test oracle` failed four snapshots on Emacs 31.1 /
org 9.8.7, identically on `main`. Neither cause is in orgo.

**org 9.8 converged on `<pre><code>`.** `ox-html.el` now wraps source
blocks the way orgo already did, so the `pre-code` divergence stopped
firing: `core` 87.9% → 91.4%, `blocks` 87.0% → 93.6%, `elements` 78.0% →
80.5%. The rule stays, because org 9.7 and earlier write a bare `<pre>`.

**`ox-html` 9.8.7 emits a malformed entity.** The heading tag prefix is
written `&nbsp;&nbsp;&nbsp` — the last reference unterminated
(`ox-html.el:2915`). `decode_entities` required the semicolon and left it
literal, so a heading with tags reduced to `A Subheading &nbsptag1 tag2`
against orgo's `A Subheading tag1 tag2`. HTML5 decodes the legacy named
set without the semicolon and a browser renders three non-breaking
spaces; the skeleton now does the same. `minimal` is back to its
committed 38/42 (90.5%).

**The `semantic-time` rule was hiding both.** It accepted any hunk where
`ours` was tags-or-text and `theirs` was timestamp-or-text — and any text
line satisfies both, so every prose difference matched. The `&nbsp`
mismatch therefore arrived labelled as a deliberate rendering choice and
`every_divergence_from_emacs_is_deliberate` kept passing. Guarded with
`any` on both sides.

That guard exposed a second divergence the rule had also been absorbing:
orgo keeps heading priority cookies, org drops them
(`org-export-with-priority` defaults to nil). It is the policy the module
docs already claim — "heading decoration is a policy difference" — but no
rule recorded it. Added as `heading-priority-cookie`.

**Reports now name the Emacs and org version they ran against.** The
oracle is whatever Emacs is on the machine, so a snapshot without that is
not interpretable, and a version bump silently rewrites the agreement
percentage — which is how this sat on `main`. The cost is that the
snapshots churn on an Emacs upgrade; the diff then says `oracle: emacs
31.1, org 9.8.7` changed, which is the thing you need to know. Pinning
Emacs instead would make it a build dependency, which the suite
deliberately avoids (it skips when Emacs is absent, as CI does).

`cargo test --locked` green, 15/15 oracle tests, `unexplained: 0` on all
twelve fixtures. `cargo clippy --all-targets --locked -- -D warnings`
clean. New unit test `unterminated_named_references_decode_like_a_browser`
pins the entity behaviour without needing any particular Emacs.

Layout: unified · split

src/skeleton.rs +47 −27

		@@ -31,6 +31,19 @@ const IGNORED: &[&str] = &["div", "span"];
31	31	/// cosmetic (`org-ul`); `href` and `src` are the content.
32	32	const KEPT_ATTRS: &[&str] = &["href", "src"];
33	33
	34	/// The named references this reduction understands. HTML5 decodes a legacy set of them
	35	/// without the closing semicolon, and org relies on that: `ox-html` 9.8 writes the
	36	/// heading tag prefix as `  &nbsp`. No name here is a prefix of another, so the
	37	/// longest match is the only match.
	38	const NAMED: &[(&str, char)] = &[
	39	("amp", '&'),
	40	("lt", '<'),
	41	("gt", '>'),
	42	("quot", '"'),
	43	("apos", '\''),
	44	("nbsp", '\u{a0}'),
	45	];
	46
34	47	/// HTML void elements, which never emit a close event.
35	48	const VOID: &[&str] = &[
36	49	"br", "hr", "img", "input", "meta", "link", "col", "area", "base", "source", "wbr",
		@@ -151,39 +164,46 @@ pub fn decode_entities(s: &str) -> String {
151	164	while let Some(amp) = rest.find('&') {
152	165	out.push_str(&rest[..amp]);
153	166	let tail = &rest[amp..];
154		let Some(semi) = tail.find(';').filter(\|s\| *s <= 12) else {
155		out.push('&');
156		rest = &tail[1..];
157		continue;
158		};
159		let entity = &tail[1..semi];
160		let decoded = match entity {
161		"amp" => Some('&'),
162		"lt" => Some('<'),
163		"gt" => Some('>'),
164		"quot" => Some('"'),
165		"apos" => Some('\''),
166		"nbsp" => Some(' '),
167		_ => entity
168		.strip_prefix('#')
169		.and_then(\|n\| match n.strip_prefix(['x', 'X']) {
170		Some(hex) => u32::from_str_radix(hex, 16).ok(),
171		None => n.parse::<u32>().ok(),
172		})
173		.and_then(char::from_u32),
174		};
175		match decoded {
176		// A non-breaking space is a space for comparison purposes.
177		Some('\u{a0}') => out.push(' '),
178		Some(c) => out.push(c),
	167	match entity_at(tail) {
	168	Some((c, len)) => {
	169	// A non-breaking space is a space for comparison purposes.
	170	out.push(if c == '\u{a0}' { ' ' } else { c });
	171	rest = &tail[len..];
	172	}
179	173	None => {
180	174	out.push('&');
181	175	rest = &tail[1..];
182		continue;
183	176	}
184	177	}
185		rest = &tail[semi + 1..];
186	178	}
187	179	out.push_str(rest);
188	180	out
189	181	}
	182
	183	/// The reference at the start of `tail`, which begins with `&`: the character it denotes
	184	/// and the bytes it occupies. The semicolon is optional for [`NAMED`], so `&nbsptag1`
	185	/// reads as a non-breaking space followed by `tag1`, the same way a browser reads it.
	186	fn entity_at(tail: &str) -> Option<(char, usize)> {
	187	if let Some(semi) = tail.find(';').filter(\|s\| *s <= 12) {
	188	if let Some(c) = named(&tail[1..semi]).or_else(\|\| numeric(&tail[1..semi])) {
	189	return Some((c, semi + 1));
	190	}
	191	}
	192	NAMED
	193	.iter()
	194	.find(\|(name, _)\| tail[1..].starts_with(name))
	195	.map(\|(name, c)\| (*c, 1 + name.len()))
	196	}
	197
	198	fn named(name: &str) -> Option<char> {
	199	NAMED.iter().find(\|(n, _)\| n == name).map(\|(_, c)\| c)
	200	}
	201
	202	fn numeric(name: &str) -> Option<char> {
	203	name.strip_prefix('#')
	204	.and_then(\|n\| match n.strip_prefix(['x', 'X']) {
	205	Some(hex) => u32::from_str_radix(hex, 16).ok(),
	206	None => n.parse::<u32>().ok(),
	207	})
	208	.and_then(char::from_u32)
	209	}

tests/oracle.rs +80 −4

		@@ -45,6 +45,27 @@ fn emacs_available() -> bool {
45	45	.unwrap_or(false)
46	46	}
47	47
	48	/// Emacs and org, as the `+` side of every report below. The ground truth moves: org
	49	/// 9.8 began wrapping source blocks in `<pre><code>`, which orgo already did, so three
	50	/// of these snapshots gained several points without orgo changing. Recorded so that
	51	/// shows in the diff as what it is rather than reading as a regression.
	52	fn oracle_version() -> &'static str {
	53	static VERSION: std::sync::OnceLock<String> = std::sync::OnceLock::new();
	54	VERSION.get_or_init(\|\| {
	55	let output = Command::new("emacs")
	56	.args([
	57	"-Q",
	58	"--batch",
	59	"--eval",
	60	"(progn (require 'org) \
	61	(princ (format \"emacs %s, org %s\" emacs-version (org-version))))",
	62	])
	63	.output()
	64	.expect("run emacs");
	65	String::from_utf8_lossy(&output.stdout).trim().to_string()
	66	})
	67	}
	68
48	69	/// Export a fixture with org's own HTML exporter.
49	70	fn org_export(fixture: &str) -> String {
50	71	let root = manifest_dir();
		@@ -105,6 +126,22 @@ fn text_of(line: &str) -> Option<&str> {
105	126	line.strip_prefix('"')?.strip_suffix('"')
106	127	}
107	128
	129	/// The one text run a hunk consists of, if that is all it is.
	130	fn one_text(lines: &[String]) -> Option<&str> {
	131	match lines {
	132	[line] => text_of(line),
	133	_ => None,
	134	}
	135	}
	136
	137	/// `TODO [#A] Write the parser` → `TODO Write the parser`.
	138	fn without_priority(text: &str) -> Option<String> {
	139	let open = text.find("[#")?;
	140	let close = open + text[open..].find(']')?;
	141	let joined = format!("{}{}", &text[..open], text[close + 1..].trim_start());
	142	Some(joined.trim().to_string())
	143	}
	144
108	145	/// Does this text run contain an org timestamp, `<2024-01-15 Mon>` or `[2024-01-15]`?
109	146	fn has_timestamp(line: &str) -> bool {
110	147	let t = match text_of(line) {
		@@ -120,14 +157,31 @@ const DELIBERATE: &[Deliberate] = &[
120	157	Deliberate {
121	158	name: "semantic-time",
122	159	matches: \|ours, theirs\| {
123		!ours.is_empty()
	160	// Both sides are allowed to carry text, so without the `any` guards this
	161	// rule matches any hunk of differing prose and files it under a rendering
	162	// choice nobody made. That is how org 9.8's malformed `&nbsp` in the heading
	163	// tag prefix arrived here labelled `semantic-time`.
	164	ours.iter().any(\|l\| is_tag(l, &["time"]))
124	165	&& ours
125	166	.iter()
126	167	.all(\|l\| is_tag(l, &["time"]) \|\| text_of(l).is_some())
	168	&& theirs.iter().any(\|l\| has_timestamp(l))
127	169	&& theirs.iter().all(\|l\| has_timestamp(l) \|\| text_of(l).is_some())
128	170	},
129	171	in_notes_only: false,
130	172	},
	173	// The priority cookie stays in the heading: the author wrote it, and a reader
	174	// skimming a task list wants to see it. Org drops it, because
	175	// `org-export-with-priority` defaults to nil. Heading decoration is a policy
	176	// difference, not a structural one.
	177	Deliberate {
	178	name: "heading-priority-cookie",
	179	matches: \|ours, theirs\| match (one_text(ours), one_text(theirs)) {
	180	(Some(ours), Some(theirs)) => without_priority(ours).as_deref() == Some(theirs),
	181	_ => false,
	182	},
	183	in_notes_only: false,
	184	},
131	185	// `<figure>`/`<figcaption>` instead of two paragraphs in a div.
132	186	Deliberate {
133	187	name: "figure-element",
		@@ -150,8 +204,10 @@ const DELIBERATE: &[Deliberate] = &[
150	204	},
151	205	in_notes_only: false,
152	206	},
153		// `<pre><code>` instead of a bare `<pre>`: the nested element is what every syntax
154		// highlighter and every reader's stylesheet expects.
	207	// `<pre><code>`: the nested element is what every syntax highlighter and every
	208	// reader's stylesheet expects. Org 9.8 emits it too, so this no longer fires — kept
	209	// because org 9.7 and earlier write a bare `<pre>`, and the oracle is whatever Emacs
	210	// is on the machine.
155	211	Deliberate {
156	212	name: "pre-code",
157	213	matches: \|ours, theirs\| {
		@@ -407,7 +463,11 @@ fn compare(fixture: &str) -> Option<String> {
407	463	}
408	464	let ours = skeleton(&our_export(fixture));
409	465	let theirs = skeleton(&org_export(fixture));
410		Some(divergence(&ours, &theirs))
	466	Some(format!(
	467	"oracle: {}\n{}",
	468	oracle_version(),
	469	divergence(&ours, &theirs)
	470	))
411	471	}
412	472
413	473	macro_rules! oracle_test {
		@@ -480,6 +540,22 @@ fn every_divergence_from_emacs_is_deliberate() {
480	540	// Invariants that must hold against the oracle, not merely be snapshotted
481	541	// ---------------------------------------------------------------------------
482	542
	543	/// org 9.8 writes the heading tag prefix as `  &nbsp` — the last reference
	544	/// unterminated. HTML5 decodes the legacy named set without the semicolon, so a browser
	545	/// reads that as three non-breaking spaces; the skeleton has to as well, or a heading
	546	/// carrying tags reads as `A Subheading &nbsptag1 tag2` and the whole report shifts.
	547	/// Asserted here rather than left to the snapshots because it must hold whichever Emacs
	548	/// is installed, including none.
	549	#[test]
	550	fn unterminated_named_references_decode_like_a_browser() {
	551	assert_eq!(
	552	decode_entities("A Subheading  &nbsptag1 tag2"),
	553	"A Subheading tag1 tag2"
	554	);
	555	assert_eq!(decode_entities("a &amp b"), "a & b");
	556	assert_eq!(decode_entities("Q&A"), "Q&A");
	557	}
	558
483	559	/// How many headings a document has and at what depth is the shape of the document.
484	560	/// Getting it wrong reorganizes someone's writing, so it is asserted rather than
485	561	/// snapshotted. Heading decoration (priority cookies, tag markup) is a policy

tests/snapshots/oracle__oracle_blocks.snap +9 −8

		@@ -2,8 +2,9 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
5		agreement: 67/77 skeleton lines (87.0%)
6		deliberate: 11 line(s) — pre-code ×6, semantic-emphasis ×4, verse-trailing-break ×1
	5	oracle: emacs 31.1, org 9.8.7
	6	agreement: 73/78 skeleton lines (93.6%)
	7	deliberate: 5 line(s) — semantic-emphasis ×4, verse-trailing-break ×1
7	8	unexplained: 0 line(s)
8	9	(- orgo, + emacs, ~ a difference we mean to have)
9	10
		@@ -44,14 +45,14 @@ unexplained: 0 line(s)
44	45	"Source"
45	46	</h2>
46	47	<pre>
47		~ pre-code (1 line(s))
	48	<code>
48	49	"def greet(name): return f\"hello {name}\""
49		~ pre-code (1 line(s))
	50	</code>
50	51	</pre>
51	52	<pre>
52		~ pre-code (1 line(s))
	53	<code>
53	54	"plain block, no language"
54		~ pre-code (1 line(s))
	55	</code>
55	56	</pre>
56	57	<h2>
57	58	"Nested"
		@@ -61,9 +62,9 @@ unexplained: 0 line(s)
61	62	"A quote containing a source block:"
62	63	</p>
63	64	<pre>
64		~ pre-code (1 line(s))
	65	<code>
65	66	"echo hi"
66		~ pre-code (1 line(s))
	67	</code>
67	68	</pre>
68	69	</blockquote>
69	70	<h2>

tests/snapshots/oracle__oracle_core.snap +5 −4

		@@ -2,8 +2,9 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
5		agreement: 51/58 skeleton lines (87.9%)
6		deliberate: 7 line(s) — list-per-bullet-type ×2, pre-code ×2, semantic-emphasis ×2
	5	oracle: emacs 31.1, org 9.8.7
	6	agreement: 53/58 skeleton lines (91.4%)
	7	deliberate: 5 line(s) — list-per-bullet-type ×2, semantic-emphasis ×2
7	8	unexplained: 0 line(s)
8	9	(- orgo, + emacs, ~ a difference we mean to have)
9	10
		@@ -60,7 +61,7 @@ unexplained: 0 line(s)
60	61	"."
61	62	</p>
62	63	<pre>
63		~ pre-code (1 line(s))
	64	<code>
64	65	"fn main() { println!(\"hello\"); }"
65		~ pre-code (1 line(s))
	66	</code>
66	67	</pre>

tests/snapshots/oracle__oracle_elements.snap +5 −4

		@@ -2,8 +2,9 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
5		agreement: 64/82 skeleton lines (78.0%)
6		deliberate: 26 line(s) — footnote-anchor-naming ×1, footnote-section-shape ×2, id-link-resolution ×1, no-colgroup ×1, pre-code ×2
	5	oracle: emacs 31.1, org 9.8.7
	6	agreement: 66/82 skeleton lines (80.5%)
	7	deliberate: 24 line(s) — footnote-anchor-naming ×1, footnote-section-shape ×2, id-link-resolution ×1, no-colgroup ×1
7	8	unexplained: 0 line(s)
8	9	(- orgo, + emacs, ~ a difference we mean to have)
9	10
		@@ -11,9 +12,9 @@ unexplained: 0 line(s)
11	12	"Code and tables"
12	13	</h2>
13	14	<pre>
14		~ pre-code (1 line(s))
	15	<code>
15	16	"fn main() { println!(\"hello\"); }"
16		~ pre-code (1 line(s))
	17	</code>
17	18	</pre>
18	19	<table>
19	20	~ no-colgroup (4 line(s))

tests/snapshots/oracle__oracle_footnote.snap +1

		@@ -2,6 +2,7 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 30/53 skeleton lines (56.6%)
6	7	deliberate: 29 line(s) — footnote-anchor-naming ×3, footnote-section-shape ×6, semantic-emphasis ×2
7	8	unexplained: 0 line(s)

tests/snapshots/oracle__oracle_headings.snap +4 −3

		@@ -2,13 +2,14 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 28/30 skeleton lines (93.3%)
6		deliberate: 2 line(s) — semantic-time ×2
	7	deliberate: 2 line(s) — heading-priority-cookie ×2
7	8	unexplained: 0 line(s)
8	9	(- orgo, + emacs, ~ a difference we mean to have)
9	10
10	11	<h2>
11		~ semantic-time (1 line(s))
	12	~ heading-priority-cookie (1 line(s))
12	13	</h2>
13	14	<p>
14	15	"A heading carrying a keyword, a priority, tags and a property drawer."
		@@ -20,7 +21,7 @@ unexplained: 0 line(s)
20	21	"Sub-headings nest by star count."
21	22	</p>
22	23	<h3>
23		~ semantic-time (1 line(s))
	24	~ heading-priority-cookie (1 line(s))
24	25	</h3>
25	26	<p>
26	27	"A priority cookie can stand alone."

tests/snapshots/oracle__oracle_images.snap +1

		@@ -2,6 +2,7 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 30/42 skeleton lines (71.4%)
6	7	deliberate: 14 line(s) — figure-element ×8, semantic-emphasis ×2
7	8	unexplained: 0 line(s)

tests/snapshots/oracle__oracle_lists.snap +1

		@@ -2,6 +2,7 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 109/111 skeleton lines (98.2%)
6	7	deliberate: 2 line(s) — semantic-emphasis ×2
7	8	unexplained: 0 line(s)

tests/snapshots/oracle__oracle_minimal.snap +1

		@@ -2,6 +2,7 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 38/42 skeleton lines (90.5%)
6	7	deliberate: 4 line(s) — semantic-emphasis ×4
7	8	unexplained: 0 line(s)

tests/snapshots/oracle__oracle_table.snap +1

		@@ -2,6 +2,7 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 30/34 skeleton lines (88.2%)
6	7	deliberate: 4 line(s) — no-colgroup ×1
7	8	unexplained: 0 line(s)

tests/snapshots/oracle__oracle_timestamps.snap +1

		@@ -2,6 +2,7 @@
2	2	source: tests/oracle.rs
3	3	expression: report
4	4	---
	5	oracle: emacs 31.1, org 9.8.7
5	6	agreement: 25/62 skeleton lines (40.3%)
6	7	deliberate: 37 line(s) — semantic-time ×5
7	8	unexplained: 0 line(s)

Commit bb6ef333ca

Commit `bb6ef333ca`