Commit `542fdaea80`

542fdaea80e82f481ec3633ecf675aefd08e34e3

parent: b45a8e5a99

Verified · cmc

cmc <hello@cleberg.net> · 2026-08-29T01:13:07Z

AST prototype: close the conformance gaps to 11/12

Three fixes bring the tree renderer to parity with the shipped one:
- a list ends at a top-level marker of the other kind, so an ordered list and
  the bullet list after it stay separate
- footnote references carry inline text (replacing a placeholder number, which
  is the renderer's concern), and inline definitions render without the <p> a
  reference-style definition gets
- timestamps model an end (endTime same-day, endDate multi-day) so a range is
  one timestamp rendered as two <time> joined by an en-dash

The conformance test now asserts instead of reporting: only outofscope diverges,
matching the shipped renderer, and any change to that set fails.

AST-PROTOTYPE.md +38 −20

		@@ -48,36 +48,54 @@ never needs to know about fonts or colors.
48	48	The tree-based HTML renderer, measured against the same `org-conformance` corpus and the same
49	49	skeleton reduction the shipped renderer is held to:
50	50
51		8 / 12 — `blocks`, `elements`, `headings`, `images`, `lists`, `minimal`, `table`, `tblfm`.
	51	11 / 12 — parity with the shipped renderer. Everything matches except `outofscope`, which
	52	is `scope: out` in the corpus (deliberately unsupported constructs, where orgo itself may
	53	differ); the shipped renderer diverges there too.
52	54
53		Reported, not gated: `ASTConformanceReportTests` prints the score and does not fail the suite,
54		because the prototype is being measured rather than defended. Run with `ORG_DUMP=1` to print
55		the first divergence per case.
	55	Because it is at parity, `ASTConformanceTests` now asserts rather than reports: a
	56	regression fails, and `outofscope` starting to match fails as well, forcing the record to be
	57	updated. Run with `ORG_DUMP=1` to print the first divergence per case.
56	58
57		Remaining gaps, all small parser work rather than anything architectural:
	59	Getting from the first draft (8/12) to parity took three fixes, all small parser work rather
	60	than anything architectural — which is the useful signal about where the remaining effort
	61	sits:
58	62
59		\| Case \| First divergence \|
60		\|---\|---\|
61		\| `core` \| consecutive ordered lists group into one `<ol>` \|
62		\| `footnote` \| inline footnotes (`[fn:x:text]`) lose their text — the definition needs to come from the reference \|
63		\| `timestamps` \| multi-day ranges (`<a>--<b>`) render as two stamps without the en-dash join \|
64		\| `outofscope` \| deliberately unsupported constructs; the shipped renderer diverges here too \|
	63	\| Case \| Was \| Fix \|
	64	\|---\|---\|---\|
	65	\| `core` \| an ordered list and the bullet list after it merged into one list \| stop a list at a top-level marker of the other kind \|
	66	\| `footnote` \| inline footnotes (`[fn:x:text]`) dropped their text \| carry it on the reference; the notes section renders inline definitions without the `<p>` a reference-style definition gets \|
	67	\| `timestamps` \| ranges rendered one `<time>`, losing the end \| model an end (`endTime` same-day, `endDate` multi-day) and emit two `<time>` joined by an en-dash, as org exports it \|
65	68
66		Two gaps were closed while writing this, each ~10 lines, which is the useful signal about
67		where the effort sits: property drawers (`:PROPERTIES:` … `:END:`) are dropped as heading
68		metadata, and a bare image renders as `<p><img></p>` while a caption or `#+ATTR_HTML`
69		promotes it to `<figure>`.
	69	Two more were closed earlier, each ~10 lines: property drawers (`:PROPERTIES:` … `:END:`) are
	70	dropped as heading metadata, and a bare image renders as `<p><img></p>` while a caption or
	71	`#+ATTR_HTML` promotes it to `<figure>`.
	72
	73	Two model refinements came out of this, both making the tree more faithful to org rather than
	74	to a particular output:
	75
	76	`OrgObject.footnoteRef` carries `inline: [OrgObject]?` instead of a placeholder `number`.
	77	Numbering is document-wide, so it belongs to the renderer, not the parse.
	78	`OrgTimestamp` carries `endTime` and `endDate`, so a range is one timestamp with an end —
	79	matching how orgo models it — rather than two stamps with punctuation between them.
70	80
71	81	## What a migration would look like
72	82
73		1. Close the four gaps above until the tree renderer also scores 11/12.
74		2. Point `OrgRenderer.renderToHTML` at `parse` + `OrgHTMLTreeRenderer` internally, keeping the
	83	The conformance gap is closed, so the remaining steps are integration rather than parsing:
	84
	85	1. ~~Close the gaps until the tree renderer also scores 11/12.~~ Done.
	86	2. Reconcile the render-time options the shipped renderer carries and this one does not yet:
	87	`metadataHeader`, the repository-relative link/image resolution (`host`, `owner`,
	88	`imagePathSegment`/`linkPathSegment`), and URL sanitising. The tree keeps link targets
	89	typed (`.external` / `.file` / `.id`), so resolution becomes a renderer concern applied to
	90	`.file` targets rather than a closure threaded through the parse — a simplification, but
	91	it is the real work left before a swap.
	92	3. Point `OrgRenderer.renderToHTML` at `parse` + `OrgHTMLTreeRenderer` internally, keeping the
75	93	public API identical. The corpus test is the proof it is safe; consumers do not change.
76		3. Delete the single-pass renderer.
77		4. Add `OrgSwiftUI` as a separate product depending on the core, so the parser and HTML
	94	4. Delete the single-pass renderer.
	95	5. Add `OrgSwiftUI` as a separate product depending on the core, so the parser and HTML
78	96	renderer stay Foundation-only and consumers who want HTML never import SwiftUI.
79	97
80		Step 4 is where tables get built: `Grid`/`GridRow` with `.gridColumnAlignment()`, wrapped in a
	98	Step 5 is where tables get built: `Grid`/`GridRow` with `.gridColumnAlignment()`, wrapped in a
81	99	horizontal `ScrollView` for phone-width overflow — the approach MarkdownUI uses, and the
82	100	`OrgTable` node already carries the rows, the rule position, and per-column alignments it
83	101	needs.

Sources/OrgSwift/AST/OrgDocument.swift +27 −2

		@@ -179,7 +179,11 @@ public indirect enum OrgObject: Sendable, Equatable {
179	179	case code(String)
180	180	case link(OrgLink)
181	181	case image(OrgFigure)
182		case footnoteRef(label: String, number: Int)
	182	/// A footnote reference. `inline` carries the text of an inline footnote
	183	/// (`[fn:label:text]`), which defines the note at the point of use; nil for a plain
	184	/// reference whose definition appears elsewhere. Numbering is document-wide, so it is the
	185	/// renderer's job, not the parser's.
	186	case footnoteRef(label: String, inline: [OrgObject]?)
183	187	case timestamp(OrgTimestamp)
184	188	case superscript([OrgObject])
185	189	case lineBreak
		@@ -208,16 +212,37 @@ public enum OrgLinkTarget: Sendable, Equatable {
208	212	public struct OrgTimestamp: Sendable, Equatable {
209	213	public var date: String
210	214	public var time: String?
	215	/// End of a same-day time range (`<… 10:00-11:45>`).
211	216	public var endTime: String?
	217	/// End of a multi-day range (`<a>--<b>`), which org models as one timestamp with an end.
	218	public var endDate: String?
212	219	public var active: Bool
213	220
214		public init(date: String, time: String? = nil, endTime: String? = nil, active: Bool) {
	221	public init(date: String, time: String? = nil, endTime: String? = nil,
	222	endDate: String? = nil, active: Bool) {
215	223	self.date = date
216	224	self.time = time
217	225	self.endTime = endTime
	226	self.endDate = endDate
218	227	self.active = active
219	228	}
220	229
	230	/// True when this stamp spans a range, in either form.
	231	public var isRange: Bool { endTime != nil \|\| endDate != nil }
	232
	233	/// The machine value and display text of the range's end, when there is one. A multi-day
	234	/// end shows its whole date; a same-day end shows only the time, since the date is already
	235	/// on the start.
	236	public var end: (machineValue: String, displayValue: String)? {
	237	if let endDate {
	238	return (endDate, endDate)
	239	}
	240	if let endTime {
	241	return ("\(date)T\(endTime)", endTime)
	242	}
	243	return nil
	244	}
	245
221	246	/// The `datetime` attribute value / sort key: `2024-01-15` or `2024-01-15T10:30`.
222	247	public var machineValue: String {
223	248	time.map { "\(date)T\($0)" } ?? date

Sources/OrgSwift/AST/OrgHTMLTreeRenderer.swift +24 −15

		@@ -188,9 +188,16 @@ public struct OrgHTMLTreeRenderer: Sendable {
188	188	case .image(let figure): html += imageTag(figure, caption: nil, &notes)
189	189	case .timestamp(let stamp):
190	190	let cssClass = stamp.active ? "timestamp" : "timestamp inactive"
191		html += #"<time class="\#(cssClass)" datetime="\#(stamp.machineValue)">\#(stamp.displayValue)</time>"#
192		case .footnoteRef(let label, _):
193		let number = notes.number(for: label)
	191	func time(_ machine: String, _ display: String) -> String {
	192	#"<time class="\#(cssClass)" datetime="\#(machine)">\#(display)</time>"#
	193	}
	194	html += time(stamp.machineValue, stamp.displayValue)
	195	// A range is two <time> elements joined by an en-dash, as org exports it.
	196	if let end = stamp.end {
	197	html += "–" + time(end.machineValue, end.displayValue)
	198	}
	199	case .footnoteRef(let label, let inline):
	200	let number = notes.number(for: label, inline: inline)
194	201	html += ##"<sup class="footnote-ref"><a id="fnr-\##(number)" href="#fn-\##(number)">\##(number)</a></sup>"##
195	202	case .link(let link):
196	203	let href = escapeHTMLAttribute(hrefValue(link.target))
		@@ -240,7 +247,10 @@ public struct OrgHTMLTreeRenderer: Sendable {
240	247	struct FootnoteNumbering {
241	248	private var numbers: [String: Int] = [:]
242	249	private var order: [String] = []
	250	/// Reference-style definitions, gathered from the document's `[fn:x] …` lines.
243	251	private var definitions: [String: [OrgObject]] = [:]
	252	/// Inline definitions, gathered from `[fn:x:text]` references as they are rendered.
	253	private var inlineDefinitions: [String: [OrgObject]] = [:]
244	254	var figureNumber = 0
245	255
246	256	init(document: OrgDocument) {
		@@ -251,7 +261,8 @@ struct FootnoteNumbering {
251	261	}
252	262	}
253	263
254		mutating func number(for label: String) -> Int {
	264	mutating func number(for label: String, inline: [OrgObject]? = nil) -> Int {
	265	if let inline, inlineDefinitions[label] == nil { inlineDefinitions[label] = inline }
255	266	if let existing = numbers[label] { return existing }
256	267	let next = order.count + 1
257	268	numbers[label] = next
		@@ -259,23 +270,21 @@ struct FootnoteNumbering {
259	270	return next
260	271	}
261	272
262		func renderSection(_ renderer: OrgHTMLTreeRenderer) -> String {
	273	mutating func renderSection(_ renderer: OrgHTMLTreeRenderer) -> String {
263	274	guard !order.isEmpty else { return "" }
264	275	var html = "<section class=\"footnotes\" aria-label=\"Footnotes\">\n<hr>\n<ol>\n"
265	276	for label in order {
266	277	let n = numbers[label] ?? 0
267		var copy = self
268		let body = definitions[label].map { renderer.inlineForNotes($0, &copy) } ?? ""
269	278	let back = ##"<a class="footnote-back" href="#fnr-\##(n)" aria-label="Back to reference \##(n)">↩</a>"##
270		html += "<li id=\"fn-\(n)\"><p>\(body)</p>\n \(back)</li>\n"
	279	// An inline footnote's text sits directly in the item; a reference-style
	280	// definition is a paragraph, matching org's exporter.
	281	if let inline = inlineDefinitions[label] {
	282	html += "<li id=\"fn-\(n)\">\(renderer.renderInline(inline, &self)) \(back)</li>\n"
	283	} else {
	284	let body = definitions[label].map { renderer.renderInline($0, &self) } ?? ""
	285	html += "<li id=\"fn-\(n)\"><p>\(body)</p>\n \(back)</li>\n"
	286	}
271	287	}
272	288	return html + "</ol>\n</section>\n"
273	289	}
274	290	}
275
276		extension OrgHTMLTreeRenderer {
277		/// Renders a footnote definition's own inline content, reusing the body inline walk.
278		func inlineForNotes(_ objects: [OrgObject], _ notes: inout FootnoteNumbering) -> String {
279		renderInline(objects, &notes)
280		}
281		}

Sources/OrgSwift/AST/OrgInlineParser.swift +21 −3

		@@ -219,9 +219,10 @@ extension OrgParser {
219	219	let rest = String(chars[start...])
220	220	guard let match = rest.firstMatch(of: /^\[fn:([A-Za-z0-9_-]+)(?::([^\]]*))?\]/) else { return nil }
221	221	let label = String(match.1)
	222	// An inline footnote defines its note where it is used; parse that text as content.
	223	let inline = match.2.map { parseInline(String($0)) }
222	224	let consumed = rest.distance(from: rest.startIndex, to: match.range.upperBound)
223		// Numbering is assigned by the renderer, which sees the whole document; 0 is a placeholder.
224		return (.footnoteRef(label: label, number: 0), start + consumed)
	225	return (.footnoteRef(label: label, inline: inline), start + consumed)
225	226	}
226	227
227	228	private static func scanTimestamp(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {
		@@ -243,7 +244,24 @@ extension OrgParser {
243	244	time = String(timeMatch.1)
244	245	if let end = timeMatch.2 { endTime = String(end) }
245	246	}
246		return (.timestamp(OrgTimestamp(date: date, time: time, endTime: endTime, active: active)), j + 1)
	247	var next = j + 1
	248
	249	// A multi-day range joins two stamps with `--`; org models that as one timestamp
	250	// carrying an end, so consume the second stamp here rather than leaving `--` as text.
	251	var endDate: String?
	252	let opening: Character = active ? "<" : "["
	253	if next + 2 < chars.count, chars[next] == "-", chars[next + 1] == "-", chars[next + 2] == opening {
	254	var k = next + 3
	255	var second = ""
	256	while k < chars.count, chars[k] != closing { second.append(chars[k]); k += 1 }
	257	if k < chars.count, let endMatch = second.firstMatch(of: /(\d{4}-\d{2}-\d{2})/) {
	258	endDate = String(endMatch.1)
	259	next = k + 1
	260	}
	261	}
	262
	263	return (.timestamp(OrgTimestamp(date: date, time: time, endTime: endTime,
	264	endDate: endDate, active: active)), next)
247	265	}
248	266
249	267	private static func scanBareURL(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {

Sources/OrgSwift/AST/OrgParser.swift +4

		@@ -246,6 +246,9 @@ public enum OrgParser {
246	246	var block: [String] = []
247	247	var index = start
248	248	var pendingBlanks: [String] = []
	249	// A top-level marker of the other kind starts a separate list: an ordered list
	250	// followed by a bullet list is two lists, not one with mixed items.
	251	let startsOrdered = orderedListItem(in: lines[start].trimmingCharacters(in: .whitespaces)) != nil
249	252
250	253	while index < lines.count {
251	254	let line = lines[index]
		@@ -254,6 +257,7 @@ public enum OrgParser {
254	257	pendingBlanks.append(line); index += 1; continue
255	258	}
256	259	if isListMarkerLine(trimmed), !isIndentedContinuationLine(line) {
	260	guard (orderedListItem(in: trimmed) != nil) == startsOrdered else { break }
257	261	block.append(contentsOf: pendingBlanks); pendingBlanks = []
258	262	block.append(line); index += 1; continue
259	263	}

Tests/OrgSwiftTests/ASTTests.swift +74 −8

		@@ -133,6 +133,63 @@ struct ASTRendererTests {
133	133	#expect(roles.contains(.timestamp))
134	134	}
135	135
	136	@Test
	137	func consecutiveListsOfDifferentKindsStaySeparate() {
	138	// An ordered list followed by a bullet list is two lists, not one with mixed items.
	139	let doc = OrgParser.parse("""
	140	1. first
	141	2. second
	142
	143	- [ ] todo
	144	- [X] done
	145	""")
	146	let lists = doc.elements.compactMap { element -> OrgList? in
	147	if case .list(let list) = element { return list } else { return nil }
	148	}
	149	#expect(lists.count == 2)
	150	#expect(lists.first?.kind == .ordered)
	151	#expect(lists.last?.kind == .unordered)
	152	#expect(lists.last?.items.first?.checkbox == .off)
	153
	154	let html = OrgHTMLTreeRenderer().render(doc)
	155	#expect(html.contains("</ol>"))
	156	#expect(html.contains("<ul>"))
	157	}
	158
	159	@Test
	160	func inlineFootnoteDefinesItsNoteAtTheReference() {
	161	let doc = OrgParser.parse("A claim.[fn:x:defined right here]")
	162	let html = OrgHTMLTreeRenderer().render(doc)
	163	#expect(html.contains(##"href="#fn-1">1</a>"##))
	164	// Inline note text sits directly in the item; only reference-style notes get a <p>.
	165	#expect(html.contains(#"<li id="fn-1">defined right here "#))
	166	#expect(!html.contains(#"<li id="fn-1"><p>"#))
	167	}
	168
	169	@Test
	170	func referenceStyleFootnoteKeepsItsParagraph() {
	171	let doc = OrgParser.parse("A claim.[fn:1]\n\n[fn:1] The definition.")
	172	let html = OrgHTMLTreeRenderer().render(doc)
	173	#expect(html.contains(#"<li id="fn-1"><p>The definition.</p>"#))
	174	}
	175
	176	@Test
	177	func timestampRangesRenderAsTwoTimeElements() {
	178	// Same-day: the end shows only its time, since the start carries the date.
	179	let sameDay = OrgHTMLTreeRenderer().render(OrgParser.parse("Range <2024-01-15 Mon 10:00-11:45>."))
	180	#expect(sameDay.contains(#"datetime="2024-01-15T10:00">2024-01-15 10:00</time>"#))
	181	#expect(sameDay.contains("–"))
	182	#expect(sameDay.contains(#"datetime="2024-01-15T11:45">11:45</time>"#))
	183
	184	// Multi-day: one timestamp carrying an end date, rendered as two stamps.
	185	let multiDay = OrgHTMLTreeRenderer().render(OrgParser.parse("Span <2024-01-15 Mon>--<2024-01-20 Sat>."))
	186	#expect(multiDay.contains(#"datetime="2024-01-15">2024-01-15</time>"#))
	187	#expect(multiDay.contains("–"))
	188	#expect(multiDay.contains(#"datetime="2024-01-20">2024-01-20</time>"#))
	189	// The `--` join is consumed, not left as stray text.
	190	#expect(!multiDay.contains("--"))
	191	}
	192
136	193	@Test
137	194	func treeRendererProducesStructuralHTML() {
138	195	let document = OrgParser.parse("""
		@@ -155,14 +212,22 @@ struct ASTRendererTests {
155	212	}
156	213	}
157	214
158		/// How far the tree-based HTML renderer already agrees with orgo, measured with the same
159		/// corpus and the same skeleton reduction the shipped renderer is held to. This is the
160		/// prototype's honest scorecard — it is not expected to match the shipped renderer's 11/12
161		/// yet, and the number here is what a migration would have to close.
162		struct ASTConformanceReportTests {
	215	/// The tree-based HTML renderer against the same corpus and the same skeleton reduction the
	216	/// shipped renderer is held to. It now matches on every case the shipped renderer matches, so
	217	/// this asserts rather than merely reports: a regression fails, and `outofscope` starting to
	218	/// match fails too, forcing this record to be updated.
	219	struct ASTConformanceTests {
	220
	221	/// Cases the tree renderer must match. `outofscope` is `scope: out` in the corpus —
	222	/// deliberately unsupported constructs, where orgo itself may differ — so it is expected
	223	/// to diverge, exactly as it does for the shipped renderer.
	224	private static let expectedToMatch: Set<String> = [
	225	"blocks", "core", "elements", "footnote", "headings", "images",
	226	"lists", "minimal", "table", "tblfm", "timestamps",
	227	]
163	228
164	229	@Test
165		func reportCorpusAgreement() throws {
	230	func matchesTheCorpusWhereTheShippedRendererDoes() throws {
166	231	guard let dir = astCorpusCasesDir() else {
167	232	print("org-conformance corpus not found — skipping")
168	233	return
		@@ -195,8 +260,9 @@ struct ASTConformanceReportTests {
195	260	}
196	261	print("AST-CONFORMANCE matched=\(matched.count)/\(matched.count + diverged.count) \(matched.sorted())")
197	262	print("AST-CONFORMANCE diverged=\(diverged.sorted())")
198		// The prototype is measured, not gated: this test reports, it does not fail.
199		#expect(matched.count + diverged.count > 0)
	263
	264	#expect(Set(matched) == Self.expectedToMatch,
	265	"tree renderer conformance changed — matched \(matched.sorted()), expected \(Self.expectedToMatch.sorted())")
200	266	}
201	267	}
202	268

Commit 542fdaea80

Commit `542fdaea80`