Commit `b59af9ae9a`

b59af9ae9a9979f788e832e184f5b7f6b978ee66

parent: 19ab2f6aa1

Unsigned

cmc <hello@cleberg.net> · 2026-08-29 01:13 UTC

AST prototype: close the conformance gaps to 11/12

Three fixes bring the tree renderer to parity with the shipped one:
- a list ends at a top-level marker of the other kind, so an ordered list and
  the bullet list after it stay separate
- footnote references carry inline text (replacing a placeholder number, which
  is the renderer's concern), and inline definitions render without the <p> a
  reference-style definition gets
- timestamps model an end (endTime same-day, endDate multi-day) so a range is
  one timestamp rendered as two <time> joined by an en-dash

The conformance test now asserts instead of reporting: only outofscope diverges,
matching the shipped renderer, and any change to that set fails.

Layout: unified · split

AST-PROTOTYPE.md +38 −20

	@@ -48,36 +48,54 @@ never needs to know about fonts or colors.
48	The tree-based HTML renderer, measured against the same `org-conformance` corpus and the same	48	The tree-based HTML renderer, measured against the same `org-conformance` corpus and the same
49	skeleton reduction the shipped renderer is held to:	49	skeleton reduction the shipped renderer is held to:
50		50
51	8 / 12 — `blocks`, `elements`, `headings`, `images`, `lists`, `minimal`, `table`, `tblfm`.	51	11 / 12 — parity with the shipped renderer. Everything matches except `outofscope`, which
		52	is `scope: out` in the corpus (deliberately unsupported constructs, where orgo itself may
		53	differ); the shipped renderer diverges there too.
52		54
53	Reported, not gated: `ASTConformanceReportTests` prints the score and does not fail the suite,	55	Because it is at parity, `ASTConformanceTests` now asserts rather than reports: a
54	because the prototype is being measured rather than defended. Run with `ORG_DUMP=1` to print	56	regression fails, and `outofscope` starting to match fails as well, forcing the record to be
55	the first divergence per case.	57	updated. Run with `ORG_DUMP=1` to print the first divergence per case.
56		58
57	Remaining gaps, all small parser work rather than anything architectural:	59	Getting from the first draft (8/12) to parity took three fixes, all small parser work rather
		60	than anything architectural — which is the useful signal about where the remaining effort
		61	sits:
58		62
59	\| Case \| First divergence \|	63	\| Case \| Was \| Fix \|
60	\|---\|---\|	64	\|---\|---\|---\|
61	\| `core` \| consecutive ordered lists group into one `<ol>` \|	65	\| `core` \| an ordered list and the bullet list after it merged into one list \| stop a list at a top-level marker of the other kind \|
62	\| `footnote` \| inline footnotes (`[fn:x:text]`) lose their text — the definition needs to come from the reference \|	66	\| `footnote` \| inline footnotes (`[fn:x:text]`) dropped their text \| carry it on the reference; the notes section renders inline definitions without the `<p>` a reference-style definition gets \|
63	\| `timestamps` \| multi-day ranges (`<a>--<b>`) render as two stamps without the en-dash join \|	67	\| `timestamps` \| ranges rendered one `<time>`, losing the end \| model an end (`endTime` same-day, `endDate` multi-day) and emit two `<time>` joined by an en-dash, as org exports it \|
64	\| `outofscope` \| deliberately unsupported constructs; the shipped renderer diverges here too \|
65		68
66	Two gaps were closed while writing this, each ~10 lines, which is the useful signal about	69	Two more were closed earlier, each ~10 lines: property drawers (`:PROPERTIES:` … `:END:`) are
67	where the effort sits: property drawers (`:PROPERTIES:` … `:END:`) are dropped as heading	70	dropped as heading metadata, and a bare image renders as `<p><img></p>` while a caption or
68	metadata, and a bare image renders as `<p><img></p>` while a caption or `#+ATTR_HTML`	71	`#+ATTR_HTML` promotes it to `<figure>`.
69	promotes it to `<figure>`.	72
		73	Two model refinements came out of this, both making the tree more faithful to org rather than
		74	to a particular output:
		75
		76	- `OrgObject.footnoteRef` carries `inline: [OrgObject]?` instead of a placeholder `number`.
		77	Numbering is document-wide, so it belongs to the renderer, not the parse.
		78	- `OrgTimestamp` carries `endTime` and `endDate`, so a range is one timestamp with an end —
		79	matching how orgo models it — rather than two stamps with punctuation between them.
70		80
71	## What a migration would look like	81	## What a migration would look like
72		82
73	1. Close the four gaps above until the tree renderer also scores 11/12.	83	The conformance gap is closed, so the remaining steps are integration rather than parsing:
74	2. Point `OrgRenderer.renderToHTML` at `parse` + `OrgHTMLTreeRenderer` internally, keeping the	84
		85	1. ~~Close the gaps until the tree renderer also scores 11/12.~~ Done.
		86	2. Reconcile the render-time options the shipped renderer carries and this one does not yet:
		87	`metadataHeader`, the repository-relative link/image resolution (`host`, `owner`,
		88	`imagePathSegment`/`linkPathSegment`), and URL sanitising. The tree keeps link targets
		89	typed (`.external` / `.file` / `.id`), so resolution becomes a renderer concern applied to
		90	`.file` targets rather than a closure threaded through the parse — a simplification, but
		91	it is the real work left before a swap.
		92	3. Point `OrgRenderer.renderToHTML` at `parse` + `OrgHTMLTreeRenderer` internally, keeping the
75	public API identical. The corpus test is the proof it is safe; consumers do not change.	93	public API identical. The corpus test is the proof it is safe; consumers do not change.
76	3. Delete the single-pass renderer.	94	4. Delete the single-pass renderer.
77	4. Add `OrgSwiftUI` as a separate product depending on the core, so the parser and HTML	95	5. Add `OrgSwiftUI` as a separate product depending on the core, so the parser and HTML
78	renderer stay Foundation-only and consumers who want HTML never import SwiftUI.	96	renderer stay Foundation-only and consumers who want HTML never import SwiftUI.
79		97
80	Step 4 is where tables get built: `Grid`/`GridRow` with `.gridColumnAlignment()`, wrapped in a	98	Step 5 is where tables get built: `Grid`/`GridRow` with `.gridColumnAlignment()`, wrapped in a
81	horizontal `ScrollView` for phone-width overflow — the approach MarkdownUI uses, and the	99	horizontal `ScrollView` for phone-width overflow — the approach MarkdownUI uses, and the
82	`OrgTable` node already carries the rows, the rule position, and per-column alignments it	100	`OrgTable` node already carries the rows, the rule position, and per-column alignments it
83	needs.	101	needs.

Sources/OrgSwift/AST/OrgDocument.swift +27 −2

	@@ -179,7 +179,11 @@ public indirect enum OrgObject: Sendable, Equatable {
179	case code(String)	179	case code(String)
180	case link(OrgLink)	180	case link(OrgLink)
181	case image(OrgFigure)	181	case image(OrgFigure)
182	case footnoteRef(label: String, number: Int)	182	/// A footnote reference. `inline` carries the text of an inline footnote
		183	/// (`[fn:label:text]`), which defines the note at the point of use; nil for a plain
		184	/// reference whose definition appears elsewhere. Numbering is document-wide, so it is the
		185	/// renderer's job, not the parser's.
		186	case footnoteRef(label: String, inline: [OrgObject]?)
183	case timestamp(OrgTimestamp)	187	case timestamp(OrgTimestamp)
184	case superscript([OrgObject])	188	case superscript([OrgObject])
185	case lineBreak	189	case lineBreak
	@@ -208,16 +212,37 @@ public enum OrgLinkTarget: Sendable, Equatable {
208	public struct OrgTimestamp: Sendable, Equatable {	212	public struct OrgTimestamp: Sendable, Equatable {
209	public var date: String	213	public var date: String
210	public var time: String?	214	public var time: String?
		215	/// End of a same-day time range (`<… 10:00-11:45>`).
211	public var endTime: String?	216	public var endTime: String?
		217	/// End of a multi-day range (`<a>--<b>`), which org models as one timestamp with an end.
		218	public var endDate: String?
212	public var active: Bool	219	public var active: Bool
213		220
214	public init(date: String, time: String? = nil, endTime: String? = nil, active: Bool) {	221	public init(date: String, time: String? = nil, endTime: String? = nil,
		222	endDate: String? = nil, active: Bool) {
215	self.date = date	223	self.date = date
216	self.time = time	224	self.time = time
217	self.endTime = endTime	225	self.endTime = endTime
		226	self.endDate = endDate
218	self.active = active	227	self.active = active
219	}	228	}
220		229
		230	/// True when this stamp spans a range, in either form.
		231	public var isRange: Bool { endTime != nil \|\| endDate != nil }
		232
		233	/// The machine value and display text of the range's end, when there is one. A multi-day
		234	/// end shows its whole date; a same-day end shows only the time, since the date is already
		235	/// on the start.
		236	public var end: (machineValue: String, displayValue: String)? {
		237	if let endDate {
		238	return (endDate, endDate)
		239	}
		240	if let endTime {
		241	return ("\(date)T\(endTime)", endTime)
		242	}
		243	return nil
		244	}
		245
221	/// The `datetime` attribute value / sort key: `2024-01-15` or `2024-01-15T10:30`.	246	/// The `datetime` attribute value / sort key: `2024-01-15` or `2024-01-15T10:30`.
222	public var machineValue: String {	247	public var machineValue: String {
223	time.map { "\(date)T\($0)" } ?? date	248	time.map { "\(date)T\($0)" } ?? date

Sources/OrgSwift/AST/OrgHTMLTreeRenderer.swift +24 −15

             case .image(let figure): html += imageTag(figure, caption: nil, &notes)
             case .timestamp(let stamp):
                 let cssClass = stamp.active ? "timestamp" : "timestamp inactive"
-                html += #"<time class="\#(cssClass)" datetime="\#(stamp.machineValue)">\#(stamp.displayValue)</time>"#
+                func time(_ machine: String, _ display: String) -> String {
-            case .footnoteRef(let label, _):
+                    #"<time class="\#(cssClass)" datetime="\#(machine)">\#(display)</time>"#
-                let number = notes.number(for: label)
+                }
+                html += time(stamp.machineValue, stamp.displayValue)
+                // A range is two <time> elements joined by an en-dash, as org exports it.
+                if let end = stamp.end {
+                    html += "&#8211;" + time(end.machineValue, end.displayValue)
+                }
+            case .footnoteRef(let label, let inline):
+                let number = notes.number(for: label, inline: inline)
                 html += ##"<sup class="footnote-ref"><a id="fnr-\##(number)" href="#fn-\##(number)">\##(number)</a></sup>"##
             case .link(let link):
                 let href = escapeHTMLAttribute(hrefValue(link.target))
 struct FootnoteNumbering {
     private var numbers: [String: Int] = [:]
     private var order: [String] = []
+    /// Reference-style definitions, gathered from the document's `[fn:x] …` lines.
     private var definitions: [String: [OrgObject]] = [:]
+    /// Inline definitions, gathered from `[fn:x:text]` references as they are rendered.
+    private var inlineDefinitions: [String: [OrgObject]] = [:]
     var figureNumber = 0
     init(document: OrgDocument) {
         }
     }
-    mutating func number(for label: String) -> Int {
+    mutating func number(for label: String, inline: [OrgObject]? = nil) -> Int {
+        if let inline, inlineDefinitions[label] == nil { inlineDefinitions[label] = inline }
         if let existing = numbers[label] { return existing }
         let next = order.count + 1
         numbers[label] = next
         return next
     }
-    func renderSection(_ renderer: OrgHTMLTreeRenderer) -> String {
+    mutating func renderSection(_ renderer: OrgHTMLTreeRenderer) -> String {
         guard !order.isEmpty else { return "" }
         var html = "<section class=\"footnotes\" aria-label=\"Footnotes\">\n<hr>\n<ol>\n"
         for label in order {
             let n = numbers[label] ?? 0
-            var copy = self
-            let body = definitions[label].map { renderer.inlineForNotes($0, &copy) } ?? ""
             let back = ##"<a class="footnote-back" href="#fnr-\##(n)" aria-label="Back to reference \##(n)">&#8617;</a>"##
-            html += "<li id=\"fn-\(n)\"><p>\(body)</p>\n \(back)</li>\n"
+            // An inline footnote's text sits directly in the item; a reference-style
+            // definition is a paragraph, matching org's exporter.
+            if let inline = inlineDefinitions[label] {
+                html += "<li id=\"fn-\(n)\">\(renderer.renderInline(inline, &self)) \(back)</li>\n"
+            } else {
+                let body = definitions[label].map { renderer.renderInline($0, &self) } ?? ""
+                html += "<li id=\"fn-\(n)\"><p>\(body)</p>\n \(back)</li>\n"
+            }
         }
         return html + "</ol>\n</section>\n"
     }
 }
-extension OrgHTMLTreeRenderer {
-    /// Renders a footnote definition's own inline content, reusing the body inline walk.
-    func inlineForNotes(_ objects: [OrgObject], _ notes: inout FootnoteNumbering) -> String {
-        renderInline(objects, &notes)
-    }
-}

Sources/OrgSwift/AST/OrgInlineParser.swift +21 −3

	@@ -219,9 +219,10 @@ extension OrgParser {
219	let rest = String(chars[start...])	219	let rest = String(chars[start...])
220	guard let match = rest.firstMatch(of: /^\[fn:([A-Za-z0-9_-]+)(?::([^\]]*))?\]/) else { return nil }	220	guard let match = rest.firstMatch(of: /^\[fn:([A-Za-z0-9_-]+)(?::([^\]]*))?\]/) else { return nil }
221	let label = String(match.1)	221	let label = String(match.1)
		222	// An inline footnote defines its note where it is used; parse that text as content.
		223	let inline = match.2.map { parseInline(String($0)) }
222	let consumed = rest.distance(from: rest.startIndex, to: match.range.upperBound)	224	let consumed = rest.distance(from: rest.startIndex, to: match.range.upperBound)
223	// Numbering is assigned by the renderer, which sees the whole document; 0 is a placeholder.	225	return (.footnoteRef(label: label, inline: inline), start + consumed)
224	return (.footnoteRef(label: label, number: 0), start + consumed)
225	}	226	}
226		227
227	private static func scanTimestamp(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {	228	private static func scanTimestamp(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {
	@@ -243,7 +244,24 @@ extension OrgParser {
243	time = String(timeMatch.1)	244	time = String(timeMatch.1)
244	if let end = timeMatch.2 { endTime = String(end) }	245	if let end = timeMatch.2 { endTime = String(end) }
245	}	246	}
246	return (.timestamp(OrgTimestamp(date: date, time: time, endTime: endTime, active: active)), j + 1)	247	var next = j + 1
		248
		249	// A multi-day range joins two stamps with `--`; org models that as one timestamp
		250	// carrying an end, so consume the second stamp here rather than leaving `--` as text.
		251	var endDate: String?
		252	let opening: Character = active ? "<" : "["
		253	if next + 2 < chars.count, chars[next] == "-", chars[next + 1] == "-", chars[next + 2] == opening {
		254	var k = next + 3
		255	var second = ""
		256	while k < chars.count, chars[k] != closing { second.append(chars[k]); k += 1 }
		257	if k < chars.count, let endMatch = second.firstMatch(of: /(\d{4}-\d{2}-\d{2})/) {
		258	endDate = String(endMatch.1)
		259	next = k + 1
		260	}
		261	}
		262
		263	return (.timestamp(OrgTimestamp(date: date, time: time, endTime: endTime,
		264	endDate: endDate, active: active)), next)
247	}	265	}
248		266
249	private static func scanBareURL(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {	267	private static func scanBareURL(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {

Sources/OrgSwift/AST/OrgParser.swift +4

         var block: [String] = []
         var index = start
         var pendingBlanks: [String] = []
+        // A top-level marker of the other kind starts a *separate* list: an ordered list
+        // followed by a bullet list is two lists, not one with mixed items.
+        let startsOrdered = orderedListItem(in: lines[start].trimmingCharacters(in: .whitespaces)) != nil
         while index < lines.count {
             let line = lines[index]
                 pendingBlanks.append(line); index += 1; continue
             }
             if isListMarkerLine(trimmed), !isIndentedContinuationLine(line) {
+                guard (orderedListItem(in: trimmed) != nil) == startsOrdered else { break }
                 block.append(contentsOf: pendingBlanks); pendingBlanks = []
                 block.append(line); index += 1; continue
             }

Tests/OrgSwiftTests/ASTTests.swift +74 −8

	@@ -133,6 +133,63 @@ struct ASTRendererTests {
133	#expect(roles.contains(.timestamp))	133	#expect(roles.contains(.timestamp))
134	}	134	}
135		135
		136	@Test
		137	func consecutiveListsOfDifferentKindsStaySeparate() {
		138	// An ordered list followed by a bullet list is two lists, not one with mixed items.
		139	let doc = OrgParser.parse("""
		140	1. first
		141	2. second
		142
		143	- [ ] todo
		144	- [X] done
		145	""")
		146	let lists = doc.elements.compactMap { element -> OrgList? in
		147	if case .list(let list) = element { return list } else { return nil }
		148	}
		149	#expect(lists.count == 2)
		150	#expect(lists.first?.kind == .ordered)
		151	#expect(lists.last?.kind == .unordered)
		152	#expect(lists.last?.items.first?.checkbox == .off)
		153
		154	let html = OrgHTMLTreeRenderer().render(doc)
		155	#expect(html.contains("</ol>"))
		156	#expect(html.contains("<ul>"))
		157	}
		158
		159	@Test
		160	func inlineFootnoteDefinesItsNoteAtTheReference() {
		161	let doc = OrgParser.parse("A claim.[fn:x:defined right here]")
		162	let html = OrgHTMLTreeRenderer().render(doc)
		163	#expect(html.contains(##"href="#fn-1">1</a>"##))
		164	// Inline note text sits directly in the item; only reference-style notes get a <p>.
		165	#expect(html.contains(#"<li id="fn-1">defined right here "#))
		166	#expect(!html.contains(#"<li id="fn-1"><p>"#))
		167	}
		168
		169	@Test
		170	func referenceStyleFootnoteKeepsItsParagraph() {
		171	let doc = OrgParser.parse("A claim.[fn:1]\n\n[fn:1] The definition.")
		172	let html = OrgHTMLTreeRenderer().render(doc)
		173	#expect(html.contains(#"<li id="fn-1"><p>The definition.</p>"#))
		174	}
		175
		176	@Test
		177	func timestampRangesRenderAsTwoTimeElements() {
		178	// Same-day: the end shows only its time, since the start carries the date.
		179	let sameDay = OrgHTMLTreeRenderer().render(OrgParser.parse("Range <2024-01-15 Mon 10:00-11:45>."))
		180	#expect(sameDay.contains(#"datetime="2024-01-15T10:00">2024-01-15 10:00</time>"#))
		181	#expect(sameDay.contains("–"))
		182	#expect(sameDay.contains(#"datetime="2024-01-15T11:45">11:45</time>"#))
		183
		184	// Multi-day: one timestamp carrying an end date, rendered as two stamps.
		185	let multiDay = OrgHTMLTreeRenderer().render(OrgParser.parse("Span <2024-01-15 Mon>--<2024-01-20 Sat>."))
		186	#expect(multiDay.contains(#"datetime="2024-01-15">2024-01-15</time>"#))
		187	#expect(multiDay.contains("–"))
		188	#expect(multiDay.contains(#"datetime="2024-01-20">2024-01-20</time>"#))
		189	// The `--` join is consumed, not left as stray text.
		190	#expect(!multiDay.contains("--"))
		191	}
		192
136	@Test	193	@Test
137	func treeRendererProducesStructuralHTML() {	194	func treeRendererProducesStructuralHTML() {
138	let document = OrgParser.parse("""	195	let document = OrgParser.parse("""
	@@ -155,14 +212,22 @@ struct ASTRendererTests {
155	}	212	}
156	}	213	}
157		214
158	/// How far the tree-based HTML renderer already agrees with orgo, measured with the same	215	/// The tree-based HTML renderer against the same corpus and the same skeleton reduction the
159	/// corpus and the same skeleton reduction the shipped renderer is held to. This is the	216	/// shipped renderer is held to. It now matches on every case the shipped renderer matches, so
160	/// prototype's honest scorecard — it is not expected to match the shipped renderer's 11/12	217	/// this asserts rather than merely reports: a regression fails, and `outofscope` starting to
161	/// yet, and the number here is what a migration would have to close.	218	/// match fails too, forcing this record to be updated.
162	struct ASTConformanceReportTests {	219	struct ASTConformanceTests {
		220
		221	/// Cases the tree renderer must match. `outofscope` is `scope: out` in the corpus —
		222	/// deliberately unsupported constructs, where orgo itself may differ — so it is expected
		223	/// to diverge, exactly as it does for the shipped renderer.
		224	private static let expectedToMatch: Set<String> = [
		225	"blocks", "core", "elements", "footnote", "headings", "images",
		226	"lists", "minimal", "table", "tblfm", "timestamps",
		227	]
163		228
164	@Test	229	@Test
165	func reportCorpusAgreement() throws {	230	func matchesTheCorpusWhereTheShippedRendererDoes() throws {
166	guard let dir = astCorpusCasesDir() else {	231	guard let dir = astCorpusCasesDir() else {
167	print("org-conformance corpus not found — skipping")	232	print("org-conformance corpus not found — skipping")
168	return	233	return
	@@ -195,8 +260,9 @@ struct ASTConformanceReportTests {
195	}	260	}
196	print("AST-CONFORMANCE matched=\(matched.count)/\(matched.count + diverged.count) \(matched.sorted())")	261	print("AST-CONFORMANCE matched=\(matched.count)/\(matched.count + diverged.count) \(matched.sorted())")
197	print("AST-CONFORMANCE diverged=\(diverged.sorted())")	262	print("AST-CONFORMANCE diverged=\(diverged.sorted())")
198	// The prototype is measured, not gated: this test reports, it does not fail.	263
199	#expect(matched.count + diverged.count > 0)	264	#expect(Set(matched) == Self.expectedToMatch,
		265	"tree renderer conformance changed — matched \(matched.sorted()), expected \(Self.expectedToMatch.sorted())")
200	}	266	}
201	}	267	}
202		268

Commit b59af9ae9a

Commit `b59af9ae9a`