Commit `19ab2f6aa1`

19ab2f6aa1652de074ce0d137dd747468c077a5f

parent: 72d3e10034

Unsigned

cmc <hello@cleberg.net> · 2026-08-29 01:05 UTC

Prototype the AST split: source -> OrgDocument -> renderers

Add a parallel pipeline alongside the shipped single-pass renderer, which is
untouched: an element tree mirroring orgo's model.rs, a block and inline parser
that builds it, an HTML renderer that walks it, and a Foundation-only
AttributedString renderer that proves a second output format is a walk over the
same parse rather than a second parser.

The tree renderer scores 8/12 on the conformance corpus, reported rather than
gated; AST-PROTOTYPE.md records the remaining gaps and what a migration would
take. Also widens three table helpers from private so the tree parser can reuse
them, including the separator-row splitter that divides on + rather than |.

Layout: unified · split

AST-PROTOTYPE.md added +83

@@ -0,0 +1,83 @@
	1	# AST split — prototype
	2
	3	A working prototype of `source → OrgDocument → {renderers}`, on the `ast-prototype` branch.
	4	Nothing shipped changes: `OrgRenderer` (the single-pass source→HTML renderer hutch and
	5	gitbay-ios use) is untouched, and its 11/12 corpus conformance is unaffected. This is a
	6	parallel pipeline built to answer one question — *is a second output format a walk over a
	7	tree, or a second parser?*
	8
	9	## What's here
	10
	11	```
	12	Sources/OrgSwift/AST/
	13	OrgDocument.swift the element tree
	14	OrgParser.swift source → OrgDocument (blocks)
	15	OrgInlineParser.swift text → [OrgObject] (inlines)
	16	OrgHTMLTreeRenderer.swift OrgDocument → HTML
	17	OrgAttributedStringRenderer.swift [OrgObject] → AttributedString
	18	```
	19
	20	The types deliberately mirror orgo's `model.rs` — `OrgElement`/`OrgObject` against orgo's
	21	`Element`/`Object`, `OrgTableRow.{cells,rule}` against `TableRow::{Cells,Rule}`, the same
	22	`ListKind`/`Checkbox` vocabulary. That is what makes future tree-level conformance possible:
	23	today the corpus compares rendered HTML reduced to a skeleton, which is a string-level proxy
	24	for "do these two agree on structure". With matching trees, that question can be asked
	25	directly.
	26
	27	## The result
	28
	29	Two renderers, one parse. `OrgAttributedStringRenderer` is Foundation-only — no SwiftUI —
	30	so it works server-side, in a CLI, anywhere, and a SwiftUI block renderer would sit on top of
	31	it for the inline runs inside each block.
	32
	33	```swift
	34	let document = OrgParser.parse(source)
	35	let html = OrgHTMLTreeRenderer().render(document) // markup
	36	let text = OrgAttributedStringRenderer().inline(objects) // native, real attributes
	37	```
	38
	39	The AttributedString path carries `inlinePresentationIntent` (`.stronglyEmphasized`,
	40	`.emphasized`, `.code`) and real `link` attributes — no markup in the string. Constructs
	41	`AttributedString` has no portable attribute for (superscript, timestamps, footnote refs,
	42	images, underline, strikethrough — the last two live only in the UIKit/AppKit scopes) travel
	43	as an `OrgRole` custom attribute the UI layer reads to decide presentation, so the renderer
	44	never needs to know about fonts or colors.
	45
	46	## Conformance scorecard
	47
	48	The tree-based HTML renderer, measured against the same `org-conformance` corpus and the same
	49	skeleton reduction the shipped renderer is held to:
	50
	51	8 / 12 — `blocks`, `elements`, `headings`, `images`, `lists`, `minimal`, `table`, `tblfm`.
	52
	53	Reported, not gated: `ASTConformanceReportTests` prints the score and does not fail the suite,
	54	because the prototype is being measured rather than defended. Run with `ORG_DUMP=1` to print
	55	the first divergence per case.
	56
	57	Remaining gaps, all small parser work rather than anything architectural:
	58
	59	\| Case \| First divergence \|
	60	\|---\|---\|
	61	\| `core` \| consecutive ordered lists group into one `<ol>` \|
	62	\| `footnote` \| inline footnotes (`[fn:x:text]`) lose their text — the definition needs to come from the reference \|
	63	\| `timestamps` \| multi-day ranges (`<a>--<b>`) render as two stamps without the en-dash join \|
	64	\| `outofscope` \| deliberately unsupported constructs; the shipped renderer diverges here too \|
	65
	66	Two gaps were closed while writing this, each ~10 lines, which is the useful signal about
	67	where the effort sits: property drawers (`:PROPERTIES:` … `:END:`) are dropped as heading
	68	metadata, and a bare image renders as `<p><img></p>` while a caption or `#+ATTR_HTML`
	69	promotes it to `<figure>`.
	70
	71	## What a migration would look like
	72
	73	1. Close the four gaps above until the tree renderer also scores 11/12.
	74	2. Point `OrgRenderer.renderToHTML` at `parse` + `OrgHTMLTreeRenderer` internally, keeping the
	75	public API identical. The corpus test is the proof it is safe; consumers do not change.
	76	3. Delete the single-pass renderer.
	77	4. Add `OrgSwiftUI` as a separate product depending on the core, so the parser and HTML
	78	renderer stay Foundation-only and consumers who want HTML never import SwiftUI.
	79
	80	Step 4 is where tables get built: `Grid`/`GridRow` with `.gridColumnAlignment()`, wrapped in a
	81	horizontal `ScrollView` for phone-width overflow — the approach MarkdownUI uses, and the
	82	`OrgTable` node already carries the rows, the rule position, and per-column alignments it
	83	needs.

Sources/OrgSwift/AST/OrgAttributedStringRenderer.swift added +144

@@ -0,0 +1,144 @@
	1	import Foundation
	2
	3	/// Renders inline org content to `AttributedString` — the native counterpart to the HTML
	4	/// renderer, walking the same tree.
	5	///
	6	/// This is the prototype's argument: a second output format is a walk over the parsed tree,
	7	/// not a second parser. It stays Foundation-only (no SwiftUI), so it is usable anywhere; a
	8	/// SwiftUI block renderer would sit on top, using this for the inline runs inside each block.
	9	public struct OrgAttributedStringRenderer: Sendable {
	10
	11	public init() {}
	12
	13	/// Render one run of inline objects, carrying intents a UI layer can style.
	14	public func inline(_ objects: [OrgObject]) -> AttributedString {
	15	var result = AttributedString()
	16	for object in objects {
	17	switch object {
	18	case .text(let text):
	19	result += AttributedString(text)
	20
	21	case .bold(let children):
	22	var part = inline(children)
	23	part.inlinePresentationIntent = .stronglyEmphasized
	24	result += part
	25
	26	case .italic(let children):
	27	var part = inline(children)
	28	part.inlinePresentationIntent = .emphasized
	29	result += part
	30
	31	// Underline and strikethrough have no Foundation-portable attribute (the
	32	// underlineStyle/strikethroughStyle keys live in the UIKit/AppKit scopes), so they
	33	// travel as roles the UI layer applies.
	34	case .underline(let children):
	35	var part = inline(children)
	36	part.orgRole = .underline
	37	result += part
	38
	39	case .strikeThrough(let children):
	40	var part = inline(children)
	41	part.orgRole = .strikeThrough
	42	result += part
	43
	44	case .verbatim(let text), .code(let text):
	45	var part = AttributedString(text)
	46	part.inlinePresentationIntent = .code
	47	result += part
	48
	49	case .superscript(let children):
	50	// No portable superscript attribute; mark it so a UI layer can raise it.
	51	var part = inline(children)
	52	part.orgRole = .superscript
	53	result += part
	54
	55	case .lineBreak:
	56	result += AttributedString("\n")
	57
	58	case .timestamp(let stamp):
	59	var part = AttributedString(stamp.displayValue)
	60	part.orgRole = .timestamp
	61	result += part
	62
	63	case .footnoteRef(let label, _):
	64	var part = AttributedString("[\(label)]")
	65	part.orgRole = .footnoteReference
	66	result += part
	67
	68	case .image(let figure):
	69	var part = AttributedString(figure.alt ?? figure.source)
	70	part.orgRole = .image
	71	result += part
	72
	73	case .link(let link):
	74	var part = link.description.map { inline($0) } ?? AttributedString(displayValue(link.target))
	75	if let url = URL(string: hrefValue(link.target)) {
	76	part.link = url
	77	}
	78	result += part
	79	}
	80	}
	81	return result
	82	}
	83
	84	/// Flatten a whole document to attributed paragraphs — a convenience for callers that
	85	/// want text without building block views (a share sheet, a plain-text export).
	86	public func paragraphs(_ document: OrgDocument) -> [AttributedString] {
	87	document.elements.compactMap { element in
	88	switch element {
	89	case .paragraph(let objects): return inline(objects)
	90	case .heading(let heading): return inline(heading.title)
	91	default: return nil
	92	}
	93	}
	94	}
	95
	96	private func hrefValue(_ target: OrgLinkTarget) -> String {
	97	switch target {
	98	case .external(let url): return url
	99	case .file(let path): return path
	100	case .id(let identifier): return "#\(identifier)"
	101	}
	102	}
	103
	104	private func displayValue(_ target: OrgLinkTarget) -> String {
	105	switch target {
	106	case .external(let url): return url
	107	case .file(let path): return path
	108	case .id(let identifier): return identifier
	109	}
	110	}
	111	}
	112
	113	// MARK: - Custom attribute
	114
	115	/// Org roles that `AttributedString` has no standard attribute for. A UI layer reads these to
	116	/// decide presentation (raise a superscript, tint a timestamp, make a footnote ref tappable)
	117	/// without the renderer needing to know about fonts or colors.
	118	public enum OrgRole: String, Sendable, Codable {
	119	case superscript
	120	case timestamp
	121	case footnoteReference
	122	case image
	123	case underline
	124	case strikeThrough
	125	}
	126
	127	public enum OrgRoleAttribute: AttributedStringKey {
	128	public typealias Value = OrgRole
	129	public static let name = "orgRole"
	130	}
	131
	132	public extension AttributeScopes {
	133	struct OrgAttributes: AttributeScope {
	134	public let orgRole: OrgRoleAttribute
	135	}
	136	var org: OrgAttributes.Type { OrgAttributes.self }
	137	}
	138
	139	public extension AttributedString {
	140	var orgRole: OrgRole? {
	141	get { self[OrgRoleAttribute.self] }
	142	set { self[OrgRoleAttribute.self] = newValue }
	143	}
	144	}

Sources/OrgSwift/AST/OrgDocument.swift added +229

@@ -0,0 +1,229 @@
	1	import Foundation
	2
	3	// The org element tree — the document model, deliberately mirroring orgo's `model.rs`
	4	// (`Element`/`Object`, `TableRow::{Cells, Rule}`, `ListKind`, `Checkbox`) so the two can be
	5	// compared tree-to-tree later, instead of only through rendered HTML.
	6	//
	7	// Why a tree at all: the shipped renderer is a single pass from source straight to an HTML
	8	// string, so every new output format would mean re-deriving the parse. With a tree, parsing
	9	// happens once and each renderer is a walk — the HTML renderer stays the one the conformance
	10	// corpus measures, and a native renderer rides on the same proven parse.
	11
	12	/// A parsed org document: metadata keywords plus the block elements of the body.
	13	public struct OrgDocument: Sendable, Equatable {
	14	/// `#+TITLE:`, `#+AUTHOR:`, `#+DATE:` and any other `#+KEY: value`, in source order.
	15	public var keywords: [(key: String, value: String)]
	16	public var elements: [OrgElement]
	17
	18	public init(keywords: [(key: String, value: String)] = [], elements: [OrgElement] = []) {
	19	self.keywords = keywords
	20	self.elements = elements
	21	}
	22
	23	public func keyword(_ name: String) -> String? {
	24	keywords.first { $0.key.caseInsensitiveCompare(name) == .orderedSame }?.value
	25	}
	26
	27	public static func == (lhs: OrgDocument, rhs: OrgDocument) -> Bool {
	28	lhs.elements == rhs.elements
	29	&& lhs.keywords.count == rhs.keywords.count
	30	&& zip(lhs.keywords, rhs.keywords).allSatisfy { $0.key == $1.key && $0.value == $1.value }
	31	}
	32	}
	33
	34	/// A block-level element.
	35	public indirect enum OrgElement: Sendable, Equatable {
	36	case heading(OrgHeading)
	37	case paragraph([OrgObject])
	38	case list(OrgList)
	39	case table(OrgTable)
	40	case srcBlock(language: String?, code: String)
	41	case exampleBlock(String)
	42	case quoteBlock([OrgElement])
	43	case centerBlock([OrgElement])
	44	case verseBlock([[OrgObject]])
	45	case specialBlock(name: String, content: [OrgElement])
	46	case exportBlock(backend: String, raw: String)
	47	case figure(OrgFigure)
	48	case horizontalRule
	49	case footnoteDefinition(label: String, content: [OrgObject])
	50	}
	51
	52	public struct OrgHeading: Sendable, Equatable {
	53	/// Star count, before any render-time level offset.
	54	public var level: Int
	55	public var todo: String?
	56	public var priority: Character?
	57	public var title: [OrgObject]
	58	public var tags: [String]
	59
	60	public init(level: Int, todo: String? = nil, priority: Character? = nil,
	61	title: [OrgObject], tags: [String] = []) {
	62	self.level = level
	63	self.todo = todo
	64	self.priority = priority
	65	self.title = title
	66	self.tags = tags
	67	}
	68	}
	69
	70	// MARK: - Lists
	71
	72	public struct OrgList: Sendable, Equatable {
	73	public var kind: OrgListKind
	74	public var items: [OrgListItem]
	75
	76	public init(kind: OrgListKind, items: [OrgListItem]) {
	77	self.kind = kind
	78	self.items = items
	79	}
	80	}
	81
	82	public enum OrgListKind: Sendable, Equatable {
	83	case unordered
	84	case ordered
	85	case description
	86	}
	87
	88	public struct OrgListItem: Sendable, Equatable {
	89	public var checkbox: OrgCheckbox?
	90	/// The term of a description-list item (`term :: definition`).
	91	public var term: [OrgObject]?
	92	/// The item's own content: one paragraph normally, several for a multi-paragraph item.
	93	public var content: [[OrgObject]]
	94	/// A nested list, when the item has one.
	95	public var sublist: OrgList?
	96
	97	public init(checkbox: OrgCheckbox? = nil, term: [OrgObject]? = nil,
	98	content: [[OrgObject]], sublist: OrgList? = nil) {
	99	self.checkbox = checkbox
	100	self.term = term
	101	self.content = content
	102	self.sublist = sublist
	103	}
	104	}
	105
	106	public enum OrgCheckbox: Sendable, Equatable {
	107	case off
	108	case on
	109	case partial
	110	}
	111
	112	// MARK: - Tables
	113
	114	public struct OrgTable: Sendable, Equatable {
	115	public var rows: [OrgTableRow]
	116	/// Per-column alignment from the separator row, when it carries `:` markers.
	117	public var alignments: [OrgAlignment?]
	118
	119	public init(rows: [OrgTableRow], alignments: [OrgAlignment?] = []) {
	120	self.rows = rows
	121	self.alignments = alignments
	122	}
	123
	124	/// A rule row separates the header band from the body, as in org.
	125	public var headerRowCount: Int {
	126	guard let ruleIndex = rows.firstIndex(where: { if case .rule = $0 { return true } else { return false } })
	127	else { return 0 }
	128	return ruleIndex
	129	}
	130	}
	131
	132	public enum OrgTableRow: Sendable, Equatable {
	133	case cells([[OrgObject]])
	134	case rule
	135	}
	136
	137	public enum OrgAlignment: String, Sendable, Equatable {
	138	case left, center, right
	139	}
	140
	141	// MARK: - Figures
	142
	143	public struct OrgFigure: Sendable, Equatable {
	144	public var source: String
	145	public var caption: [OrgObject]?
	146	/// `#+ATTR_HTML:` pairs, kept as parsed so an HTML renderer can emit them and a native
	147	/// renderer can read the ones it understands (`:alt`, `:width`).
	148	public var attributes: [(key: String, value: String)]
	149
	150	public init(source: String, caption: [OrgObject]? = nil, attributes: [(key: String, value: String)] = []) {
	151	self.source = source
	152	self.caption = caption
	153	self.attributes = attributes
	154	}
	155
	156	public var alt: String? {
	157	attributes.first { $0.key == "alt" }?.value
	158	}
	159
	160	public static func == (lhs: OrgFigure, rhs: OrgFigure) -> Bool {
	161	lhs.source == rhs.source && lhs.caption == rhs.caption
	162	&& lhs.attributes.count == rhs.attributes.count
	163	&& zip(lhs.attributes, rhs.attributes).allSatisfy { $0.key == $1.key && $0.value == $1.value }
	164	}
	165	}
	166
	167	// MARK: - Inline objects
	168
	169	/// An inline object. Text-bearing cases carry their own children so a renderer can nest
	170	/// styling (`bold /and italic/`) rather than receiving pre-formatted markup.
	171	public indirect enum OrgObject: Sendable, Equatable {
	172	case text(String)
	173	case bold([OrgObject])
	174	case italic([OrgObject])
	175	case underline([OrgObject])
	176	case strikeThrough([OrgObject])
	177	/// Non-nesting by definition in org: `=verbatim=` and `~code~` hold literal text.
	178	case verbatim(String)
	179	case code(String)
	180	case link(OrgLink)
	181	case image(OrgFigure)
	182	case footnoteRef(label: String, number: Int)
	183	case timestamp(OrgTimestamp)
	184	case superscript([OrgObject])
	185	case lineBreak
	186	}
	187
	188	public struct OrgLink: Sendable, Equatable {
	189	public var target: OrgLinkTarget
	190	/// Nil description means the link shows its target.
	191	public var description: [OrgObject]?
	192
	193	public init(target: OrgLinkTarget, description: [OrgObject]? = nil) {
	194	self.target = target
	195	self.description = description
	196	}
	197	}
	198
	199	public enum OrgLinkTarget: Sendable, Equatable {
	200	/// An absolute URL (`https:`, `mailto:`) or a bare autolinked URL.
	201	case external(String)
	202	/// A repository-relative path, from `[[file:…]]` or a bare relative target.
	203	case file(String)
	204	/// `[[id:…]]` — an in-page fragment.
	205	case id(String)
	206	}
	207
	208	public struct OrgTimestamp: Sendable, Equatable {
	209	public var date: String
	210	public var time: String?
	211	public var endTime: String?
	212	public var active: Bool
	213
	214	public init(date: String, time: String? = nil, endTime: String? = nil, active: Bool) {
	215	self.date = date
	216	self.time = time
	217	self.endTime = endTime
	218	self.active = active
	219	}
	220
	221	/// The `datetime` attribute value / sort key: `2024-01-15` or `2024-01-15T10:30`.
	222	public var machineValue: String {
	223	time.map { "\(date)T\($0)" } ?? date
	224	}
	225
	226	public var displayValue: String {
	227	time.map { "\(date) \($0)" } ?? date
	228	}
	229	}

Sources/OrgSwift/AST/OrgHTMLTreeRenderer.swift added +281

@@ -0,0 +1,281 @@
	1	import Foundation
	2
	3	/// Renders an ``OrgDocument`` to HTML by walking the tree.
	4	///
	5	/// The point of the prototype: this produces the same shape of output as the shipped
	6	/// single-pass ``OrgRenderer``, but from a parsed tree rather than from source, so a second
	7	/// renderer (see ``OrgAttributedStringRenderer``) can share the parse instead of re-deriving it.
	8	public struct OrgHTMLTreeRenderer: Sendable {
	9	public var headingLevelOffset: Int
	10	public var highlighter: CodeHighlighter
	11
	12	public init(headingLevelOffset: Int = 0, highlighter: CodeHighlighter = PlainCodeHighlighter()) {
	13	self.headingLevelOffset = headingLevelOffset
	14	self.highlighter = highlighter
	15	}
	16
	17	public func render(_ document: OrgDocument) -> String {
	18	var footnotes = FootnoteNumbering(document: document)
	19	var html = document.elements.map { element( $0, &footnotes) }.joined()
	20	html += footnotes.renderSection(self)
	21	return html
	22	}
	23
	24	// MARK: - Blocks
	25
	26	private func element(_ element: OrgElement, _ notes: inout FootnoteNumbering) -> String {
	27	switch element {
	28	case .heading(let heading):
	29	let level = min(6, max(1, heading.level + headingLevelOffset))
	30	var inner = ""
	31	if let todo = heading.todo {
	32	inner += #"<span class="\#(todo.lowercased()) \#(todo)">\#(todo)</span> "#
	33	}
	34	if let priority = heading.priority {
	35	inner += #"<span class="priority">[#\#(priority)]</span> "#
	36	}
	37	inner += renderInline(heading.title, &notes)
	38	for tag in heading.tags {
	39	inner += #" <span class="tag">\#(escapeHTML(tag))</span>"#
	40	}
	41	return "<h\(level)>\(inner)</h\(level)>\n"
	42
	43	case .paragraph(let objects):
	44	return "<p>" + renderInline(objects, &notes) + "</p>\n"
	45
	46	case .list(let list):
	47	return renderList(list, &notes)
	48
	49	case .table(let table):
	50	return renderTable(table, &notes)
	51
	52	case .srcBlock(let language, let code):
	53	let classAttribute = language.map { #" class="language-\#(escapeHTMLAttribute($0))""# } ?? ""
	54	let body = highlighter.highlightedHTML(code: code, language: language) ?? escapeHTML(code)
	55	return "<pre><code\(classAttribute)>\(body)</code></pre>\n"
	56
	57	case .exampleBlock(let text):
	58	return "<pre>\(escapeHTML(text))</pre>\n"
	59
	60	case .quoteBlock(let children):
	61	return "<blockquote>\n" + children.map { self.element($0, &notes) }.joined() + "</blockquote>\n"
	62
	63	case .centerBlock(let children):
	64	return #"<div class="center">"# + "\n" + children.map { self.element($0, &notes) }.joined() + "</div>\n"
	65
	66	case .verseBlock(let lines):
	67	let body = lines.map { renderInline($0, &notes) }.joined(separator: "<br>\n")
	68	return #"<p class="verse">"# + "\n" + body + "\n</p>\n"
	69
	70	case .specialBlock(let name, let children):
	71	return #"<div class="\#(escapeHTMLAttribute(name))">"# + "\n"
	72	+ children.map { self.element($0, &notes) }.joined() + "</div>\n"
	73
	74	case .exportBlock(let backend, let raw):
	75	return backend == "html" ? raw + "\n" : ""
	76
	77	case .figure(let figure):
	78	// A bare image is a paragraph; a caption or explicit attributes promote it to a
	79	// <figure>, matching org's exporter.
	80	guard figure.caption != nil \|\| !figure.attributes.isEmpty else {
	81	return "<p>" + imageTag(figure, caption: nil, &notes) + "</p>\n"
	82	}
	83	var html = "<figure>" + imageTag(figure, caption: figure.caption, &notes)
	84	if let caption = figure.caption {
	85	notes.figureNumber += 1
	86	html += #"<figcaption><span class="figure-number">Figure \#(notes.figureNumber): </span>"#
	87	+ renderInline(caption, &notes) + "</figcaption>"
	88	}
	89	return html + "</figure>\n"
	90
	91	case .horizontalRule:
	92	return "<hr>\n"
	93
	94	case .footnoteDefinition:
	95	return "" // collected and emitted in the notes section
	96	}
	97	}
	98
	99	private func renderList(_ list: OrgList, _ notes: inout FootnoteNumbering) -> String {
	100	if list.kind == .description {
	101	var html = "<dl>\n"
	102	for item in list.items {
	103	if let term = item.term {
	104	html += "<dt>" + renderInline(term, &notes) + "</dt>\n"
	105	}
	106	if let first = item.content.first {
	107	html += "<dd>" + renderInline(first, &notes) + "</dd>\n"
	108	}
	109	}
	110	return html + "</dl>\n"
	111	}
	112
	113	let tag = list.kind == .ordered ? "ol" : "ul"
	114	var html = "<\(tag)>\n"
	115	for item in list.items {
	116	html += "<li>"
	117	if let checkbox = item.checkbox {
	118	switch checkbox {
	119	case .off: html += "<code>[ ]</code> "
	120	case .on: html += "<code>[X]</code> "
	121	case .partial: html += "<code>[-]</code> "
	122	}
	123	}
	124	if item.content.count <= 1 {
	125	html += renderInline(item.content.first ?? [], &notes)
	126	} else {
	127	html += item.content.map { "<p>" + renderInline($0, &notes) + "</p>" }.joined(separator: "\n")
	128	}
	129	if let sublist = item.sublist {
	130	html += "\n" + renderList(sublist, &notes)
	131	}
	132	html += "</li>\n"
	133	}
	134	return html + "</\(tag)>\n"
	135	}
	136
	137	private func renderTable(_ table: OrgTable, _ notes: inout FootnoteNumbering) -> String {
	138	var html = "<table>\n"
	139	var wroteHeader = false
	140	var inBody = false
	141	let headerCount = table.headerRowCount
	142
	143	for (index, row) in table.rows.enumerated() {
	144	switch row {
	145	case .rule:
	146	if wroteHeader, !inBody { html += "</thead>\n<tbody>\n"; inBody = true }
	147	case .cells(let cells):
	148	let isHeader = headerCount > 0 && index < headerCount
	149	if isHeader, !wroteHeader { html += "<thead>\n"; wroteHeader = true }
	150	if !isHeader, !inBody { html += "<tbody>\n"; inBody = true }
	151	html += "<tr>\n"
	152	for (column, cell) in cells.enumerated() {
	153	let tag = isHeader ? "th" : "td"
	154	let alignment = column < table.alignments.count ? table.alignments[column] : nil
	155	let style = alignment.map { #" style="text-align: \#($0.rawValue);""# } ?? ""
	156	html += "<\(tag)\(style)>" + renderInline(cell, &notes) + "</\(tag)>\n"
	157	}
	158	html += "</tr>\n"
	159	}
	160	}
	161	if inBody { html += "</tbody>\n" }
	162	return html + "</table>\n"
	163	}
	164
	165	private func imageTag(_ figure: OrgFigure, caption: [OrgObject]?, _ notes: inout FootnoteNumbering) -> String {
	166	let alt = figure.alt ?? caption.map { plainText($0) } ?? ""
	167	var html = #"<img src="\#(escapeHTMLAttribute(figure.source))" alt="\#(escapeHTMLAttribute(alt))""#
	168	for (key, value) in figure.attributes where key != "alt" {
	169	html += " \(escapeHTMLAttribute(key))=\"\(escapeHTMLAttribute(value))\""
	170	}
	171	return html + ">"
	172	}
	173
	174	// MARK: - Inline
	175
	176	func renderInline(_ objects: [OrgObject], _ notes: inout FootnoteNumbering) -> String {
	177	var html = ""
	178	for object in objects {
	179	switch object {
	180	case .text(let text): html += escapeHTML(text)
	181	case .bold(let children): html += "<strong>" + renderInline(children, &notes) + "</strong>"
	182	case .italic(let children): html += "<em>" + renderInline(children, &notes) + "</em>"
	183	case .underline(let children): html += "<u>" + renderInline(children, &notes) + "</u>"
	184	case .strikeThrough(let children): html += "<del>" + renderInline(children, &notes) + "</del>"
	185	case .verbatim(let text), .code(let text): html += "<code>" + escapeHTML(text) + "</code>"
	186	case .superscript(let children): html += "<sup>" + renderInline(children, &notes) + "</sup>"
	187	case .lineBreak: html += "<br>"
	188	case .image(let figure): html += imageTag(figure, caption: nil, &notes)
	189	case .timestamp(let stamp):
	190	let cssClass = stamp.active ? "timestamp" : "timestamp inactive"
	191	html += #"<time class="\#(cssClass)" datetime="\#(stamp.machineValue)">\#(stamp.displayValue)</time>"#
	192	case .footnoteRef(let label, _):
	193	let number = notes.number(for: label)
	194	html += ##"<sup class="footnote-ref"><a id="fnr-\##(number)" href="#fn-\##(number)">\##(number)</a></sup>"##
	195	case .link(let link):
	196	let href = escapeHTMLAttribute(hrefValue(link.target))
	197	let text = link.description.map { renderInline($0, &notes) } ?? escapeHTML(displayValue(link.target))
	198	html += #"<a href="\#(href)">\#(text)</a>"#
	199	}
	200	}
	201	return html
	202	}
	203
	204	private func hrefValue(_ target: OrgLinkTarget) -> String {
	205	switch target {
	206	case .external(let url): return url
	207	case .file(let path): return path
	208	case .id(let identifier): return "#\(identifier)"
	209	}
	210	}
	211
	212	private func displayValue(_ target: OrgLinkTarget) -> String {
	213	switch target {
	214	case .external(let url): return url
	215	case .file(let path): return path
	216	case .id(let identifier): return identifier
	217	}
	218	}
	219
	220	/// Inline objects reduced to plain text, for an `alt` attribute.
	221	func plainText(_ objects: [OrgObject]) -> String {
	222	objects.map { object in
	223	switch object {
	224	case .text(let text): return text
	225	case .verbatim(let text), .code(let text): return text
	226	case .bold(let c), .italic(let c), .underline(let c), .strikeThrough(let c), .superscript(let c):
	227	return plainText(c)
	228	case .link(let link): return link.description.map { plainText($0) } ?? displayValue(link.target)
	229	case .timestamp(let stamp): return stamp.displayValue
	230	case .image(let figure): return figure.alt ?? ""
	231	case .footnoteRef, .lineBreak: return ""
	232	}
	233	}.joined()
	234	}
	235	}
	236
	237	// MARK: - Footnote numbering
	238
	239	/// Assigns footnote numbers in first-reference order and renders the notes section.
	240	struct FootnoteNumbering {
	241	private var numbers: [String: Int] = [:]
	242	private var order: [String] = []
	243	private var definitions: [String: [OrgObject]] = [:]
	244	var figureNumber = 0
	245
	246	init(document: OrgDocument) {
	247	for element in document.elements {
	248	if case .footnoteDefinition(let label, let content) = element {
	249	definitions[label] = content
	250	}
	251	}
	252	}
	253
	254	mutating func number(for label: String) -> Int {
	255	if let existing = numbers[label] { return existing }
	256	let next = order.count + 1
	257	numbers[label] = next
	258	order.append(label)
	259	return next
	260	}
	261
	262	func renderSection(_ renderer: OrgHTMLTreeRenderer) -> String {
	263	guard !order.isEmpty else { return "" }
	264	var html = "<section class=\"footnotes\" aria-label=\"Footnotes\">\n<hr>\n<ol>\n"
	265	for label in order {
	266	let n = numbers[label] ?? 0
	267	var copy = self
	268	let body = definitions[label].map { renderer.inlineForNotes($0, &copy) } ?? ""
	269	let back = ##"<a class="footnote-back" href="#fnr-\##(n)" aria-label="Back to reference \##(n)">↩</a>"##
	270	html += "<li id=\"fn-\(n)\"><p>\(body)</p>\n \(back)</li>\n"
	271	}
	272	return html + "</ol>\n</section>\n"
	273	}
	274	}
	275
	276	extension OrgHTMLTreeRenderer {
	277	/// Renders a footnote definition's own inline content, reusing the body inline walk.
	278	func inlineForNotes(_ objects: [OrgObject], _ notes: inout FootnoteNumbering) -> String {
	279	renderInline(objects, &notes)
	280	}
	281	}

Sources/OrgSwift/AST/OrgInlineParser.swift added +271

@@ -0,0 +1,271 @@
	1	import Foundation
	2
	3	// Inline parsing into `[OrgObject]`.
	4	//
	5	// The shipped renderer does inline work by regex-substituting HTML into an escaped string,
	6	// using placeholder tokens to protect what must not be re-scanned. That works for one output
	7	// format but bakes HTML into the parse. Here the same constructs become a tree, so `*bold
	8	// /italic/*` nests properly and every renderer decides its own representation.
	9
	10	extension OrgParser {
	11
	12	/// Inline objects reduced to their plain text — for alt text, previews, or assertions.
	13	public static func plain(_ objects: [OrgObject]) -> String {
	14	objects.map { object in
	15	switch object {
	16	case .text(let text): return text
	17	case .verbatim(let text), .code(let text): return text
	18	case .bold(let c), .italic(let c), .underline(let c), .strikeThrough(let c), .superscript(let c):
	19	return plain(c)
	20	case .link(let link):
	21	if let description = link.description { return plain(description) }
	22	switch link.target {
	23	case .external(let value), .file(let value), .id(let value): return value
	24	}
	25	case .image(let figure): return figure.alt ?? ""
	26	case .timestamp(let stamp): return stamp.displayValue
	27	case .footnoteRef, .lineBreak: return ""
	28	}
	29	}.joined()
	30	}
	31
	32	/// Parse a run of inline org text into objects.
	33	public static func parseInline(_ text: String) -> [OrgObject] {
	34	var objects: [OrgObject] = []
	35	var plain = ""
	36	let chars = Array(text)
	37	var i = 0
	38
	39	func flushPlain() {
	40	if !plain.isEmpty { objects.append(.text(plain)); plain = "" }
	41	}
	42
	43	while i < chars.count {
	44	// Bracket links: [[target]] or [[target][description]]
	45	if chars[i] == "[", i + 1 < chars.count, chars[i + 1] == "[",
	46	let link = scanLink(chars, from: i) {
	47	flushPlain()
	48	objects.append(link.object)
	49	i = link.next
	50	continue
	51	}
	52
	53	// Footnote reference: [fn:label] or [fn:label:inline]
	54	if chars[i] == "[", let note = scanFootnote(chars, from: i) {
	55	flushPlain()
	56	objects.append(note.object)
	57	i = note.next
	58	continue
	59	}
	60
	61	// Timestamps: <2024-01-15 Mon 10:30> or [2024-01-15 Mon]
	62	if chars[i] == "<" \|\| chars[i] == "[", let stamp = scanTimestamp(chars, from: i) {
	63	flushPlain()
	64	objects.append(stamp.object)
	65	i = stamp.next
	66	continue
	67	}
	68
	69	// Bare URL autolink.
	70	if chars[i] == "h", let url = scanBareURL(chars, from: i) {
	71	flushPlain()
	72	objects.append(url.object)
	73	i = url.next
	74	continue
	75	}
	76
	77	// Emphasis: bold /italic/ _underline_ +strike+ =verbatim= ~code~
	78	if let marker = emphasisMarker(chars[i]), boundaryBefore(chars, i),
	79	let span = scanEmphasis(chars, from: i, marker: chars[i]) {
	80	flushPlain()
	81	switch marker {
	82	case .bold: objects.append(.bold(parseInline(span.body)))
	83	case .italic: objects.append(.italic(parseInline(span.body)))
	84	case .underline: objects.append(.underline(parseInline(span.body)))
	85	case .strike: objects.append(.strikeThrough(parseInline(span.body)))
	86	case .verbatim: objects.append(.verbatim(span.body))
	87	case .code: objects.append(.code(span.body))
	88	}
	89	i = span.next
	90	continue
	91	}
	92
	93	// Superscript: x^2 or x^{group}
	94	if chars[i] == "^", i > 0, isWordCharacter(chars[i - 1]),
	95	let sup = scanSuperscript(chars, from: i) {
	96	flushPlain()
	97	objects.append(.superscript(parseInline(sup.body)))
	98	i = sup.next
	99	continue
	100	}
	101
	102	plain.append(chars[i])
	103	i += 1
	104	}
	105	flushPlain()
	106	return objects
	107	}
	108
	109	// MARK: - Scanners
	110
	111	private enum Emphasis { case bold, italic, underline, strike, verbatim, code }
	112
	113	private static func emphasisMarker(_ c: Character) -> Emphasis? {
	114	switch c {
	115	case "*": return .bold
	116	case "/": return .italic
	117	case "_": return .underline
	118	case "+": return .strike
	119	case "=": return .verbatim
	120	case "~": return .code
	121	default: return nil
	122	}
	123	}
	124
	125	/// org requires the opening marker to follow whitespace or start the run.
	126	private static func boundaryBefore(_ chars: [Character], _ i: Int) -> Bool {
	127	i == 0 \|\| chars[i - 1].isWhitespace \|\| "([{'\"".contains(chars[i - 1])
	128	}
	129
	130	private static func isWordCharacter(_ c: Character) -> Bool {
	131	c.isLetter \|\| c.isNumber
	132	}
	133
	134	private static func scanEmphasis(_ chars: [Character], from start: Int, marker: Character)
	135	-> (body: String, next: Int)? {
	136	var j = start + 1
	137	var body = ""
	138	while j < chars.count {
	139	if chars[j] == marker {
	140	// The closer must end the run or be followed by space/punctuation.
	141	let after = j + 1 < chars.count ? chars[j + 1] : " "
	142	if !body.isEmpty, after.isWhitespace \|\| ".,;:!?)]}'\"".contains(after) \|\| j + 1 == chars.count {
	143	return (body, j + 1)
	144	}
	145	}
	146	if chars[j] == "\n" { return nil }
	147	body.append(chars[j])
	148	j += 1
	149	}
	150	return nil
	151	}
	152
	153	private static func scanLink(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {
	154	var j = start + 2
	155	var target = ""
	156	while j < chars.count, !(chars[j] == "]" && j + 1 < chars.count && (chars[j + 1] == "]" \|\| chars[j + 1] == "[")) {
	157	target.append(chars[j]); j += 1
	158	}
	159	guard j < chars.count else { return nil }
	160
	161	var description: String?
	162	if chars[j + 1] == "[" {
	163	j += 2
	164	var text = ""
	165	var depth = 0
	166	while j < chars.count {
	167	if chars[j] == "[" { depth += 1 }
	168	if chars[j] == "]" {
	169	if depth == 0 { break }
	170	depth -= 1
	171	}
	172	text.append(chars[j]); j += 1
	173	}
	174	description = text
	175	}
	176	// Consume the closing ]]
	177	while j < chars.count, chars[j] == "]" { j += 1 }
	178
	179	let object = makeLinkObject(target: target, description: description)
	180	return (object, j)
	181	}
	182
	183	private static func makeLinkObject(target rawTarget: String, description: String?) -> OrgObject {
	184	let target = rawTarget.hasPrefix("file:") ? String(rawTarget.dropFirst(5)) : rawTarget
	185
	186	// A description that is itself an image makes the image the link's content.
	187	if let description {
	188	let inner = description.hasPrefix("[[") && description.hasSuffix("]]")
	189	? String(description.dropFirst(2).dropLast(2)) : description
	190	let imageSource = inner.hasPrefix("file:") ? String(inner.dropFirst(5)) : inner
	191	if isImagePath(imageSource), inner != description \|\| inner.hasPrefix("file:")
	192	\|\| inner.hasPrefix("http://") \|\| inner.hasPrefix("https://") {
	193	return .link(OrgLink(target: linkTarget(target),
	194	description: [.image(OrgFigure(source: imageSource))]))
	195	}
	196	return .link(OrgLink(target: linkTarget(target), description: parseInline(description)))
	197	}
	198
	199	if isImagePath(target) { return .image(OrgFigure(source: target)) }
	200	return .link(OrgLink(target: linkTarget(target), description: nil))
	201	}
	202
	203	private static func linkTarget(_ target: String) -> OrgLinkTarget {
	204	if target.hasPrefix("id:") { return .id(String(target.dropFirst(3))) }
	205	if target.hasPrefix("http://") \|\| target.hasPrefix("https://")
	206	\|\| target.hasPrefix("mailto:") \|\| target.hasPrefix("#") {
	207	return .external(target)
	208	}
	209	return .file(target)
	210	}
	211
	212	private static func isImagePath(_ path: String) -> Bool {
	213	let lower = path.lowercased()
	214	return [".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg", ".bmp", ".heic"]
	215	.contains { lower.hasSuffix($0) }
	216	}
	217
	218	private static func scanFootnote(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {
	219	let rest = String(chars[start...])
	220	guard let match = rest.firstMatch(of: /^\[fn:([A-Za-z0-9_-]+)(?::([^\]]*))?\]/) else { return nil }
	221	let label = String(match.1)
	222	let consumed = rest.distance(from: rest.startIndex, to: match.range.upperBound)
	223	// Numbering is assigned by the renderer, which sees the whole document; 0 is a placeholder.
	224	return (.footnoteRef(label: label, number: 0), start + consumed)
	225	}
	226
	227	private static func scanTimestamp(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {
	228	let active = chars[start] == "<"
	229	let closing: Character = active ? ">" : "]"
	230	var j = start + 1
	231	var body = ""
	232	while j < chars.count, chars[j] != closing {
	233	if chars[j] == "\n" { return nil }
	234	body.append(chars[j]); j += 1
	235	}
	236	guard j < chars.count else { return nil }
	237	guard let dateMatch = body.firstMatch(of: /(\d{4}-\d{2}-\d{2})/) else { return nil }
	238
	239	let date = String(dateMatch.1)
	240	var time: String?
	241	var endTime: String?
	242	if let timeMatch = body.firstMatch(of: /(\d{2}:\d{2})(?:-(\d{2}:\d{2}))?/) {
	243	time = String(timeMatch.1)
	244	if let end = timeMatch.2 { endTime = String(end) }
	245	}
	246	return (.timestamp(OrgTimestamp(date: date, time: time, endTime: endTime, active: active)), j + 1)
	247	}
	248
	249	private static func scanBareURL(_ chars: [Character], from start: Int) -> (object: OrgObject, next: Int)? {
	250	let rest = String(chars[start...])
	251	guard let match = rest.firstMatch(of: /^https?:\/\/[^\s<>()\[\]]+/) else { return nil }
	252	var url = String(rest[match.range])
	253	while let last = url.last, ".,;:!?".contains(last) { url.removeLast() }
	254	return (.link(OrgLink(target: .external(url), description: nil)), start + url.count)
	255	}
	256
	257	private static func scanSuperscript(_ chars: [Character], from start: Int) -> (body: String, next: Int)? {
	258	var j = start + 1
	259	guard j < chars.count else { return nil }
	260	if chars[j] == "{" {
	261	j += 1
	262	var body = ""
	263	while j < chars.count, chars[j] != "}" { body.append(chars[j]); j += 1 }
	264	guard j < chars.count else { return nil }
	265	return (body, j + 1)
	266	}
	267	var body = ""
	268	while j < chars.count, isWordCharacter(chars[j]) { body.append(chars[j]); j += 1 }
	269	return body.isEmpty ? nil : (body, j)
	270	}
	271	}

Sources/OrgSwift/AST/OrgParser.swift added +387

@@ -0,0 +1,387 @@
	1	import Foundation
	2
	3	/// Parses org source into an ``OrgDocument``.
	4	///
	5	/// This is the prototype of the AST split: parsing happens once, here, and renderers walk the
	6	/// result. It is deliberately a separate pipeline from the shipped ``OrgRenderer`` (which goes
	7	/// straight from source to an HTML string) so the two can be compared before anything migrates.
	8	public enum OrgParser {
	9
	10	public static func parse(_ source: String) -> OrgDocument {
	11	var document = OrgDocument()
	12	let lines = source
	13	.replacingOccurrences(of: "\r\n", with: "\n")
	14	.replacingOccurrences(of: "\r", with: "\n")
	15	.components(separatedBy: "\n")
	16	var index = 0
	17	var pendingCaption: String?
	18	var pendingAttrs: [(key: String, value: String)] = []
	19
	20	func flushPending() {
	21	pendingCaption = nil
	22	pendingAttrs = []
	23	}
	24
	25	while index < lines.count {
	26	let line = lines[index]
	27	let trimmed = line.trimmingCharacters(in: .whitespaces)
	28
	29	if trimmed.isEmpty { index += 1; continue }
	30
	31	// Comments.
	32	if trimmed == "#" \|\| trimmed.hasPrefix("# ") { index += 1; continue }
	33
	34	// Property drawers are heading metadata; org's exporter drops them.
	35	if trimmed == ":PROPERTIES:" {
	36	index += 1
	37	while index < lines.count,
	38	lines[index].trimmingCharacters(in: .whitespaces) != ":END:" {
	39	index += 1
	40	}
	41	if index < lines.count { index += 1 }
	42	continue
	43	}
	44
	45	// Affiliated keywords and document metadata.
	46	if let directive = orgKeywordDirective(in: trimmed) {
	47	switch directive.keyword {
	48	case "caption": pendingCaption = directive.value
	49	case "attr_html": pendingAttrs = parseAttributes(directive.value)
	50	default: document.keywords.append((directive.keyword, directive.value))
	51	}
	52	index += 1
	53	continue
	54	}
	55
	56	// Blocks: #+begin_… / #+end_…
	57	if trimmed.lowercased().hasPrefix("#+begin_") {
	58	let (element, next) = parseBlock(lines, from: index)
	59	if let element { document.elements.append(element) }
	60	index = next
	61	flushPending()
	62	continue
	63	}
	64
	65	// Heading.
	66	if let match = trimmed.firstMatch(of: /^(\*{1,6})\s+(.+)$/) {
	67	document.elements.append(.heading(parseHeading(stars: match.1.count, rest: String(match.2))))
	68	index += 1
	69	flushPending()
	70	continue
	71	}
	72
	73	// Horizontal rule.
	74	if isOrgHorizontalRule(trimmed) {
	75	document.elements.append(.horizontalRule)
	76	index += 1
	77	flushPending()
	78	continue
	79	}
	80
	81	// Footnote definition.
	82	if let def = orgFootnoteDefinition(in: trimmed) {
	83	document.elements.append(.footnoteDefinition(label: def.label, content: parseInline(def.text)))
	84	index += 1
	85	flushPending()
	86	continue
	87	}
	88
	89	// A standalone image link, promoted to a figure by an affiliated caption/attrs.
	90	if let path = standaloneOrgImage(in: trimmed) {
	91	document.elements.append(.figure(OrgFigure(
	92	source: path,
	93	caption: pendingCaption.map(parseInline),
	94	attributes: pendingAttrs
	95	)))
	96	index += 1
	97	flushPending()
	98	continue
	99	}
	100
	101	// Table.
	102	if isTableLine(trimmed) {
	103	let (table, next) = parseTable(lines, from: index)
	104	document.elements.append(.table(table))
	105	index = next
	106	flushPending()
	107	continue
	108	}
	109
	110	// List.
	111	if isListMarkerLine(trimmed), !isIndentedContinuationLine(line) {
	112	let (list, next) = parseList(lines, from: index)
	113	document.elements.append(.list(list))
	114	index = next
	115	flushPending()
	116	continue
	117	}
	118
	119	// Paragraph: consume until a blank line or a line that starts another construct.
	120	var paragraph: [String] = []
	121	while index < lines.count {
	122	let candidate = lines[index]
	123	let candidateTrimmed = candidate.trimmingCharacters(in: .whitespaces)
	124	if candidateTrimmed.isEmpty \|\| startsNewConstruct(candidateTrimmed, raw: candidate) { break }
	125	paragraph.append(candidateTrimmed)
	126	index += 1
	127	}
	128	if !paragraph.isEmpty {
	129	document.elements.append(.paragraph(parseInline(paragraph.joined(separator: " "))))
	130	}
	131	flushPending()
	132	}
	133
	134	return document
	135	}
	136
	137	/// Would this line begin a construct other than the paragraph currently being consumed?
	138	private static func startsNewConstruct(_ trimmed: String, raw: String) -> Bool {
	139	if trimmed.hasPrefix("#+") \|\| trimmed.hasPrefix("#") { return true }
	140	if trimmed.firstMatch(of: /^\*{1,6}\s+/) != nil { return true }
	141	if isOrgHorizontalRule(trimmed) { return true }
	142	if isTableLine(trimmed) { return true }
	143	if orgFootnoteDefinition(in: trimmed) != nil { return true }
	144	if isListMarkerLine(trimmed), !isIndentedContinuationLine(raw) { return true }
	145	return false
	146	}
	147
	148	// MARK: - Heading
	149
	150	private static func parseHeading(stars: Int, rest: String) -> OrgHeading {
	151	var body = rest
	152	var todo: String?
	153	var priority: Character?
	154
	155	for keyword in ["TODO", "DONE"] where body == keyword \|\| body.hasPrefix("\(keyword) ") {
	156	todo = keyword
	157	body = String(body.dropFirst(keyword.count)).trimmingCharacters(in: .whitespaces)
	158	break
	159	}
	160	if let match = body.firstMatch(of: /^\[#([A-Z])\]\s*/) {
	161	priority = Character(String(match.1))
	162	body = String(body[match.range.upperBound...])
	163	}
	164	let (title, tags) = splitHeadingTags(body)
	165	return OrgHeading(level: stars, todo: todo, priority: priority,
	166	title: parseInline(title), tags: tags)
	167	}
	168
	169	// MARK: - Blocks
	170
	171	private static func parseBlock(_ lines: [String], from start: Int) -> (OrgElement?, Int) {
	172	let opener = lines[start].trimmingCharacters(in: .whitespaces)
	173	let lower = opener.lowercased()
	174	let name = String(lower.dropFirst("#+begin_".count)).split(separator: " ").first.map(String.init) ?? ""
	175	let argument = opener
	176	.split(separator: " ", maxSplits: 1, omittingEmptySubsequences: true)
	177	.dropFirst().first.map { String($0).trimmingCharacters(in: .whitespaces) }
	178
	179	var body: [String] = []
	180	var index = start + 1
	181	while index < lines.count {
	182	let trimmed = lines[index].trimmingCharacters(in: .whitespaces).lowercased()
	183	if trimmed == "#+end_\(name)" { index += 1; break }
	184	body.append(lines[index])
	185	index += 1
	186	}
	187
	188	switch name {
	189	case "src":
	190	return (.srcBlock(language: argument?.isEmpty == false ? argument : nil,
	191	code: body.joined(separator: "\n")), index)
	192	case "example":
	193	return (.exampleBlock(body.joined(separator: "\n")), index)
	194	case "quote":
	195	return (.quoteBlock(parse(body.joined(separator: "\n")).elements), index)
	196	case "center":
	197	return (.centerBlock(parse(body.joined(separator: "\n")).elements), index)
	198	case "verse":
	199	return (.verseBlock(body.map(parseInline)), index)
	200	case "export":
	201	return (.exportBlock(backend: (argument ?? "").lowercased(),
	202	raw: body.joined(separator: "\n")), index)
	203	default:
	204	return (.specialBlock(name: name, content: parse(body.joined(separator: "\n")).elements), index)
	205	}
	206	}
	207
	208	// MARK: - Table
	209
	210	private static func parseTable(_ lines: [String], from start: Int) -> (OrgTable, Int) {
	211	var rows: [OrgTableRow] = []
	212	var alignments: [OrgAlignment?] = []
	213	var index = start
	214
	215	while index < lines.count {
	216	let trimmed = lines[index].trimmingCharacters(in: .whitespaces)
	217	guard isTableLine(trimmed) else { break }
	218
	219	// A separator row's columns are divided by `+`, not `\|`, so it needs its own
	220	// split — `\|:---+---:\|` is two columns, which splitting on `\|` would miss.
	221	let separatorCells = parseOrgTableSeparatorRow(trimmed)
	222	let cells = separatorCells.allSatisfy(isTableSeparatorCell)
	223	? separatorCells : parseTableRow(trimmed)
	224	if !cells.isEmpty, cells.allSatisfy(isTableSeparatorCell) {
	225	rows.append(.rule)
	226	let parsed = cells.map { cell -> OrgAlignment? in
	227	switch tableAlignment(for: cell) {
	228	case "left": return .left
	229	case "center": return .center
	230	case "right": return .right
	231	default: return nil
	232	}
	233	}
	234	if alignments.isEmpty \|\| alignments.allSatisfy({ $0 == nil }) { alignments = parsed }
	235	} else {
	236	rows.append(.cells(cells.map(parseInline)))
	237	}
	238	index += 1
	239	}
	240	return (OrgTable(rows: rows, alignments: alignments), index)
	241	}
	242
	243	// MARK: - List
	244
	245	private static func parseList(_ lines: [String], from start: Int) -> (OrgList, Int) {
	246	var block: [String] = []
	247	var index = start
	248	var pendingBlanks: [String] = []
	249
	250	while index < lines.count {
	251	let line = lines[index]
	252	let trimmed = line.trimmingCharacters(in: .whitespaces)
	253	if trimmed.isEmpty {
	254	pendingBlanks.append(line); index += 1; continue
	255	}
	256	if isListMarkerLine(trimmed), !isIndentedContinuationLine(line) {
	257	block.append(contentsOf: pendingBlanks); pendingBlanks = []
	258	block.append(line); index += 1; continue
	259	}
	260	if isIndentedContinuationLine(line) {
	261	block.append(contentsOf: pendingBlanks); pendingBlanks = []
	262	block.append(line); index += 1; continue
	263	}
	264	break
	265	}
	266	return (buildList(block), index)
	267	}
	268
	269	/// Group a list block's lines into items, recursing for nested lists.
	270	private static func buildList(_ lines: [String]) -> OrgList {
	271	let base = lines.filter { !$0.trimmingCharacters(in: .whitespaces).isEmpty }
	272	.map(leadingWidth).min() ?? 0
	273	let normalized = lines.map { dropLeading($0, base) }
	274
	275	var groups: [[String]] = []
	276	var current: [String] = []
	277	for line in normalized {
	278	if isListMarkerLine(line) {
	279	if !current.isEmpty { groups.append(current) }
	280	current = [line]
	281	} else if !current.isEmpty {
	282	current.append(line)
	283	}
	284	}
	285	if !current.isEmpty { groups.append(current) }
	286
	287	let firstMarker = groups.first?.first ?? ""
	288	var kind: OrgListKind = orderedListItem(in: firstMarker) != nil ? .ordered : .unordered
	289	if stripMarker(firstMarker).contains(" :: ") { kind = .description }
	290
	291	let items = groups.map { buildItem($0, kind: kind) }
	292	return OrgList(kind: kind, items: items)
	293	}
	294
	295	private static func buildItem(_ lines: [String], kind: OrgListKind) -> OrgListItem {
	296	var head = stripMarker(lines[0])
	297	var checkbox: OrgCheckbox?
	298	if head.hasPrefix("[ ] ") { checkbox = .off; head = String(head.dropFirst(4)) }
	299	else if head.hasPrefix("[X] ") \|\| head.hasPrefix("[x] ") { checkbox = .on; head = String(head.dropFirst(4)) }
	300	else if head.hasPrefix("[-] ") { checkbox = .partial; head = String(head.dropFirst(4)) }
	301
	302	let rest = Array(lines.dropFirst())
	303	let childIndent = rest.filter { !$0.trimmingCharacters(in: .whitespaces).isEmpty }
	304	.map(leadingWidth).min() ?? 0
	305	let outdented = rest.map { dropLeading($0, childIndent) }
	306
	307	var paragraphs: [String] = []
	308	var currentParagraph = [head]
	309	var sublistLines: [String] = []
	310	var inSublist = false
	311
	312	func flush() {
	313	let joined = currentParagraph.joined(separator: " ").trimmingCharacters(in: .whitespaces)
	314	if !joined.isEmpty { paragraphs.append(joined) }
	315	currentParagraph = []
	316	}
	317
	318	for line in outdented {
	319	let trimmed = line.trimmingCharacters(in: .whitespaces)
	320	if isListMarkerLine(line) \|\| inSublist {
	321	if !inSublist { flush() }
	322	inSublist = true
	323	sublistLines.append(line)
	324	} else if trimmed.isEmpty {
	325	flush()
	326	} else {
	327	currentParagraph.append(trimmed)
	328	}
	329	}
	330	flush()
	331
	332	var term: [OrgObject]?
	333	var content = paragraphs
	334	if kind == .description, let first = paragraphs.first, let range = first.range(of: " :: ") {
	335	term = parseInline(String(first[..<range.lowerBound]))
	336	content[0] = String(first[range.upperBound...])
	337	}
	338
	339	return OrgListItem(
	340	checkbox: checkbox,
	341	term: term,
	342	content: content.map(parseInline),
	343	sublist: sublistLines.isEmpty ? nil : buildList(sublistLines)
	344	)
	345	}
	346
	347	// MARK: - Helpers
	348
	349	private static func parseAttributes(_ value: String) -> [(key: String, value: String)] {
	350	guard let regex = try? NSRegularExpression(pattern: #":([A-Za-z_][A-Za-z0-9_-])\s+("[^"]"\|\S+)"#) else {
	351	return []
	352	}
	353	let ns = value as NSString
	354	return regex.matches(in: value, range: NSRange(location: 0, length: ns.length)).map { m in
	355	var raw = ns.substring(with: m.range(at: 2))
	356	if raw.count >= 2, raw.hasPrefix("\""), raw.hasSuffix("\"") { raw = String(raw.dropFirst().dropLast()) }
	357	return (ns.substring(with: m.range(at: 1)).lowercased(), raw)
	358	}
	359	}
	360
	361	private static func stripMarker(_ line: String) -> String {
	362	let trimmed = line.trimmingCharacters(in: .whitespaces)
	363	if trimmed.hasPrefix("- ") \|\| trimmed.hasPrefix("+ ") { return String(trimmed.dropFirst(2)) }
	364	if let match = trimmed.firstMatch(of: /^\d+[.)]\s+(.*)$/) { return String(match.1) }
	365	return trimmed
	366	}
	367
	368	private static func leadingWidth(_ line: String) -> Int {
	369	var count = 0
	370	for ch in line {
	371	if ch == " " { count += 1 } else if ch == "\t" { count += 8 } else { break }
	372	}
	373	return count
	374	}
	375
	376	private static func dropLeading(_ line: String, _ n: Int) -> String {
	377	var dropped = 0
	378	var index = line.startIndex
	379	while index < line.endIndex, dropped < n {
	380	if line[index] == " " { dropped += 1 }
	381	else if line[index] == "\t" { dropped += 8 }
	382	else { break }
	383	index = line.index(after: index)
	384	}
	385	return String(line[index...])
	386	}
	387	}

Sources/OrgSwift/Tables.swift +3 −3

	@@ -12,7 +12,7 @@ func parseTableRow(_ line: String) -> [String] {
12	.map { String($0).trimmingCharacters(in: .whitespaces) }	12	.map { String($0).trimmingCharacters(in: .whitespaces) }
13	}	13	}
14		14
15	private func parseOrgTableSeparatorRow(_ line: String) -> [String] {	15	func parseOrgTableSeparatorRow(_ line: String) -> [String] {
16	var content = line.trimmingCharacters(in: .whitespaces)	16	var content = line.trimmingCharacters(in: .whitespaces)
17	if content.hasPrefix("\|") {	17	if content.hasPrefix("\|") {
18	content.removeFirst()	18	content.removeFirst()
	@@ -25,11 +25,11 @@ private func parseOrgTableSeparatorRow(_ line: String) -> [String] {
25	.map { String($0).trimmingCharacters(in: .whitespaces) }	25	.map { String($0).trimmingCharacters(in: .whitespaces) }
26	}	26	}
27		27
28	private func isTableSeparatorCell(_ cell: String) -> Bool {	28	func isTableSeparatorCell(_ cell: String) -> Bool {
29	tableAlignment(for: cell) != nil	29	tableAlignment(for: cell) != nil
30	}	30	}
31		31
32	private func tableAlignment(for cell: String) -> String? {	32	func tableAlignment(for cell: String) -> String? {
33	let trimmed = cell.trimmingCharacters(in: .whitespaces)	33	let trimmed = cell.trimmingCharacters(in: .whitespaces)
34	guard !trimmed.isEmpty else { return nil }	34	guard !trimmed.isEmpty else { return nil }
35		35

Tests/OrgSwiftTests/ASTTests.swift added +218

@@ -0,0 +1,218 @@
	1	import Foundation
	2	import Testing
	3	@testable import OrgSwift
	4
	5	/// Tests for the AST prototype: `source → OrgDocument → {HTML, AttributedString}`.
	6	///
	7	/// Two things are being proven. First, that the tree carries the structure faithfully (the
	8	/// parse tests). Second — the actual argument for the split — that a second output format is
	9	/// a walk over the same tree rather than a second parser (the AttributedString tests), and
	10	/// that the tree-based HTML renderer can be held to the same conformance corpus as the
	11	/// shipped single-pass renderer.
	12	struct ASTParseTests {
	13
	14	@Test
	15	func headingCarriesTodoPriorityAndTags() {
	16	let doc = OrgParser.parse("* TODO [#A] Write the parser :work:rust:")
	17	guard case .heading(let heading) = doc.elements.first else {
	18	Issue.record("expected a heading"); return
	19	}
	20	#expect(heading.level == 1)
	21	#expect(heading.todo == "TODO")
	22	#expect(heading.priority == "A")
	23	#expect(heading.tags == ["work", "rust"])
	24	#expect(OrgParser.plain(heading.title) == "Write the parser")
	25	}
	26
	27	@Test
	28	func emphasisNestsRatherThanFlattening() {
	29	// The point of a tree: bold containing italic is structure, not a markup string.
	30	let objects = OrgParser.parseInline("bold /inner/ rest")
	31	guard case .bold(let children) = objects.first else {
	32	Issue.record("expected bold"); return
	33	}
	34	#expect(children.contains { if case .italic = $0 { return true } else { return false } })
	35	}
	36
	37	@Test
	38	func documentKeywordsAreMetadataNotContent() {
	39	let doc = OrgParser.parse("#+TITLE: My Doc\n#+AUTHOR: Someone\n\nBody.")
	40	#expect(doc.keyword("title") == "My Doc")
	41	#expect(doc.keyword("author") == "Someone")
	42	// Metadata does not appear as a body element.
	43	#expect(doc.elements.count == 1)
	44	guard case .paragraph = doc.elements.first else {
	45	Issue.record("expected a single paragraph"); return
	46	}
	47	}
	48
	49	@Test
	50	func nestedListsBecomeNestedItems() {
	51	let doc = OrgParser.parse("""
	52	- outer
	53	- inner
	54	- deepest
	55	- second
	56	""")
	57	guard case .list(let list) = doc.elements.first else {
	58	Issue.record("expected a list"); return
	59	}
	60	#expect(list.items.count == 2)
	61	let inner = list.items[0].sublist
	62	#expect(inner != nil)
	63	#expect(inner?.items.first?.sublist?.items.count == 1)
	64	}
	65
	66	@Test
	67	func tableKeepsRuleRowAndAlignments() {
	68	let doc = OrgParser.parse("""
	69	\| Name \| Score \|
	70	\|:------+------:\|
	71	\| alpha \| 10 \|
	72	""")
	73	guard case .table(let table) = doc.elements.first else {
	74	Issue.record("expected a table"); return
	75	}
	76	#expect(table.rows.count == 3)
	77	#expect(table.headerRowCount == 1)
	78	#expect(table.alignments == [.left, .right])
	79	if case .rule = table.rows[1] {} else { Issue.record("row 1 should be the rule") }
	80	}
	81
	82	@Test
	83	func timestampsAndLinksBecomeTypedObjects() {
	84	let objects = OrgParser.parseInline("due <2024-01-15 Mon 10:30> see [[id:abc][the thing]]")
	85	let hasTimestamp = objects.contains {
	86	if case .timestamp(let stamp) = $0 { return stamp.machineValue == "2024-01-15T10:30" }
	87	return false
	88	}
	89	#expect(hasTimestamp)
	90	let hasIDLink = objects.contains {
	91	if case .link(let link) = $0, case .id(let identifier) = link.target { return identifier == "abc" }
	92	return false
	93	}
	94	#expect(hasIDLink)
	95	}
	96	}
	97
	98	struct ASTRendererTests {
	99
	100	/// The payoff: one parse, two output formats, neither re-deriving the other's work.
	101	@Test
	102	func oneParseFeedsTwoRenderers() {
	103	let document = OrgParser.parse("A bold claim with ~code~ and a [[https://example.com][link]].")
	104
	105	let html = OrgHTMLTreeRenderer().render(document)
	106	#expect(html.contains("<strong>bold</strong>"))
	107	#expect(html.contains("<code>code</code>"))
	108	#expect(html.contains(#"<a href="https://example.com">link</a>"#))
	109
	110	let attributed = OrgAttributedStringRenderer().inline({
	111	if case .paragraph(let objects) = document.elements[0] { return objects }
	112	return []
	113	}())
	114	// Same content, native representation: no markup, real attributes.
	115	let plain = String(attributed.characters)
	116	#expect(plain == "A bold claim with code and a link.")
	117	#expect(!plain.contains("<"))
	118
	119	let boldRun = attributed.runs.first { $0.inlinePresentationIntent == .stronglyEmphasized }
	120	#expect(boldRun != nil)
	121	let codeRun = attributed.runs.first { $0.inlinePresentationIntent == .code }
	122	#expect(codeRun != nil)
	123	let linkRun = attributed.runs.first { $0.link != nil }
	124	#expect(linkRun?.link?.absoluteString == "https://example.com")
	125	}
	126
	127	@Test
	128	func attributedStringCarriesRolesForNonStandardIntents() {
	129	let objects = OrgParser.parseInline("x^2 and <2024-01-15 Mon>")
	130	let attributed = OrgAttributedStringRenderer().inline(objects)
	131	let roles: [OrgRole] = attributed.runs.compactMap { $0[OrgRoleAttribute.self] }
	132	#expect(roles.contains(.superscript))
	133	#expect(roles.contains(.timestamp))
	134	}
	135
	136	@Test
	137	func treeRendererProducesStructuralHTML() {
	138	let document = OrgParser.parse("""
	139	* Heading
	140
	141	\| a \| b \|
	142	\|---+---\|
	143	\| 1 \| 2 \|
	144
	145	- [ ] todo
	146	- [X] done
	147	""")
	148	let html = OrgHTMLTreeRenderer().render(document)
	149	#expect(html.contains("<h1>Heading</h1>"))
	150	#expect(html.contains("<thead>"))
	151	#expect(html.contains("<th>a</th>"))
	152	#expect(html.contains("<td>1</td>"))
	153	#expect(html.contains("<code>[ ]</code> todo"))
	154	#expect(html.contains("<code>[X]</code> done"))
	155	}
	156	}
	157
	158	/// How far the tree-based HTML renderer already agrees with orgo, measured with the same
	159	/// corpus and the same skeleton reduction the shipped renderer is held to. This is the
	160	/// prototype's honest scorecard — it is not expected to match the shipped renderer's 11/12
	161	/// yet, and the number here is what a migration would have to close.
	162	struct ASTConformanceReportTests {
	163
	164	@Test
	165	func reportCorpusAgreement() throws {
	166	guard let dir = astCorpusCasesDir() else {
	167	print("org-conformance corpus not found — skipping")
	168	return
	169	}
	170	let renderer = OrgHTMLTreeRenderer(headingLevelOffset: 1)
	171	var matched: [String] = []
	172	var diverged: [String] = []
	173
	174	for name in astCaseNames(dir) {
	175	let source = (try? String(contentsOf: dir.appendingPathComponent("\(name).org"), encoding: .utf8)) ?? ""
	176	let goldenRaw = (try? String(contentsOf: dir.appendingPathComponent("\(name).skeleton"), encoding: .utf8)) ?? ""
	177	let trimmed = goldenRaw.hasSuffix("\n") ? String(goldenRaw.dropLast()) : goldenRaw
	178	let golden = trimmed.isEmpty ? [] : trimmed.components(separatedBy: "\n")
	179
	180	let html = renderer.render(OrgParser.parse(source))
	181	let got = OrgSkeleton.skeleton(html)
	182	if got == golden {
	183	matched.append(name)
	184	} else {
	185	diverged.append(name)
	186	if ProcessInfo.processInfo.environment["ORG_DUMP"] != nil {
	187	let firstDiff = (0..<max(got.count, golden.count)).first {
	188	($0 < golden.count ? golden[$0] : "∅") != ($0 < got.count ? got[$0] : "∅")
	189	}
	190	if let k = firstDiff {
	191	print("AST-DIFF \(name) @\(k): orgo=\(k < golden.count ? golden[k] : "∅") ours=\(k < got.count ? got[k] : "∅")")
	192	}
	193	}
	194	}
	195	}
	196	print("AST-CONFORMANCE matched=\(matched.count)/\(matched.count + diverged.count) \(matched.sorted())")
	197	print("AST-CONFORMANCE diverged=\(diverged.sorted())")
	198	// The prototype is measured, not gated: this test reports, it does not fail.
	199	#expect(matched.count + diverged.count > 0)
	200	}
	201	}
	202
	203	private func astCorpusCasesDir() -> URL? {
	204	if let env = ProcessInfo.processInfo.environment["ORG_CONFORMANCE_DIR"] {
	205	let cases = URL(fileURLWithPath: env).appendingPathComponent("cases")
	206	if FileManager.default.fileExists(atPath: cases.path) { return cases }
	207	}
	208	let pkgRoot = URL(fileURLWithPath: #filePath)
	209	.deletingLastPathComponent().deletingLastPathComponent().deletingLastPathComponent()
	210	let sibling = pkgRoot.deletingLastPathComponent()
	211	.appendingPathComponent("org-conformance").appendingPathComponent("cases")
	212	return FileManager.default.fileExists(atPath: sibling.path) ? sibling : nil
	213	}
	214
	215	private func astCaseNames(_ dir: URL) -> [String] {
	216	let items = (try? FileManager.default.contentsOfDirectory(atPath: dir.path)) ?? []
	217	return items.filter { $0.hasSuffix(".org") }.map { String($0.dropLast(4)) }.sorted()
	218	}

Commit 19ab2f6aa1

Commit `19ab2f6aa1`