# Phase 1 Performance Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Meet the phase 1 speed gates (open a 1 MB file under 100 ms; keystroke to restyled text under 16 ms p95, including a 1 MB single-section file and a 5,000-row table) and record the memory gate the user set. **Architecture:** Four changes, each found by measuring first. (1) The inline scanner works on unicode scalars, rejects characters that can't start an object before trying recognizers, and slices token text from the source; this was most of the parse time. (2) A region strategy reparses a window of elements inside the innermost section's body instead of the whole top-level section, verifying the end of the window by reparsing one element past it, and reuses unchanged table rows. (3) The editor loads text through the text storage directly, styles the first screen before the first render and the rest in later run-loop turns, and restyles only the rows or items around an edit inside large tables and lists. (4) Tree queries on hot paths create only the child nodes they need. **Tech Stack:** Swift 6.2 tools, Swift Testing, AppKit, TextKit 2. **Spec:** `docs/design.md`, "Phases" (exit gates). ## Global Constraints - Every reparse equals a full parse: 100,000 random edits per seed, several seeds, release build. - Every incremental restyle equals a fresh editor's full styling: 1,500 random edits per seed, several seeds. - Memory gate (user decision): under 10x the file size for typical documents; under 45x for worst-case dense markup. ## Results Release build, M-series Mac; the M1 Air reference machine is still to be measured. | Measure | Before | After | Gate | | --- | --- | --- | --- | | Full parse, `csg.org` (1.5 MB) | 313 ms | 62 ms | | | Full parse, `org-manual.org` (0.86 MB) | 126 ms | 64 ms | | | Open, `csg.org` | ~440 ms | 75 ms | 100 ms per MB | | Typing p95, `csg.org` | 5.8 ms | 6.5 ms | 16 ms | | Blank line p95, `csg.org` | 256 ms | 7.3 ms | 16 ms | | Synthetic 1 MB single section: open / typing / blank line | | 76 / 7.0 / 8.1 ms | 100 / 16 / 16 ms | | Synthetic 5,000-row table: open / typing / blank line | | 38 / 3.2 / 8.8 ms | 100 / 16 / 16 ms | | Memory: `csg.org` / single section / table | | 9.6x / 30.4x / 42.9x | 10x / 45x / 45x | What the measurements showed along the way: - Line splitting and classification were never the problem (12 ms for 1.5 MB); inline scanning at about 170 ns per character was. - `NSTextView.string` was not the problem either: the first text loaded in a process pays a one-time 40 ms set-up, after which loading 1 MB takes under 1 ms through the text storage. - Restyling a whole table or list for one edit inside it cost 60 ms on 5,000 rows. - Memory is mostly the tree (19x to 28x on dense markup: one token or node per piece of markup) plus about 10x for the text view. A compact tree storing token text as offsets is possible later if memory becomes a real problem. The differential tests found three bugs during this work, each fixed and covered: the element before a reparse window can absorb the window's first line when a drawer loses its `:END:`; an emptied zeroth section must disappear as in a full parse; and the initial-styling watermark must move with edits. Run the gates: `ORGSTAR_GATES=1 ORGSTAR_GATE_FILE= swift test -c release --filter GateTests`. --- ### Task 1: Inline scanning on unicode scalars **Files:** - Modify: `Sources/OrgCore/Parser/Inline.swift`, `Sources/OrgCore/Timestamp.swift` - [ ] **Step 1: Rewrite the scanner and the timestamp recognizer over `[Unicode.Scalar]`** The existing inline, timestamp and round-trip tests are the specification; they don't change. ```swift /// Characters allowed before an emphasis opener, besides whitespace and the start of the run. private let emphasisPre: Set = ["-", "(", "{", "'", "\""] /// Characters allowed after an emphasis closer, besides whitespace and the end of the run. private let emphasisPost: Set = ["-", ".", ",", ":", "!", "?", ";", "'", "\"", ")", "}", "\\", "["] private let angleLinkSchemes: Set = [ "http", "https", "mailto", "file", "id", "doi", "ftp", "news", "shell", "elisp", "info", "help", "attachment", ] private let plainLinkPrefixes: [[Unicode.Scalar]] = ["https://", "http://", "mailto:", "file:"].map { Array($0.unicodeScalars) } private func emphasisKind(_ c: Unicode.Scalar) -> SyntaxKind? { switch c { case "*": .bold case "/": .italic case "_": .underline case "+": .strikeThrough case "=": .verbatim case "~": .code default: nil } } func isNewline(_ c: Unicode.Scalar) -> Bool { c == "\n" } func isSpace(_ c: Unicode.Scalar) -> Bool { c.isASCII ? (c == " " || c == "\t" || c == "\n" || c == "\r" || c == "\u{0B}" || c == "\u{0C}") : c.properties.isWhitespace } func isWordScalar(_ c: Unicode.Scalar) -> Bool { if c.isASCII { let v = c.value return (v >= 48 && v <= 57) || (v >= 65 && v <= 90) || (v >= 97 && v <= 122) } return c.properties.isAlphabetic || c.properties.numericType != nil } enum InlineMatch { case emphasis(SyntaxKind, open: Int, close: Int) case link(path: Range, description: Range?, whole: Range) case object(SyntaxKind, Range) var end: Int { switch self { case .emphasis(_, _, let close): close + 1 case .link(_, _, let whole): whole.upperBound case .object(_, let range): range.upperBound } } } /// Turns a run of text into text, newline and object tokens. Every scalar ends up in exactly /// one token. Works on unicode scalars: every recognizer starts on an ASCII character, so /// grapheme clusters never need to be formed. struct InlineScanner { let chars: [Unicode.Scalar] /// Where each scalar starts in `source`, plus its end, so token text is sliced from the /// source instead of rebuilt scalar by scalar. let source: Substring.UnicodeScalarView let starts: [String.Index] init(_ text: Substring) { source = text.unicodeScalars var chars: [Unicode.Scalar] = [] var starts: [String.Index] = [] chars.reserveCapacity(text.utf8.count) starts.reserveCapacity(text.utf8.count + 1) var index = source.startIndex while index < source.endIndex { chars.append(source[index]) starts.append(index) index = source.index(after: index) } starts.append(source.endIndex) self.chars = chars self.starts = starts } func scan(_ range: Range, into b: inout GreenBuilder, inLink: Bool = false) { var textStart = range.lowerBound var i = range.lowerBound while i < range.upperBound { if Self.mayStartObject(chars[i]), let match = match(at: i, in: range, inLink: inLink) { emitText(textStart.. Bool { switch c { case "[", "<", "{", "\\", "^", "s", "h", "m", "f", "*", "/", "_", "+", "=", "~": true default: false } } /// Text with each line break as its own newline token; "\r\n" stays one token. func emitText(_ range: Range, into b: inout GreenBuilder) { var start = range.lowerBound var k = range.lowerBound while k < range.upperBound { if chars[k] == "\n" { let breakStart = k > start && chars[k - 1] == "\r" ? k - 1 : k if start < breakStart { b.token(.text, string(start..) -> String { String(source[starts[range.lowerBound].. Bool { guard i + s.count <= limit else { return false } for (offset, c) in s.enumerated() where chars[i + offset] != c { return false } return true } func hasPrefix(_ s: String, at i: Int, _ limit: Int) -> Bool { hasPrefix(Array(s.unicodeScalars), at: i, limit) } // MARK: - Recognizers func match(at i: Int, in range: Range, inLink: Bool) -> InlineMatch? { let limit = range.upperBound let previous: Unicode.Scalar? = i > range.lowerBound ? chars[i - 1] : nil let afterWord = previous.map(isWordScalar) ?? false switch chars[i] { case "[": if !inLink, let link = bracketLink(i, limit) { return link } if let end = footnoteReference(i, limit) { return .object(.footnoteReference, i.. Int? { let marker = chars[i] guard i + 1 < limit, !isSpace(chars[i + 1]) else { return nil } var newlines = 0 var j = i + 1 while j < limit { if isNewline(chars[j]) { newlines += 1 if newlines > 1 { return nil } } else if chars[j] == marker, j > i + 1, !isSpace(chars[j - 1]) { if j + 1 == limit || isSpace(chars[j + 1]) || emphasisPost.contains(chars[j + 1]) { return j } } j += 1 } return nil } /// `[[path]]` or `[[path][description]]`. func bracketLink(_ i: Int, _ limit: Int) -> InlineMatch? { guard i + 1 < limit, chars[i + 1] == "[" else { return nil } var j = i + 2 while j < limit, chars[j] != "]" { if chars[j] == "[" || isNewline(chars[j]) { return nil } if chars[j] == "\\", j + 1 < limit { j += 1 } j += 1 } guard j > i + 2, j + 1 < limit else { return nil } let path = (i + 2).. descriptionStart, k + 1 < limit, chars[k + 1] == "]" else { return nil } return .link(path: path, description: descriptionStart.. Int? { guard hasPrefix("[fn:", at: i, limit) else { return nil } var j = i + 4 while j < limit, isWordScalar(chars[j]) || chars[j] == "_" || chars[j] == "-" { j += 1 } guard j < limit else { return nil } if chars[j] == "]" { return j > i + 4 ? j + 1 : nil } guard chars[j] == ":" else { return nil } var depth = 0 j += 1 while j < limit { if chars[j] == "[" { depth += 1 } else if chars[j] == "]" { if depth == 0 { return j + 1 } depth -= 1 } j += 1 } return nil } /// `[1/3]`, `[/]`, `[50%]` or `[%]`. func statisticsCookie(_ i: Int, _ limit: Int) -> Int? { var j = i + 1 while j < limit, isASCIIDigit(chars[j]) { j += 1 } guard j < limit else { return nil } if chars[j] == "%" { j += 1 } else if chars[j] == "/" { j += 1 while j < limit, isASCIIDigit(chars[j]) { j += 1 } } else { return nil } guard j < limit, chars[j] == "]" else { return nil } return j + 1 } /// `<>`. func target(_ i: Int, _ limit: Int) -> Int? { guard hasPrefix("<<", at: i, limit), i + 2 < limit, chars[i + 2] != "<" else { return nil } let start = i + 2 var j = start while j < limit, chars[j] != ">" { if chars[j] == "<" || isNewline(chars[j]) { return nil } j += 1 } guard j > start, j + 1 < limit, chars[j + 1] == ">", !isSpace(chars[start]), !isSpace(chars[j - 1]) else { return nil } return j + 2 } /// `` for a known scheme. func angleLink(_ i: Int, _ limit: Int) -> Int? { var j = i + 1 while j < limit, chars[j].properties.isAlphabetic { j += 1 } guard j < limit, chars[j] == ":", angleLinkSchemes.contains(string((i + 1).. bodyStart else { return nil } return j + 1 } /// A bare URL. Trailing sentence punctuation stays outside the link. func plainLink(_ i: Int, _ limit: Int) -> Int? { guard let prefix = plainLinkPrefixes.first(where: { hasPrefix($0, at: i, limit) }) else { return nil } let bodyStart = i + prefix.count var j = bodyStart while j < limit, !isSpace(chars[j]), !"()<>[]\"".unicodeScalars.contains(chars[j]) { j += 1 } while j > bodyStart, ".,;:!?'".unicodeScalars.contains(chars[j - 1]) { j -= 1 } return j > bodyStart ? j : nil } /// `{{{name}}}` or `{{{name(arguments)}}}`. func macro(_ i: Int, _ limit: Int) -> Int? { guard hasPrefix("{{{", at: i, limit) else { return nil } var j = i + 3 guard j < limit, chars[j].properties.isAlphabetic else { return nil } while j < limit, isWordScalar(chars[j]) || chars[j] == "-" || chars[j] == "_" { j += 1 } if j < limit, chars[j] == "(" { var k = j + 1 while k < limit, !hasPrefix(")}}}", at: k, limit) { if isNewline(chars[k]) { return nil } k += 1 } return k < limit ? k + 4 : nil } return hasPrefix("}}}", at: j, limit) ? j + 3 : nil } /// `\\` at the end of a line, before optional trailing blanks. The line break stays outside. func lineBreak(_ i: Int, _ limit: Int) -> Int? { guard hasPrefix("\\\\", at: i, limit) else { return nil } var j = i + 2 while j < limit, chars[j] == " " || chars[j] == "\t" { j += 1 } if j + 1 < limit, chars[j] == "\r", chars[j + 1] == "\n" { return j } guard j == limit || isNewline(chars[j]) else { return nil } return j } /// `\(...\)` or `\[...\]`. func latexFragment(_ i: Int, _ limit: Int) -> Int? { guard i + 1 < limit else { return nil } let closer: String switch chars[i + 1] { case "(": closer = "\\)" case "[": closer = "\\]" default: return nil } let closing = Array(closer.unicodeScalars) var j = i + 2 while j < limit { if hasPrefix(closing, at: j, limit) { return j + 2 } j += 1 } return nil } /// `^word` or `^{group}` after a letter or digit. func superscript(_ i: Int, _ limit: Int) -> Int? { var j = i + 1 guard j < limit else { return nil } if chars[j] == "{" { j += 1 while j < limit, chars[j] != "}" { if isNewline(chars[j]) { return nil } j += 1 } return j < limit ? j + 1 : nil } let start = j while j < limit, isWordScalar(chars[j]) { j += 1 } return j > start ? j : nil } /// `src_lang{body}` or `src_lang[headers]{body}`, on one line, with balanced braces. func inlineSourceBlock(_ i: Int, _ limit: Int) -> Int? { guard hasPrefix("src_", at: i, limit) else { return nil } var j = i + 4 let languageStart = j while j < limit, !isSpace(chars[j]), chars[j] != "[", chars[j] != "{" { j += 1 } guard j > languageStart, j < limit else { return nil } if chars[j] == "[" { while j < limit, chars[j] != "]" { if isNewline(chars[j]) { return nil } j += 1 } guard j < limit else { return nil } j += 1 } guard j < limit, chars[j] == "{" else { return nil } var depth = 0 while j < limit { if isNewline(chars[j]) { return nil } if chars[j] == "{" { depth += 1 } else if chars[j] == "}" { depth -= 1 if depth == 0 { return j + 1 } } j += 1 } return nil } } extension Parser { mutating func inline(_ text: Substring) { let scanner = InlineScanner(text) scanner.scan(0.. Timestamp? { - let chars = Array(text) + let chars = Array(text.unicodeScalars) guard let result = scanTimestamp(chars, at: 0, limit: chars.count), result.end == chars.count else { return nil } return result.stamp } } /// A timestamp or `--` range starting at `start`, and the index after it. -func scanTimestamp(_ chars: [Character], at start: Int, limit: Int) -> (stamp: Timestamp, end: Int)? { +func scanTimestamp(_ chars: [Unicode.Scalar], at start: Int, limit: Int) -> (stamp: Timestamp, end: Int)? { guard let first = scanSingleTimestamp(chars, at: start, limit: limit) else { return nil } if first.stamp.end == nil, first.end + 2 < limit, chars[first.end] == "-", chars[first.end + 1] == "-", let second = scanSingleTimestamp(chars, at: first.end + 2, limit: limit), @@ -62,22 +62,23 @@ func scanTimestamp(_ chars: [Character], at start: Int, limit: Int) -> (stamp: T } /// ``, or the same in `[...]` for inactive. -private func scanSingleTimestamp(_ chars: [Character], at start: Int, limit: Int) -> (stamp: Timestamp, end: Int)? { +private func scanSingleTimestamp(_ chars: [Unicode.Scalar], at start: Int, limit: Int) -> (stamp: Timestamp, end: Int)? { guard start < limit, chars[start] == "<" || chars[start] == "[" else { return nil } let active = chars[start] == "<" - let close: Character = active ? ">" : "]" + let close: Unicode.Scalar = active ? ">" : "]" var j = start + 1 func number(_ minDigits: Int, _ maxDigits: Int) -> Int? { var k = j - while k < limit, k - j < maxDigits, chars[k].isASCII, chars[k].isNumber { k += 1 } + while k < limit, k - j < maxDigits, isASCIIDigit(chars[k]) { k += 1 } guard k - j >= minDigits else { return nil } - let value = Int(String(chars[j.. Bool { + func take(_ c: Unicode.Scalar) -> Bool { guard j < limit, chars[j] == c else { return false } j += 1 return true @@ -85,7 +86,7 @@ private func scanSingleTimestamp(_ chars: [Character], at start: Int, limit: Int func interval() -> Timestamp.Interval? { let before = j - guard let value = number(1, 9), j < limit, let unit = Timestamp.Unit(rawValue: chars[j]) else { + guard let value = number(1, 9), j < limit, let unit = Timestamp.Unit(rawValue: Character(chars[j])) else { j = before return nil } @@ -145,7 +146,7 @@ private func scanSingleTimestamp(_ chars: [Character], at start: Int, limit: Int // Day name: anything but digits, whitespace, `+`, `-`, `]` and `>`, in any language. if j < limit, chars[j] == " " { var k = j + 1 - while k < limit, !(chars[k].isNumber || chars[k].isWhitespace || "+-]>".contains(chars[k])) { k += 1 } + while k < limit, !(isDigit(chars[k]) || chars[k].properties.isWhitespace || "+-]>".unicodeScalars.contains(chars[k])) { k += 1 } if k > j + 1 { j = k } } @@ -179,3 +180,12 @@ private func scanSingleTimestamp(_ chars: [Character], at start: Int, limit: Int guard take(close) else { return nil } return (stamp, j) } + +func isASCIIDigit(_ c: Unicode.Scalar) -> Bool { + c.value >= 48 && c.value <= 57 +} + +/// Any numeric character, as `Character.isNumber` would see it in a day name. +func isDigit(_ c: Unicode.Scalar) -> Bool { + c.properties.numericType != nil +} ``` - [ ] **Step 2: Run tests, the corpus and the reparse gate; commit** Run: `swift test`; `ORGSTAR_CORPUS= swift test -c release --filter corpusRoundTrips` for each local folder; `ORGSTAR_FUZZ_EDITS=100000 swift test -c release --filter incrementalEqualsFull`. ```bash git add Sources/OrgCore/Parser/Inline.swift Sources/OrgCore/Timestamp.swift git commit -m "Scan inline objects over unicode scalars" ``` --- ### Task 2: Region reparse **Files:** - Modify: `Sources/OrgCore/Syntax/SyntaxNode.swift`, `Sources/OrgCore/Syntax/GreenTree.swift`, `Sources/OrgCore/Parser/Parser.swift`, `Sources/OrgCore/Parser/Incremental.swift` - Test: `Tests/OrgCoreTests/IncrementalTests.swift` **Interfaces:** - Produces: `ReparseStrategy.region`; `SyntaxNode.children(overlapping:)`, `child(containing:)`, `firstChild(_:)`; `GreenBuilder.node(_:)`; `Parser.reusableRows`. - [ ] **Step 1: Write the failing tests** ```diff @@ -49,8 +49,23 @@ struct IncrementalTests { #expect(checkReparse("* a\nx\ny\n* b\n", 6..<6, "* c\n") == .sections) } - @Test func joiningParagraphsReparsesSections() { - #expect(checkReparse("a\n\nb\n", 1..<2, "") == .sections) + @Test func joiningParagraphsReparsesARegion() { + #expect(checkReparse("a\n\nb\n", 1..<2, "") == .region) + #expect(checkReparse("* h\nfirst\nsecond\n\nthird\n** child\n", 15..<15, "\n\n") == .region) + } + + @Test func regionGrowsUntilABoundaryHolds() { + // Splitting a paragraph next to a table and a list. + #expect(checkReparse("* h\nx\ntext\nmore\n| a |\n- i\n", 9..<9, "\n") == .region) + // A new begin line without its end runs to the end of the body. + #expect(checkReparse("* h\nx\none\ntwo\n#+end_src\nthree\n", 6..<6, "\n#+begin_src") == .region) + } + + @Test func regionDefersToSectionsNearHeadings() { + // The first body line could become a planning line. + #expect(checkReparse("* h\nbody\n", 4..<4, "\n") != .region) + // An added end delimiter could close a block that starts before the window. + #expect(checkReparse("x\n#+begin_src\na\n\nb\n", 16..<16, "#+end_src\n") != .region) } @Test func runGrowsUntilAHeadingEndsIt() { ``` - [ ] **Step 2: Implement** ```diff diff --git a/Sources/OrgCore/Parser/Incremental.swift b/Sources/OrgCore/Parser/Incremental.swift index b34284a..9f8347a 100644 --- a/Sources/OrgCore/Parser/Incremental.swift +++ b/Sources/OrgCore/Parser/Incremental.swift @@ -22,6 +22,8 @@ public struct TextEdit: Sendable, Equatable { enum ReparseStrategy: Equatable { /// One element reparsed in place; everything else reused. case element + /// A run of elements in one section's body reparsed; everything else reused. + case region /// A run of top-level sections reparsed; the sections before and after reused. case sections case full @@ -41,6 +43,7 @@ extension OrgParser { let context = EditContext(oldText: oldText, newText: newText, edit: edit) if context.touchesSettings() { return (parse(newText, defaults: defaults), .full) } if let tree = context.reparseElement(old) { return (tree, .element) } + if let tree = context.reparseRegion(old) { return (tree, .region) } if let tree = context.reparseSections(old) { return (tree, .sections) } return (parse(newText, defaults: defaults), .full) } @@ -113,10 +116,7 @@ private struct EditContext { func leaf(in root: SyntaxNode) -> SyntaxNode? { let range = edit.range var current = root - while let child = current.children.first(where: { - $0.range.lowerBound <= range.lowerBound && range.lowerBound < $0.range.upperBound - && range.upperBound <= $0.range.upperBound - }) { + while let child = current.child(containing: range.lowerBound), range.upperBound <= child.range.upperBound { if Self.leafKinds.contains(child.kind) { return child } current = child } @@ -140,6 +140,151 @@ private struct EditContext { return replacement } + // MARK: - Body region + + /// Reparses a window of the innermost section's own body (the elements between its heading + /// lines and its first child section) and splices it into the old section. + /// + /// The window starts one element before the element holding the line before the edit. It ends at the first + /// later element boundary that the reparse reproduces at the same shifted offset. The + /// window is parsed one element past that boundary, so every decision about where an element + /// ends was made with the following line in view; from a reproduced boundary on, elements + /// parse the same as before. Edits that touch heading lines, the first line after a heading + /// (where planning lines and property drawers are only recognized), or that add an end + /// delimiter (which could pair with a begin line before the window) are left to the + /// section strategy. + func reparseRegion(_ old: OrgTree) -> OrgTree? { + let start = lineStart(oldText, edit.range.lowerBound) + let oldLines = classes(slice(oldText, start, lineEnd(oldText, edit.range.upperBound))) + let newLines = classes(slice(newText, start, lineEnd(newText, edit.range.lowerBound + edit.replacement.utf16.count))) + for line in oldLines + newLines { + if case .heading = line.cls { return nil } + } + for line in newLines { + switch line.cls { + case .blockEnd, .dynamicEnd, .drawerEnd: return nil + default: break + } + } + + // The innermost section or zeroth section holding the edited lines. + let damaged = start..= firstEditable, edit.range.upperBound <= bodyEnd, !body.isEmpty else { return nil } + + // Window start: the body element holding the line before the edit, and one more before + // it, which could absorb the window's first line if that line changes meaning (a drawer + // that loses its end turns into paragraph text). + let anchor = max(bodyStart, start - 1) + guard var first = body.lastIndex(where: { $0.offset <= anchor }) else { return nil } + while first > 0, case .token = body[first].element { first -= 1 } + if first > 0 { + first -= 1 + while first > 0, case .token = body[first].element { first -= 1 } + } + let windowStart = body[first].offset + + var next = body.firstIndex { $0.offset > edit.range.upperBound } ?? body.count + var attempts = 0 + while attempts < 8 { + attempts += 1 + // Parse through one node past the candidate boundary. + var verifyEnd = next + while verifyEnd < body.count, case .token = body[verifyEnd].element { verifyEnd += 1 } + let boundary = next < body.count ? body[next].offset : bodyEnd + let oldWindowEnd = verifyEnd < body.count ? body[verifyEnd].offset + body[verifyEnd].element.length : bodyEnd + let rows = reusableRows(body[first.. body.count { return nil } + continue + } + let before = container.green.children[..<(prefixCount + first)] + let after = next < body.count ? Array(body[next...].map(\.element)) : [] + let tail = container.green.children[(prefixCount + body.count)...] + let green = GreenNode(kind: container.kind, children: Array(before) + reparsed + after + Array(tail)) + // A full parse has no zeroth section for empty text. + if green.children.isEmpty { return nil } + return OrgTree(green: replacing(container, with: green), settings: old.settings) + } + return nil + } + + /// Rows of the tables among `elements`, keyed by their text. + func reusableRows(_ elements: [GreenElement]) -> [Substring: GreenNode] { + var rows: [Substring: GreenNode] = [:] + for case .node(let table) in elements where table.kind == .table { + for case .node(let row) in table.children where row.kind == .tableRow { + rows[Substring(row.text)] = row + } + } + return rows + } + + /// Body elements parsed from `text`, or nil if the text holds a heading. + func parseBody(_ text: Substring, settings: OrgSettings, rows: [Substring: GreenNode] = [:]) -> (elements: [GreenElement], unmatchedBegin: Bool)? { + var parser = Parser(text: String(text), settings: settings) + parser.reusableRows = rows + let count = parser.lines.count + parser.builder.start(.zerothSection) + parser.parseContent(limit: count) + guard parser.i == count else { return nil } + parser.builder.finish() + var unmatched = false + for (index, line) in parser.info.enumerated() { + switch line.cls { + case .blockBegin, .dynamicBegin, .drawerBegin: + if parser.blockEnds[index] == nil { unmatched = true } + default: + break + } + } + return (parser.builder.build().children, unmatched) + } + // MARK: - Sections /// Reparses from the top-level section before the edit up to the first later top-level diff --git a/Sources/OrgCore/Parser/Parser.swift b/Sources/OrgCore/Parser/Parser.swift index 8478c72..2f2addf 100644 --- a/Sources/OrgCore/Parser/Parser.swift +++ b/Sources/OrgCore/Parser/Parser.swift @@ -14,6 +14,9 @@ struct Parser { let settings: OrgSettings var builder = GreenBuilder() var i = 0 + /// Table rows from a previous version, by their line text. A row parses from its own line + /// alone, so a row with the same text can be reused instead of scanned again. + var reusableRows: [Substring: GreenNode] = [:] init(text: String, defaults: OrgSettings) { let lines = splitRawLines(text) @@ -230,6 +233,14 @@ struct Parser { /// A rule row (`|---+---|`) is one text token. Other rows alternate `|` markers and cells; /// every pair of pipes gets a cell, even an empty one, so columns line up. mutating func tableRow() { + if !reusableRows.isEmpty { + let content = lines[i].content + if let row = reusableRows[content.base[content.startIndex..) -> [SyntaxNode] { + var result: [SyntaxNode] = [] + var at = offset + for child in green.children { + let childRange = at..<(at + child.length) + if childRange.lowerBound >= range.upperBound, !range.isEmpty { break } + if case .node(let node) = child, childRange.overlaps(range) { + result.append(SyntaxNode(green: node, offset: at, parent: self)) + } + at = childRange.upperBound + } + return result + } + + /// The child node whose range contains `position`. + public func child(containing position: Int) -> SyntaxNode? { + var at = offset + for child in green.children { + let end = at + child.length + if position < end { + if case .node(let node) = child, position >= at { return SyntaxNode(green: node, offset: at, parent: self) } + return nil + } + at = end + } + return nil + } + + /// The first child node of `kind`. + public func firstChild(_ kind: SyntaxKind) -> SyntaxNode? { + var at = offset + for child in green.children { + if case .node(let node) = child, node.kind == kind { return SyntaxNode(green: node, offset: at, parent: self) } + at += child.length + } + return nil + } + /// This node and every node below it, in document order. public func descendants() -> [SyntaxNode] { [self] + children.flatMap { $0.descendants() } ``` - [ ] **Step 3: Run the gate on several seeds; commit** Run: `ORGSTAR_FUZZ_SEED= ORGSTAR_FUZZ_EDITS=100000 swift test -c release --filter incrementalEqualsFull` for at least five seeds. ```bash git add Sources/OrgCore Tests/OrgCoreTests/IncrementalTests.swift git commit -m "Reparse structural edits within one section's body" ``` --- ### Task 3: Editor styling **Files:** - Modify: `Sources/OrgEditorAppKit/OrgEditor.swift`, `Sources/OrgPresentation/Presentation.swift`, `Sources/OrgDocument/ViewState.swift` - Test: `Tests/OrgEditorAppKitTests/EditorTests.swift` **Interfaces:** - Produces: `OrgEditor.finishStyling()`, `OrgEditor.styledUpTo`, `OrgEditor.firstStyleChunk`, `OrgEditor.styleChunk`; test `Harness(_:layout:)`, `Harness.drainStyling()`. - [ ] **Step 1: Write the failing tests** ```diff @@ -10,13 +10,24 @@ final class Harness { let editor: OrgEditor let window: NSWindow - init(_ text: String) { + convenience init(_ text: String) { + self.init(DocumentState(bytes: Array(text.utf8))) + drainStyling() + layout() + } + + /// The editor as a window would show it first: background styling not yet run. + init(_ document: DocumentState, layout: Bool = true) { _ = NSApplication.shared - editor = OrgEditor(document: DocumentState(bytes: Array(text.utf8))) + editor = OrgEditor(document: document) window = NSWindow(contentRect: NSRect(x: 0, y: 0, width: 600, height: 800), styleMask: [.titled], backing: .buffered, defer: false) window.contentView = editor.makeScrollView() window.makeFirstResponder(editor.textView) - layout() + if layout { self.layout() } + } + + func drainStyling() { + editor.finishStyling() } var textView: NSTextView { editor.textView } @@ -246,3 +257,26 @@ struct RestyleFuzzTests { h.checkInSync() } } + +@MainActor +struct InitialStylingTests { + @Test func stylesTheFirstScreenThenTheRest() throws { + let line = "* heading *bold*\nbody text\n" + let text = String(repeating: line, count: 4_000) + let h = Harness(DocumentState(bytes: Array(text.utf8)), layout: false) + let length = (text as NSString).length + #expect(h.editor.styledUpTo >= OrgEditor.firstStyleChunk && h.editor.styledUpTo < length) + h.textView.insertText("x", replacementRange: NSRange(location: length - 3, length: 0)) + h.drainStyling() + #expect(h.editor.styledUpTo == length + 1) + #expect(h.textView.textStorage!.isEqual(to: Harness(h.string).textView.textStorage!)) + } + + @Test func editsBeforeTheWatermarkShiftIt() { + let text = String(repeating: "line of text\n", count: 5_000) + let h = Harness(DocumentState(bytes: Array(text.utf8)), layout: false) + let before = h.editor.styledUpTo + h.textView.insertText("abc", replacementRange: NSRange(location: 0, length: 0)) + #expect(h.editor.styledUpTo == before + 3) + } +} ``` - [ ] **Step 2: Implement** ```diff diff --git a/Sources/OrgDocument/ViewState.swift b/Sources/OrgDocument/ViewState.swift index 7590ffe..9cdf35a 100644 --- a/Sources/OrgDocument/ViewState.swift +++ b/Sources/OrgDocument/ViewState.swift @@ -30,7 +30,7 @@ public struct ViewState: Sendable, Equatable { /// Descends only through the sections containing `offset`. func isHeadingStart(_ offset: Int, _ root: SyntaxNode) -> Bool { var node = root - while let child = node.children.first(where: { $0.range.contains(offset) }) { + while let child = node.child(containing: offset) { if child.kind == .heading { return child.range.lowerBound == offset } guard child.kind == .section || child.kind == .zerothSection else { return false } node = child diff --git a/Sources/OrgEditorAppKit/OrgEditor.swift b/Sources/OrgEditorAppKit/OrgEditor.swift index ecea416..b7c3dc4 100644 --- a/Sources/OrgEditorAppKit/OrgEditor.swift +++ b/Sources/OrgEditorAppKit/OrgEditor.swift @@ -55,6 +55,15 @@ public final class OrgEditor: NSObject { let theme = Theme() private var isLoading = false private var previousSelection = 0 + /// Text before this offset is styled; after it, only base attributes until the background + /// passes reach it. Always at a line start. + private(set) var styledUpTo = 0 + private var stylingScheduled = false + + /// Styled before the first render: more than a screenful. + static let firstStyleChunk = 30_000 + /// Styled per run-loop turn afterwards. + static let styleChunk = 200_000 public init(document: DocumentState, frame: NSRect = NSRect(x: 0, y: 0, width: 600, height: 800)) { self.document = document @@ -77,10 +86,46 @@ public final class OrgEditor: NSObject { textLayoutManager.delegate = self textContentStorage.delegate = self + load(document.text) + } + + /// Replaces the storage's text. Setting the attributed string directly is about 100x + /// faster than `NSTextView.string` for large files. + private func load(_ text: String) { isLoading = true - textView.string = document.text + textView.textStorage?.setAttributedString(NSAttributedString(string: text, attributes: theme.base)) isLoading = false - restyleOutsideEditing(0.. [SyntaxNode] { + nodes.filter { $0.range.overlaps(around) }.flatMap { node -> [SyntaxNode] in + guard [.table, .plainList, .item].contains(node.kind) else { return [node] } + let inner = overlapping(node.children(overlapping: around)) + return inner.isEmpty ? [node] : inner + } + } + for child in overlapping(container.children(overlapping: around)) { lower = min(lower, child.range.lowerBound) upper = max(upper, child.range.upperBound) } - let heading = children.first { $0.kind == .heading } + let heading = container.firstChild(.heading) if touchedHeadings || heading.map({ $0.range.overlaps(lineRange) }) == true { lower = min(lower, container.range.lowerBound) - upper = max(upper, children.first { $0.kind == .section }?.range.lowerBound ?? container.range.upperBound) + upper = max(upper, container.firstChild(.section)?.range.lowerBound ?? container.range.upperBound) } return lower..= edit.range.upperBound { + styledUpTo += edit.replacement.utf16.count - edit.range.count + } else if styledUpTo > edit.range.lowerBound { + styledUpTo = (storage.string as NSString).lineRange(for: NSRange(location: edit.range.lowerBound, length: 0)).location + } let oldHidden = hidden.all view = view.mapped(through: [edit]).pruned(to: document.tree) folded.set(view.folds) diff --git a/Sources/OrgPresentation/Presentation.swift b/Sources/OrgPresentation/Presentation.swift index 312d7f8..504e0b5 100644 --- a/Sources/OrgPresentation/Presentation.swift +++ b/Sources/OrgPresentation/Presentation.swift @@ -77,7 +77,7 @@ public enum Presentation { default: break } - for child in node.children where child.range.overlaps(range) { + for child in node.children(overlapping: range) { visit(child, range, settings, &runs) } } @@ -118,12 +118,11 @@ public enum Presentation { } private static func collectIndents(_ node: SyntaxNode, _ range: Range, _ runs: inout [IndentRun]) { - for section in node.children where section.kind == .section && section.range.overlaps(range) { - let children = section.children - guard let heading = children.first(where: { $0.kind == .heading }) else { continue } + for section in node.children(overlapping: range) where section.kind == .section { + guard let heading = section.firstChild(.heading) else { continue } let level = heading.tokens.first { $0.kind == .stars }?.text.count ?? 1 runs.append(IndentRun(range: heading.range, firstLine: 0, wrapped: level + 1)) - let bodyEnd = children.first { $0.kind == .section }?.range.lowerBound ?? section.range.upperBound + let bodyEnd = section.firstChild(.section)?.range.lowerBound ?? section.range.upperBound if heading.range.upperBound < bodyEnd { runs.append(IndentRun(range: heading.range.upperBound.. Int? { var node = tree.root - while let child = node.children.first(where: { $0.range.contains(offset) }) { + while let child = node.child(containing: offset) { if child.kind == .heading { return child.range.lowerBound } guard child.kind == .section || child.kind == .zerothSection else { return nil } node = child ``` - [ ] **Step 3: Run the restyle fuzz on several seeds; commit** Run: `ORGSTAR_FUZZ_SEED= ORGSTAR_RESTYLE_EDITS=1500 swift test -c release --filter RestyleFuzzTests` for at least three seeds. ```bash git add Sources/OrgEditorAppKit Sources/OrgPresentation Sources/OrgDocument Tests/OrgEditorAppKitTests/EditorTests.swift git commit -m "Style the first screen first and narrow restyles" ``` --- ### Task 4: Gate tests **Files:** - Create: `Tests/OrgEditorAppKitTests/GateTests.swift` - Modify: `docs/design.md` (memory gate) - [ ] **Step 1: Add the gate suite** ```swift import AppKit import Foundation import OrgDocument import Testing @testable import OrgEditorAppKit /// The phase 1 performance gates (design, "Phases"). Skipped unless `ORGSTAR_GATES` is set; /// run in a release build on the reference machine: /// /// ORGSTAR_GATES=1 swift test -c release --filter GateTests /// /// `ORGSTAR_GATE_FILE` adds a real file to the synthetic ones. Memory limits follow the design: /// 10x the file size for typical documents, 45x for the synthetic worst cases. @MainActor @Suite(.enabled(if: ProcessInfo.processInfo.environment["ORGSTAR_GATES"] != nil)) struct GateTests { /// About 1 MB of body text under a single top-level heading. static let singleSection: String = { var text = "* One big section\n" var n = 0 while text.utf8.count < 1_000_000 { text += "Paragraph \(n) with *bold*, /italic/, a [[https://example.com/\(n)][link]] and <2026-10-04 Sun>.\n" text += "It continues on a second line with =code= and ~more~.\n\n" text += "- item \(n)\n - nested [ ] task\n\n" n += 1 } return text }() /// A 5,000-row table. static let bigTable: String = { var text = "* Table\n| id | name | note |\n|----+------+------|\n" for n in 0..<5_000 { text += "| \(n) | row \(n) | *bold* and [[https://example.com][link]] |\n" } return text }() struct Result: CustomStringConvertible { var name: String var bytes: Int var open: Duration var typingP95: Duration var newlineP95: Duration var memoryRatio: Double var description: String { "\(name) (\(bytes / 1000) kB): open \(open), typing p95 \(typingP95), blank line p95 \(newlineP95), memory \(String(format: "%.1f", memoryRatio))x" } } /// Heap bytes in use, which unlike resident size isn't hidden by memory freed earlier. static func heapBytes() -> Int { var stats = malloc_statistics_t() malloc_zone_statistics(nil, &stats) return Int(stats.size_in_use) } func measure(_ name: String, _ text: String) -> Result { let clock = ContinuousClock() let bytes = Array(text.utf8) let before = Self.heapBytes() var harness: Harness! let open = clock.measure { harness = Harness(DocumentState(bytes: bytes), layout: false) } harness.drainStyling() let memoryRatio = Double(Self.heapBytes() - before) / Double(bytes.count) var timings: [Duration] = [] harness.editor.onEditTiming = { timings.append($0) } let paragraphs = harness.editor.document.tree.root.descendants().filter { $0.kind == .paragraph || $0.kind == .tableCell } harness.caret(at: paragraphs[paragraphs.count / 2].range.lowerBound + 1) for _ in 0..<100 { harness.textView.insertText("x", replacementRange: harness.textView.selectedRange()) } let typing = timings.sorted()[95] timings = [] for _ in 0..<20 { harness.textView.insertNewline(nil) harness.textView.insertNewline(nil) harness.textView.deleteBackward(nil) harness.textView.deleteBackward(nil) } let sorted = timings.sorted() return Result(name: name, bytes: bytes.count, open: open, typingP95: typing, newlineP95: sorted[sorted.count * 95 / 100], memoryRatio: memoryRatio) } @Test func gates() throws { // The first text view, and the first text loaded into one, set up the text system once // per process; an app pays that at launch, not per file. NSTextView(usingTextLayoutManager: true).textStorage?.setAttributedString(NSAttributedString(string: "warm up")) // Memory limits: worst-case dense markup for the synthetic files, typical for real ones. var inputs = [("single 1 MB section", Self.singleSection, 45.0), ("5,000-row table", Self.bigTable, 45.0)] if let path = ProcessInfo.processInfo.environment["ORGSTAR_GATE_FILE"] { inputs.append((URL(fileURLWithPath: path).lastPathComponent, try String(contentsOfFile: path, encoding: .utf8), 10.0)) } for (name, text, memoryLimit) in inputs { let result = measure(name, text) print("GATE \(result)") let perMegabyte = Double(result.bytes) / 1_000_000 #expect(result.open < .milliseconds(100) * max(1, perMegabyte), "\(name): open") #expect(result.typingP95 < .milliseconds(16), "\(name): typing") #expect(result.newlineP95 < .milliseconds(16), "\(name): blank line") #expect(result.memoryRatio < memoryLimit, "\(name): memory") } } } ``` - [ ] **Step 2: Run the gates; commit** Run: `ORGSTAR_GATES=1 ORGSTAR_GATE_FILE= swift test -c release --filter GateTests` ```bash git add Tests/OrgEditorAppKitTests/GateTests.swift git commit -m "Add performance gate tests" ```