krz/org-conformance

A language-neutral conformance corpus for org-mode to HTML renderers.

conformance html org-mode testing

Clone: git clone https://gitbay.org/krz/org-conformance.git

1 commit 1 branch 0 tags

HTML 49.1%Markdown 26.6%Org 20.3%Shell 4%

1 contributorcmc

org-conformance

A language-neutral conformance corpus for org-mode → HTML renderers.

The problem it solves: the same .org file is rendered by several independent codebases — a Go forge (gitbay via go-org), a Rust static-site generator (orgo), a Swift package (OrgSwift, shared by the iOS clients), a browser previewer (org-live). They drift. A footnote renders in one and vanishes in another; a :tag: styles here and prints raw there. Nobody notices until they look at the same document on two surfaces.

This corpus is the shared answer to "what is the right output". Each implementation runs it in its own test suite and asserts against the same goldens.

What's here

cases/
  <name>.org        the input
  <name>.html       orgo's rendered HTML — the reference rendering, for humans
  <name>.skeleton   the golden semantic skeleton — the assertion target
manifest.json       every case, the constructs it exercises, and its scope (in/out)
SKELETON.md         the exact reduction each implementation must port
tools/generate.sh   regenerate .html and .skeleton from orgo

The reference

The goldens are produced by orgo, and orgo alone. orgo is not self-appointed: it carries an oracle test that exports every fixture with Emacs's own ox-html under emacs --batch and diffs the two semantic skeletons, so its output is anchored to org's reference exporter rather than to itself. When orgo and Emacs deliberately differ (orgo emits <time datetime> for a timestamp where Emacs writes plain text, <em> where Emacs writes <i>, and so on), those divergences are enumerated and reviewed in orgo's oracle. Everything downstream conforms to orgo.

How an implementation conforms

Comparing raw HTML across languages is pointless — every renderer wraps and classes things differently. Instead, both sides are reduced to a skeleton: the ordered sequence of element opens, closes, and text runs, with div/span and all attributes except href/src dropped, whitespace collapsed, and entities decoded. See SKELETON.md for the exact algorithm.

Each implementation:

  1. Ports the ~150-line skeleton reduction. Verify the port first by feeding each cases/<name>.html through it and checking it reproduces cases/<name>.skeleton exactly — this separates a skeleton-port bug from a renderer bug.
  2. Renders each cases/<name>.org with its own renderer, reduces the output with its port, and asserts equality against cases/<name>.skeleton.
  3. For constructs it does not yet support, keeps an explicit allowlist of expected divergences (mirroring orgo's own "deliberate divergence" list against Emacs). A green suite with a documented allowlist beats a red suite nobody acts on — the point is that new, unintended drift shows up as a diff.

Cases with "scope": "out" in the manifest are deliberately-unsupported constructs (babel, LaTeX, macros, arbitrary drawers). The contract there is only that they degrade predictably and never crash; an implementation may legitimately differ from orgo, so treat those as documentation rather than hard assertions.

Consumers

Implementation Language Engine Status
orgo Rust own element-tree renderer reference (generates the goldens)
OrgSwift Swift extracted from hutch conformance test in swift test
gitbay Go go-org planned
org-live JS (to migrate off vendored org-js) planned

Regenerating

The goldens are derived, not hand-written. After an intended change to orgo's renderer:

ORGO_DIR=../orgo tools/generate.sh

Review the diff, and only commit it once orgo's own oracle agrees the change is right.