Commit `1e25fb947d`

1e25fb947d3c01fc983b481fff5b064ed82349ab

parent: ea57052e99

Unsigned

cmc <hello@cleberg.net> · 2026-07-20 22:50 UTC

ci: split accessibility coverage between local runtimes and CI

The two-job CI matrix was buying two near-identical iOS 26.x runs at
double the macOS minutes. GitHub's macos-26 image ships only 26.x
simulator runtimes, so it cannot test the 17.6 floor at all, and floor
coverage was the entire justification for the second job.

Split the work by what each side can uniquely do instead:

CI keeps one job on the newest runtime. Its real value is not the
runtime — it is building a clean checkout of the merge result, which
catches a file that was never committed. A local run cannot, and that
failure mode is live here: DomainDig.xcodeproj is hand-edited and uses
file-system-synchronized groups, where an entire missing folder still
builds locally. sr.ht cannot run macOS, so this is the only place that
check exists. Collapsing the matrix also removed the deployment-target
math, since "newest" is always above the floor.

Scripts/audit-a11y.sh runs the audit against real runtimes, defaulting
to floor + current. It reads the deployment target from the project
rather than hard-coding it, selects the oldest runtime at or above it
(one below is useless — the app cannot install), and says so plainly
when the nearest installed runtime is a major version above the target
rather than implying floor coverage it does not have.

.githooks/pre-push runs the floor tier, and only when Swift, asset, or
project files changed. Pre-push rather than pre-commit because the suite
takes ~85s: at pre-commit that blocks every commit, and a hook routinely
bypassed with --no-verify is worse than none. Opt in per clone with
'git config core.hooksPath .githooks'.

Docs/ACCESSIBILITY.md records the split, the measured non-nested
coverage that motivates it, and the enforcement ratchet.

Layout: unified · split

.githooks/pre-push added +68

@@ -0,0 +1,68 @@
	1	#!/usr/bin/env bash
	2	#
	3	# Run the accessibility audit against the oldest supported simulator runtime
	4	# before pushing.
	5	#
	6	# Enable once per clone:
	7	# git config core.hooksPath .githooks
	8	#
	9	# Why pre-push and not pre-commit: the suite takes ~85s. At pre-commit that
	10	# blocks every commit, and a hook you routinely bypass with --no-verify is worse
	11	# than no hook, because it trains you to ignore it. Pushes are far less frequent
	12	# and map to the unit of work that actually reaches CI.
	13	#
	14	# Why only the floor tier: CI already audits the newest runtime on a clean
	15	# checkout. The GitHub image has no old runtimes, so the floor is the one thing
	16	# CI structurally cannot cover — and it is the one this machine can. Running
	17	# both here would just double the wait to re-check what CI already does.
	18	#
	19	# Skip deliberately with: git push --no-verify
	20
	21	set -euo pipefail
	22
	23	repo_root=$(git rev-parse --show-toplevel)
	24	cd "$repo_root"
	25
	26	zero='0000000000000000000000000000000000000000'
	27	changed=''
	28
	29	# stdin: <local ref> <local sha> <remote ref> <remote sha>, one line per ref.
	30	while read -r _local_ref local_sha _remote_ref remote_sha; do
	31	[ "$local_sha" = "$zero" ] && continue # branch deletion
	32
	33	if [ "$remote_sha" = "$zero" ]; then
	34	# New branch: diff against the default branch rather than the whole history.
	35	base=$(git merge-base origin/main "$local_sha" 2>/dev/null \|\| echo '')
	36	range="${base:+$base..}$local_sha"
	37	else
	38	range="$remote_sha..$local_sha"
	39	fi
	40
	41	changed="$changed$(git diff --name-only "$range" 2>/dev/null \|\| true)"$'\n'
	42	done
	43
	44	if [ -z "$(printf '%s' "$changed" \| tr -d '[:space:]')" ]; then
	45	exit 0
	46	fi
	47
	48	# Only pay the ~85s when something could actually change the rendered UI.
	49	if ! printf '%s' "$changed" \| grep -qE '\.(swift\|xcassets\|xcodeproj)\|\.pbxproj\|xcscheme'; then
	50	echo "pre-push: no Swift/project changes, skipping accessibility audit"
	51	exit 0
	52	fi
	53
	54	echo "pre-push: running accessibility audit on the floor runtime (~85s)"
	55	echo " skip with 'git push --no-verify'"
	56
	57	if ! ./Scripts/audit-a11y.sh floor; then
	58	echo
	59	echo "pre-push: audit could not run. Push aborted." >&2
	60	echo " Re-run with './Scripts/audit-a11y.sh floor' to see why," >&2
	61	echo " or bypass with 'git push --no-verify'." >&2
	62	exit 1
	63	fi
	64
	65	# Note: findings are reported, not enforced, so a clean exit here does not mean
	66	# zero findings — read the list above. Enforcement is controlled by
	67	# AccessibilityAuditHarness.enforcedAuditTypes.
	68	exit 0

.github/workflows/build.yml +31 −61

 # dependencies). The test target is DomainDigUITests — an accessibility audit
 # suite; see DomainDigUITests/AccessibilityAuditHarness.swift.
 #
+# WHAT THIS JOB IS FOR, given the audit also runs locally:
+# a clean checkout of the merge result. A local hook runs against the working
+# tree and therefore cannot catch a file that was never committed — the failure
+# mode that matters most here, since DomainDig.xcodeproj is hand-edited and uses
+# file-system-synchronized groups where a whole missing folder still builds fine
+# locally. This job is the only place that check exists; sr.ht cannot run it.
+#
+# DELIBERATELY ONE JOB, NEWEST RUNTIME ONLY. Audit coverage is not nested across
+# OS versions, so the oldest supported OS genuinely needs its own run — but the
+# macos-26 image ships only iOS 26.x runtimes, so CI *cannot* provide it. Asking
+# for two jobs here bought two near-identical 26.x runs at double the macOS
+# minutes. Floor coverage lives in Scripts/audit-a11y.sh, run from a machine that
+# actually has an 18.x runtime installed, and is wired to the pre-push hook in
+# .githooks/. See Docs/ACCESSIBILITY.md for the split.
+#
 # The audit REPORTS but does not FAIL by default. It surfaces violations that
 # exist today, so gating on it would block every unrelated PR until the
 # accessibility pass in issue #21 completes. Findings land in the job log and in
-# the uploaded .xcresult bundle, tagged [report] or [FAIL].
+# the uploaded .xcresult bundle, tagged [report] or [FAIL]. Enforcement is a
-#
+# committed constant: widen `AccessibilityAuditHarness.enforcedAuditTypes` as
-# Enforcement is a committed constant, not a CI setting: widen
+# each phase clears a category. (Env vars were tried first — neither a plain
-# `AccessibilityAuditHarness.enforcedAuditTypes` as each phase clears a category.
+# xcodebuild env var nor a TEST_RUNNER_-prefixed build setting reaches the UI
-# (Env vars were tried first — neither a plain xcodebuild env var nor a
+# test process.)
-# TEST_RUNNER_-prefixed build setting reaches the UI test process.)
-#
-# The matrix runs two simulators because audit coverage is NOT nested — each
-# runtime reports findings the other misses, in both directions. Measured
-# locally on the Tracked Domains screen, iOS 18.6 reported 2 issues and iOS 27.0
-# reported 6 (including contrast and element-detection issues 18.6 never
-# raised); at accessibility text sizes the Dashboard produced a hit-region
-# finding on 18.6 that 27.0 did not.
-#
-# CAVEAT — this image cannot test the real support floor. The app's deployment
-# target is 17.6, but the macos-26 runner ships only iOS 26.x simulator
-# runtimes, so "floor" resolves to ~26.2 here rather than an 18.x image. CI
-# therefore compares two 26.x runtimes; genuine oldest-supported-OS coverage has
-# to come from a local run or a self-hosted runner with older runtimes
-# installed. The "Select simulator" step emits a warning annotation when the
-# resolved floor sits well above the deployment target, so this gap stays
-# visible instead of being silently assumed away.
-#
-# Installing an older runtime in CI (xcodebuild -downloadPlatform iOS
-# -buildVersion 18.6) is possible but costs several GB and minutes per job; not
-# done by default.
-#
-# Runtimes are resolved dynamically rather than pinned. The previous selector
-# took the first iPhone from any runtime, which could pick a simulator BELOW the
-# deployment target, where the app cannot install.
 #
 # pull_request only, plus manual dispatch. GitHub builds the merge result (PR
 # merged into main), so a green PR validates exactly what will land on main.
   group: build-${{ github.ref }}
   cancel-in-progress: true
-env:
-  # Keep in sync with IPHONEOS_DEPLOYMENT_TARGET in DomainDig.xcodeproj.
-  DEPLOYMENT_TARGET_MAJOR: '17'
-  DEPLOYMENT_TARGET_MINOR: '6'
 jobs:
   test:
-    name: xcodebuild test (${{ matrix.tier }})
+    name: xcodebuild test
     # macos-latest still points at macOS 15, which lacks the iOS 26+ SDK this app
     # is built against.
     runs-on: macos-26
-    strategy:
-      fail-fast: false
-      matrix:
-        # floor   = oldest runtime the app actually supports
-        # current = newest runtime available on the image
-        tier: [floor, current]
     steps:
       - uses: actions/checkout@v7
         id: sim
         run: |
           set -euo pipefail
-          floor=$(( DEPLOYMENT_TARGET_MAJOR * 1000 + DEPLOYMENT_TARGET_MINOR ))
+          # Newest available iPhone runtime. No deployment-target filtering is
+          # needed for "newest" — it is always at or above the floor. The
+          # previous selector took the first iPhone from ANY runtime, which on a
+          # machine with an older runtime installed could pick a simulator below
+          # the deployment target, where the app cannot install.
           selected=$(xcrun simctl list devices available --json \
-            | jq -c --argjson floor "$floor" --arg tier "${{ matrix.tier }}" '
+            | jq -c '
                 [ .devices | to_entries[]
                   | (.key | capture("SimRuntime\\.iOS-(?<maj>[0-9]+)-(?<min>[0-9]+)$")) as $v
                   | (($v.maj | tonumber) * 1000 + ($v.min | tonumber)) as $rank
-                  | select($rank >= $floor)
                   | .value[]
                   | select(.name | startswith("iPhone"))
                   | { rank: $rank, udid: .udid, name: .name, os: "\($v.maj).\($v.min)" }
                 ]
                 | sort_by(.rank, .name)
-                | if length == 0 then empty
+                | last
-                  elif $tier == "floor" then .[0]
-                  else .[-1] end
               ')
-          if [ -z "$selected" ]; then
+          if [ -z "$selected" ] || [ "$selected" = "null" ]; then
-            echo "::error::No iPhone simulator at or above iOS ${DEPLOYMENT_TARGET_MAJOR}.${DEPLOYMENT_TARGET_MINOR} on this image"
+            echo "::error::No iPhone simulator available on this image"
             xcrun simctl list devices available >&2
             exit 1
           fi
           echo "udid=$(echo "$selected" | jq -r .udid)" >> "$GITHUB_OUTPUT"
           echo "label=$label" >> "$GITHUB_OUTPUT"
-          # Surface the floor-coverage gap rather than letting the matrix imply
-          # coverage it does not have. One major version of slack is tolerated.
-          if [ "${{ matrix.tier }}" = "floor" ]; then
-            rank=$(echo "$selected" | jq -r .rank)
-            if [ "$rank" -ge $(( (DEPLOYMENT_TARGET_MAJOR + 1) * 1000 )) ]; then
-              echo "::warning::Floor tier resolved to $label, well above the ${DEPLOYMENT_TARGET_MAJOR}.${DEPLOYMENT_TARGET_MINOR} deployment target. This image has no older runtime, so the oldest supported OS is NOT covered by this run."
-            fi
-          fi
       - name: Test on ${{ steps.sim.outputs.label }}
         run: |
           set -o pipefail
         if: always()
         uses: actions/upload-artifact@v4
         with:
-          name: test-results-${{ matrix.tier }}
+          name: test-results
           path: TestResults.xcresult
           retention-days: 7

Docs/ACCESSIBILITY.md added +96

@@ -0,0 +1,96 @@
	1	# Accessibility Audit
	2
	3	`DomainDigUITests` runs Apple's `performAccessibilityAudit()` across every
	4	primary screen. The audit checks contrast, hit-region size, clipped text at
	5	large Dynamic Type, element descriptions, trait correctness, and Dynamic Type
	6	support — the same ground the accessibility pass tracked in
	7	[issue #21](https://github.com/zerolabsco/domain-dig/issues/21) covers.
	8
	9	## Findings are reported, not enforced
	10
	11	The audit surfaces violations that exist today, so failing on all of them would
	12	block every unrelated change until the whole pass lands. Instead, findings are
	13	logged and attached to the result bundle tagged `[report]` or `[FAIL]`.
	14
	15	Enforcement is the committed constant
	16	`AccessibilityAuditHarness.enforcedAuditTypes`. Widen it as each phase clears a
	17	category:
	18
	19	\| After phase \| Enforce \|
	20	\| --- \| --- \|
	21	\| 2 — semantic colors + light mode \| `.contrast` \|
	22	\| 3 — Dynamic Type + reflow \| `.textClipped`, `.dynamicType`, `.hitRegion` \|
	23	\| 4 — VoiceOver \| `.elementDetection`, `.sufficientElementDescription`, `.trait` \|
	24
	25	A constant rather than a CI setting, for two reasons. Environment variables do
	26	not work: neither a plain `xcodebuild` env var nor a `TEST_RUNNER_`-prefixed
	27	build setting reaches the UI test process, so the toggle silently did nothing.
	28	And a committed value makes "when did contrast become enforced?" answerable with
	29	`git blame` instead of CI tribal knowledge.
	30
	31	## Why coverage is split between local and CI
	32
	33	Audit coverage is not nested across OS versions. Each runtime reports
	34	findings the others miss, in both directions. Measured on this project:
	35
	36	\| Screen \| iOS 18.6 \| iOS 27.0 \|
	37	\| --- \| --- \| --- \|
	38	\| Tracked Domains \| 2 (text clipped) \| 6 (+ contrast ×3, element detection) \|
	39	\| Settings \| 2 contrast \| `dynamicType` finding 18.6 missed \|
	40	\| Dashboard @ `AccessibilityXXXL` \| hit region + 2 clipped \| 1 clipped only \|
	41
	42	Neither runtime is a superset, so the oldest supported OS needs its own run.
	43	This also rules out committing per-screen baseline counts as a regression guard:
	44	no single number is correct on both.
	45
	46	The catch is that **GitHub's `macos-26` image ships only iOS 26.x simulator
	47	runtimes.** It cannot test the 17.6 floor at all. A two-job CI matrix was tried
	48	and produced two near-identical 26.x runs at double the macOS minutes.
	49
	50	So the work is split by what each side can uniquely do:
	51
	52	\| \| Runtime \| Uniquely provides \|
	53	\| --- \| --- \| --- \|
	54	\| CI (`.github/workflows/build.yml`) \| newest available \| A clean checkout of the merge result — catches a file that was never committed, which a local run cannot. Matters here because `DomainDig.xcodeproj` is hand-edited and uses file-system-synchronized groups, where a whole missing folder still builds locally. \|
	55	\| Local (`Scripts/audit-a11y.sh`) \| oldest supported + newest \| Real floor coverage, on a machine that actually has an 18.x runtime installed. \|
	56
	57	Together they cover both ends; neither duplicates the other.
	58
	59	## Running it
	60
	61	```sh
	62	./Scripts/audit-a11y.sh # floor + current
	63	./Scripts/audit-a11y.sh floor # oldest supported only (~85s)
	64	./Scripts/audit-a11y.sh current # newest installed only
	65	```
	66
	67	The script reads the deployment target from the project rather than hard-coding
	68	it, and selects the oldest installed runtime at or above it — a runtime
	69	below the deployment target is useless, because the app cannot install there.
	70	If the nearest installed runtime is a major version above the target, it says
	71	so rather than implying floor coverage it does not have.
	72
	73	### Pre-push hook
	74
	75	```sh
	76	git config core.hooksPath .githooks
	77	```
	78
	79	Runs the floor audit before a push, and only when Swift, asset, or project files
	80	changed. Bypass with `git push --no-verify`.
	81
	82	Pre-push rather than pre-commit deliberately: the suite takes ~85s, and at
	83	pre-commit that blocks every commit. A hook routinely bypassed with
	84	`--no-verify` is worse than no hook, because it trains you to ignore it.
	85
	86	## Notes
	87
	88	- Audits retry up to three times. Slower machines can miss the audit's internal
	89	deadline (`Audit failed to complete in time`, code `-56`), which is a tooling
	90	timeout, not an app defect. A screen that still cannot be audited is reported
	91	as an `XCTSkip`, never a pass — skips are visually distinct in CI, so an
	92	unaudited screen stays visible instead of being silently counted as clean.
	93	- The suite launches with `DOMAIN_DIG_FORCE_PRO_PLUS` so Pro-gated screens are
	94	reachable. `PurchaseService` honours that argument in `DEBUG` builds only.
	95	- Everything used is available at the iOS 17.6 deployment floor;
	96	`performAccessibilityAudit` is `ios(17.0)`.

README.md +24

	@@ -53,6 +53,30 @@ xcodebuild -project DomainDig.xcodeproj -scheme DomainDig -destination 'platform
53		53
54	The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`.	54	The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`.
55		55
		56	### Accessibility Audit
		57
		58	`DomainDigUITests` runs `performAccessibilityAudit()` over every primary screen,
		59	at default and at the largest accessibility text size.
		60
		61	```sh
		62	./Scripts/audit-a11y.sh # oldest supported + newest runtime
		63	./Scripts/audit-a11y.sh floor # oldest supported runtime only (~85s)
		64	```
		65
		66	Findings are reported, not enforced — they are the burndown list for the
		67	accessibility pass. Widen `AccessibilityAuditHarness.enforcedAuditTypes` to turn
		68	a category into a build failure as each phase lands.
		69
		70	Optional pre-push hook, which runs the floor audit when Swift or project files
		71	change:
		72
		73	```sh
		74	git config core.hooksPath .githooks
		75	```
		76
		77	See [Docs/ACCESSIBILITY.md](Docs/ACCESSIBILITY.md) for why coverage is split
		78	between this script and CI.
		79
56	## Release Planning	80	## Release Planning
57		81
58	See `RELEASE_ROADMAP.md` for the semver release plan from `v4.4.1` through the planned `v5.0.0` stabilization milestone.	82	See `RELEASE_ROADMAP.md` for the semver release plan from `v4.4.1` through the planned `v5.0.0` stabilization milestone.

Scripts/audit-a11y.sh added +129

@@ -0,0 +1,129 @@
	1	#!/usr/bin/env bash
	2	#
	3	# Run the accessibility audit suite against real simulator runtimes.
	4	#
	5	# ./Scripts/audit-a11y.sh # floor + current
	6	# ./Scripts/audit-a11y.sh floor # oldest supported runtime only
	7	# ./Scripts/audit-a11y.sh current # newest installed runtime only
	8	#
	9	# Why this exists rather than living entirely in CI: audit coverage is NOT
	10	# nested across OS versions — each runtime reports findings the others miss, in
	11	# both directions. Measured on Tracked Domains, iOS 18.6 reported 2 findings and
	12	# iOS 27.0 reported 6; at accessibility text sizes the Dashboard produced a
	13	# hit-region finding on 18.6 that 27.0 did not. The GitHub macos-26 image ships
	14	# only iOS 26.x runtimes, so it structurally cannot cover the floor. This machine
	15	# can.
	16	#
	17	# CI covers "current" on a clean checkout (which a local run cannot, since it
	18	# would miss an uncommitted file). This script covers "floor" (which CI cannot).
	19	# Together they cover both; neither duplicates the other.
	20	#
	21	# The deployment target is read from the project rather than hard-coded, so it
	22	# cannot drift out of sync.
	23
	24	set -euo pipefail
	25
	26	cd "$(dirname "$0")/.."
	27
	28	PROJECT="DomainDig.xcodeproj"
	29	SCHEME="DomainDig"
	30	TIER="${1:-both}"
	31
	32	case "$TIER" in
	33	floor \| current \| both) ;;
	34	*)
	35	echo "usage: $0 [floor\|current\|both]" >&2
	36	exit 2
	37	;;
	38	esac
	39
	40	echo "==> Reading deployment target from $PROJECT"
	41	DEPLOYMENT_TARGET=$(
	42	xcodebuild -project "$PROJECT" -scheme "$SCHEME" -showBuildSettings 2>/dev/null \
	43	\| awk -F' = ' '/ IPHONEOS_DEPLOYMENT_TARGET = /{print $2; exit}'
	44	)
	45
	46	if [ -z "${DEPLOYMENT_TARGET:-}" ]; then
	47	echo "error: could not read IPHONEOS_DEPLOYMENT_TARGET" >&2
	48	exit 1
	49	fi
	50
	51	DT_MAJOR="${DEPLOYMENT_TARGET%%.*}"
	52	DT_MINOR="${DEPLOYMENT_TARGET##*.}"
	53	[ "$DT_MINOR" = "$DEPLOYMENT_TARGET" ] && DT_MINOR=0
	54	FLOOR_RANK=$(( DT_MAJOR * 1000 + DT_MINOR ))
	55
	56	echo " deployment target: $DEPLOYMENT_TARGET (rank $FLOOR_RANK)"
	57
	58	# Pick an iPhone simulator at or above the deployment target. A runtime BELOW it
	59	# is useless — the app cannot install there — which is why this filters rather
	60	# than just taking the oldest installed runtime.
	61	select_sim() {
	62	local which="$1"
	63	xcrun simctl list devices available --json \
	64	\| jq -c --argjson floor "$FLOOR_RANK" --arg which "$which" '
	65	[ .devices \| to_entries[]
	66	\| (.key \| capture("SimRuntime\\.iOS-(?<maj>[0-9]+)-(?<min>[0-9]+)$")) as $v
	67	\| (($v.maj \| tonumber) * 1000 + ($v.min \| tonumber)) as $rank
	68	\| select($rank >= $floor)
	69	\| .value[]
	70	\| select(.name \| startswith("iPhone"))
	71	\| { rank: $rank, udid: .udid, name: .name, os: "\($v.maj).\($v.min)" }
	72	]
	73	\| sort_by(.rank, .name)
	74	\| if length == 0 then empty
	75	elif $which == "floor" then first
	76	else last end
	77	'
	78	}
	79
	80	run_tier() {
	81	local which="$1"
	82	local sim udid label
	83
	84	sim=$(select_sim "$which")
	85	if [ -z "$sim" ]; then
	86	echo "error: no iPhone simulator at or above iOS $DEPLOYMENT_TARGET installed" >&2
	87	echo "hint: install one with 'xcodebuild -downloadPlatform iOS'" >&2
	88	return 1
	89	fi
	90
	91	udid=$(echo "$sim" \| jq -r .udid)
	92	label=$(echo "$sim" \| jq -r '"\(.name) (iOS \(.os))"')
	93
	94	echo
	95	echo "==> $which: $label"
	96
	97	if [ "$which" = "floor" ] && [ "$(echo "$sim" \| jq -r .rank)" -ge $(( (DT_MAJOR + 1) * 1000 )) ]; then
	98	echo " NOTE: nearest installed runtime is a major version above the $DEPLOYMENT_TARGET"
	99	echo " deployment target, so this is not true floor coverage."
	100	fi
	101
	102	# Findings are printed by the suite itself; surface them plus the verdict.
	103	set -o pipefail
	104	xcodebuild test \
	105	-project "$PROJECT" \
	106	-scheme "$SCHEME" \
	107	-destination "id=$udid" \
	108	CODE_SIGNING_ALLOWED=NO 2>&1 \
	109	\| grep -E 'finding$s$\|no accessibility findings\|^ • \|did not complete in time\|Executed [0-9]+ test\|\\ TEST (SUCCEEDED\|FAILED)' \
	110	\|\| true
	111
	112	return 0
	113	}
	114
	115	status=0
	116	if [ "$TIER" = "both" ]; then
	117	run_tier floor \|\| status=1
	118	run_tier current \|\| status=1
	119	else
	120	run_tier "$TIER" \|\| status=1
	121	fi
	122
	123	echo
	124	if [ "$status" -eq 0 ]; then
	125	echo "==> Done. Findings above are the burndown list for issue #21."
	126	echo " They are reported, not enforced — widen"
	127	echo " AccessibilityAuditHarness.enforcedAuditTypes as each phase lands."
	128	fi
	129	exit "$status"

Commit 1e25fb947d

Commit `1e25fb947d`