Commit `99e4623af1`

99e4623af1b08f36120a01b67cbe60df99668651

parent: a52dee116d

Unsigned

cmc <hello@cleberg.net> · 2026-07-25 04:35 UTC

feat: versioned store-migration policy for persisted data (v5 step 2)

Third v5.0.0 roadmap item: define and implement a migration policy for the
on-device persisted store (tracked domains, history/snapshots, audits,
workflows, monitoring, settings), so data upgrades cleanly across app versions
instead of relying on a one-shot marker.

- DataMigrationService is reworked from a single boolean marker
  (`data.migrations.v3_4_0`) into a versioned runner keyed by an integer store
  schema version (`data.storeSchemaVersion`). It runs each step once in
  ascending order up to `currentStoreSchemaVersion`, stamping the version as it
  goes. Adding a future migration is now a `case N:` plus a version bump.

  Policy guarantees, all covered by tests:
  - Forward-only and idempotent; every step must be safe on an empty/older store.
  - Never downgrades: a store written by a newer build (higher version) is left
    byte-for-byte untouched.
  - Pre-versioning installs are handled: a set legacy boolean marker reads as
    "already at v1", so the v1 normalization never re-runs for them.

  v1 is the existing normalization pass (dedup + drop the legacy `watchedDomains`
  key + sanitize monitoring settings), now expressed as migration step 1.

- Docs/data-migration.md documents the persisted surface, the two independent
  version lines (store vs. backup export), when to use lenient decoding vs. a
  migration step, the runner contract, an "adding a migration" checklist, and
  backup-import compatibility. Linked from the README.

- DataMigrationServiceTests: 6 tests over legacy fixtures — fresh-store stamping,
  legacy `watchedDomains` migration + key drop, in-place dedup of the stored
  blob, idempotence, legacy-marker-as-v1, and the no-downgrade guard. Full unit
  suite: 58 passing.

Layout: unified · split

Docs/data-migration.md added +94

		@@ -0,0 +1,94 @@
	1	# DomainDig Data Migration Policy
	2
	3	How DomainDig's persisted data evolves across app versions without losing or
	4	corrupting a user's on-device store.
	5
	6	## What is persisted
	7
	8	The store is a set of independent JSON blobs in `UserDefaults`, each under a
	9	stable key (see `DomainDataPortabilityService.StorageKey`):
	10
	11	\| Data \| Key \|
	12	\|------\|-----\|
	13	\| Tracked domains \| `trackedDomains` (legacy: `watchedDomains`) \|
	14	\| Lookup history (snapshots) \| `lookupHistory` \|
	15	\| Audit sessions \| `domainAudits` \|
	16	\| Workflows \| `domainWorkflows` \|
	17	\| Monitoring settings / logs \| `monitoring.settings`, `monitoring.logs` \|
	18	\| App settings \| `recentSearches`, `savedDomains`, resolver URL, density \|
	19	\| Feature metadata \| `purchase.cachedEntitlement`, `usageCredits.ledger` \|
	20
	21	A backup export (`DomainDigBackup`) is a separate, self-describing file that
	22	bundles all of the above with its own `schemaVersion`.
	23
	24	## Two version lines
	25
	26	- Store schema version — `DataMigrationService.currentStoreSchemaVersion`,
	27	persisted under `data.storeSchemaVersion`. Describes the shape of the
	28	on-device `UserDefaults` store. Advanced by the migration runner.
	29	- Backup schema version — `DomainDigBackup.currentSchemaVersion`, written
	30	into every exported file. Describes the shape of an export. Checked on import
	31	by `DataValidationService`.
	32
	33	They advance independently: a store migration that doesn't change the export
	34	shape need not bump the backup version, and vice versa.
	35
	36	## How models evolve
	37
	38	Prefer additive, lenient decoding — it needs no migration:
	39
	40	- New optional field → add it with `decodeIfPresent(...) ?? default` in the
	41	model's `init(from:)`. Old data simply lacks the key and falls back.
	42	- New value in a `String`-backed enum → decode unknown values to a safe default
	43	rather than throwing.
	44
	45	Reach for a migration step only when lenient decoding can't express the
	46	change:
	47
	48	- Renaming or removing a storage key (e.g. `watchedDomains` → `trackedDomains`).
	49	- Re-normalizing existing rows (dedup, canonicalizing domain casing).
	50	- Reshaping a blob in a way old readers would misread.
	51
	52	## The migration runner
	53
	54	`DataMigrationService.migrateIfNeeded(defaults:)` runs at launch (and before any
	55	backup export/import). Its contract:
	56
	57	1. Forward-only. It reads the stored version and runs each step with a target
	58	greater than it, in ascending order, up to `currentStoreSchemaVersion`,
	59	stamping the new version after each step.
	60	2. Never downgrades. A store stamped at a version higher than this build
	61	understands (a user who ran a newer build first) is left untouched — no
	62	rewrite, no data loss.
	63	3. Idempotent & safe on any state. Every step must be safe to run on an empty
	64	store and to re-run, because a downgrade-then-upgrade or a partial run can
	65	replay it. v1 (the `watchedDomains` drop + dedup normalization) satisfies this
	66	by loading through the deduplicating loaders and writing back.
	67	4. Pre-versioning installs. Before this framework, a boolean marker
	68	(`data.migrations.v3_4_0`) recorded that the v1 normalization had run. A set
	69	marker is read as "already at version 1," so v1 never re-runs for those users.
	70
	71	## Adding a migration
	72
	73	1. Add a `case N:` to `DataMigrationService.runMigration(to:defaults:)` and a
	74	private helper that performs the change.
	75	2. Bump `currentStoreSchemaVersion` to `N`.
	76	3. Make the helper idempotent and safe on an empty/older store.
	77	4. Add a `DataMigrationServiceTests` case that seeds a pre-`N` fixture, runs
	78	`migrateIfNeeded`, and asserts the upgrade plus the version stamp.
	79	5. If the change also alters the export shape, bump
	80	`DomainDigBackup.currentSchemaVersion` and update `Docs/local-api.md` /
	81	backup validation as needed.
	82
	83	## Backup import compatibility
	84
	85	On import, `DataValidationService.validate(backup:)` compares the file's
	86	`schemaVersion` to the current one:
	87
	88	- Newer than this build → surfaced as an error (the build can't safely read
	89	it).
	90	- Older → imported under the same lenient decoders and merge/dedup rules that
	91	govern the live store; a note is surfaced, not an error.
	92
	93	Imported data flows through `migrateIfNeeded` and the same `save*` deduplication
	94	as everything else, so an old backup lands in the store already normalized.

DomainDataPortabilityService.swift +59 −4

		@@ -342,12 +342,69 @@ enum DataPortabilityCSV {
342	342	}
343	343	}
344	344
	345	/// Versioned migration runner for the on-device persisted store.
	346	///
	347	/// The store is a set of independent JSON blobs in `UserDefaults` (tracked
	348	/// domains, history, audits, workflows, monitoring settings/logs, app settings).
	349	/// Most model evolution is handled additively by the models' own lenient
	350	/// decoders (`decodeIfPresent` with defaults), which need no migration at all.
	351	/// This runner exists only for changes lenient decoding can't express: dropping
	352	/// a renamed storage key, re-normalizing existing rows, or reshaping a blob.
	353	///
	354	/// `currentStoreSchemaVersion` is bumped whenever such a step is added. Each step
	355	/// runs exactly once, in ascending order, and must be safe to run on any prior
	356	/// state — including an empty store. A store written by a newer build (a higher
	357	/// version than this build knows) is left untouched; migrations never downgrade.
	358	/// The policy is documented in `Docs/data-migration.md`.
345	359	enum DataMigrationService {
346		private static let migrationMarkerKey = "data.migrations.v3_4_0"
	360	/// The schema version this build expects the on-device store to be at.
	361	static let currentStoreSchemaVersion = 1
	362
	363	/// UserDefaults key holding the store's current schema version.
	364	static let storeSchemaVersionKey = "data.storeSchemaVersion"
	365
	366	/// Pre-versioning installs recorded that the one-shot v1 normalization had
	367	/// run using this boolean marker; `true` means the store is already at v1.
	368	private static let legacyNormalizationMarkerKey = "data.migrations.v3_4_0"
	369
	370	/// The store's current schema version. Absent on pre-versioning installs: a
	371	/// set legacy marker means v1 already ran, otherwise the store is fresh or
	372	/// never-migrated at v0.
	373	static func storeSchemaVersion(defaults: UserDefaults = .standard) -> Int {
	374	if let version = defaults.object(forKey: storeSchemaVersionKey) as? Int {
	375	return version
	376	}
	377	return defaults.bool(forKey: legacyNormalizationMarkerKey) ? 1 : 0
	378	}
347	379
348	380	static func migrateIfNeeded(defaults: UserDefaults = .standard) {
349		guard !defaults.bool(forKey: migrationMarkerKey) else { return }
	381	// At or ahead of this build's version: nothing to do, and never rewrite
	382	// a store a newer build may have reshaped (forward compatibility).
	383	guard storeSchemaVersion(defaults: defaults) < currentStoreSchemaVersion else { return }
	384
	385	var version = storeSchemaVersion(defaults: defaults)
	386	while version < currentStoreSchemaVersion {
	387	let target = version + 1
	388	runMigration(to: target, defaults: defaults)
	389	defaults.set(target, forKey: storeSchemaVersionKey)
	390	version = target
	391	}
	392	}
	393
	394	private static func runMigration(to version: Int, defaults: UserDefaults) {
	395	switch version {
	396	case 1:
	397	normalizeAllStores(defaults: defaults)
	398	default:
	399	break
	400	}
	401	}
350	402
	403	/// v1: load every store through its deduplicating loader and write it back.
	404	/// This consolidates duplicate rows, drops the legacy `watchedDomains` key
	405	/// (via `saveTrackedDomains`), and sanitizes monitoring settings against the
	406	/// surviving tracked domains. Safe on an empty store (every step is a no-op).
	407	private static func normalizeAllStores(defaults: UserDefaults) {
351	408	let trackedDomains = DomainDataPortabilityService.loadTrackedDomains(defaults: defaults)
352	409	DomainDataPortabilityService.saveTrackedDomains(trackedDomains, defaults: defaults)
353	410
		@@ -366,8 +423,6 @@ enum DataMigrationService {
366	423
367	424	let monitoringLogs = DomainDataPortabilityService.loadMonitoringLogs(defaults: defaults)
368	425	DomainDataPortabilityService.saveMonitoringLogs(monitoringLogs, defaults: defaults)
369
370		defaults.set(true, forKey: migrationMarkerKey)
371	426	}
372	427	}
373	428

DomainDig.xcodeproj/project.pbxproj +4

		@@ -10,6 +10,7 @@
10	10	05F775C7E0743AF727B008EE /* LocalAPIContract.swift in Sources / = {isa = PBXBuildFile; fileRef = 7572DA0A838E0A044F045260 / LocalAPIContract.swift */; };
11	11	38316D90539394C2CC7C12BE /* Foundation.framework in Frameworks / = {isa = PBXBuildFile; fileRef = DE8B269A01CC5E593DA3DFC2 / Foundation.framework */; };
12	12	47CD3BB1AE143733A73E0E5B /* DomainReportExporterTests.swift in Sources / = {isa = PBXBuildFile; fileRef = 54D7C7D97A4006831572F468 / DomainReportExporterTests.swift */; };
	13	4F96CEB875EC3501E784CE39 /* DataMigrationServiceTests.swift in Sources / = {isa = PBXBuildFile; fileRef = 435AFB3F99D579D3D7B58279 / DataMigrationServiceTests.swift */; };
13	14	81359F63C7A23454B8FA0141 /* DomainReportBuilderTests.swift in Sources / = {isa = PBXBuildFile; fileRef = E82320955416797CFE59464A / DomainReportBuilderTests.swift */; };
14	15	8BBFEF092F9874AE00E8E144 /* DomainInspectionService.swift in Sources / = {isa = PBXBuildFile; fileRef = 8BBFEF032F9874AE00E8E144 / DomainInspectionService.swift */; };
15	16	8BBFEF0A2F9874AE00E8E144 /* DomainReportBuilder.swift in Sources / = {isa = PBXBuildFile; fileRef = 8BBFEF042F9874AE00E8E144 / DomainReportBuilder.swift */; };
		@@ -75,6 +76,7 @@
75	76	/* Begin PBXFileReference section */
76	77	0D85A44C5F315A1644AC9073 /* LocalAPIContractTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = LocalAPIContractTests.swift; sourceTree = "<group>"; };
77	78	15A25DF2B8BB52589D49986B /* DiffServiceTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DiffServiceTests.swift; sourceTree = "<group>"; };
	79	435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DataMigrationServiceTests.swift; sourceTree = "<group>"; };
78	80	54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = DomainReportExporterTests.swift; sourceTree = "<group>"; };
79	81	5CA789D3607B55E3612E6FFA /* SnapshotFixture.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = SnapshotFixture.swift; sourceTree = "<group>"; };
80	82	7572DA0A838E0A044F045260 /* LocalAPIContract.swift */ = {isa = PBXFileReference; includeInIndex = 1; lastKnownFileType = sourcecode.swift; path = LocalAPIContract.swift; sourceTree = "<group>"; };
		@@ -229,6 +231,7 @@
229	231	54D7C7D97A4006831572F468 /* DomainReportExporterTests.swift */,
230	232	CC948A02EC0184228BC4630E /* DomainDataPortabilityServiceTests.swift */,
231	233	0D85A44C5F315A1644AC9073 /* LocalAPIContractTests.swift */,
	234	435AFB3F99D579D3D7B58279 /* DataMigrationServiceTests.swift */,
232	235	);
233	236	name = DomainDigTests;
234	237	path = DomainDigTests;
		@@ -487,6 +490,7 @@
487	490	C7CA9E02B0DC2708DE7A8563 /* DomainDataPortabilityServiceTests.swift in Sources */,
488	491	A5AF921BC1C2E6E940CC05DC /* SnapshotFixture.swift in Sources */,
489	492	EA12012CDB4C21ABB4217D86 /* LocalAPIContractTests.swift in Sources */,
	493	4F96CEB875EC3501E784CE39 /* DataMigrationServiceTests.swift in Sources */,
490	494	);
491	495	runOnlyForDeploymentPostprocessing = 0;
492	496	};

DomainDigTests/DataMigrationServiceTests.swift added +109

		@@ -0,0 +1,109 @@
	1	import XCTest
	2	@testable import DomainDig
	3
	4	/// Characterization tests for the versioned store-migration runner. They drive
	5	/// `migrateIfNeeded` against legacy on-disk fixtures in an ephemeral
	6	/// `UserDefaults` suite and pin the policy: forward-only, idempotent, and
	7	/// non-destructive to a store written by a newer build.
	8	///
	9	/// The raw storage-key strings ("trackedDomains", "watchedDomains", the legacy
	10	/// marker) are duplicated here on purpose — they are the on-disk contract, and
	11	/// hard-coding them means an accidental rename shows up as a failing migration.
	12	final class DataMigrationServiceTests: XCTestCase {
	13	private let suiteName = "DomainDigTests.migration"
	14	private var defaults: UserDefaults!
	15	private let base = Date(timeIntervalSince1970: 1_700_000_000)
	16
	17	override func setUp() {
	18	super.setUp()
	19	defaults = UserDefaults(suiteName: suiteName)
	20	defaults.removePersistentDomain(forName: suiteName)
	21	}
	22
	23	override func tearDown() {
	24	defaults.removePersistentDomain(forName: suiteName)
	25	defaults = nil
	26	super.tearDown()
	27	}
	28
	29	func testFreshStoreIsStampedAtCurrentVersion() {
	30	XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 0)
	31
	32	DataMigrationService.migrateIfNeeded(defaults: defaults)
	33
	34	XCTAssertEqual(
	35	DataMigrationService.storeSchemaVersion(defaults: defaults),
	36	DataMigrationService.currentStoreSchemaVersion
	37	)
	38	}
	39
	40	func testLegacyWatchedDomainsAreMigratedAndTheOldKeyIsDropped() throws {
	41	let legacy = [WatchedDomain(domain: "legacy.example", createdAt: base, lastKnownAvailability: .registered)]
	42	defaults.set(try JSONEncoder().encode(legacy), forKey: "watchedDomains")
	43
	44	DataMigrationService.migrateIfNeeded(defaults: defaults)
	45
	46	let tracked = DomainDataPortabilityService.loadTrackedDomains(defaults: defaults)
	47	XCTAssertEqual(tracked.map(\.domain), ["legacy.example"])
	48	XCTAssertNil(defaults.data(forKey: "watchedDomains"), "legacy key is dropped after migration")
	49	XCTAssertNotNil(defaults.data(forKey: "trackedDomains"), "data is rewritten under the current key")
	50	XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 1)
	51	}
	52
	53	func testMigrationDeduplicatesTheStoredBlobInPlace() throws {
	54	let dupes = [
	55	TrackedDomain(domain: "dupe.example", updatedAt: base.addingTimeInterval(-10)),
	56	TrackedDomain(domain: "DUPE.example", updatedAt: base)
	57	]
	58	defaults.set(try JSONEncoder().encode(dupes), forKey: "trackedDomains")
	59
	60	DataMigrationService.migrateIfNeeded(defaults: defaults)
	61
	62	let data = try XCTUnwrap(defaults.data(forKey: "trackedDomains"))
	63	let stored = try JSONDecoder().decode([TrackedDomain].self, from: data)
	64	XCTAssertEqual(stored.count, 1, "the persisted blob is deduplicated, not just the load result")
	65	}
	66
	67	func testMigrationIsIdempotent() throws {
	68	defaults.set(
	69	try JSONEncoder().encode([TrackedDomain(domain: "a.example", updatedAt: base)]),
	70	forKey: "trackedDomains"
	71	)
	72
	73	DataMigrationService.migrateIfNeeded(defaults: defaults)
	74	let afterFirst = defaults.data(forKey: "trackedDomains")
	75
	76	DataMigrationService.migrateIfNeeded(defaults: defaults)
	77	let afterSecond = defaults.data(forKey: "trackedDomains")
	78
	79	XCTAssertEqual(afterFirst, afterSecond, "a second run makes no further changes")
	80	XCTAssertEqual(
	81	DataMigrationService.storeSchemaVersion(defaults: defaults),
	82	DataMigrationService.currentStoreSchemaVersion
	83	)
	84	}
	85
	86	func testLegacyBooleanMarkerCountsAsVersionOne() {
	87	defaults.set(true, forKey: "data.migrations.v3_4_0")
	88
	89	XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), 1)
	90
	91	// Already at v1, so the v1 step must not re-run: a leftover legacy blob
	92	// is left exactly as found.
	93	defaults.set(Data("x".utf8), forKey: "watchedDomains")
	94	DataMigrationService.migrateIfNeeded(defaults: defaults)
	95	XCTAssertNotNil(defaults.data(forKey: "watchedDomains"), "v1 is treated as already applied; no re-run")
	96	}
	97
	98	func testNewerStoreVersionIsNeverDowngradedOrRewritten() throws {
	99	let future = DataMigrationService.currentStoreSchemaVersion + 1
	100	defaults.set(future, forKey: DataMigrationService.storeSchemaVersionKey)
	101	let blob = try JSONEncoder().encode([TrackedDomain(domain: "keep.example", updatedAt: base)])
	102	defaults.set(blob, forKey: "trackedDomains")
	103
	104	DataMigrationService.migrateIfNeeded(defaults: defaults)
	105
	106	XCTAssertEqual(DataMigrationService.storeSchemaVersion(defaults: defaults), future, "must never downgrade")
	107	XCTAssertEqual(defaults.data(forKey: "trackedDomains"), blob, "future-version data is left byte-for-byte")
	108	}
	109	}

README.md +1 −1

		@@ -51,7 +51,7 @@ Network inspection requests are made only to perform the requested domain checks
51	51	xcodebuild -project DomainDig.xcodeproj -scheme DomainDig -destination 'platform=iOS Simulator,name=iPhone 16' build
52	52	```
53	53
54		The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`. The Local API's endpoints, response envelope, and `v1` compatibility policy are documented in [Docs/local-api.md](Docs/local-api.md).
	54	The app and local API share the canonical report pipeline through `DomainInspectionService`, `DomainReportBuilder`, and `DomainReportExporter`. The Local API's endpoints, response envelope, and `v1` compatibility policy are documented in [Docs/local-api.md](Docs/local-api.md). How the on-device store evolves across app versions is documented in [Docs/data-migration.md](Docs/data-migration.md).
55	55
56	56	### Accessibility Audit
57	57

Commit 99e4623af1

Commit `99e4623af1`