Commit `cd892b8d63`

cd892b8d6390321c8cc880452433f7342a3a51f4

parent: 23df422b45

Verified · cmc

cmc <hello@cleberg.net> · 2026-08-31 22:21 UTC

Convert README.nfo to README.org

Layout: unified · split

README.nfo deleted −201

		@@ -1,201 +0,0 @@
1		┌──────────────────────────────────────────────────────────────┐
2		│ O M A H A M E T R O B L O T T E R [ KRZ ] krz.sh │
3		└──────────────────────────────────────────────────────────────┘
4
5		WHAT
6		an archive of police activity and alpr surveillance across the
7		omaha metro, pulled from the agencies' own feeds. sarpy county,
8		council bluffs and the flock portals all serve rolling windows and
9		delete what falls outside them; this keeps it.
10
11		COVERAGE
12		omaha pd dcgis arcgis view, 2022-01-01 onward,
13		nibrs offence records, updated daily.
14		no stop or disposition data.
15		bellevue pd sarpy county publiccrimemap, cad calls
16		papillion pd for service with stop type, disposition
17		la vista pd and category. rolling 12-month window:
18		sarpy county so records age out of the feed, so the local
19		archive is the only long-term copy. gretna
20		and springfield appear as fire only; their
21		police departments do not report to it.
22		council bluffs pd cbpd public cfs feed, refreshed every ten
23		minutes, rolling 12-month window. stop
24		type, disposition, priority, response time.
25		street addresses withheld; points exact.
26		ralston pd no machine-readable feed. absent.
27
28		alpr cameras openstreetmap via overpass, the same data
29		deflock renders. 169 nodes in the metro
30		bbox. odbl, attribution required.
31
32		alpr searches flock transparency portals. council bluffs
33		publishes a downloadable 30-day search
34		audit; sarpy county and douglas county
35		publish counts but no export. omaha pd has
36		no portal.
37
38		SETUP
39		uv venv .venv
40		uv pip install --python .venv/bin/python -r requirements.txt
41
42		USE
43		.venv/bin/python ingest.py --full # first run, backfill
44		.venv/bin/python ingest.py # daily, last 30 days
45		.venv/bin/python ingest.py cbpd sarpy # one source at a time
46		.venv/bin/python ingest.py opd_archive # omaha 2015-2021 backfill
47		.venv/bin/python app.py
48
49		SITE
50		build_site.py precomputes every figure into one self-contained
51		site/index.html -- no server, no fetch, no dependencies, 38 kb of
52		data. the workflow rebuilds it after each pull and deploys it to
53		github pages. app.py stays as the exploration tool; it needs a live
54		process and refilters 300k rows per interaction, which is fine for
55		one person and wrong for the public.
56
57		.venv/bin/python build_site.py && open site/index.html
58
59		the map's outlines come from raw_data/boundaries.geojson: douglas
60		county city limits and boundary, plus sarpy county municipal
61		boundaries. static reference geometry, so refreshing it is a rare
62		manual step rather than part of the daily pull.
63
64		.venv/bin/python fetch_boundaries.py
65
66		pottawattamie county, iowa publishes neither, so council bluffs is
67		unoutlined. douglas county sheriff publishes no incident or calls
68		feed at all -- only a flock portal with counts -- so the area
69		inside omaha's limits carries cameras and no stops.
70
71		ARCHIVE
72		.github/workflows/daily-pull.yml runs at 11:17 and 23:17 utc and
73		keeps the database as metro.db.gz on the "archive" release, so the
74		archive does not depend on any one machine. each run restores that
75		asset, pulls, refuses to publish if any source came back with fewer
76		rows than it started with, then uploads and fails loudly if a feed
77		has not moved in seven days.
78
79		sundays it sweeps every feed in full instead of the last 30 days,
80		because a 30-day window cannot see an agency amending a record it
81		filed months ago, and omaha does that.
82
83		first run: trigger it manually with bootstrap enabled, which pulls
84		every feed in full and creates the release. after that the restore
85		step is mandatory -- a bootstrap over a live archive throws away
86		whatever has already aged out of the sarpy and council bluffs
87		feeds.
88
89		github disables scheduled workflows after 60 days without repo
90		activity, and emails first. that is the most likely way this stops
91		quietly.
92
93		to run the pull locally instead:
94
95		0 6 * * * cd /path/to/omaha-metro-blotter && .venv/bin/python ingest.py
96
97		RAW
98		raw_records keeps the feed's own json for every version of every
99		record, keyed the same way amendments are. a parse that turns out
100		wrong, or a field a feed adds later, can only be applied to history
101		if the bytes were kept, and the rolling feeds mean there is no
102		second chance to fetch them. the payloads already carry fields
103		ingest does not map: council bluffs response times and priority,
104		sarpy case status.
105
106		it costs about 0.7 mb gzipped a day and roughly triples the
107		database: 20 mb published without it, 52 mb with. 319 records
108		predate it and their raw is gone; the feeds no longer serve them.
109
110		FLOCK SEARCH AUDIT
111		transparency.flocksafety.com/<slug> publishes camera counts, plate
112		reads, search counts and each agency's sharing network. council
113		bluffs also offers the search audit itself as a csv: one row per
114		search, with a timestamp, how many camera networks it reached and
115		a free-text reason.
116
117		cloudflare serves a challenge to every non-browser client, so
118		ingest.py cannot fetch it. collecting is manual: open the portal,
119		click "download csv", then
120
121		.venv/bin/python ingest.py --import-flock ~/Downloads/public_search_audit.csv
122
123		which checks the columns, files it under the right slug and loads
124		it. commit what it writes. every export is named
125		public_search_audit.csv with the agency nowhere inside, so pass
126		--agency for any portal other than council bluffs.
127
128		loading is not manual -- the flock source runs in the daily job and
129		picks up whatever is committed. search ids are stable uuids, so
130		overlapping exports dedupe and re-importing the same window is a
131		no-op.
132
133		the portals keep 30 days. miss a month and that month is gone, so
134		the workflow warns at 21 days since the last export and fails the
135		run at 27, while there is still time to act.
136
137		as of the first export, 100 of 442 council bluffs searches carried
138		any reason at all, against an access policy stating that all access
139		requires one. userid is redacted upstream, so no search can be
140		attributed to a person. the median search reached 466 camera
141		networks; the largest reached 6072.
142
143		all three portals list traffic enforcement under prohibited uses,
144		which is what the camera-proximity panel is measuring against.
145
146		AMENDMENTS
147		agencies edit records after publishing them: a disposition changes,
148		a case reopens, a record is withdrawn. nothing in the incidents
149		table is ever updated, so what an agency published first stays
150		readable. every later version the feed serves lands in
151		incident_amendments, and incidents_current is the newest version of
152		each record. analysis.changed_stop_outcomes() lists stops whose
153		disposition changed after filing.
154
155		a version is keyed on the hash of its payload, so a record that
156		reverts to a payload already on file is not recorded again. this
157		holds the set of distinct states observed, not a strict timeline.
158
159		the hash has to survive a round trip through sqlite. a lon of -96
160		arrives from the feed as a json int and comes back out of a REAL
161		column as -96.0, so lat, lon and is_stop are coerced before
162		hashing. get this wrong and every affected record is filed as
163		amended on every run, forever. the workflow fails if any amendment
164		is byte-identical to its original.
165
166		NOTES
167		all three arcgis services return utc epochs; their where-clause
168		literals do not agree (opd and council bluffs utc, sarpy central).
169		ingest.py stores occurred_at in local time.
170
171		each feed has its own taxonomy and none of them are comparable, so
172		ingest.py derives one cross-agency flag, is_stop, per source. stop
173		outcomes compare citation and arrest rate by substring, which is
174		all the two disposition vocabularies support: an agency that
175		records warnings less thoroughly shows a higher citation rate for
176		that reason alone.
177
178		colour scheme follows prefers-color-scheme. plotly writes colours
179		into the figure, so assets/theme.js reports the media query into a
180		store and app.py builds each figure from it. restyling after the
181		fact does not work: swapping a maplibre basemap at runtime leaves
182		it rebuilding with no data layers.
183
184		the camera-proximity panel compares stops against other calls from
185		the same agencies. the baseline has to be restricted that way: run
186		against the whole archive it shows stops 2.6x more likely to be
187		within 200m of a camera, but most of the archive is omaha, which
188		reports no stops, so that number measures geography rather than
189		enforcement. like for like it is 1.19x, and median distance is
190		805m for stops against 780m for everything else -- no meaningful
191		separation.
192
193		raw_data/ingress.db is the old 2015-2023 sqlite build. nothing
194		reads it any more.
195
196		SCREENSHOTS
197		screenshots/*.png are from the previous 2015-2023 dashboard.
198
199		┌──────────────────────────────────────────────────────────────┐
200		│ krz.sh │
201		└──────────────────────────────────────────────────────────────┘

README.org added +213

		@@ -0,0 +1,213 @@
	1	#+title: omaha metro blotter
	2
	3	* what
	4	an archive of police activity and alpr surveillance across the
	5	omaha metro, pulled from the agencies' own feeds. sarpy county,
	6	council bluffs and the flock portals all serve rolling windows and
	7	delete what falls outside them; this keeps it.
	8
	9	* coverage
	10	#+begin_example
	11	omaha pd dcgis arcgis view, 2022-01-01 onward,
	12	nibrs offence records, updated daily.
	13	no stop or disposition data.
	14	bellevue pd sarpy county publiccrimemap, cad calls
	15	papillion pd for service with stop type, disposition
	16	la vista pd and category. rolling 12-month window:
	17	sarpy county so records age out of the feed, so the local
	18	archive is the only long-term copy. gretna
	19	and springfield appear as fire only; their
	20	police departments do not report to it.
	21	council bluffs pd cbpd public cfs feed, refreshed every ten
	22	minutes, rolling 12-month window. stop
	23	type, disposition, priority, response time.
	24	street addresses withheld; points exact.
	25	ralston pd no machine-readable feed. absent.
	26	#+end_example
	27
	28	#+begin_example
	29	alpr cameras openstreetmap via overpass, the same data
	30	deflock renders. 169 nodes in the metro
	31	bbox. odbl, attribution required.
	32	#+end_example
	33
	34	#+begin_example
	35	alpr searches flock transparency portals. council bluffs
	36	publishes a downloadable 30-day search
	37	audit; sarpy county and douglas county
	38	publish counts but no export. omaha pd has
	39	no portal.
	40	#+end_example
	41
	42	* setup
	43	#+begin_src sh
	44	uv venv .venv
	45	uv pip install --python .venv/bin/python -r requirements.txt
	46	#+end_src
	47
	48	* use
	49	#+begin_src sh
	50	.venv/bin/python ingest.py --full # first run, backfill
	51	.venv/bin/python ingest.py # daily, last 30 days
	52	.venv/bin/python ingest.py cbpd sarpy # one source at a time
	53	.venv/bin/python ingest.py opd_archive # omaha 2015-2021 backfill
	54	.venv/bin/python app.py
	55	#+end_src
	56
	57	* site
	58	build_site.py precomputes every figure into one self-contained
	59	site/index.html -- no server, no fetch, no dependencies, 38 kb of
	60	data. the workflow rebuilds it after each pull and deploys it to
	61	github pages. app.py stays as the exploration tool; it needs a live
	62	process and refilters 300k rows per interaction, which is fine for
	63	one person and wrong for the public.
	64
	65	#+begin_src sh
	66	.venv/bin/python build_site.py && open site/index.html
	67	#+end_src
	68
	69	the map's outlines come from raw_data/boundaries.geojson: douglas
	70	county city limits and boundary, plus sarpy county municipal
	71	boundaries. static reference geometry, so refreshing it is a rare
	72	manual step rather than part of the daily pull.
	73
	74	#+begin_src sh
	75	.venv/bin/python fetch_boundaries.py
	76	#+end_src
	77
	78	pottawattamie county, iowa publishes neither, so council bluffs is
	79	unoutlined. douglas county sheriff publishes no incident or calls
	80	feed at all -- only a flock portal with counts -- so the area
	81	inside omaha's limits carries cameras and no stops.
	82
	83	* archive
	84	.github/workflows/daily-pull.yml runs at 11:17 and 23:17 utc and
	85	keeps the database as metro.db.gz on the "archive" release, so the
	86	archive does not depend on any one machine. each run restores that
	87	asset, pulls, refuses to publish if any source came back with fewer
	88	rows than it started with, then uploads and fails loudly if a feed
	89	has not moved in seven days.
	90
	91	sundays it sweeps every feed in full instead of the last 30 days,
	92	because a 30-day window cannot see an agency amending a record it
	93	filed months ago, and omaha does that.
	94
	95	first run: trigger it manually with bootstrap enabled, which pulls
	96	every feed in full and creates the release. after that the restore
	97	step is mandatory -- a bootstrap over a live archive throws away
	98	whatever has already aged out of the sarpy and council bluffs
	99	feeds.
	100
	101	github disables scheduled workflows after 60 days without repo
	102	activity, and emails first. that is the most likely way this stops
	103	quietly.
	104
	105	to run the pull locally instead:
	106
	107	#+begin_src sh
	108	0 6 * * * cd /path/to/omaha-metro-blotter && .venv/bin/python ingest.py
	109	#+end_src
	110
	111	* raw
	112	raw_records keeps the feed's own json for every version of every
	113	record, keyed the same way amendments are. a parse that turns out
	114	wrong, or a field a feed adds later, can only be applied to history
	115	if the bytes were kept, and the rolling feeds mean there is no
	116	second chance to fetch them. the payloads already carry fields
	117	ingest does not map: council bluffs response times and priority,
	118	sarpy case status.
	119
	120	it costs about 0.7 mb gzipped a day and roughly triples the
	121	database: 20 mb published without it, 52 mb with. 319 records
	122	predate it and their raw is gone; the feeds no longer serve them.
	123
	124	* flock search audit
	125	transparency.flocksafety.com/<slug> publishes camera counts, plate
	126	reads, search counts and each agency's sharing network. council
	127	bluffs also offers the search audit itself as a csv: one row per
	128	search, with a timestamp, how many camera networks it reached and
	129	a free-text reason.
	130
	131	cloudflare serves a challenge to every non-browser client, so
	132	ingest.py cannot fetch it. collecting is manual: open the portal,
	133	click "download csv", then
	134
	135	#+begin_src sh
	136	.venv/bin/python ingest.py --import-flock ~/Downloads/public_search_audit.csv
	137	#+end_src
	138
	139	which checks the columns, files it under the right slug and loads
	140	it. commit what it writes. every export is named
	141	public_search_audit.csv with the agency nowhere inside, so pass
	142	--agency for any portal other than council bluffs.
	143
	144	loading is not manual -- the flock source runs in the daily job and
	145	picks up whatever is committed. search ids are stable uuids, so
	146	overlapping exports dedupe and re-importing the same window is a
	147	no-op.
	148
	149	the portals keep 30 days. miss a month and that month is gone, so
	150	the workflow warns at 21 days since the last export and fails the
	151	run at 27, while there is still time to act.
	152
	153	as of the first export, 100 of 442 council bluffs searches carried
	154	any reason at all, against an access policy stating that all access
	155	requires one. userid is redacted upstream, so no search can be
	156	attributed to a person. the median search reached 466 camera
	157	networks; the largest reached 6072.
	158
	159	all three portals list traffic enforcement under prohibited uses,
	160	which is what the camera-proximity panel is measuring against.
	161
	162	* amendments
	163	agencies edit records after publishing them: a disposition changes,
	164	a case reopens, a record is withdrawn. nothing in the incidents
	165	table is ever updated, so what an agency published first stays
	166	readable. every later version the feed serves lands in
	167	incident_amendments, and incidents_current is the newest version of
	168	each record. analysis.changed_stop_outcomes() lists stops whose
	169	disposition changed after filing.
	170
	171	a version is keyed on the hash of its payload, so a record that
	172	reverts to a payload already on file is not recorded again. this
	173	holds the set of distinct states observed, not a strict timeline.
	174
	175	the hash has to survive a round trip through sqlite. a lon of -96
	176	arrives from the feed as a json int and comes back out of a REAL
	177	column as -96.0, so lat, lon and is_stop are coerced before
	178	hashing. get this wrong and every affected record is filed as
	179	amended on every run, forever. the workflow fails if any amendment
	180	is byte-identical to its original.
	181
	182	* notes
	183	all three arcgis services return utc epochs; their where-clause
	184	literals do not agree (opd and council bluffs utc, sarpy central).
	185	ingest.py stores occurred_at in local time.
	186
	187	each feed has its own taxonomy and none of them are comparable, so
	188	ingest.py derives one cross-agency flag, is_stop, per source. stop
	189	outcomes compare citation and arrest rate by substring, which is
	190	all the two disposition vocabularies support: an agency that
	191	records warnings less thoroughly shows a higher citation rate for
	192	that reason alone.
	193
	194	colour scheme follows prefers-color-scheme. plotly writes colours
	195	into the figure, so assets/theme.js reports the media query into a
	196	store and app.py builds each figure from it. restyling after the
	197	fact does not work: swapping a maplibre basemap at runtime leaves
	198	it rebuilding with no data layers.
	199
	200	the camera-proximity panel compares stops against other calls from
	201	the same agencies. the baseline has to be restricted that way: run
	202	against the whole archive it shows stops 2.6x more likely to be
	203	within 200m of a camera, but most of the archive is omaha, which
	204	reports no stops, so that number measures geography rather than
	205	enforcement. like for like it is 1.19x, and median distance is
	206	805m for stops against 780m for everything else -- no meaningful
	207	separation.
	208
	209	raw_data/ingress.db is the old 2015-2023 sqlite build. nothing
	210	reads it any more.
	211
	212	* screenshots
	213	screenshots/*.png are from the previous 2015-2023 dashboard.

Commit cd892b8d63

Commit `cd892b8d63`