Commit `37a1b042db`

37a1b042dba0037a4bd6cecd3ab786d2be4c8ae9

parent: e24593c6e2

Unsigned ci/daily-pull: error

cmc <hello@cleberg.net> · 2026-08-31 22:41 UTC

Convert README.nfo to README.org (!1)

Section headings become org headings, indented command blocks become
src blocks, and column-aligned content stays in example blocks.

Layout: unified · split

README.nfo deleted −201

	@@ -1,201 +0,0 @@
1	┌──────────────────────────────────────────────────────────────┐
2	│ O M A H A M E T R O B L O T T E R [ KRZ ] krz.sh │
3	└──────────────────────────────────────────────────────────────┘
4
5	WHAT
6	an archive of police activity and alpr surveillance across the
7	omaha metro, pulled from the agencies' own feeds. sarpy county,
8	council bluffs and the flock portals all serve rolling windows and
9	delete what falls outside them; this keeps it.
10
11	COVERAGE
12	omaha pd dcgis arcgis view, 2022-01-01 onward,
13	nibrs offence records, updated daily.
14	no stop or disposition data.
15	bellevue pd sarpy county publiccrimemap, cad calls
16	papillion pd for service with stop type, disposition
17	la vista pd and category. rolling 12-month window:
18	sarpy county so records age out of the feed, so the local
19	archive is the only long-term copy. gretna
20	and springfield appear as fire only; their
21	police departments do not report to it.
22	council bluffs pd cbpd public cfs feed, refreshed every ten
23	minutes, rolling 12-month window. stop
24	type, disposition, priority, response time.
25	street addresses withheld; points exact.
26	ralston pd no machine-readable feed. absent.
27
28	alpr cameras openstreetmap via overpass, the same data
29	deflock renders. 169 nodes in the metro
30	bbox. odbl, attribution required.
31
32	alpr searches flock transparency portals. council bluffs
33	publishes a downloadable 30-day search
34	audit; sarpy county and douglas county
35	publish counts but no export. omaha pd has
36	no portal.
37
38	SETUP
39	uv venv .venv
40	uv pip install --python .venv/bin/python -r requirements.txt
41
42	USE
43	.venv/bin/python ingest.py --full # first run, backfill
44	.venv/bin/python ingest.py # daily, last 30 days
45	.venv/bin/python ingest.py cbpd sarpy # one source at a time
46	.venv/bin/python ingest.py opd_archive # omaha 2015-2021 backfill
47	.venv/bin/python app.py
48
49	SITE
50	build_site.py precomputes every figure into one self-contained
51	site/index.html -- no server, no fetch, no dependencies, 38 kb of
52	data. the workflow rebuilds it after each pull and deploys it to
53	github pages. app.py stays as the exploration tool; it needs a live
54	process and refilters 300k rows per interaction, which is fine for
55	one person and wrong for the public.
56
57	.venv/bin/python build_site.py && open site/index.html
58
59	the map's outlines come from raw_data/boundaries.geojson: douglas
60	county city limits and boundary, plus sarpy county municipal
61	boundaries. static reference geometry, so refreshing it is a rare
62	manual step rather than part of the daily pull.
63
64	.venv/bin/python fetch_boundaries.py
65
66	pottawattamie county, iowa publishes neither, so council bluffs is
67	unoutlined. douglas county sheriff publishes no incident or calls
68	feed at all -- only a flock portal with counts -- so the area
69	inside omaha's limits carries cameras and no stops.
70
71	ARCHIVE
72	.github/workflows/daily-pull.yml runs at 11:17 and 23:17 utc and
73	keeps the database as metro.db.gz on the "archive" release, so the
74	archive does not depend on any one machine. each run restores that
75	asset, pulls, refuses to publish if any source came back with fewer
76	rows than it started with, then uploads and fails loudly if a feed
77	has not moved in seven days.
78
79	sundays it sweeps every feed in full instead of the last 30 days,
80	because a 30-day window cannot see an agency amending a record it
81	filed months ago, and omaha does that.
82
83	first run: trigger it manually with bootstrap enabled, which pulls
84	every feed in full and creates the release. after that the restore
85	step is mandatory -- a bootstrap over a live archive throws away
86	whatever has already aged out of the sarpy and council bluffs
87	feeds.
88
89	github disables scheduled workflows after 60 days without repo
90	activity, and emails first. that is the most likely way this stops
91	quietly.
92
93	to run the pull locally instead:
94
95	0 6 * * * cd /path/to/omaha-metro-blotter && .venv/bin/python ingest.py
96
97	RAW
98	raw_records keeps the feed's own json for every version of every
99	record, keyed the same way amendments are. a parse that turns out
100	wrong, or a field a feed adds later, can only be applied to history
101	if the bytes were kept, and the rolling feeds mean there is no
102	second chance to fetch them. the payloads already carry fields
103	ingest does not map: council bluffs response times and priority,
104	sarpy case status.
105
106	it costs about 0.7 mb gzipped a day and roughly triples the
107	database: 20 mb published without it, 52 mb with. 319 records
108	predate it and their raw is gone; the feeds no longer serve them.
109
110	FLOCK SEARCH AUDIT
111	transparency.flocksafety.com/<slug> publishes camera counts, plate
112	reads, search counts and each agency's sharing network. council
113	bluffs also offers the search audit itself as a csv: one row per
114	search, with a timestamp, how many camera networks it reached and
115	a free-text reason.
116
117	cloudflare serves a challenge to every non-browser client, so
118	ingest.py cannot fetch it. collecting is manual: open the portal,
119	click "download csv", then
120
121	.venv/bin/python ingest.py --import-flock ~/Downloads/public_search_audit.csv
122
123	which checks the columns, files it under the right slug and loads
124	it. commit what it writes. every export is named
125	public_search_audit.csv with the agency nowhere inside, so pass
126	--agency for any portal other than council bluffs.
127
128	loading is not manual -- the flock source runs in the daily job and
129	picks up whatever is committed. search ids are stable uuids, so
130	overlapping exports dedupe and re-importing the same window is a
131	no-op.
132
133	the portals keep 30 days. miss a month and that month is gone, so
134	the workflow warns at 21 days since the last export and fails the
135	run at 27, while there is still time to act.
136
137	as of the first export, 100 of 442 council bluffs searches carried
138	any reason at all, against an access policy stating that all access
139	requires one. userid is redacted upstream, so no search can be
140	attributed to a person. the median search reached 466 camera
141	networks; the largest reached 6072.
142
143	all three portals list traffic enforcement under prohibited uses,
144	which is what the camera-proximity panel is measuring against.
145
146	AMENDMENTS
147	agencies edit records after publishing them: a disposition changes,
148	a case reopens, a record is withdrawn. nothing in the incidents
149	table is ever updated, so what an agency published first stays
150	readable. every later version the feed serves lands in
151	incident_amendments, and incidents_current is the newest version of
152	each record. analysis.changed_stop_outcomes() lists stops whose
153	disposition changed after filing.
154
155	a version is keyed on the hash of its payload, so a record that
156	reverts to a payload already on file is not recorded again. this
157	holds the set of distinct states observed, not a strict timeline.
158
159	the hash has to survive a round trip through sqlite. a lon of -96
160	arrives from the feed as a json int and comes back out of a REAL
161	column as -96.0, so lat, lon and is_stop are coerced before
162	hashing. get this wrong and every affected record is filed as
163	amended on every run, forever. the workflow fails if any amendment
164	is byte-identical to its original.
165
166	NOTES
167	all three arcgis services return utc epochs; their where-clause
168	literals do not agree (opd and council bluffs utc, sarpy central).
169	ingest.py stores occurred_at in local time.
170
171	each feed has its own taxonomy and none of them are comparable, so
172	ingest.py derives one cross-agency flag, is_stop, per source. stop
173	outcomes compare citation and arrest rate by substring, which is
174	all the two disposition vocabularies support: an agency that
175	records warnings less thoroughly shows a higher citation rate for
176	that reason alone.
177
178	colour scheme follows prefers-color-scheme. plotly writes colours
179	into the figure, so assets/theme.js reports the media query into a
180	store and app.py builds each figure from it. restyling after the
181	fact does not work: swapping a maplibre basemap at runtime leaves
182	it rebuilding with no data layers.
183
184	the camera-proximity panel compares stops against other calls from
185	the same agencies. the baseline has to be restricted that way: run
186	against the whole archive it shows stops 2.6x more likely to be
187	within 200m of a camera, but most of the archive is omaha, which
188	reports no stops, so that number measures geography rather than
189	enforcement. like for like it is 1.19x, and median distance is
190	805m for stops against 780m for everything else -- no meaningful
191	separation.
192
193	raw_data/ingress.db is the old 2015-2023 sqlite build. nothing
194	reads it any more.
195
196	SCREENSHOTS
197	screenshots/*.png are from the previous 2015-2023 dashboard.
198
199	┌──────────────────────────────────────────────────────────────┐
200	│ krz.sh │
201	└──────────────────────────────────────────────────────────────┘

README.org added +213

@@ -0,0 +1,213 @@
	1	#+title: omaha metro blotter
	2
	3	* what
	4	an archive of police activity and alpr surveillance across the
	5	omaha metro, pulled from the agencies' own feeds. sarpy county,
	6	council bluffs and the flock portals all serve rolling windows and
	7	delete what falls outside them; this keeps it.
	8
	9	* coverage
	10	#+begin_example
	11	omaha pd dcgis arcgis view, 2022-01-01 onward,
	12	nibrs offence records, updated daily.
	13	no stop or disposition data.
	14	bellevue pd sarpy county publiccrimemap, cad calls
	15	papillion pd for service with stop type, disposition
	16	la vista pd and category. rolling 12-month window:
	17	sarpy county so records age out of the feed, so the local
	18	archive is the only long-term copy. gretna
	19	and springfield appear as fire only; their
	20	police departments do not report to it.
	21	council bluffs pd cbpd public cfs feed, refreshed every ten
	22	minutes, rolling 12-month window. stop
	23	type, disposition, priority, response time.
	24	street addresses withheld; points exact.
	25	ralston pd no machine-readable feed. absent.
	26	#+end_example
	27
	28	#+begin_example
	29	alpr cameras openstreetmap via overpass, the same data
	30	deflock renders. 169 nodes in the metro
	31	bbox. odbl, attribution required.
	32	#+end_example
	33
	34	#+begin_example
	35	alpr searches flock transparency portals. council bluffs
	36	publishes a downloadable 30-day search
	37	audit; sarpy county and douglas county
	38	publish counts but no export. omaha pd has
	39	no portal.
	40	#+end_example
	41
	42	* setup
	43	#+begin_src sh
	44	uv venv .venv
	45	uv pip install --python .venv/bin/python -r requirements.txt
	46	#+end_src
	47
	48	* use
	49	#+begin_src sh
	50	.venv/bin/python ingest.py --full # first run, backfill
	51	.venv/bin/python ingest.py # daily, last 30 days
	52	.venv/bin/python ingest.py cbpd sarpy # one source at a time
	53	.venv/bin/python ingest.py opd_archive # omaha 2015-2021 backfill
	54	.venv/bin/python app.py
	55	#+end_src
	56
	57	* site
	58	build_site.py precomputes every figure into one self-contained
	59	site/index.html -- no server, no fetch, no dependencies, 38 kb of
	60	data. the workflow rebuilds it after each pull and deploys it to
	61	github pages. app.py stays as the exploration tool; it needs a live
	62	process and refilters 300k rows per interaction, which is fine for
	63	one person and wrong for the public.
	64
	65	#+begin_src sh
	66	.venv/bin/python build_site.py && open site/index.html
	67	#+end_src
	68
	69	the map's outlines come from raw_data/boundaries.geojson: douglas
	70	county city limits and boundary, plus sarpy county municipal
	71	boundaries. static reference geometry, so refreshing it is a rare
	72	manual step rather than part of the daily pull.
	73
	74	#+begin_src sh
	75	.venv/bin/python fetch_boundaries.py
	76	#+end_src
	77
	78	pottawattamie county, iowa publishes neither, so council bluffs is
	79	unoutlined. douglas county sheriff publishes no incident or calls
	80	feed at all -- only a flock portal with counts -- so the area
	81	inside omaha's limits carries cameras and no stops.
	82
	83	* archive
	84	.github/workflows/daily-pull.yml runs at 11:17 and 23:17 utc and
	85	keeps the database as metro.db.gz on the "archive" release, so the
	86	archive does not depend on any one machine. each run restores that
	87	asset, pulls, refuses to publish if any source came back with fewer
	88	rows than it started with, then uploads and fails loudly if a feed
	89	has not moved in seven days.
	90
	91	sundays it sweeps every feed in full instead of the last 30 days,
	92	because a 30-day window cannot see an agency amending a record it
	93	filed months ago, and omaha does that.
	94
	95	first run: trigger it manually with bootstrap enabled, which pulls
	96	every feed in full and creates the release. after that the restore
	97	step is mandatory -- a bootstrap over a live archive throws away
	98	whatever has already aged out of the sarpy and council bluffs
	99	feeds.
	100
	101	github disables scheduled workflows after 60 days without repo
	102	activity, and emails first. that is the most likely way this stops
	103	quietly.
	104
	105	to run the pull locally instead:
	106
	107	#+begin_src sh
	108	0 6 * * * cd /path/to/omaha-metro-blotter && .venv/bin/python ingest.py
	109	#+end_src
	110
	111	* raw
	112	raw_records keeps the feed's own json for every version of every
	113	record, keyed the same way amendments are. a parse that turns out
	114	wrong, or a field a feed adds later, can only be applied to history
	115	if the bytes were kept, and the rolling feeds mean there is no
	116	second chance to fetch them. the payloads already carry fields
	117	ingest does not map: council bluffs response times and priority,
	118	sarpy case status.
	119
	120	it costs about 0.7 mb gzipped a day and roughly triples the
	121	database: 20 mb published without it, 52 mb with. 319 records
	122	predate it and their raw is gone; the feeds no longer serve them.
	123
	124	* flock search audit
	125	transparency.flocksafety.com/<slug> publishes camera counts, plate
	126	reads, search counts and each agency's sharing network. council
	127	bluffs also offers the search audit itself as a csv: one row per
	128	search, with a timestamp, how many camera networks it reached and
	129	a free-text reason.
	130
	131	cloudflare serves a challenge to every non-browser client, so
	132	ingest.py cannot fetch it. collecting is manual: open the portal,
	133	click "download csv", then
	134
	135	#+begin_src sh
	136	.venv/bin/python ingest.py --import-flock ~/Downloads/public_search_audit.csv
	137	#+end_src
	138
	139	which checks the columns, files it under the right slug and loads
	140	it. commit what it writes. every export is named
	141	public_search_audit.csv with the agency nowhere inside, so pass
	142	--agency for any portal other than council bluffs.
	143
	144	loading is not manual -- the flock source runs in the daily job and
	145	picks up whatever is committed. search ids are stable uuids, so
	146	overlapping exports dedupe and re-importing the same window is a
	147	no-op.
	148
	149	the portals keep 30 days. miss a month and that month is gone, so
	150	the workflow warns at 21 days since the last export and fails the
	151	run at 27, while there is still time to act.
	152
	153	as of the first export, 100 of 442 council bluffs searches carried
	154	any reason at all, against an access policy stating that all access
	155	requires one. userid is redacted upstream, so no search can be
	156	attributed to a person. the median search reached 466 camera
	157	networks; the largest reached 6072.
	158
	159	all three portals list traffic enforcement under prohibited uses,
	160	which is what the camera-proximity panel is measuring against.
	161
	162	* amendments
	163	agencies edit records after publishing them: a disposition changes,
	164	a case reopens, a record is withdrawn. nothing in the incidents
	165	table is ever updated, so what an agency published first stays
	166	readable. every later version the feed serves lands in
	167	incident_amendments, and incidents_current is the newest version of
	168	each record. analysis.changed_stop_outcomes() lists stops whose
	169	disposition changed after filing.
	170
	171	a version is keyed on the hash of its payload, so a record that
	172	reverts to a payload already on file is not recorded again. this
	173	holds the set of distinct states observed, not a strict timeline.
	174
	175	the hash has to survive a round trip through sqlite. a lon of -96
	176	arrives from the feed as a json int and comes back out of a REAL
	177	column as -96.0, so lat, lon and is_stop are coerced before
	178	hashing. get this wrong and every affected record is filed as
	179	amended on every run, forever. the workflow fails if any amendment
	180	is byte-identical to its original.
	181
	182	* notes
	183	all three arcgis services return utc epochs; their where-clause
	184	literals do not agree (opd and council bluffs utc, sarpy central).
	185	ingest.py stores occurred_at in local time.
	186
	187	each feed has its own taxonomy and none of them are comparable, so
	188	ingest.py derives one cross-agency flag, is_stop, per source. stop
	189	outcomes compare citation and arrest rate by substring, which is
	190	all the two disposition vocabularies support: an agency that
	191	records warnings less thoroughly shows a higher citation rate for
	192	that reason alone.
	193
	194	colour scheme follows prefers-color-scheme. plotly writes colours
	195	into the figure, so assets/theme.js reports the media query into a
	196	store and app.py builds each figure from it. restyling after the
	197	fact does not work: swapping a maplibre basemap at runtime leaves
	198	it rebuilding with no data layers.
	199
	200	the camera-proximity panel compares stops against other calls from
	201	the same agencies. the baseline has to be restricted that way: run
	202	against the whole archive it shows stops 2.6x more likely to be
	203	within 200m of a camera, but most of the archive is omaha, which
	204	reports no stops, so that number measures geography rather than
	205	enforcement. like for like it is 1.19x, and median distance is
	206	805m for stops against 780m for everything else -- no meaningful
	207	separation.
	208
	209	raw_data/ingress.db is the old 2015-2023 sqlite build. nothing
	210	reads it any more.
	211
	212	* screenshots
	213	screenshots/*.png are from the previous 2015-2023 dashboard.

Commit 37a1b042db

Commit `37a1b042db`