README.org

  1#+title: omaha metro blotter
  2
  3* what
  4an archive of police activity and alpr surveillance across the
  5omaha metro, pulled from the agencies' own feeds. sarpy county,
  6council bluffs and the flock portals all serve rolling windows and
  7delete what falls outside them; this keeps it.
  8
  9* coverage
 10#+begin_example
 11omaha pd                dcgis arcgis view, 2022-01-01 onward,
 12                        nibrs offence records, updated daily.
 13                        no stop or disposition data.
 14bellevue pd             sarpy county publiccrimemap, cad calls
 15papillion pd            for service with stop type, disposition
 16la vista pd             and category. rolling 12-month window:
 17sarpy county so         records age out of the feed, so the local
 18                        archive is the only long-term copy. gretna
 19                        and springfield appear as fire only; their
 20                        police departments do not report to it.
 21council bluffs pd       cbpd public cfs feed, refreshed every ten
 22                        minutes, rolling 12-month window. stop
 23                        type, disposition, priority, response time.
 24                        street addresses withheld; points exact.
 25ralston pd              no machine-readable feed. absent.
 26#+end_example
 27
 28#+begin_example
 29alpr cameras            openstreetmap via overpass, the same data
 30                        deflock renders. 169 nodes in the metro
 31                        bbox. odbl, attribution required.
 32#+end_example
 33
 34#+begin_example
 35alpr searches           flock transparency portals. council bluffs
 36                        publishes a downloadable 30-day search
 37                        audit; sarpy county and douglas county
 38                        publish counts but no export. omaha pd has
 39                        no portal.
 40#+end_example
 41
 42* setup
 43#+begin_src sh
 44uv venv .venv
 45uv pip install --python .venv/bin/python -r requirements.txt
 46#+end_src
 47
 48* use
 49#+begin_src sh
 50.venv/bin/python ingest.py --full        # first run, backfill
 51.venv/bin/python ingest.py               # daily, last 30 days
 52.venv/bin/python ingest.py cbpd sarpy    # one source at a time
 53.venv/bin/python ingest.py opd_archive   # omaha 2015-2021 backfill
 54.venv/bin/python app.py
 55#+end_src
 56
 57* site
 58build_site.py precomputes every figure into one self-contained
 59site/index.html -- no server, no fetch, no dependencies, 38 kb of
 60data. the workflow rebuilds it after each pull and deploys it to
 61github pages. app.py stays as the exploration tool; it needs a live
 62process and refilters 300k rows per interaction, which is fine for
 63one person and wrong for the public.
 64
 65#+begin_src sh
 66.venv/bin/python build_site.py && open site/index.html
 67#+end_src
 68
 69the map's outlines come from raw_data/boundaries.geojson: douglas
 70county city limits and boundary, plus sarpy county municipal
 71boundaries. static reference geometry, so refreshing it is a rare
 72manual step rather than part of the daily pull.
 73
 74#+begin_src sh
 75.venv/bin/python fetch_boundaries.py
 76#+end_src
 77
 78pottawattamie county, iowa publishes neither, so council bluffs is
 79unoutlined. douglas county sheriff publishes no incident or calls
 80feed at all -- only a flock portal with counts -- so the area
 81inside omaha's limits carries cameras and no stops.
 82
 83* archive
 84.github/workflows/daily-pull.yml runs at 11:17 and 23:17 utc and
 85keeps the database as metro.db.gz on the "archive" release, so the
 86archive does not depend on any one machine. each run restores that
 87asset, pulls, refuses to publish if any source came back with fewer
 88rows than it started with, then uploads and fails loudly if a feed
 89has not moved in seven days.
 90
 91sundays it sweeps every feed in full instead of the last 30 days,
 92because a 30-day window cannot see an agency amending a record it
 93filed months ago, and omaha does that.
 94
 95first run: trigger it manually with bootstrap enabled, which pulls
 96every feed in full and creates the release. after that the restore
 97step is mandatory -- a bootstrap over a live archive throws away
 98whatever has already aged out of the sarpy and council bluffs
 99feeds.
100
101github disables scheduled workflows after 60 days without repo
102activity, and emails first. that is the most likely way this stops
103quietly.
104
105to run the pull locally instead:
106
107#+begin_src sh
1080 6 * * *  cd /path/to/omaha-metro-blotter && .venv/bin/python ingest.py
109#+end_src
110
111* raw
112raw_records keeps the feed's own json for every version of every
113record, keyed the same way amendments are. a parse that turns out
114wrong, or a field a feed adds later, can only be applied to history
115if the bytes were kept, and the rolling feeds mean there is no
116second chance to fetch them. the payloads already carry fields
117ingest does not map: council bluffs response times and priority,
118sarpy case status.
119
120it costs about 0.7 mb gzipped a day and roughly triples the
121database: 20 mb published without it, 52 mb with. 319 records
122predate it and their raw is gone; the feeds no longer serve them.
123
124* flock search audit
125transparency.flocksafety.com/<slug> publishes camera counts, plate
126reads, search counts and each agency's sharing network. council
127bluffs also offers the search audit itself as a csv: one row per
128search, with a timestamp, how many camera networks it reached and
129a free-text reason.
130
131cloudflare serves a challenge to every non-browser client, so
132ingest.py cannot fetch it. collecting is manual: open the portal,
133click "download csv", then
134
135#+begin_src sh
136.venv/bin/python ingest.py --import-flock ~/Downloads/public_search_audit.csv
137#+end_src
138
139which checks the columns, files it under the right slug and loads
140it. commit what it writes. every export is named
141public_search_audit.csv with the agency nowhere inside, so pass
142--agency for any portal other than council bluffs.
143
144loading is not manual -- the flock source runs in the daily job and
145picks up whatever is committed. search ids are stable uuids, so
146overlapping exports dedupe and re-importing the same window is a
147no-op.
148
149the portals keep 30 days. miss a month and that month is gone, so
150the workflow warns at 21 days since the last export and fails the
151run at 27, while there is still time to act.
152
153as of the first export, 100 of 442 council bluffs searches carried
154any reason at all, against an access policy stating that all access
155requires one. userid is redacted upstream, so no search can be
156attributed to a person. the median search reached 466 camera
157networks; the largest reached 6072.
158
159all three portals list traffic enforcement under prohibited uses,
160which is what the camera-proximity panel is measuring against.
161
162* amendments
163agencies edit records after publishing them: a disposition changes,
164a case reopens, a record is withdrawn. nothing in the incidents
165table is ever updated, so what an agency published first stays
166readable. every later version the feed serves lands in
167incident_amendments, and incidents_current is the newest version of
168each record. analysis.changed_stop_outcomes() lists stops whose
169disposition changed after filing.
170
171a version is keyed on the hash of its payload, so a record that
172reverts to a payload already on file is not recorded again. this
173holds the set of distinct states observed, not a strict timeline.
174
175the hash has to survive a round trip through sqlite. a lon of -96
176arrives from the feed as a json int and comes back out of a REAL
177column as -96.0, so lat, lon and is_stop are coerced before
178hashing. get this wrong and every affected record is filed as
179amended on every run, forever. the workflow fails if any amendment
180is byte-identical to its original.
181
182* notes
183all three arcgis services return utc epochs; their where-clause
184literals do not agree (opd and council bluffs utc, sarpy central).
185ingest.py stores occurred_at in local time.
186
187each feed has its own taxonomy and none of them are comparable, so
188ingest.py derives one cross-agency flag, is_stop, per source. stop
189outcomes compare citation and arrest rate by substring, which is
190all the two disposition vocabularies support: an agency that
191records warnings less thoroughly shows a higher citation rate for
192that reason alone.
193
194colour scheme follows prefers-color-scheme. plotly writes colours
195into the figure, so assets/theme.js reports the media query into a
196store and app.py builds each figure from it. restyling after the
197fact does not work: swapping a maplibre basemap at runtime leaves
198it rebuilding with no data layers.
199
200the camera-proximity panel compares stops against other calls from
201the same agencies. the baseline has to be restricted that way: run
202against the whole archive it shows stops 2.6x more likely to be
203within 200m of a camera, but most of the archive is omaha, which
204reports no stops, so that number measures geography rather than
205enforcement. like for like it is 1.19x, and median distance is
206805m for stops against 780m for everything else -- no meaningful
207separation.
208
209raw_data/ingress.db, the old 2015-2023 sqlite build, and the yearly
210Incidents_*.csv downloads are gone from the tree: their LFS objects
211survive nowhere, nothing reads them, and ingest.py fetches the city's
212CSVs itself.
213
214* screenshots
215screenshots/*.png are from the previous 2015-2023 dashboard.