krz/omaha-metro-blotter

 1┌──────────────────────────────────────────────────────────────┐
 2│  O M A H A   I N C I D E N T S             [ KRZ ]   krz.sh  │
 3└──────────────────────────────────────────────────────────────┘
 4
 5WHAT
 6  police activity across the omaha metro, pulled from the agencies'
 7  own feeds and joined against alpr camera locations from
 8  openstreetmap.
 9
10COVERAGE
11  omaha pd                dcgis arcgis view, 2022-01-01 onward,
12                          nibrs offence records, updated daily.
13                          no stop or disposition data.
14  bellevue pd             sarpy county publiccrimemap, cad calls
15  papillion pd            for service with stop type, disposition
16  la vista pd             and category. rolling 12-month window:
17  sarpy county so         records age out of the feed, so the local
18                          archive is the only long-term copy. gretna
19                          and springfield appear as fire only; their
20                          police departments do not report to it.
21  council bluffs pd       cbpd public cfs feed, refreshed every ten
22                          minutes, rolling 12-month window. stop
23                          type, disposition, priority, response time.
24                          street addresses withheld; points exact.
25  ralston pd              no machine-readable feed. absent.
26
27  alpr cameras            openstreetmap via overpass, the same data
28                          deflock renders. 168 nodes in the metro
29                          bbox. odbl, attribution required.
30
31SETUP
32      uv venv .venv
33      uv pip install --python .venv/bin/python -r requirements.txt
34
35USE
36      .venv/bin/python ingest.py --full        # first run, backfill
37      .venv/bin/python ingest.py               # daily, last 30 days
38      .venv/bin/python ingest.py cbpd sarpy    # one source at a time
39      .venv/bin/python ingest.py opd_csv       # 2015-2023 csv archive
40      .venv/bin/python app.py
41
42ARCHIVE
43  .github/workflows/daily-pull.yml runs the pull at 11:17 utc and
44  keeps the database as metro.db.gz on the "archive" release, so the
45  archive does not depend on any one machine. each run restores that
46  asset, pulls, refuses to publish if any source came back with fewer
47  rows than it started with, then uploads and fails loudly if a feed
48  has not moved in seven days.
49
50  first run: trigger it manually with bootstrap enabled, which pulls
51  every feed in full and creates the release. after that the restore
52  step is mandatory -- a bootstrap over a live archive throws away
53  whatever has already aged out of the sarpy and council bluffs
54  feeds.
55
56  github disables scheduled workflows after 60 days without repo
57  activity, and emails first. that is the most likely way this stops
58  quietly.
59
60  to run the pull locally instead:
61
62      0 6 * * *  cd /path/to/omaha-incidents && .venv/bin/python ingest.py
63
64NOTES
65  all three arcgis services return utc epochs; their where-clause
66  literals do not agree (opd and council bluffs utc, sarpy central).
67  ingest.py stores occurred_at in local time.
68
69  each feed has its own taxonomy and none of them are comparable, so
70  ingest.py derives one cross-agency flag, is_stop, per source. stop
71  outcomes compare citation and arrest rate by substring, which is
72  all the two disposition vocabularies support: an agency that
73  records warnings less thoroughly shows a higher citation rate for
74  that reason alone.
75
76  colour scheme follows prefers-color-scheme. plotly writes colours
77  into the figure, so assets/theme.js reports the media query into a
78  store and app.py builds each figure from it. restyling after the
79  fact does not work: swapping a maplibre basemap at runtime leaves
80  it rebuilding with no data layers.
81
82  the camera-proximity panel compares stops against a non-stop
83  baseline. cameras and stops both concentrate on arterials, so a
84  gap between the curves is a starting point, not a finding.
85
86  raw_data/ingress.db is the old 2015-2023 sqlite build. nothing
87  reads it any more.
88
89SCREENSHOTS
90  screenshots/*.png are from the previous 2015-2023 dashboard.
91
92┌──────────────────────────────────────────────────────────────┐
93│  krz.sh                                                      │
94└──────────────────────────────────────────────────────────────┘