krz/omaha-metro-blotter
Archive of police activity and ALPR surveillance across the Omaha metro.
clone: git clone https://gitbay.org/krz/omaha-metro-blotter.git
c5db763af6af53a4c1f83217e44b3d0bc363bd8e
verified · cmc
author: Christian Cleberg <hello@cleberg.net> · 2026-08-23T21:18:06Z
.github/workflows/daily-pull.yml | 4 +- README.nfo | 11 ++++-- analysis.py | 16 ++++++-- ingest.py | 83 +++++++++++++++++++++++++++------------- site_template.html | 6 ++- 5 files changed, 84 insertions(+), 36 deletions(-) @@ -99,7 +99,9 @@ jobs: if [ "${{ inputs.full }}" = "true" ] || [ "${{ inputs.bootstrap }}" = "true" ] \ || [ "$(date -u +%u)" = "7" ]; then echo "full sweep" - python ingest.py --full + # opd_archive is closed years that never change; the weekly sweep is + # often enough to notice if OPD ever restates one. + python ingest.py --full opd sarpy cbpd alpr flock opd_archive else python ingest.py fi @@ -180,9 +180,14 @@ NOTES fact does not work: swapping a maplibre basemap at runtime leaves it rebuilding with no data layers. - the camera-proximity panel compares stops against a non-stop - baseline. cameras and stops both concentrate on arterials, so a - gap between the curves is a starting point, not a finding. + the camera-proximity panel compares stops against other calls from + the same agencies. the baseline has to be restricted that way: run + against the whole archive it shows stops 2.6x more likely to be + within 200m of a camera, but most of the archive is omaha, which + reports no stops, so that number measures geography rather than + enforcement. like for like it is 1.19x, and median distance is + 805m for stops against 780m for everything else -- no meaningful + separation. raw_data/ingress.db is the old 2015-2023 sqlite build. nothing reads it any more. @@ -113,11 +113,21 @@ def stop_outcomes(df): def camera_proximity(df, cameras, bin_m=200, max_m=2000): """Share of stops vs other incidents falling in each distance band. - Both series are normalised, so a gap between them means stops cluster - differently around cameras than the rest of the call volume does. It is not - evidence of causation: cameras and stops both concentrate on arterials.""" + The baseline is drawn only from agencies that report stops. Comparing stops + against every incident in the archive instead compares Sarpy and Council + Bluffs stops with a baseline that is mostly Omaha, a city reporting no stops + at all, and the geography alone then makes stops look far closer to cameras + than they are: 2.6x within 200 m across all agencies, 1.2x within the ones + actually being measured. + + Even restricted this way it is not evidence of causation. Cameras and + enforcement both concentrate on arterials.""" if df.empty or cameras.empty: return pd.DataFrame(columns=["distance_m", "kind", "share"]) + reporting = df.loc[df["is_stop"] == 1, "agency"].unique() + df = df[df["agency"].isin(reporting)] + if df.empty: + return pd.DataFrame(columns=["distance_m", "kind", "share"]) d = df.assign(dist=nearest_camera_m(df, cameras)) d = d[d["dist"] <= max_m] if d.empty: @@ -14,8 +14,11 @@ Sources: raw_data/flock/<portal-slug>_<date>.csv and this reads what is there. The portals keep a rolling 30 days, so a gap longer than that is permanent. - opd_csv One-time backfill of raw_data/Incidents_*.csv (2015-2023). Statute text - only, no NIBRS category. + opd_archive + OPD's own yearly incident CSVs, 2015-2021. The ArcGIS view only goes + back to 2022-01-01, so this is the only route to the earlier years, + and it carries the statute description rather than a NIBRS category. + Reported crimes only: no stops, no dispositions. All three ArcGIS services return UTC epochs. Their WHERE literals do not agree: OPD and Council Bluffs use UTC, Sarpy uses America/Chicago, so each source @@ -97,6 +100,16 @@ SARPY_AGENCIES = { # calls -- reactive, not officer-initiated. SARPY_STOP_CATEGORY = "Proactive Policing - Vehicle Stop" +# OPD publishes a CSV per year at a predictable path, updated daily. The ArcGIS +# view starts 2022-01-01, so only the years before that are taken from here -- +# ingesting the overlap would double-count every Omaha incident since 2022. +OPD_ARCHIVE = "https://police-static.cityofomaha.org/crime-data/{y}/Incidents_{y}.csv" +OPD_ARCHIVE_YEARS = range(2015, 2022) +OPD_ARCHIVE_COLUMNS = ("RB Number", "Reported Date", "Reported Time", + "Statute/Ordinance Description", "Occurred Location", + "Occurred District", "Occurred Block LAT", + "Occurred Block LON") + OVERPASS = "https://overpass-api.de/api/interpreter" # Douglas and Sarpy counties in Nebraska plus Council Bluffs across the river. BBOX = (40.95, -96.35, 41.45, -95.65) @@ -298,30 +311,46 @@ def ingest_cbpd(conn, since): return upsert(conn, rows) -def ingest_opd_csv(conn, _since): - """Backfill the 2015-2023 CSV archive. Statute text goes to offense_desc; - category stays NULL because it is not a NIBRS category.""" - rows = [] - for path in sorted((ROOT / "raw_data").glob("Incidents_*.csv")): - with path.open(newline="") as fh: - if fh.readline().startswith("version https://git-lfs"): - print(f" {path.name}: git-lfs pointer, run 'git lfs pull'") +def ingest_opd_archive(conn, _since): + """Load OPD's yearly incident CSVs for the years the ArcGIS view predates. + + Keyed on a hash of the row rather than its position in the file: these are + closed years and should be stable, but a single row inserted upstream would + otherwise shift every key below it and file fifty thousand false + amendments.""" + rows, seen = [], {} + for year in OPD_ARCHIVE_YEARS: + url = OPD_ARCHIVE.format(y=year) + req = urllib.request.Request(url, headers={"User-Agent": "omaha-incidents/1.0"}) + with urllib.request.urlopen(req, timeout=180, context=SSL_CTX) as r: + text = r.read().decode("utf-8-sig", "replace") + reader = csv.DictReader(text.splitlines()) + if tuple(reader.fieldnames or ()) != OPD_ARCHIVE_COLUMNS: + print(f" {year}: unexpected columns {reader.fieldnames}") + continue + n = 0 + for rec in reader: + when = f"{rec['Reported Date']} {rec['Reported Time']}" + try: + occurred = datetime.strptime(when, "%m/%d/%Y %H:%M:%S") + except ValueError: continue - fh.seek(0) - for i, r in enumerate(csv.reader(fh)): - if len(r) < 8 or r[0] == "RB Number": - continue - rb, date, tm, desc, loc, district, lat, lon = r[:8] - try: - when = datetime.strptime(f"{date} {tm}", "%m/%d/%Y %H:%M") - except ValueError: - continue - rows.append((("opd_csv", f"{path.stem}:{i}", "Omaha PD", rb, - when.strftime("%Y-%m-%dT%H:%M:%S"), None, None, - None, desc, 0, loc, - float(lat) if lat else None, - float(lon) if lon else None), - json.dumps(r))) + raw = json.dumps(rec, sort_keys=True) + # a few rows repeat verbatim; number them so each keeps its own key + h = hashlib.blake2b(raw.encode(), digest_size=8).hexdigest() + # No raw kept: unlike the rolling feeds, whose aged-out records can + # never be fetched again, these year files stay downloadable. The + # one field not carried into a column is Occurred District. + seen[h] = seen.get(h, 0) + 1 + lat, lon = rec["Occurred Block LAT"], rec["Occurred Block LON"] + rows.append((( + "opd_archive", f"{h}:{seen[h]}", "Omaha PD", rec["RB Number"], + occurred.strftime("%Y-%m-%dT%H:%M:%S"), None, None, None, + rec["Statute/Ordinance Description"], 0, + rec["Occurred Location"], + float(lat) if lat else None, float(lon) if lon else None), None)) + n += 1 + print(f" {year}: {n}", end="\r", file=sys.stderr, flush=True) return upsert(conn, rows) @@ -443,7 +472,7 @@ SOURCES = { "cbpd": ingest_cbpd, "alpr": ingest_alpr, "flock": ingest_flock, - "opd_csv": ingest_opd_csv, + "opd_archive": ingest_opd_archive, } @@ -454,7 +483,7 @@ def main(): help="default: opd sarpy cbpd alpr flock") p.add_argument("--since-days", type=int, default=30, help="only pull incidents this recent (default 30); " - "ignored by alpr, flock and opd_csv") + "ignored by alpr, flock and opd_archive") p.add_argument("--full", action="store_true", help="pull the complete feed instead of --since-days") p.add_argument("--import-flock", metavar="CSV", @@ -112,12 +112,14 @@ footer a { color: var(--text-secondary); } <section> <h2>Distance to the nearest ALPR camera</h2> - <p>Where vehicle stops happen relative to automated licence plate readers, against every other kind of incident as a baseline. Both lines are shares of their own series, so the gap is what matters.</p> + <p>Where vehicle stops happen relative to automated licence plate readers, against every other call handled by <em>the same agencies</em> as a baseline. Both lines are shares of their own series, so the gap between them is what matters.</p> <div class="card"> <div class="legend" id="lg-prox"></div> <div id="c-prox"></div> </div> - <p class="note"><strong>This is not evidence that cameras cause stops.</strong> Cameras get mounted on arterial roads and arterial roads are where traffic enforcement happens, so the two concentrate together for reasons that have nothing to do with each other. It is a starting point for asking the agencies a question, not an answer. That question has a stated standard: all three metro Flock transparency portals list <em>traffic enforcement</em> under prohibited uses.</p> + <p class="note"><strong>The two lines nearly overlap, and that is the finding.</strong> Stops sit marginally closer to cameras than the same agencies' other calls — 14.7% against 12.4% within 200 m — but the median distance is 805 m for stops and 780 m for everything else. There is no meaningful separation here.</p> + <p class="note">Comparing stops against <em>every</em> incident in the archive instead produces a much larger gap, 2.6× within 200 m. That gap is an artifact: most of the archive is Omaha, a city that reports no stops at all, so the comparison was measuring the difference between two geographies rather than between stops and other calls. Restricting the baseline to the five agencies that actually report stops removes it.</p> + <p class="note">All three metro Flock transparency portals list <em>traffic enforcement</em> under prohibited uses. On this measure, at this resolution, nothing here contradicts that. It is a weak instrument — cameras and enforcement both concentrate on arterial roads, and camera locations are volunteer-mapped and incomplete — so it cannot clear an agency either.</p> <details><summary>Show the numbers</summary><div id="t-prox"></div></details> </section>