Commit `e7ec619ff2`

e7ec619ff25af162e464c1118571edb28b536a48

parent: 5d258a14f0

Verified · cmc

cmc <hello@cleberg.net> · 2026-08-02 20:29 UTC

convert readme to nfo; convert docs to txt; add 0bsd license

Layout: unified · split

API.md → API.txt renamed

LICENSE added +12

@@ -0,0 +1,12 @@
	1	Copyright (C) 2026 krazy warez
	2
	3	Permission to use, copy, modify, and/or distribute this software for any
	4	purpose with or without fee is hereby granted.
	5
	6	THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
	7	WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
	8	MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
	9	ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
	10	WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
	11	ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
	12	OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

README.md deleted −367

	@@ -1,367 +0,0 @@
1	# srht-contrib
2
3	`srht-contrib` is a small Python service that polls SourceHut activity, normalizes it into one internal event model, stores it in SQLite, and exposes a contribution-calendar JSON API that an iOS app can render directly.
4
5	The current V1 is intentionally narrow and production-oriented:
6
7	- FastAPI JSON API only
8	- SQLite-backed persistence
9	- polling-based ingestion
10	- complete `todo.sr.ht` ingestion path
11	- practical `git.sr.ht` commit ingestion with automatic repository discovery
12	- public read-only contribution endpoints plus API-key protection for mutating/admin routes
13	- Alembic-managed schema migrations
14
15	## What It Does
16
17	The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs recent incremental polling, backfills the most recent 365 days for newly requested actors, and prunes older activity from storage.
18
19	Example use cases:
20
21	- render a GitHub-style contribution grid in an iOS app
22	- show total score and streak stats for a SourceHut user
23	- poll recent activity on a schedule or trigger polling manually
24
25	## Architecture Overview
26
27	The code is split into small, testable layers:
28
29	- `src/srht_contrib/config.py`: environment-driven settings and event weights
30	- `src/srht_contrib/db.py`: SQLAlchemy engine/session setup and app-scoped DB access
31	- `src/srht_contrib/models.py`: ORM models for normalized events, sync state, aliases, tracked actors, and backfill state
32	- `src/srht_contrib/services/srht_client.py`: generic SourceHut GraphQL client with error handling and simple retries
33	- `src/srht_contrib/services/todo.py`: `todo.sr.ht` ingestion and normalization
34	- `src/srht_contrib/services/git.py`: `git.sr.ht` tracked-repository commit ingestion
35	- `src/srht_contrib/services/aggregator.py`: per-day aggregation and streak/stat calculations
36	- `src/srht_contrib/jobs/poller.py`: repeated-safe polling and idempotent persistence
37	- `src/srht_contrib/api/`: FastAPI routes, auth dependencies, and repository management
38	- `alembic/`: schema migration environment and versioned migrations
39
40	## Supported sr.ht Services
41
42	### Implemented
43
44	- `todo.sr.ht`
45	- `git.sr.ht`
46
47	Current normalized event types:
48
49	- `ticket_created`
50	- `ticket_comment`
51	- `ticket_closed`
52	- `commit`
53
54	`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories.
55
56	## Canonical Event Model
57
58	All ingestion services normalize external activity into this shape:
59
60	- `service`
61	- `event_type`
62	- `actor`
63	- `repo_name`
64	- `resource_id`
65	- `external_uid`
66	- `occurred_at`
67	- `weight`
68	- `raw_payload_json`
69
70	The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat.
71
72	Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories.
73
74	## Configuration
75
76	Environment variables:
77
78	- `API_KEY`: required header token for mutating/admin routes via `X-API-Key`
79	- `ENABLE_SCHEDULER`: defaults to `false`; enables in-process polling when set to `true`
80	- `SRHT_TOKEN`: bearer token for SourceHut GraphQL
81	- `TODO_SRHT_ENDPOINT`: defaults to `https://todo.sr.ht/query`
82	- `GIT_SRHT_ENDPOINT`: defaults to `https://git.sr.ht/query`
83	- `DATABASE_URL`: defaults to `sqlite:///./srht_contrib.db`
84	- `DEFAULT_ACTOR`: actor used by the scheduled poll job
85	- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds
86	- `DISCOVERY_BATCH_SIZE`: max number of due actors to process per scheduler pass
87	- `INDEXED_ACTOR_REPOLL_SECONDS`: how long to wait before re-polling an already indexed actor
88	- `DISCOVERY_ERROR_BACKOFF_SECONDS`: base retry delay after a failed scheduled poll
89	- `DISCOVERY_ERROR_BACKOFF_MAX_SECONDS`: maximum retry delay after repeated scheduled poll failures
90	- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping
91	- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling
92
93	Example `.env`:
94
95	```env
96	API_KEY=replace-me
97	ENABLE_SCHEDULER=false
98	SRHT_TOKEN=replace-me
99	TODO_SRHT_ENDPOINT=https://todo.sr.ht/query
100	GIT_SRHT_ENDPOINT=https://git.sr.ht/query
101	DATABASE_URL=sqlite:///./srht_contrib.db
102	DEFAULT_ACTOR=~your-user
103	POLL_INTERVAL_SECONDS=300
104	DISCOVERY_BATCH_SIZE=20
105	INDEXED_ACTOR_REPOLL_SECONDS=21600
106	DISCOVERY_ERROR_BACKOFF_SECONDS=3600
107	DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600
108	ACTOR_ALIASES_JSON={"~your-user":["you@example.com","Your Name"]}
109	GIT_TRACKED_REPOSITORIES=["your-repo","~your-user/your-site"]
110	```
111
112	## Local Run Instructions
113
114	### 1. Create a virtual environment and install dependencies
115
116	Using `uv`:
117
118	```bash
119	uv venv
120	source .venv/bin/activate
121	uv pip install -e ".[dev]"
122	```
123
124	Using `pip`:
125
126	```bash
127	python3.12 -m venv .venv
128	source .venv/bin/activate
129	pip install -e ".[dev]"
130	```
131
132	### 2. Configure environment
133
134	```bash
135	cp .env.example .env
136	```
137
138	Set at least:
139
140	- `API_KEY`
141	- `SRHT_TOKEN`
142	- `DEFAULT_ACTOR`
143	- `GIT_TRACKED_REPOSITORIES` if you want to force-include extra repositories beyond the actor's owned repos
144
145	### 3. Run database migrations
146
147	```bash
148	alembic upgrade head
149	```
150
151	### 4. Run the API
152
153	```bash
154	uvicorn srht_contrib.main:app --reload
155	```
156
157	## Manual Polling
158
159	Manual polling is exposed as an API endpoint:
160
161	```bash
162	curl -X POST "http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \
163	-H "X-API-Key: replace-me"
164	```
165
166	Example response:
167
168	```json
169	{
170	"actor": "~your-user",
171	"inserted_events": 3,
172	"services": ["todo", "git"]
173	}
174	```
175
176	Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, runs one poll immediately at startup, and public contribution reads register additional actors for later background polling and one-year backfill.
177
178	The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to `DISCOVERY_BATCH_SIZE` per run, performs a fast first indexing pass before bounded one-year backfill work, then reschedules indexed actors with `INDEXED_ACTOR_REPOLL_SECONDS` and failed actors with capped backoff based on `DISCOVERY_ERROR_BACKOFF_SECONDS`.
179
180	Clients can explicitly signal that a public contribution read is for the signed-in user's own graph by sending `prioritize_self=true` on the read request. That temporarily boosts the actor to the front of the due queue for the next indexing pass, then clears the boost after the poll completes.
181
182	## Bulk Enqueue Without Immediate Indexing
183
184	To durably queue a large username list without polling it immediately:
185
186	```bash
187	srht-enqueue-actors srht_usernames.txt --stagger-seconds 60
188	```
189
190	This command stores usernames in `tracked_actors`, marks them queued, and spaces out their first eligible poll time. With `--stagger-seconds 60`, a file of 15,771 users will be spread across roughly 11 days before becoming due for first poll.
191
192	For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either:
193
194	- `"Hutch"` for a repository owned by `DEFAULT_ACTOR`
195	- `"~your-user/your-site"` for an explicit owner/repository pair
196
197	## API Endpoints
198
199	### Health
200
201	```bash
202	curl "http://127.0.0.1:8000/health"
203	```
204
205	Response:
206
207	```json
208	{"status":"ok"}
209	```
210
211	### Contribution Calendar by Year
212
213	```bash
214	curl "http://127.0.0.1:8000/api/contributions/~your-user?year=2026"
215	```
216
217	### Contribution Calendar by Date Range
218
219	```bash
220	curl "http://127.0.0.1:8000/api/contributions/~your-user?from=2026-01-01&to=2026-03-30"
221	```
222
223	Example response:
224
225	```json
226	{
227	"actor": "~your-user",
228	"from": "2026-01-01",
229	"to": "2026-03-30",
230	"is_indexed": true,
231	"last_polled_at": "2026-04-11T18:05:00Z",
232	"indexing_state": "indexed",
233	"is_recent_window_backfilled": true,
234	"recent_backfill_state": "completed",
235	"recent_backfill_completed_at": "2026-04-11T18:02:00Z",
236	"days": [
237	{"date": "2026-03-28", "count": 3, "score": 3.5},
238	{"date": "2026-03-29", "count": 0, "score": 0.0},
239	{"date": "2026-03-30", "count": 7, "score": 8.25}
240	]
241	}
242	```
243
244	### Contribution Stats
245
246	```bash
247	curl "http://127.0.0.1:8000/api/contributions/~your-user/stats?year=2026"
248	```
249
250	Example response:
251
252	```json
253	{
254	"actor": "~your-user",
255	"from": "2026-01-01",
256	"to": "2026-12-31",
257	"is_indexed": true,
258	"last_polled_at": "2026-04-11T18:05:00Z",
259	"indexing_state": "indexed",
260	"is_recent_window_backfilled": true,
261	"recent_backfill_state": "completed",
262	"recent_backfill_completed_at": "2026-04-11T18:02:00Z",
263	"total_events": 42,
264	"total_score": 37.5,
265	"active_days": 18,
266	"longest_streak": 5,
267	"current_streak": 2
268	}
269	```
270
271	### Tracked Repositories
272
273	List tracked repositories:
274
275	```bash
276	curl "http://127.0.0.1:8000/api/repositories?actor=~your-user" \
277	-H "X-API-Key: replace-me"
278	```
279
280	Create a tracked repository:
281
282	```bash
283	curl -X POST "http://127.0.0.1:8000/api/repositories" \
284	-H "X-API-Key: replace-me" \
285	-H "Content-Type: application/json" \
286	-d '{"actor":"~your-user","repo_name":"your-repo"}'
287	```
288
289	Get, update, and delete a tracked repository:
290
291	```bash
292	curl "http://127.0.0.1:8000/api/repositories/1" \
293	-H "X-API-Key: replace-me"
294
295	curl -X PATCH "http://127.0.0.1:8000/api/repositories/1" \
296	-H "X-API-Key: replace-me" \
297	-H "Content-Type: application/json" \
298	-d '{"repo_name":"~your-user/your-site"}'
299
300	curl -X DELETE "http://127.0.0.1:8000/api/repositories/1" \
301	-H "X-API-Key: replace-me"
302	```
303
304	## Event Weighting
305
306	Weights live in `src/srht_contrib/config.py` so they are easy to tune without touching aggregation code:
307
308	- `commit`: `1.0`
309	- `ticket_created`: `1.0`
310	- `ticket_comment`: `0.5`
311	- `ticket_closed`: `0.75`
312	- `build_started`: `0.25`
313	- `build_passed`: `0.25`
314
315	## Testing
316
317	Run the test suite with:
318
319	```bash
320	pytest
321	```
322
323	Covered areas:
324
325	- health endpoint
326	- public read-only contribution endpoints
327	- API key enforcement for mutating/admin routes
328	- calendar aggregation
329	- zero-filled ranges
330	- stats calculations
331	- invalid date handling
332	- idempotent ingestion
333	- todo feed fallback traversal
334	- repository CRUD and normalization
335	- SourceHut error mapping
336	- git commit alias normalization
337	- Alembic upgrade path
338
339	## SourceHut Schema Assumptions
340
341	The SourceHut-specific assumptions are isolated to the service modules:
342
343	- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery.
344	- `src/srht_contrib/services/git.py` discovers owned repositories for an actor, polls each repository `log(cursor)`, and attributes commits through the configured alias map.
345	- the service only retains the most recent 365 days of contribution history and periodically prunes older rows
346
347	## Known Limitations
348
349	- `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API
350	- scheduled polling runs in-process, so it is not a distributed scheduler
351	- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs
352	- the service is intentionally limited to a rolling one-year history window; older activity is not retained
353	- one-year backfill still runs in bounded batches, so a newly requested actor may take multiple scheduler passes before their visible graph is fully filled in
354	- alias management is config-driven; there is no alias CRUD API yet
355	- current deployment model is trusted-operator V1, not a public multi-tenant service
356
357	## Deployment Notes
358
359	- Add a `.dockerignore` when building container images so local secrets and SQLite files are never sent to the build context.
360	- For production, prefer exposing the service behind a reverse proxy instead of publishing the application port directly to the internet.
361	- Set `ENABLE_SCHEDULER=true` only for single-instance deployments where this service should own polling.
362
363	## Recommended Next Steps
364
365	1. Add alias-management APIs or seed files for stronger actor identity mapping.
366	2. Add more SourceHut services such as `builds.sr.ht` and `lists.sr.ht`.
367	3. Move scheduled polling into an external worker if the deployment grows past a single process.

README.nfo added +86

@@ -0,0 +1,86 @@
	1	┌──────────────────────────────────────────────────────────────┐
	2	│ H U T C H - S T A T S [ KRZ ] krz.sh │
	3	└──────────────────────────────────────────────────────────────┘
	4
	5	WHAT
	6	srht-contrib. python service. polls sourcehut activity, normalizes
	7	it into one event model, stores it in sqlite, and serves a
	8	contribution-calendar json api an ios app renders directly.
	9
	10	v1 scope:
	11	- fastapi json api
	12	- sqlite persistence
	13	- polling ingestion
	14	- full todo.sr.ht path
	15	- git.sr.ht commit ingestion with repo auto-discovery
	16	- public read-only endpoints, api-key auth for mutating routes
	17	- alembic migrations
	18
	19	DOES
	20	collects activity from sr.ht graphql services, aggregates by day,
	21	and returns zero-filled ranges so the client never patches missing
	22	dates. polls incrementally, backfills the last 365 days for new
	23	actors, prunes older activity.
	24
	25	SERVICES
	26	implemented: todo.sr.ht, git.sr.ht.
	27	event types: ticket_created, ticket_comment, ticket_closed, commit.
	28
	29	ENV
	30	API_KEY=replace-me
	31	ENABLE_SCHEDULER=false
	32	SRHT_TOKEN=replace-me
	33	TODO_SRHT_ENDPOINT=https://todo.sr.ht/query
	34	GIT_SRHT_ENDPOINT=https://git.sr.ht/query
	35	DATABASE_URL=sqlite:///./srht_contrib.db
	36	DEFAULT_ACTOR=~your-user
	37	POLL_INTERVAL_SECONDS=300
	38
	39	more knobs (discovery batch size, repoll and backoff intervals,
	40	actor aliases, tracked repositories) live in config.py.
	41
	42	RUN
	43	uv venv
	44	source .venv/bin/activate
	45	uv pip install -e ".[dev]"
	46	cp .env.example .env
	47	alembic upgrade head
	48	uvicorn srht_contrib.main:app --reload
	49
	50	POLL
	51	manual poll:
	52
	53	curl -X POST \
	54	"http://127.0.0.1:8000/api/contributions/poll?actor=~your-user" \
	55	-H "X-API-Key: replace-me"
	56
	57	scheduled polling runs only when ENABLE_SCHEDULER=true. it drains
	58	due actors in batches, indexes fast, then backfills a year in
	59	bounded passes.
	60
	61	bulk queue without immediate indexing:
	62
	63	srht-enqueue-actors srht_usernames.txt --stagger-seconds 60
	64
	65	ENDPOINTS
	66	calendar by year, calendar by date range, stats, and repository
	67	crud. full request and response shapes are in API.txt.
	68
	69	WEIGHTS
	70	commit 1.0, ticket_created 1.0, ticket_comment 0.5,
	71	ticket_closed 0.75, build_started 0.25, build_passed 0.25.
	72
	73	TESTS
	74	pytest
	75
	76	LIMITS
	77	- git polling assumes repos are discoverable via graphql
	78	- the scheduler runs in-process, not distributed
	79	- new actors index asynchronously; the first read may be empty
	80	- rolling one-year window; older activity is not kept
	81	- alias management is config-driven, no crud api yet
	82	- trusted-operator v1, not a public multi-tenant service
	83
	84	┌──────────────────────────────────────────────────────────────┐
	85	│ krz.sh │
	86	└──────────────────────────────────────────────────────────────┘

Commit e7ec619ff2

Commit `e7ec619ff2`