Commit `9d88cda282`

9d88cda2826224aab5cda15424453ea1cad75ae0

parent: 6149ca9c39

Verified · cmc

cmc <hello@cleberg.net> · 2026-04-11 19:13 UTC

feat: auto-discover git repositories for actor polling

Layout: unified · split

API.md +1

	@@ -51,6 +51,7 @@ Repository names:
51	- shorthand `repo-name`	51	- shorthand `repo-name`
52	- canonical `~owner/repo-name`	52	- canonical `~owner/repo-name`
53	- Stored repository names are normalized to canonical `~owner/repo-name` form.	53	- Stored repository names are normalized to canonical `~owner/repo-name` form.
		54	- Tracked repositories are optional force-includes for git polling; owned repositories are auto-discovered per actor.
54		55
55	## Health	56	## Health
56		57

README.md +8 −8

	@@ -8,7 +8,7 @@ The current V1 is intentionally narrow and production-oriented:
8	- SQLite-backed persistence	8	- SQLite-backed persistence
9	- polling-based ingestion	9	- polling-based ingestion
10	- complete `todo.sr.ht` ingestion path	10	- complete `todo.sr.ht` ingestion path
11	- practical `git.sr.ht` commit ingestion for tracked repositories	11	- practical `git.sr.ht` commit ingestion with automatic repository discovery
12	- public read-only contribution endpoints plus API-key protection for mutating/admin routes	12	- public read-only contribution endpoints plus API-key protection for mutating/admin routes
13	- Alembic-managed schema migrations	13	- Alembic-managed schema migrations
14		14
	@@ -51,7 +51,7 @@ Current normalized event types:
51	- `ticket_closed`	51	- `ticket_closed`
52	- `commit`	52	- `commit`
53		53
54	`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls tracked repositories for recent commits on the default branch.	54	`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories.
55		55
56	## Canonical Event Model	56	## Canonical Event Model
57		57
	@@ -69,7 +69,7 @@ All ingestion services normalize external activity into this shape:
69		69
70	The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat.	70	The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat.
71		71
72	Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. The poller seeds that table from `GIT_TRACKED_REPOSITORIES`, and repositories can also be created, updated, and deleted through the API.	72	Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories.
73		73
74	## Configuration	74	## Configuration
75		75
	@@ -84,7 +84,7 @@ Environment variables:
84	- `DEFAULT_ACTOR`: actor used by the scheduled poll job	84	- `DEFAULT_ACTOR`: actor used by the scheduled poll job
85	- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds	85	- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds
86	- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping	86	- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping
87	- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings for git polling	87	- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling
88		88
89	Example `.env`:	89	Example `.env`:
90		90
	@@ -132,7 +132,7 @@ Set at least:
132	- `API_KEY`	132	- `API_KEY`
133	- `SRHT_TOKEN`	133	- `SRHT_TOKEN`
134	- `DEFAULT_ACTOR`	134	- `DEFAULT_ACTOR`
135	- `GIT_TRACKED_REPOSITORIES` if you want git commit ingestion	135	- `GIT_TRACKED_REPOSITORIES` if you want to force-include extra repositories beyond the actor's owned repos
136		136
137	### 3. Run database migrations	137	### 3. Run database migrations
138		138
	@@ -167,7 +167,7 @@ Example response:
167		167
168	Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, and public contribution reads register additional actors for later background polling.	168	Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, and public contribution reads register additional actors for later background polling.
169		169
170	For `git.sr.ht`, tracked repositories are configured via `GIT_TRACKED_REPOSITORIES`. Entries may be either:	170	For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either:
171		171
172	- `"Hutch"` for a repository owned by `DEFAULT_ACTOR`	172	- `"Hutch"` for a repository owned by `DEFAULT_ACTOR`
173	- `"~your-user/your-site"` for an explicit owner/repository pair	173	- `"~your-user/your-site"` for an explicit owner/repository pair
	@@ -310,11 +310,11 @@ Covered areas:
310	The SourceHut-specific assumptions are isolated to the service modules:	310	The SourceHut-specific assumptions are isolated to the service modules:
311		311
312	- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery.	312	- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery.
313	- `src/srht_contrib/services/git.py` uses the documented repository `log(cursor)` query against tracked repositories and attributes commits through the configured alias map.	313	- `src/srht_contrib/services/git.py` discovers owned repositories for an actor, polls each repository `log(cursor)`, and attributes commits through the configured alias map.
314		314
315	## Known Limitations	315	## Known Limitations
316		316
317	- `git.sr.ht` polling is limited to repositories listed in `GIT_TRACKED_REPOSITORIES`	317	- `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API
318	- scheduled polling runs in-process, so it is not a distributed scheduler	318	- scheduled polling runs in-process, so it is not a distributed scheduler
319	- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs	319	- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs
320	- alias management is config-driven; there is no alias CRUD API yet	320	- alias management is config-driven; there is no alias CRUD API yet

src/srht_contrib/services/git.py +76 −7

	@@ -47,6 +47,23 @@ query RepositoryLog($username: String!, $repoName: String!, $cursor: Cursor) {
47	""".strip()	47	""".strip()
48		48
49		49
		50	USER_REPOSITORIES_QUERY = """
		51	query UserRepositories($username: String!, $cursor: Cursor) {
		52	user(username: $username) {
		53	repositories(cursor: $cursor) {
		54	results {
		55	name
		56	owner {
		57	canonicalName
		58	}
		59	}
		60	cursor
		61	}
		62	}
		63	}
		64	""".strip()
		65
		66
50	@dataclass(slots=True)	67	@dataclass(slots=True)
51	class GitPollResult:	68	class GitPollResult:
52	events: list[NormalizedEvent]	69	events: list[NormalizedEvent]
	@@ -54,7 +71,7 @@ class GitPollResult:
54		71
55		72
56	class GitIngestionService:	73	class GitIngestionService:
57	"""Polls tracked git.sr.ht repositories and normalizes commits for one actor."""	74	"""Polls git.sr.ht repositories and normalizes commits for one actor."""
58		75
59	service_name = "git"	76	service_name = "git"
60		77
	@@ -70,13 +87,13 @@ class GitIngestionService:
70	repositories: list[str] \| None = None,	87	repositories: list[str] \| None = None,
71	) -> GitPollResult:	88	) -> GitPollResult:
72	since_dt = ensure_utc(since or (datetime.now(tz=UTC) - timedelta(days=30)))	89	since_dt = ensure_utc(since or (datetime.now(tz=UTC) - timedelta(days=30)))
73	tracked_repositories = repositories or self._tracked_repositories(actor)	90	discovered_repositories = repositories or self._repositories_for_actor(actor)
74	if not tracked_repositories:	91	if not discovered_repositories:
75	logger.info("git poll skipped for actor=%s because no tracked repositories are configured", actor)	92	logger.info("git poll skipped for actor=%s because no repositories were discovered", actor)
76	return GitPollResult(events=[], cursor=datetime.now(tz=UTC).isoformat())	93	return GitPollResult(events=[], cursor=datetime.now(tz=UTC).isoformat())
77		94
78	events: list[NormalizedEvent] = []	95	events: list[NormalizedEvent] = []
79	for repository in tracked_repositories:	96	for repository in discovered_repositories:
80	owner, repo_name = self._split_repository(actor, repository)	97	owner, repo_name = self._split_repository(actor, repository)
81	repo_events = self._fetch_repository_commits(actor=actor, owner=owner, repo_name=repo_name, since=since_dt)	98	repo_events = self._fetch_repository_commits(actor=actor, owner=owner, repo_name=repo_name, since=since_dt)
82	events.extend(repo_events)	99	events.extend(repo_events)
	@@ -84,10 +101,62 @@ class GitIngestionService:
84	logger.info("git poll complete for actor=%s normalized_events=%s", actor, len(events))	101	logger.info("git poll complete for actor=%s normalized_events=%s", actor, len(events))
85	return GitPollResult(events=events, cursor=datetime.now(tz=UTC).isoformat())	102	return GitPollResult(events=events, cursor=datetime.now(tz=UTC).isoformat())
86		103
87	def _tracked_repositories(self, actor: str) -> list[str]:	104	def _repositories_for_actor(self, actor: str) -> list[str]:
88	repositories = self.settings.git_tracked_repositories	105	configured = {
		106	self._canonical_repository_name(actor, repository)
		107	for repository in self.settings.git_tracked_repositories
		108	}
		109	discovered = set(self._discover_owned_repositories(actor))
		110	repositories = sorted(configured \| discovered)
		111	logger.info(
		112	"git repositories selected for actor=%s count=%s configured=%s discovered=%s",
		113	actor,
		114	len(repositories),
		115	len(configured),
		116	len(discovered),
		117	)
89	return repositories	118	return repositories
90		119
		120	def _discover_owned_repositories(self, actor: str) -> list[str]:
		121	owner = actor.lstrip("~")
		122	repositories: list[str] = []
		123	cursor: str \| None = None
		124
		125	for _ in range(50):
		126	data = self.client.execute(
		127	USER_REPOSITORIES_QUERY,
		128	{"username": owner, "cursor": cursor},
		129	)
		130	user = data.get("user") or {}
		131	repositories_page = user.get("repositories") or {}
		132	results = repositories_page.get("results") or []
		133	cursor = repositories_page.get("cursor")
		134	logger.info(
		135	"git repository discovery actor=%s page_count=%s next_cursor=%s",
		136	actor,
		137	len(results),
		138	bool(cursor),
		139	)
		140
		141	for repository in results:
		142	if not isinstance(repository, dict):
		143	continue
		144	name = repository.get("name")
		145	repository_owner = ((repository.get("owner") or {}).get("canonicalName") or actor).strip()
		146	if not name or not repository_owner:
		147	continue
		148	repositories.append(f"{repository_owner}/{name}")
		149
		150	if not cursor:
		151	break
		152
		153	return repositories
		154
		155	@staticmethod
		156	def _canonical_repository_name(default_actor: str, repository: str) -> str:
		157	owner, repo_name = GitIngestionService._split_repository(default_actor, repository)
		158	return f"~{owner}/{repo_name}"
		159
91	@staticmethod	160	@staticmethod
92	def _split_repository(default_actor: str, repository: str) -> tuple[str, str]:	161	def _split_repository(default_actor: str, repository: str) -> tuple[str, str]:
93	if "/" in repository:	162	if "/" in repository:

tests/test_ingestion.py +61

	@@ -310,6 +310,67 @@ def test_git_ingestion_normalizes_commit_aliases_and_repository_names(db_session
310	assert tracked_repositories == ["~ccleberg/Hutch"]	310	assert tracked_repositories == ["~ccleberg/Hutch"]
311		311
312		312
		313	def test_git_ingestion_auto_discovers_owned_repositories(db_session) -> None:
		314	settings = make_settings(
		315	ACTOR_ALIASES_JSON={"~ccleberg": ["cmc@example.com", "Chris Cleberg"]},
		316	GIT_TRACKED_REPOSITORIES=[],
		317	)
		318	client = StubClient(
		319	payloads_by_query={
		320	"query UserRepositories": {
		321	"user": {
		322	"repositories": {
		323	"results": [
		324	{"name": "Hutch", "owner": {"canonicalName": "~ccleberg"}},
		325	],
		326	"cursor": None,
		327	}
		328	}
		329	},
		330	"query RepositoryLog": {
		331	"user": {
		332	"repository": {
		333	"name": "Hutch",
		334	"owner": {"canonicalName": "~ccleberg"},
		335	"log": {
		336	"results": [
		337	{
		338	"id": "abc123",
		339	"shortId": "abc123",
		340	"author": {
		341	"name": "Chris Cleberg",
		342	"email": "cmc@example.com",
		343	"time": "2026-03-30T12:00:00Z",
		344	},
		345	"committer": {
		346	"name": "Chris Cleberg",
		347	"email": "cmc@example.com",
		348	"time": "2026-03-30T12:00:00Z",
		349	},
		350	"message": "Auto-discovered repo commit",
		351	}
		352	],
		353	"cursor": None,
		354	},
		355	}
		356	}
		357	},
		358	}
		359	)
		360
		361	todo_service = TodoIngestionService(
		362	StubClient(payload={"me": {"canonicalName": "~ccleberg"}, "events": {"results": [], "cursor": None}}),
		363	settings,
		364	)
		365	git_service = GitIngestionService(client, settings)
		366	poller = PollerService(todo_service=todo_service, git_service=git_service)
		367
		368	inserted = poller.poll_all(db_session, "~ccleberg")
		369
		370	assert inserted == 1
		371	assert any("query UserRepositories" in call[0] for call in client.calls)
		372
		373
313	def test_sync_overlap_reuses_cursor_window_and_suppresses_duplicates(db_session) -> None:	374	def test_sync_overlap_reuses_cursor_window_and_suppresses_duplicates(db_session) -> None:
314	event = NormalizedEvent(	375	event = NormalizedEvent(
315	service="todo",	376	service="todo",

Commit 9d88cda282

Commit `9d88cda282`