Commit `9d88cda282`

9d88cda2826224aab5cda15424453ea1cad75ae0

parent: 6149ca9c39

Verified · cmc

cmc <hello@cleberg.net> · 2026-04-11 19:13 UTC

feat: auto-discover git repositories for actor polling

Layout: unified · split

API.md +1

		@@ -51,6 +51,7 @@ Repository names:
51	51	- shorthand `repo-name`
52	52	- canonical `~owner/repo-name`
53	53	- Stored repository names are normalized to canonical `~owner/repo-name` form.
	54	- Tracked repositories are optional force-includes for git polling; owned repositories are auto-discovered per actor.
54	55
55	56	## Health
56	57

README.md +8 −8

		@@ -8,7 +8,7 @@ The current V1 is intentionally narrow and production-oriented:
8	8	- SQLite-backed persistence
9	9	- polling-based ingestion
10	10	- complete `todo.sr.ht` ingestion path
11		- practical `git.sr.ht` commit ingestion for tracked repositories
	11	- practical `git.sr.ht` commit ingestion with automatic repository discovery
12	12	- public read-only contribution endpoints plus API-key protection for mutating/admin routes
13	13	- Alembic-managed schema migrations
14	14
		@@ -51,7 +51,7 @@ Current normalized event types:
51	51	- `ticket_closed`
52	52	- `commit`
53	53
54		`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls tracked repositories for recent commits on the default branch.
	54	`todo.sr.ht` uses a feed-first strategy and falls back to crawling the authenticated user’s trackers, tickets, and ticket events when the top-level activity feed is empty. `git.sr.ht` polls the actor's owned repositories for recent commits on the default branch and unions in any explicitly configured repositories.
55	55
56	56	## Canonical Event Model
57	57
		@@ -69,7 +69,7 @@ All ingestion services normalize external activity into this shape:
69	69
70	70	The database enforces uniqueness on `(service, external_uid)` so polling is safe to repeat.
71	71
72		Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. The poller seeds that table from `GIT_TRACKED_REPOSITORIES`, and repositories can also be created, updated, and deleted through the API.
	72	Tracked git repositories are persisted in the `tracked_repositories` table and stored in canonical `~owner/repo` form. They are optional overrides now: the poller auto-discovers an actor's owned repositories and unions in any configured or API-managed repositories.
73	73
74	74	## Configuration
75	75
		@@ -84,7 +84,7 @@ Environment variables:
84	84	- `DEFAULT_ACTOR`: actor used by the scheduled poll job
85	85	- `POLL_INTERVAL_SECONDS`: scheduler interval in seconds
86	86	- `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping
87		- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings for git polling
	87	- `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling
88	88
89	89	Example `.env`:
90	90
		@@ -132,7 +132,7 @@ Set at least:
132	132	- `API_KEY`
133	133	- `SRHT_TOKEN`
134	134	- `DEFAULT_ACTOR`
135		- `GIT_TRACKED_REPOSITORIES` if you want git commit ingestion
	135	- `GIT_TRACKED_REPOSITORIES` if you want to force-include extra repositories beyond the actor's owned repos
136	136
137	137	### 3. Run database migrations
138	138
		@@ -167,7 +167,7 @@ Example response:
167	167
168	168	Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, and public contribution reads register additional actors for later background polling.
169	169
170		For `git.sr.ht`, tracked repositories are configured via `GIT_TRACKED_REPOSITORIES`. Entries may be either:
	170	For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either:
171	171
172	172	- `"Hutch"` for a repository owned by `DEFAULT_ACTOR`
173	173	- `"~your-user/your-site"` for an explicit owner/repository pair
		@@ -310,11 +310,11 @@ Covered areas:
310	310	The SourceHut-specific assumptions are isolated to the service modules:
311	311
312	312	- `src/srht_contrib/services/todo.py` uses the authenticated `events(cursor)` feed first, then falls back to tracker/ticket event traversal for reliable contribution discovery.
313		- `src/srht_contrib/services/git.py` uses the documented repository `log(cursor)` query against tracked repositories and attributes commits through the configured alias map.
	313	- `src/srht_contrib/services/git.py` discovers owned repositories for an actor, polls each repository `log(cursor)`, and attributes commits through the configured alias map.
314	314
315	315	## Known Limitations
316	316
317		- `git.sr.ht` polling is limited to repositories listed in `GIT_TRACKED_REPOSITORIES`
	317	- `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API
318	318	- scheduled polling runs in-process, so it is not a distributed scheduler
319	319	- newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs
320	320	- alias management is config-driven; there is no alias CRUD API yet

src/srht_contrib/services/git.py +76 −7

		@@ -47,6 +47,23 @@ query RepositoryLog($username: String!, $repoName: String!, $cursor: Cursor) {
47	47	""".strip()
48	48
49	49
	50	USER_REPOSITORIES_QUERY = """
	51	query UserRepositories($username: String!, $cursor: Cursor) {
	52	user(username: $username) {
	53	repositories(cursor: $cursor) {
	54	results {
	55	name
	56	owner {
	57	canonicalName
	58	}
	59	}
	60	cursor
	61	}
	62	}
	63	}
	64	""".strip()
	65
	66
50	67	@dataclass(slots=True)
51	68	class GitPollResult:
52	69	events: list[NormalizedEvent]
		@@ -54,7 +71,7 @@ class GitPollResult:
54	71
55	72
56	73	class GitIngestionService:
57		"""Polls tracked git.sr.ht repositories and normalizes commits for one actor."""
	74	"""Polls git.sr.ht repositories and normalizes commits for one actor."""
58	75
59	76	service_name = "git"
60	77
		@@ -70,13 +87,13 @@ class GitIngestionService:
70	87	repositories: list[str] \| None = None,
71	88	) -> GitPollResult:
72	89	since_dt = ensure_utc(since or (datetime.now(tz=UTC) - timedelta(days=30)))
73		tracked_repositories = repositories or self._tracked_repositories(actor)
74		if not tracked_repositories:
75		logger.info("git poll skipped for actor=%s because no tracked repositories are configured", actor)
	90	discovered_repositories = repositories or self._repositories_for_actor(actor)
	91	if not discovered_repositories:
	92	logger.info("git poll skipped for actor=%s because no repositories were discovered", actor)
76	93	return GitPollResult(events=[], cursor=datetime.now(tz=UTC).isoformat())
77	94
78	95	events: list[NormalizedEvent] = []
79		for repository in tracked_repositories:
	96	for repository in discovered_repositories:
80	97	owner, repo_name = self._split_repository(actor, repository)
81	98	repo_events = self._fetch_repository_commits(actor=actor, owner=owner, repo_name=repo_name, since=since_dt)
82	99	events.extend(repo_events)
		@@ -84,10 +101,62 @@ class GitIngestionService:
84	101	logger.info("git poll complete for actor=%s normalized_events=%s", actor, len(events))
85	102	return GitPollResult(events=events, cursor=datetime.now(tz=UTC).isoformat())
86	103
87		def _tracked_repositories(self, actor: str) -> list[str]:
88		repositories = self.settings.git_tracked_repositories
	104	def _repositories_for_actor(self, actor: str) -> list[str]:
	105	configured = {
	106	self._canonical_repository_name(actor, repository)
	107	for repository in self.settings.git_tracked_repositories
	108	}
	109	discovered = set(self._discover_owned_repositories(actor))
	110	repositories = sorted(configured \| discovered)
	111	logger.info(
	112	"git repositories selected for actor=%s count=%s configured=%s discovered=%s",
	113	actor,
	114	len(repositories),
	115	len(configured),
	116	len(discovered),
	117	)
89	118	return repositories
90	119
	120	def _discover_owned_repositories(self, actor: str) -> list[str]:
	121	owner = actor.lstrip("~")
	122	repositories: list[str] = []
	123	cursor: str \| None = None
	124
	125	for _ in range(50):
	126	data = self.client.execute(
	127	USER_REPOSITORIES_QUERY,
	128	{"username": owner, "cursor": cursor},
	129	)
	130	user = data.get("user") or {}
	131	repositories_page = user.get("repositories") or {}
	132	results = repositories_page.get("results") or []
	133	cursor = repositories_page.get("cursor")
	134	logger.info(
	135	"git repository discovery actor=%s page_count=%s next_cursor=%s",
	136	actor,
	137	len(results),
	138	bool(cursor),
	139	)
	140
	141	for repository in results:
	142	if not isinstance(repository, dict):
	143	continue
	144	name = repository.get("name")
	145	repository_owner = ((repository.get("owner") or {}).get("canonicalName") or actor).strip()
	146	if not name or not repository_owner:
	147	continue
	148	repositories.append(f"{repository_owner}/{name}")
	149
	150	if not cursor:
	151	break
	152
	153	return repositories
	154
	155	@staticmethod
	156	def _canonical_repository_name(default_actor: str, repository: str) -> str:
	157	owner, repo_name = GitIngestionService._split_repository(default_actor, repository)
	158	return f"~{owner}/{repo_name}"
	159
91	160	@staticmethod
92	161	def _split_repository(default_actor: str, repository: str) -> tuple[str, str]:
93	162	if "/" in repository:

tests/test_ingestion.py +61

		@@ -310,6 +310,67 @@ def test_git_ingestion_normalizes_commit_aliases_and_repository_names(db_session
310	310	assert tracked_repositories == ["~ccleberg/Hutch"]
311	311
312	312
	313	def test_git_ingestion_auto_discovers_owned_repositories(db_session) -> None:
	314	settings = make_settings(
	315	ACTOR_ALIASES_JSON={"~ccleberg": ["cmc@example.com", "Chris Cleberg"]},
	316	GIT_TRACKED_REPOSITORIES=[],
	317	)
	318	client = StubClient(
	319	payloads_by_query={
	320	"query UserRepositories": {
	321	"user": {
	322	"repositories": {
	323	"results": [
	324	{"name": "Hutch", "owner": {"canonicalName": "~ccleberg"}},
	325	],
	326	"cursor": None,
	327	}
	328	}
	329	},
	330	"query RepositoryLog": {
	331	"user": {
	332	"repository": {
	333	"name": "Hutch",
	334	"owner": {"canonicalName": "~ccleberg"},
	335	"log": {
	336	"results": [
	337	{
	338	"id": "abc123",
	339	"shortId": "abc123",
	340	"author": {
	341	"name": "Chris Cleberg",
	342	"email": "cmc@example.com",
	343	"time": "2026-03-30T12:00:00Z",
	344	},
	345	"committer": {
	346	"name": "Chris Cleberg",
	347	"email": "cmc@example.com",
	348	"time": "2026-03-30T12:00:00Z",
	349	},
	350	"message": "Auto-discovered repo commit",
	351	}
	352	],
	353	"cursor": None,
	354	},
	355	}
	356	}
	357	},
	358	}
	359	)
	360
	361	todo_service = TodoIngestionService(
	362	StubClient(payload={"me": {"canonicalName": "~ccleberg"}, "events": {"results": [], "cursor": None}}),
	363	settings,
	364	)
	365	git_service = GitIngestionService(client, settings)
	366	poller = PollerService(todo_service=todo_service, git_service=git_service)
	367
	368	inserted = poller.poll_all(db_session, "~ccleberg")
	369
	370	assert inserted == 1
	371	assert any("query UserRepositories" in call[0] for call in client.calls)
	372
	373
313	374	def test_sync_overlap_reuses_cursor_window_and_suppresses_duplicates(db_session) -> None:
314	375	event = NormalizedEvent(
315	376	service="todo",

Commit 9d88cda282

Commit `9d88cda282`