Commit `01c1b50541`

01c1b50541b0dc1d42cbdaa90052f6b94ceba20c

parent: f0393a0b8d

Verified · cmc

cmc <hello@cleberg.net> · 2026-04-11 23:45 UTC

feat: prioritize recent-window backfill before full history

Layout: unified · split

API.md +16

 - Contribution read endpoints return zero-filled days, so clients do not need to patch missing dates.
 - Public contribution reads also register the actor for background indexing. A first lookup may therefore return an empty graph while the scheduler catches up.
 - Incremental indexing and historical backfill are separate. An actor can be recently indexed without being fully backfilled yet.
+- The service prioritizes a recent visible history window first, then continues deep-history backfill afterward.
 Background polling:
 - This endpoint also registers the actor for background indexing and updates the actor's `last_requested_at` timestamp.
 - The response is always immediate; it does not wait for SourceHut polling to finish.
 - Historical backfill runs in bounded background batches and may take multiple scheduler passes to complete.
+- The recent visible window is prioritized before full-history backfill so clients can show a useful graph sooner.
 Example by year:
   "is_indexed": false,
   "last_polled_at": null,
   "indexing_state": "pending",
+  "is_recent_window_backfilled": false,
+  "recent_backfill_state": "in_progress",
+  "recent_backfill_completed_at": null,
   "is_backfilled": false,
   "backfill_state": "in_progress",
   "backfill_completed_at": null,
 - `is_indexed` boolean: whether the service has already completed at least one successful recent/incremental poll for this actor
 - `last_polled_at` string or `null`: most recent successful poll time, if any
 - `indexing_state` string: one of `pending`, `indexed`, or `error`
+- `is_recent_window_backfilled` boolean: whether the prioritized recent history window has completed backfill
+- `recent_backfill_state` string: one of `pending`, `in_progress`, `completed`, or `error`
+- `recent_backfill_completed_at` string or `null`: when recent-window backfill completed, if it has
 - `is_backfilled` boolean: whether historical backfill has completed for this actor
 - `backfill_state` string: one of `pending`, `in_progress`, `completed`, or `error`
 - `backfill_completed_at` string or `null`: when full historical backfill completed, if it has
 Backfill state semantics:
+- recent-window fields:
+  - represent the prioritized visible-history window for client UX
 - `pending`: the actor has not started historical backfill yet
 - `in_progress`: historical backfill is actively progressing in bounded background batches
 - `completed`: historical backfill has completed for all supported services
   "is_indexed": true,
   "last_polled_at": "2026-04-11T18:05:00Z",
   "indexing_state": "indexed",
+  "is_recent_window_backfilled": true,
+  "recent_backfill_state": "completed",
+  "recent_backfill_completed_at": "2026-04-11T18:02:00Z",
   "is_backfilled": false,
   "backfill_state": "in_progress",
   "backfill_completed_at": null,
 - `is_indexed` boolean
 - `last_polled_at` string or `null`
 - `indexing_state` string
+- `is_recent_window_backfilled` boolean
+- `recent_backfill_state` string
+- `recent_backfill_completed_at` string or `null`
 - `is_backfilled` boolean
 - `backfill_state` string
 - `backfill_completed_at` string or `null`

README.md +8 −1

 ## What It Does
-The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs both recent incremental polling and bounded historical backfill.
+The service collects SourceHut activity from one or more sr.ht GraphQL services, turns those records into a canonical event shape, aggregates activity by day, and returns zero-filled calendar ranges so the client never has to patch missing dates. It performs recent incremental polling, prioritizes a recent visible-history window for faster UX, and then continues bounded historical backfill.
 Example use cases:
   "is_indexed": true,
   "last_polled_at": "2026-04-11T18:05:00Z",
   "indexing_state": "indexed",
+  "is_recent_window_backfilled": true,
+  "recent_backfill_state": "completed",
+  "recent_backfill_completed_at": "2026-04-11T18:02:00Z",
   "is_backfilled": false,
   "backfill_state": "in_progress",
   "backfill_completed_at": null,
   "is_indexed": true,
   "last_polled_at": "2026-04-11T18:05:00Z",
   "indexing_state": "indexed",
+  "is_recent_window_backfilled": true,
+  "recent_backfill_state": "completed",
+  "recent_backfill_completed_at": "2026-04-11T18:02:00Z",
   "is_backfilled": false,
   "backfill_state": "in_progress",
   "backfill_completed_at": null,
 - `git.sr.ht` polling assumes the actor's repositories are discoverable through the SourceHut GraphQL API
 - scheduled polling runs in-process, so it is not a distributed scheduler
 - newly requested actors are indexed asynchronously, so the first public read may be empty until a scheduler or manual poll runs
+- the recent visible-history window is prioritized first, but deep-history backfill can still take many scheduler passes for active users
 - full historical backfill can take many scheduler passes for active users because it runs in bounded batches
 - alias management is config-driven; there is no alias CRUD API yet
 - current deployment model is trusted-operator V1, not a public multi-tenant service

alembic/versions/20260411_0004_recent_backfill_scope.py added +89

@@ -0,0 +1,89 @@
	1	"""recent backfill scope and actor fields"""
	2
	3	from __future__ import annotations
	4
	5	from alembic import op
	6	import sqlalchemy as sa
	7	from sqlalchemy import inspect
	8
	9
	10	revision = "20260411_0004"
	11	down_revision = "20260411_0003"
	12	branch_labels = None
	13	depends_on = None
	14
	15
	16	def _table_names() -> set[str]:
	17	return set(inspect(op.get_bind()).get_table_names())
	18
	19
	20	def _column_names(table_name: str) -> set[str]:
	21	return {column["name"] for column in inspect(op.get_bind()).get_columns(table_name)}
	22
	23
	24	def _service_backfill_needs_upgrade() -> bool:
	25	if "service_backfill_states" not in _table_names():
	26	return False
	27	columns = _column_names("service_backfill_states")
	28	if "scope" not in columns:
	29	return True
	30	unique_constraints = {
	31	constraint["name"]
	32	for constraint in inspect(op.get_bind()).get_unique_constraints("service_backfill_states")
	33	}
	34	return "uq_service_backfill_state_actor_service_scope" not in unique_constraints
	35
	36
	37	def _upgrade_service_backfill_states() -> None:
	38	op.execute(
	39	sa.text(
	40	"""
	41	CREATE TABLE service_backfill_states__alembic_new (
	42	id INTEGER NOT NULL PRIMARY KEY,
	43	actor VARCHAR(255) NOT NULL,
	44	service VARCHAR(32) NOT NULL,
	45	scope VARCHAR(16) NOT NULL,
	46	cursor_json JSON,
	47	status VARCHAR(32) NOT NULL,
	48	started_at DATETIME,
	49	completed_at DATETIME,
	50	last_error TEXT,
	51	updated_at DATETIME NOT NULL,
	52	CONSTRAINT uq_service_backfill_state_actor_service_scope UNIQUE (actor, service, scope)
	53	)
	54	"""
	55	)
	56	)
	57	op.execute(
	58	sa.text(
	59	"""
	60	INSERT INTO service_backfill_states__alembic_new
	61	(id, actor, service, scope, cursor_json, status, started_at, completed_at, last_error, updated_at)
	62	SELECT id, actor, service, 'full', cursor_json, status, started_at, completed_at, last_error, updated_at
	63	FROM service_backfill_states
	64	"""
	65	)
	66	)
	67	op.execute(sa.text("DROP TABLE service_backfill_states"))
	68	op.execute(sa.text("ALTER TABLE service_backfill_states__alembic_new RENAME TO service_backfill_states"))
	69
	70
	71	def upgrade() -> None:
	72	if "tracked_actors" in _table_names():
	73	columns = _column_names("tracked_actors")
	74	if "recent_backfill_status" not in columns:
	75	op.add_column("tracked_actors", sa.Column("recent_backfill_status", sa.String(length=32), nullable=True))
	76	op.execute(sa.text("UPDATE tracked_actors SET recent_backfill_status = 'pending' WHERE recent_backfill_status IS NULL"))
	77	if "recent_backfill_started_at" not in columns:
	78	op.add_column("tracked_actors", sa.Column("recent_backfill_started_at", sa.DateTime(timezone=True), nullable=True))
	79	if "recent_backfill_completed_at" not in columns:
	80	op.add_column("tracked_actors", sa.Column("recent_backfill_completed_at", sa.DateTime(timezone=True), nullable=True))
	81	if "last_recent_backfill_error" not in columns:
	82	op.add_column("tracked_actors", sa.Column("last_recent_backfill_error", sa.Text(), nullable=True))
	83
	84	if _service_backfill_needs_upgrade():
	85	_upgrade_service_backfill_states()
	86
	87
	88	def downgrade() -> None:
	89	pass

src/srht_contrib/jobs/poller.py +128 −45

	@@ -18,6 +18,9 @@ from srht_contrib.utils.repositories import canonicalize_repository_name
18		18
19	logger = logging.getLogger(__name__)	19	logger = logging.getLogger(__name__)
20	SYNC_OVERLAP = timedelta(hours=24)	20	SYNC_OVERLAP = timedelta(hours=24)
		21	RECENT_BACKFILL_DAYS = 365
		22	RECENT_BACKFILL_BATCHES_PER_SERVICE = 5
		23	FULL_BACKFILL_BATCHES_PER_SERVICE = 1
21		24
22		25
23	class PollerService:	26	class PollerService:
	@@ -63,7 +66,12 @@ class PollerService:
63	def track_actor_request(self, db: Session, actor: str, *, update_last_requested: bool = True) -> TrackedActor:	66	def track_actor_request(self, db: Session, actor: str, *, update_last_requested: bool = True) -> TrackedActor:
64	tracked_actor = db.scalar(select(TrackedActor).where(TrackedActor.actor == actor))	67	tracked_actor = db.scalar(select(TrackedActor).where(TrackedActor.actor == actor))
65	if tracked_actor is None:	68	if tracked_actor is None:
66	tracked_actor = TrackedActor(actor=actor, is_active=True, backfill_status="pending")	69	tracked_actor = TrackedActor(
		70	actor=actor,
		71	is_active=True,
		72	recent_backfill_status="pending",
		73	backfill_status="pending",
		74	)
67	db.add(tracked_actor)	75	db.add(tracked_actor)
68		76
69	tracked_actor.is_active = True	77	tracked_actor.is_active = True
	@@ -177,32 +185,98 @@ class PollerService:
177		185
178	def _run_backfill_batches(self, db: Session, actor: str) -> int:	186	def _run_backfill_batches(self, db: Session, actor: str) -> int:
179	tracked_actor = self.track_actor_request(db, actor, update_last_requested=False)	187	tracked_actor = self.track_actor_request(db, actor, update_last_requested=False)
180	if tracked_actor.backfill_status == "completed":	188	if tracked_actor.recent_backfill_status == "completed" and tracked_actor.backfill_status == "completed":
181	return 0	189	return 0
182		190
183	if tracked_actor.backfill_started_at is None:	191	now = datetime.now(tz=UTC)
184	tracked_actor.backfill_started_at = datetime.now(tz=UTC)	192	recent_since = now - timedelta(days=RECENT_BACKFILL_DAYS)
185	tracked_actor.backfill_status = "in_progress"
186	tracked_actor.last_backfill_error = None
187	db.add(tracked_actor)	193	db.add(tracked_actor)
188	db.flush()	194	db.flush()
189	total_inserted = 0	195	total_inserted = 0
190		196
191	services = [	197	recent_services = [
192	(self.todo_service.service_name, self.todo_service.fetch_backfill_batch),	198	(self.todo_service.service_name, self.todo_service.fetch_recent_backfill_batch),
193	(self.git_service.service_name, self.git_service.fetch_backfill_batch),	199	(self.git_service.service_name, self.git_service.fetch_recent_backfill_batch),
194	]	200	]
195	all_complete = True	201	if tracked_actor.recent_backfill_status != "completed":
		202	if tracked_actor.recent_backfill_started_at is None:
		203	tracked_actor.recent_backfill_started_at = now
		204	tracked_actor.recent_backfill_status = "in_progress"
		205	tracked_actor.last_recent_backfill_error = None
		206	total_inserted += self._run_backfill_scope(
		207	db,
		208	actor=actor,
		209	scope="recent",
		210	services=recent_services,
		211	batches_per_service=RECENT_BACKFILL_BATCHES_PER_SERVICE,
		212	since=recent_since,
		213	)
		214	recent_statuses = db.scalars(
		215	select(ServiceBackfillState.status)
		216	.where(ServiceBackfillState.actor == actor)
		217	.where(ServiceBackfillState.scope == "recent")
		218	).all()
		219	if recent_statuses and all(status == "completed" for status in recent_statuses):
		220	tracked_actor.recent_backfill_status = "completed"
		221	tracked_actor.recent_backfill_completed_at = datetime.now(tz=UTC)
		222	tracked_actor.last_recent_backfill_error = None
		223	db.add(tracked_actor)
		224	db.flush()
		225
		226	if tracked_actor.recent_backfill_status == "completed" and tracked_actor.backfill_status != "completed":
		227	if tracked_actor.backfill_started_at is None:
		228	tracked_actor.backfill_started_at = now
		229	tracked_actor.backfill_status = "in_progress"
		230	tracked_actor.last_backfill_error = None
		231	total_inserted += self._run_backfill_scope(
		232	db,
		233	actor=actor,
		234	scope="full",
		235	services=[
		236	(self.todo_service.service_name, self.todo_service.fetch_backfill_batch),
		237	(self.git_service.service_name, self.git_service.fetch_backfill_batch),
		238	],
		239	batches_per_service=FULL_BACKFILL_BATCHES_PER_SERVICE,
		240	since=None,
		241	)
		242	full_statuses = db.scalars(
		243	select(ServiceBackfillState.status)
		244	.where(ServiceBackfillState.actor == actor)
		245	.where(ServiceBackfillState.scope == "full")
		246	).all()
		247	if full_statuses and all(status == "completed" for status in full_statuses):
		248	tracked_actor.backfill_status = "completed"
		249	tracked_actor.backfill_completed_at = datetime.now(tz=UTC)
		250	tracked_actor.last_backfill_error = None
		251	db.add(tracked_actor)
		252	db.flush()
		253
		254	return total_inserted
		255
		256	def _run_backfill_scope(
		257	self,
		258	db: Session,
		259	*,
		260	actor: str,
		261	scope: str,
		262	services,
		263	batches_per_service: int,
		264	since: datetime \| None,
		265	) -> int:
		266	tracked_actor = self.track_actor_request(db, actor, update_last_requested=False)
		267	total_inserted = 0
196	for service_name, fetcher in services:	268	for service_name, fetcher in services:
197	state = db.scalar(	269	state = db.scalar(
198	select(ServiceBackfillState)	270	select(ServiceBackfillState)
199	.where(ServiceBackfillState.actor == actor)	271	.where(ServiceBackfillState.actor == actor)
200	.where(ServiceBackfillState.service == service_name)	272	.where(ServiceBackfillState.service == service_name)
		273	.where(ServiceBackfillState.scope == scope)
201	)	274	)
202	if state is None:	275	if state is None:
203	state = ServiceBackfillState(	276	state = ServiceBackfillState(
204	actor=actor,	277	actor=actor,
205	service=service_name,	278	service=service_name,
		279	scope=scope,
206	cursor_json=None,	280	cursor_json=None,
207	status="pending",	281	status="pending",
208	started_at=None,	282	started_at=None,
	@@ -216,47 +290,56 @@ class PollerService:
216	if state.status == "completed":	290	if state.status == "completed":
217	continue	291	continue
218		292
219	all_complete = False
220	if state.started_at is None:	293	if state.started_at is None:
221	state.started_at = datetime.now(tz=UTC)	294	state.started_at = datetime.now(tz=UTC)
222	state.status = "in_progress"	295	state.status = "in_progress"
223	state.updated_at = datetime.now(tz=UTC)	296	state.updated_at = datetime.now(tz=UTC)
224	try:	297
225	result = fetcher(actor=actor, cursor_state=state.cursor_json)	298	for _ in range(batches_per_service):
226	inserted = self._insert_events(db, result.events)	299	try:
227	total_inserted += inserted	300	if since is None:
228	state.cursor_json = copy.deepcopy(result.cursor_state)	301	result = fetcher(actor=actor, cursor_state=state.cursor_json)
229	state.last_error = None	302	else:
230	state.updated_at = datetime.now(tz=UTC)	303	result = fetcher(actor=actor, cursor_state=state.cursor_json, since=since)
231	if result.complete:	304	inserted = self._insert_events(db, result.events)
232	state.status = "completed"	305	total_inserted += inserted
233	state.completed_at = datetime.now(tz=UTC)	306	state.cursor_json = copy.deepcopy(result.cursor_state)
234	logger.info("Backfill complete for service=%s actor=%s inserted=%s", service_name, actor, inserted)	307	state.last_error = None
235	else:	308	state.updated_at = datetime.now(tz=UTC)
236	logger.info("Backfill batch complete for service=%s actor=%s inserted=%s", service_name, actor, inserted)	309	if result.complete:
237	except Exception as exc:	310	state.status = "completed"
238	state.status = "error"	311	state.completed_at = datetime.now(tz=UTC)
239	state.last_error = str(exc)	312	logger.info(
240	state.updated_at = datetime.now(tz=UTC)	313	"Backfill complete for scope=%s service=%s actor=%s inserted=%s",
241	tracked_actor.backfill_status = "error"	314	scope,
242	tracked_actor.last_backfill_error = str(exc)	315	service_name,
243	db.add(state)	316	actor,
244	db.add(tracked_actor)	317	inserted,
245	db.flush()	318	)
246	raise	319	break
		320	logger.info(
		321	"Backfill batch complete for scope=%s service=%s actor=%s inserted=%s",
		322	scope,
		323	service_name,
		324	actor,
		325	inserted,
		326	)
		327	except Exception as exc:
		328	state.status = "error"
		329	state.last_error = str(exc)
		330	state.updated_at = datetime.now(tz=UTC)
		331	if scope == "recent":
		332	tracked_actor.recent_backfill_status = "error"
		333	tracked_actor.last_recent_backfill_error = str(exc)
		334	else:
		335	tracked_actor.backfill_status = "error"
		336	tracked_actor.last_backfill_error = str(exc)
		337	db.add(state)
		338	db.add(tracked_actor)
		339	db.flush()
		340	raise
247		341
248	db.add(state)	342	db.add(state)
249	db.flush()	343	db.flush()
250		344
251	completed = db.scalars(
252	select(ServiceBackfillState.status).where(ServiceBackfillState.actor == actor)
253	).all()
254	if completed and all(status == "completed" for status in completed):
255	tracked_actor.backfill_status = "completed"
256	tracked_actor.backfill_completed_at = datetime.now(tz=UTC)
257	tracked_actor.last_backfill_error = None
258	elif tracked_actor.backfill_status != "error":
259	tracked_actor.backfill_status = "in_progress"
260	db.add(tracked_actor)
261	db.flush()
262	return total_inserted	345	return total_inserted

src/srht_contrib/models.py +6 −1

     last_polled_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
     last_poll_status: Mapped[str | None] = mapped_column(String(32), nullable=True)
     last_poll_error: Mapped[str | None] = mapped_column(Text, nullable=True)
+    recent_backfill_status: Mapped[str] = mapped_column(String(32), nullable=False, default="pending")
+    recent_backfill_started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    recent_backfill_completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
+    last_recent_backfill_error: Mapped[str | None] = mapped_column(Text, nullable=True)
     backfill_status: Mapped[str] = mapped_column(String(32), nullable=False, default="pending")
     backfill_started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
     backfill_completed_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)
 class ServiceBackfillState(Base):
     __tablename__ = "service_backfill_states"
-    __table_args__ = (UniqueConstraint("actor", "service", name="uq_service_backfill_state_actor_service"),)
+    __table_args__ = (UniqueConstraint("actor", "service", "scope", name="uq_service_backfill_state_actor_service_scope"),)
     id: Mapped[int] = mapped_column(Integer, primary_key=True)
     actor: Mapped[str] = mapped_column(String(255), nullable=False)
     service: Mapped[str] = mapped_column(String(32), nullable=False)
+    scope: Mapped[str] = mapped_column(String(16), nullable=False, default="full")
     cursor_json: Mapped[dict | None] = mapped_column(JSON, nullable=True)
     status: Mapped[str] = mapped_column(String(32), nullable=False, default="pending")
     started_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)

src/srht_contrib/schemas.py +3

     is_indexed: bool
     last_polled_at: datetime | None = None
     indexing_state: Literal["pending", "indexed", "error"]
+    is_recent_window_backfilled: bool = False
+    recent_backfill_state: Literal["pending", "in_progress", "completed", "error"] = "pending"
+    recent_backfill_completed_at: datetime | None = None
     is_backfilled: bool = False
     backfill_state: Literal["pending", "in_progress", "completed", "error"] = "pending"
     backfill_completed_at: datetime | None = None

src/srht_contrib/services/aggregator.py +9

             is_indexed=calendar.is_indexed,
             last_polled_at=calendar.last_polled_at,
             indexing_state=calendar.indexing_state,
+            is_recent_window_backfilled=calendar.is_recent_window_backfilled,
+            recent_backfill_state=calendar.recent_backfill_state,
+            recent_backfill_completed_at=calendar.recent_backfill_completed_at,
+            is_backfilled=calendar.is_backfilled,
+            backfill_state=calendar.backfill_state,
+            backfill_completed_at=calendar.backfill_completed_at,
         )
     def _index_metadata(self, db: Session, actor: str) -> ContributionIndexMetadata:
             is_indexed=is_indexed,
             last_polled_at=tracked_actor.last_polled_at if tracked_actor is not None else None,
             indexing_state=indexing_state,
+            is_recent_window_backfilled=(tracked_actor.recent_backfill_status == "completed") if tracked_actor is not None else False,
+            recent_backfill_state=(tracked_actor.recent_backfill_status if tracked_actor is not None else "pending"),
+            recent_backfill_completed_at=tracked_actor.recent_backfill_completed_at if tracked_actor is not None else None,
             is_backfilled=(tracked_actor.backfill_status == "completed") if tracked_actor is not None else False,
             backfill_state=(tracked_actor.backfill_status if tracked_actor is not None else "pending"),
             backfill_completed_at=tracked_actor.backfill_completed_at if tracked_actor is not None else None,

src/srht_contrib/services/git.py +31 −8

         return repositories
     def fetch_backfill_batch(self, actor: str, cursor_state: dict | None = None) -> BackfillBatchResult:
+        return self._fetch_backfill_batch(actor, cursor_state, since=None)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return self._fetch_backfill_batch(actor, cursor_state, since=since)
+    def _fetch_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None,
+        *,
+        since: datetime | None,
+    ) -> BackfillBatchResult:
         state = {
             "discovery_cursor": None,
             "discovery_complete": False,
         log_page = repository.get("log") or {}
         commits = log_page.get("results") or []
         next_cursor = log_page.get("cursor")
-        events = [
+        events: list[NormalizedEvent] = []
-            normalized
+        stop_repository = False
-            for commit in commits
+        for commit in commits:
-            if isinstance(commit, dict)
+            if not isinstance(commit, dict):
-            for normalized in [self._normalize_commit(actor=actor, repo_name=repo_name, commit=commit)]
+                continue
-            if normalized is not None
+            commit_time = parse_datetime((commit.get("author") or {}).get("time"))
-        ]
+            if since is not None and commit_time < since:
+                stop_repository = True
+                break
+            normalized = self._normalize_commit(actor=actor, repo_name=repo_name, commit=commit)
+            if normalized is not None:
+                events.append(normalized)
         logger.info(
             "git backfill actor=%s repository=%s commits=%s next_cursor=%s",
             actor,
             len(commits),
             bool(next_cursor),
         )
-        if next_cursor:
+        if next_cursor and not stop_repository:
             state["current_repository"]["cursor"] = next_cursor
         else:
             state["current_repository"] = None

src/srht_contrib/services/todo.py +37 −5

         return TodoPollResult(events=tracker_events, cursor=cursor_time)
     def fetch_backfill_batch(self, actor: str, cursor_state: dict | None = None) -> BackfillBatchResult:
+        return self._fetch_backfill_batch(actor, cursor_state, since=None)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return self._fetch_backfill_batch(actor, cursor_state, since=since)
+    def _fetch_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None,
+        *,
+        since: datetime | None,
+    ) -> BackfillBatchResult:
         state = {
             "trackers_cursor": None,
             "tracker_queue": [],
             tracker = data.get("tracker") or {}
             tickets_page = tracker.get("tickets") or {}
             tickets = tickets_page.get("results") or []
-            current_tracker["tickets_cursor"] = tickets_page.get("cursor")
+            next_tickets_cursor = tickets_page.get("cursor")
-            current_tracker["pending_tickets"].extend(
+            stop_tracker_paging = False
-                [{"id": int(ticket["id"]), "ref": str(ticket.get("ref") or ticket["id"])} for ticket in tickets if isinstance(ticket, dict)]
+            for ticket in tickets:
-            )
+                if not isinstance(ticket, dict):
+                    continue
+                if since is not None:
+                    updated = parse_datetime(ticket["updated"])
+                    if updated < since:
+                        stop_tracker_paging = True
+                        continue
+                current_tracker["pending_tickets"].append(
+                    {"id": int(ticket["id"]), "ref": str(ticket.get("ref") or ticket["id"])}
+                )
+            current_tracker["tickets_cursor"] = None if stop_tracker_paging else next_tickets_cursor
             logger.info(
                 "todo backfill tracker=%s ticket page count=%s next_cursor=%s pending_tickets=%s",
                 current_tracker.get("name") or current_tracker.get("id"),
         )
         events: list[NormalizedEvent] = []
+        stop_ticket_paging = False
         for event in page_events:
             if not isinstance(event, dict):
                 continue
                 "tracker": {"name": current_tracker.get("name")},
             }
             occurred_at = parse_datetime(event["created"])
+            if since is not None and occurred_at < since:
+                stop_ticket_paging = True
+                continue
             for change in event.get("changes") or []:
                 if not isinstance(change, dict):
                     continue
                 if normalized is not None:
                     events.append(normalized)
-        if next_cursor:
+        if next_cursor and not stop_ticket_paging:
             state["current_ticket"]["cursor"] = next_cursor
         else:
             state["current_ticket"] = None

tests/test_contributions_api.py +7

     assert response.status_code == 200
     assert response.json()["is_indexed"] is True
     assert response.json()["indexing_state"] == "indexed"
+    assert response.json()["is_recent_window_backfilled"] is False
+    assert response.json()["recent_backfill_state"] == "pending"
     assert response.json()["days"] == [
         {"date": "2026-03-28", "count": 0, "score": 0.0},
         {"date": "2026-03-29", "count": 0, "score": 0.0},
     assert response.status_code == 200
     assert response.json()["is_indexed"] is False
     assert response.json()["indexing_state"] == "pending"
+    assert response.json()["is_recent_window_backfilled"] is False
+    assert response.json()["recent_backfill_state"] == "pending"
+    assert response.json()["recent_backfill_completed_at"] is None
     assert response.json()["is_backfilled"] is False
     assert response.json()["backfill_state"] == "pending"
     assert response.json()["backfill_completed_at"] is None
     assert response.json()["current_streak"] == 2
     assert response.json()["is_indexed"] is True
     assert response.json()["indexing_state"] == "indexed"
+    assert response.json()["is_recent_window_backfilled"] is False
+    assert response.json()["recent_backfill_state"] == "pending"
     assert response.json()["is_backfilled"] is False

tests/test_ingestion.py +57 −4

     def fetch_backfill_batch(self, actor: str, cursor_state: dict | None = None) -> BackfillBatchResult:
         return BackfillBatchResult(events=[], cursor_state=None, complete=True)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return BackfillBatchResult(events=[], cursor_state=None, complete=True)
 class EmptyGitService:
     service_name = "git"
     def fetch_backfill_batch(self, actor: str, cursor_state: dict | None = None) -> BackfillBatchResult:
         return BackfillBatchResult(events=[], cursor_state=None, complete=True)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return BackfillBatchResult(events=[], cursor_state=None, complete=True)
 class BackfillingTodoService:
     service_name = "todo"
         )
         return BackfillBatchResult(events=[event], cursor_state=None, complete=True)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return self.fetch_backfill_batch(actor, cursor_state)
 class QueueShrinkingTodoService:
     service_name = "todo"
         state["tracker_queue"].pop(0)
         return BackfillBatchResult(events=[], cursor_state=state, complete=False)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return self.fetch_backfill_batch(actor, cursor_state)
 class QueueShrinkingGitService:
     service_name = "git"
         state["repository_queue"].pop(0)
         return BackfillBatchResult(events=[], cursor_state=state, complete=False)
+    def fetch_recent_backfill_batch(
+        self,
+        actor: str,
+        cursor_state: dict | None = None,
+        *,
+        since: datetime,
+    ) -> BackfillBatchResult:
+        return self.fetch_backfill_batch(actor, cursor_state)
 def make_settings(**overrides) -> Settings:
     values = {
     tracked_actor = db_session.scalar(select(TrackedActor).where(TrackedActor.actor == "~ccleberg"))
     service_states = db_session.scalars(
-        select(ServiceBackfillState).where(ServiceBackfillState.actor == "~ccleberg").order_by(ServiceBackfillState.service)
+        select(ServiceBackfillState)
+        .where(ServiceBackfillState.actor == "~ccleberg")
+        .order_by(ServiceBackfillState.scope, ServiceBackfillState.service)
     ).all()
     assert inserted == 1
     assert tracked_actor is not None
+    assert tracked_actor.recent_backfill_status == "completed"
+    assert tracked_actor.recent_backfill_completed_at is not None
     assert tracked_actor.backfill_status == "completed"
     assert tracked_actor.backfill_completed_at is not None
-    assert [state.service for state in service_states] == ["git", "todo"]
+    assert [f"{state.scope}:{state.service}" for state in service_states] == ["full:git", "full:todo", "recent:git", "recent:todo"]
     assert all(state.status == "completed" for state in service_states)
     first_states = {
         state.service: state.cursor_json
         for state in db_session.scalars(
-            select(ServiceBackfillState).where(ServiceBackfillState.actor == "~ccleberg")
+            select(ServiceBackfillState)
+            .where(ServiceBackfillState.actor == "~ccleberg")
+            .where(ServiceBackfillState.scope == "full")
         ).all()
     }
     second_states = {
         state.service: state.cursor_json
         for state in db_session.scalars(
-            select(ServiceBackfillState).where(ServiceBackfillState.actor == "~ccleberg")
+            select(ServiceBackfillState)
+            .where(ServiceBackfillState.actor == "~ccleberg")
+            .where(ServiceBackfillState.scope == "full")
         ).all()
     }

Commit 01c1b50541

Commit `01c1b50541`