Commit `6289a28296`

6289a28296bd0863d373bb47c1c85038f4bc6a2b

parent: 79182c1290

Verified · cmc

cmc <hello@cleberg.net> · 2026-05-07 04:19 UTC

feat: improve scheduled indexing throughput and capped backoff

Layout: unified · split

.env.example +6 −1

 GIT_SRHT_ENDPOINT=https://git.sr.ht/query
 DATABASE_URL=sqlite:///./srht_contrib.db
 DEFAULT_ACTOR=~your-user
-POLL_INTERVAL_SECONDS=900
+POLL_INTERVAL_SECONDS=300
+DISCOVERY_BATCH_SIZE=20
+INDEXED_ACTOR_REPOLL_SECONDS=21600
+DISCOVERY_ERROR_BACKOFF_SECONDS=3600
+DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600
+SRHT_REQUEST_DELAY_SECONDS=0.5
 # Optional JSON object. Example:
 # {"~your-user":["you@example.com","Your Name"]}
 ACTOR_ALIASES_JSON={}

API.md +1

	@@ -54,6 +54,7 @@ Background polling:
54	- The scheduler always seeds `DEFAULT_ACTOR` as a known actor.	54	- The scheduler always seeds `DEFAULT_ACTOR` as a known actor.
55	- Public contribution reads register additional actors for later background polling.	55	- Public contribution reads register additional actors for later background polling.
56	- The scheduler only processes due actors, up to `DISCOVERY_BATCH_SIZE` per pass.	56	- The scheduler only processes due actors, up to `DISCOVERY_BATCH_SIZE` per pass.
		57	- Scheduled first indexing skips bounded one-year backfill work, then drains backfill in later scheduled passes so newly requested actors become indexed sooner.
57	- Manual polling remains available through `POST /api/contributions/poll`.	58	- Manual polling remains available through `POST /api/contributions/poll`.
58		59
59	Repository names:	60	Repository names:

README.md +7 −5

 - `DISCOVERY_BATCH_SIZE`: max number of due actors to process per scheduler pass
 - `INDEXED_ACTOR_REPOLL_SECONDS`: how long to wait before re-polling an already indexed actor
 - `DISCOVERY_ERROR_BACKOFF_SECONDS`: base retry delay after a failed scheduled poll
+- `DISCOVERY_ERROR_BACKOFF_MAX_SECONDS`: maximum retry delay after repeated scheduled poll failures
 - `ACTOR_ALIASES_JSON`: optional JSON object for actor/email/display-name alias mapping
 - `GIT_TRACKED_REPOSITORIES`: optional JSON array of repository names or `owner/repo` strings to union into git polling
 GIT_SRHT_ENDPOINT=https://git.sr.ht/query
 DATABASE_URL=sqlite:///./srht_contrib.db
 DEFAULT_ACTOR=~your-user
-POLL_INTERVAL_SECONDS=900
+POLL_INTERVAL_SECONDS=300
-DISCOVERY_BATCH_SIZE=5
+DISCOVERY_BATCH_SIZE=20
 INDEXED_ACTOR_REPOLL_SECONDS=21600
 DISCOVERY_ERROR_BACKOFF_SECONDS=3600
+DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=21600
 ACTOR_ALIASES_JSON={"~your-user":["you@example.com","Your Name"]}
 GIT_TRACKED_REPOSITORIES=["your-repo","~your-user/your-site"]
 ```
 Scheduled polling only runs when `ENABLE_SCHEDULER=true`. The scheduler seeds `DEFAULT_ACTOR` as an initial known actor, runs one poll immediately at startup, and public contribution reads register additional actors for later background polling and one-year backfill.
-The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to `DISCOVERY_BATCH_SIZE` per run, then reschedules indexed actors with `INDEXED_ACTOR_REPOLL_SECONDS` and failed actors with backoff based on `DISCOVERY_ERROR_BACKOFF_SECONDS`.
+The scheduler now drains actors gradually instead of polling every tracked actor on every pass. It only claims due actors, up to `DISCOVERY_BATCH_SIZE` per run, performs a fast first indexing pass before bounded one-year backfill work, then reschedules indexed actors with `INDEXED_ACTOR_REPOLL_SECONDS` and failed actors with capped backoff based on `DISCOVERY_ERROR_BACKOFF_SECONDS`.
 Clients can explicitly signal that a public contribution read is for the signed-in user's own graph by sending `prioritize_self=true` on the read request. That temporarily boosts the actor to the front of the due queue for the next indexing pass, then clears the boost after the poll completes.
 To durably queue a large username list without polling it immediately:
 ```bash
-srht-enqueue-actors srht_usernames.txt --stagger-seconds 300
+srht-enqueue-actors srht_usernames.txt --stagger-seconds 60
 ```
-This command stores usernames in `tracked_actors`, marks them queued, and spaces out their first eligible poll time. With `--stagger-seconds 300`, a file of 15,771 users will be spread across roughly 54.8 days before becoming due for first poll.
+This command stores usernames in `tracked_actors`, marks them queued, and spaces out their first eligible poll time. With `--stagger-seconds 60`, a file of 15,771 users will be spread across roughly 11 days before becoming due for first poll.
 For `git.sr.ht`, owned repositories are auto-discovered for the actor. `GIT_TRACKED_REPOSITORIES` can still be used to union in extra repositories. Entries may be either:

compose.yml +3 −2

       GIT_SRHT_ENDPOINT: ${GIT_SRHT_ENDPOINT:-https://git.sr.ht/query}
       DATABASE_URL: sqlite:////data/srht_contrib.db
       DEFAULT_ACTOR: ${DEFAULT_ACTOR}
-      POLL_INTERVAL_SECONDS: ${POLL_INTERVAL_SECONDS:-900}
+      POLL_INTERVAL_SECONDS: ${POLL_INTERVAL_SECONDS:-300}
-      DISCOVERY_BATCH_SIZE: ${DISCOVERY_BATCH_SIZE:-5}
+      DISCOVERY_BATCH_SIZE: ${DISCOVERY_BATCH_SIZE:-20}
       INDEXED_ACTOR_REPOLL_SECONDS: ${INDEXED_ACTOR_REPOLL_SECONDS:-21600}
       DISCOVERY_ERROR_BACKOFF_SECONDS: ${DISCOVERY_ERROR_BACKOFF_SECONDS:-3600}
+      DISCOVERY_ERROR_BACKOFF_MAX_SECONDS: ${DISCOVERY_ERROR_BACKOFF_MAX_SECONDS:-21600}
       SRHT_REQUEST_DELAY_SECONDS: ${SRHT_REQUEST_DELAY_SECONDS:-0.5}
       ACTOR_ALIASES_JSON: ${ACTOR_ALIASES_JSON:-{}}
       GIT_TRACKED_REPOSITORIES: ${GIT_TRACKED_REPOSITORIES:-[]}

src/srht_contrib/config.py +3 −2

         alias="DATABASE_URL",
     )
     default_actor: str = Field(default="~unknown", alias="DEFAULT_ACTOR")
-    poll_interval_seconds: int = Field(default=900, alias="POLL_INTERVAL_SECONDS")
+    poll_interval_seconds: int = Field(default=300, alias="POLL_INTERVAL_SECONDS")
     sync_overlap_hours: int = Field(default=1, alias="SYNC_OVERLAP_HOURS")
     srht_request_delay_seconds: float = Field(default=0.5, alias="SRHT_REQUEST_DELAY_SECONDS")
     sqlite_busy_timeout_seconds: float = Field(default=30.0, alias="SQLITE_BUSY_TIMEOUT_SECONDS")
-    discovery_batch_size: int = Field(default=5, alias="DISCOVERY_BATCH_SIZE")
+    discovery_batch_size: int = Field(default=20, alias="DISCOVERY_BATCH_SIZE")
     indexed_actor_repoll_seconds: int = Field(default=21600, alias="INDEXED_ACTOR_REPOLL_SECONDS")
     discovery_error_backoff_seconds: int = Field(default=3600, alias="DISCOVERY_ERROR_BACKOFF_SECONDS")
+    discovery_error_backoff_max_seconds: int = Field(default=21600, alias="DISCOVERY_ERROR_BACKOFF_MAX_SECONDS")
     git_repo_discovery_ttl_seconds: int = Field(default=3600, alias="GIT_REPO_DISCOVERY_TTL_SECONDS")
     actor_aliases_json: dict[str, list[str]] = Field(
         default_factory=dict,

src/srht_contrib/jobs/poller.py +49 −7

         self.settings = settings
         self._sync_overlap = timedelta(hours=settings.sync_overlap_hours)
-    def poll_all(self, db: Session, actor: str) -> int:
+    def poll_all(self, db: Session, actor: str, *, run_backfill: bool = True) -> int:
         self.track_actor_request(db, actor, update_last_requested=False)
         try:
             inserted = self._poll_actor(db, actor)
             raise
         self._update_tracked_actor_poll_state(db, actor, status="indexed", error=None)
-        inserted += self._run_backfill_batches(db, actor)
+        if run_backfill:
+            inserted += self._run_backfill_batches(db, actor)
         db.commit()
         return inserted
         results: dict[str, int] = {}
         actors = db.scalars(
-            select(TrackedActor.actor)
+            select(TrackedActor)
             .where(TrackedActor.is_active.is_(True))
             .where(
                 or_(
             )
             .limit(self.settings.discovery_batch_size)
         ).all()
-        for actor in actors:
+        for tracked_actor in actors:
+            actor = tracked_actor.actor
+            run_backfill_only = (
+                tracked_actor.last_polled_at is not None
+                and tracked_actor.recent_backfill_status != "completed"
+            )
             claimed_actor = self.track_actor_request(db, actor, update_last_requested=False)
             claimed_actor.discovery_state = "in_progress"
             claimed_actor.last_claimed_at = datetime.now(tz=UTC)
             db.add(claimed_actor)
             db.commit()
             try:
-                results[actor] = self.poll_all(db, actor)
+                if run_backfill_only:
+                    results[actor] = self.poll_recent_backfill(db, actor)
+                else:
+                    results[actor] = self.poll_all(db, actor, run_backfill=False)
+                    self._schedule_pending_backfill_or_repoll(db, actor)
+                    db.commit()
             except SourceHutClientError:
                 logger.exception("Scheduled poll failed for actor=%s", actor)
             except Exception:
             tracked_actor.last_polled_at = now
             tracked_actor.next_poll_after = now + timedelta(seconds=self.settings.indexed_actor_repoll_seconds)
             tracked_actor.priority_boosted_at = None
+            tracked_actor.poll_attempts = 0
         elif status == "error":
             tracked_actor.discovery_state = "error"
-            tracked_actor.next_poll_after = now + timedelta(
+            backoff_seconds = min(
-                seconds=self.settings.discovery_error_backoff_seconds * max(tracked_actor.poll_attempts, 1)
+                self.settings.discovery_error_backoff_seconds * max(tracked_actor.poll_attempts, 1),
+                self.settings.discovery_error_backoff_max_seconds,
             )
+            tracked_actor.next_poll_after = now + timedelta(seconds=backoff_seconds)
             tracked_actor.priority_boosted_at = None
         db.add(tracked_actor)
         db.flush()
+    def poll_recent_backfill(self, db: Session, actor: str) -> int:
+        try:
+            inserted = self._run_backfill_batches(db, actor)
+        except Exception as exc:
+            db.rollback()
+            self._update_tracked_actor_poll_state(db, actor, status="error", error=str(exc))
+            db.commit()
+            raise
+        self._schedule_pending_backfill_or_repoll(db, actor)
+        db.commit()
+        return inserted
+    def _schedule_pending_backfill_or_repoll(self, db: Session, actor: str) -> None:
+        tracked_actor = self.track_actor_request(db, actor, update_last_requested=False)
+        now = datetime.now(tz=UTC)
+        tracked_actor.discovery_state = "indexed"
+        tracked_actor.last_poll_status = "indexed"
+        tracked_actor.last_poll_error = None
+        tracked_actor.priority_boosted_at = None
+        tracked_actor.poll_attempts = 0
+        if tracked_actor.recent_backfill_status == "completed":
+            tracked_actor.next_poll_after = now + timedelta(seconds=self.settings.indexed_actor_repoll_seconds)
+        else:
+            tracked_actor.next_poll_after = now
+        db.add(tracked_actor)
+        db.flush()
     def _run_backfill_batches(self, db: Session, actor: str) -> int:
         tracked_actor = self.track_actor_request(db, actor, update_last_requested=False)
         if tracked_actor.recent_backfill_status == "completed":

src/srht_contrib/scripts/enqueue_actors.py +2 −2

	@@ -40,7 +40,7 @@ def _iter_usernames(path: Path) -> list[str]:
40	return usernames	40	return usernames
41		41
42		42
43	def enqueue_actors(username_file: Path, *, stagger_seconds: int = 300, start_at: datetime \| None = None) -> int:	43	def enqueue_actors(username_file: Path, *, stagger_seconds: int = 60, start_at: datetime \| None = None) -> int:
44	settings = Settings()	44	settings = Settings()
45	session_factory = make_session_factory(settings)	45	session_factory = make_session_factory(settings)
46	usernames = _iter_usernames(username_file)	46	usernames = _iter_usernames(username_file)
	@@ -95,7 +95,7 @@ def main() -> None:
95	parser.add_argument(	95	parser.add_argument(
96	"--stagger-seconds",	96	"--stagger-seconds",
97	type=int,	97	type=int,
98	default=300,	98	default=60,
99	help="Seconds to space out each actor's first eligible poll time.",	99	help="Seconds to space out each actor's first eligible poll time.",
100	)	100	)
101	args = parser.parse_args()	101	args = parser.parse_args()

tests/test_ingestion.py +116 −2

	@@ -9,6 +9,7 @@ from srht_contrib.models import ContributionEvent, ServiceBackfillState, SyncSta
9	from srht_contrib.scripts.enqueue_actors import enqueue_actors	9	from srht_contrib.scripts.enqueue_actors import enqueue_actors
10	from srht_contrib.schemas import NormalizedEvent	10	from srht_contrib.schemas import NormalizedEvent
11	from srht_contrib.services.git import GitIngestionService, GitPollResult	11	from srht_contrib.services.git import GitIngestionService, GitPollResult
		12	from srht_contrib.services.srht_client import SourceHutClientError
12	from srht_contrib.services.todo import TodoIngestionService, TodoPollResult	13	from srht_contrib.services.todo import TodoIngestionService, TodoPollResult
13	from srht_contrib.services.types import BackfillBatchResult	14	from srht_contrib.services.types import BackfillBatchResult
14		15
	@@ -112,6 +113,25 @@ class BackfillingTodoService:
112	return self.fetch_backfill_batch(actor, cursor_state)	113	return self.fetch_backfill_batch(actor, cursor_state)
113		114
114		115
		116	class FailingTodoService:
		117	service_name = "todo"
		118
		119	def fetch_recent_events(self, actor: str, since: datetime \| None = None) -> TodoPollResult:
		120	raise SourceHutClientError("temporary upstream failure")
		121
		122	def fetch_backfill_batch(self, actor: str, cursor_state: dict \| None = None) -> BackfillBatchResult:
		123	return BackfillBatchResult(events=[], cursor_state=None, complete=True)
		124
		125	def fetch_recent_backfill_batch(
		126	self,
		127	actor: str,
		128	cursor_state: dict \| None = None,
		129	*,
		130	since: datetime,
		131	) -> BackfillBatchResult:
		132	return BackfillBatchResult(events=[], cursor_state=None, complete=True)
		133
		134
115	class QueueShrinkingTodoService:	135	class QueueShrinkingTodoService:
116	service_name = "todo"	136	service_name = "todo"
117		137
	@@ -658,6 +678,50 @@ def test_poll_marks_backfill_complete_and_persists_service_state(db_session) ->
658	assert all(state.status == "completed" for state in service_states)	678	assert all(state.status == "completed" for state in service_states)
659		679
660		680
		681	def test_scheduled_poll_indexes_before_draining_recent_backfill(db_session) -> None:
		682	settings = make_settings(DISCOVERY_BATCH_SIZE=1, INDEXED_ACTOR_REPOLL_SECONDS=3600)
		683	poller = PollerService(todo_service=BackfillingTodoService(), git_service=EmptyGitService(), settings=settings)
		684	now = datetime.now(tz=UTC)
		685	db_session.add(
		686	TrackedActor(
		687	actor="~ccleberg",
		688	is_active=True,
		689	discovery_state="queued",
		690	queued_for_discovery_at=now - timedelta(minutes=1),
		691	next_poll_after=now - timedelta(minutes=1),
		692	recent_backfill_status="pending",
		693	)
		694	)
		695	db_session.commit()
		696
		697	first_results = poller.poll_tracked_actors(db_session)
		698	service_states_after_first_poll = db_session.scalars(select(ServiceBackfillState)).all()
		699	tracked_after_first_poll = db_session.scalar(select(TrackedActor).where(TrackedActor.actor == "~ccleberg"))
		700
		701	assert first_results == {"~ccleberg": 0}
		702	assert service_states_after_first_poll == []
		703	assert tracked_after_first_poll is not None
		704	assert tracked_after_first_poll.discovery_state == "indexed"
		705	assert tracked_after_first_poll.recent_backfill_status == "pending"
		706	assert tracked_after_first_poll.next_poll_after is not None
		707	first_due_at = tracked_after_first_poll.next_poll_after
		708	if first_due_at.tzinfo is None:
		709	first_due_at = first_due_at.replace(tzinfo=UTC)
		710	assert first_due_at <= datetime.now(tz=UTC)
		711
		712	second_results = poller.poll_tracked_actors(db_session)
		713
		714	tracked_after_second_poll = db_session.scalar(select(TrackedActor).where(TrackedActor.actor == "~ccleberg"))
		715	service_states_after_second_poll = db_session.scalars(
		716	select(ServiceBackfillState).order_by(ServiceBackfillState.scope, ServiceBackfillState.service)
		717	).all()
		718
		719	assert second_results == {"~ccleberg": 1}
		720	assert tracked_after_second_poll is not None
		721	assert tracked_after_second_poll.recent_backfill_status == "completed"
		722	assert [f"{state.scope}:{state.service}" for state in service_states_after_second_poll] == ["recent:git", "recent:todo"]
		723
		724
661	def test_backfill_cursor_state_shrinks_across_repeated_polls(db_session) -> None:	725	def test_backfill_cursor_state_shrinks_across_repeated_polls(db_session) -> None:
662	settings = make_settings()	726	settings = make_settings()
663	poller = PollerService(	727	poller = PollerService(
	@@ -779,11 +843,61 @@ def test_poll_tracked_actors_limits_to_due_batch_size(db_session) -> None:
779	assert actors["~a"].discovery_state == "indexed"	843	assert actors["~a"].discovery_state == "indexed"
780	assert actors["~b"].discovery_state == "indexed"	844	assert actors["~b"].discovery_state == "indexed"
781	assert actors["~c"].discovery_state == "queued"	845	assert actors["~c"].discovery_state == "queued"
782	assert actors["~a"].poll_attempts == 1	846	assert actors["~a"].poll_attempts == 0
783	assert actors["~b"].poll_attempts == 1	847	assert actors["~b"].poll_attempts == 0
784	assert actors["~c"].poll_attempts == 0	848	assert actors["~c"].poll_attempts == 0
785		849
786		850
		851	def test_scheduled_error_backoff_is_capped_and_success_resets_attempts(db_session) -> None:
		852	settings = make_settings(
		853	DISCOVERY_BATCH_SIZE=1,
		854	DISCOVERY_ERROR_BACKOFF_SECONDS=3600,
		855	DISCOVERY_ERROR_BACKOFF_MAX_SECONDS=7200,
		856	)
		857	now = datetime.now(tz=UTC)
		858	db_session.add(
		859	TrackedActor(
		860	actor="~flaky",
		861	is_active=True,
		862	discovery_state="queued",
		863	queued_for_discovery_at=now - timedelta(minutes=1),
		864	next_poll_after=now - timedelta(minutes=1),
		865	poll_attempts=10,
		866	recent_backfill_status="completed",
		867	)
		868	)
		869	db_session.commit()
		870	failing_poller = PollerService(todo_service=FailingTodoService(), git_service=EmptyGitService(), settings=settings)
		871
		872	failing_poller.poll_tracked_actors(db_session)
		873
		874	failed_actor = db_session.scalar(select(TrackedActor).where(TrackedActor.actor == "~flaky"))
		875	assert failed_actor is not None
		876	assert failed_actor.discovery_state == "error"
		877	assert failed_actor.poll_attempts == 11
		878	assert failed_actor.next_poll_after is not None
		879	failed_next_poll_after = failed_actor.next_poll_after
		880	if failed_next_poll_after.tzinfo is None:
		881	failed_next_poll_after = failed_next_poll_after.replace(tzinfo=UTC)
		882	assert failed_next_poll_after <= datetime.now(tz=UTC) + timedelta(seconds=7200, minutes=1)
		883
		884	failed_actor.next_poll_after = datetime.now(tz=UTC)
		885	db_session.add(failed_actor)
		886	db_session.commit()
		887	successful_poller = PollerService(
		888	todo_service=RecordingTodoService(events_by_call=[[]]),
		889	git_service=EmptyGitService(),
		890	settings=settings,
		891	)
		892
		893	successful_poller.poll_tracked_actors(db_session)
		894
		895	recovered_actor = db_session.scalar(select(TrackedActor).where(TrackedActor.actor == "~flaky"))
		896	assert recovered_actor is not None
		897	assert recovered_actor.discovery_state == "indexed"
		898	assert recovered_actor.poll_attempts == 0
		899
		900
787	def test_track_actor_request_prioritize_marks_actor_boosted_and_due_now(db_session) -> None:	901	def test_track_actor_request_prioritize_marks_actor_boosted_and_due_now(db_session) -> None:
788	settings = make_settings(INDEXED_ACTOR_REPOLL_SECONDS=3600)	902	settings = make_settings(INDEXED_ACTOR_REPOLL_SECONDS=3600)
789	poller = PollerService(todo_service=RecordingTodoService(events_by_call=[[]]), git_service=EmptyGitService(), settings=settings)	903	poller = PollerService(todo_service=RecordingTodoService(events_by_call=[[]]), git_service=EmptyGitService(), settings=settings)

Commit 6289a28296

Commit `6289a28296`