Serve /robots.txt disallowing /search, /api, /group_user,
/media and any path with ?p=. Add a per-client-IP token bucket ahead of
the upstream throttle so one crawler cannot consume the whole DA budget and
turn it into latency for everyone else. Honour X-Forwarded-For only when
the request came from a configured trusted proxy.
Files: app/router.go, new app/ratelimit.go, app/config.go,
SETUP.md.
Verify: test that N+1 requests from one address within the window get 429.
Roadmap item 1.3, size S. See ROADMAP.md.
closed by commit 4261b937e4 by cmc: Add robots.txt and a per-client rate limit
2026-09-11 15:13 UTC