Protect AI endpoints with sliding-window rate limits
Prevent traffic bursts and abusive scrapers from draining token budgets with smooth per-IP, per-key, and per-user sliding limits.
Checklist for smooth sliding-window limits
- 01
Prefer sliding over fixed windows
Avoid boundary-minute double bursts where callers fire 2x capacity at :59 and :01 seconds.
- 02
Combine burst and sustained caps
Allow short 5-request interactive bursts while capping total sustained calls per hour.
- 03
Key behind trusted proxy headers
Extract real client IPs cleanly when traffic arrives via Cloudflare or application proxies.
- 04
Separate limit buckets by route
Give cheap classification routes generous caps while guarding expensive generation tightly.
Common questions
Why are sliding windows better than fixed 60-second buckets?
Fixed buckets let bursts fire twice the limit across a minute boundary. Sliding windows enforce smooth continuous limits.
Can I limit by tokens consumed instead of just request count?
Yes. Token-bucket and TPM rate limits prevent callers from sneaking huge prompts under a small request-count limit.
Do I need an external Redis instance for distributed limits?
No. APIkit enforces distributed rate limits globally across edge nodes automatically.
Related tasks in this category
See all in Response Caching & Rate Limiting-
Cache identical AI responses at the edge for instant hits
Serve repeated identical prompts in under 20 milliseconds while slashing token expenses to zero for deterministic endpoints.
-
Cut LLM costs with semantic similarity response caching
Match paraphrased user questions like 'reset password' and 'forgot login' using vector similarity thresholds to reuse verified answers.
-
Throttle AI endpoints automatically by customer plan tier
Give free users 10 daily runs, Pro teams 120 RPM, and Enterprise workspaces custom headroom without coding metering middleware.
-
Manage cache TTLs and purge stale AI outputs instantly
Set smart time-to-live schedules and invalidate tagged cache entries via webhook the second underlying product docs or pricing change.
-
Stop infinite client retry loops from draining AI spend
Detect bugged client loops, runaway webhooks, and daily spend velocity anomalies at the gateway before they rack up huge bills.
-
Coalesce simultaneous identical AI calls into one run
Collapse a thundering herd of 200 users opening the same dashboard at 9:00 AM into a single upstream model execution.