Cap per-request token budgets inside routing rules
Prevent massive accidental documents from blowing out per-call spend by enforcing strict input token limits and max-output caps.
Checklist for tight per-request token budgets
- 01
Calculate real unit cost limits
Pick max input and output token counts that keep every single call profitable under your pricing.
- 02
Choose reject vs truncate policy
Return clear HTTP 413 payload errors for API partners or smart tail-trimming for UI widgets.
- 03
Prevent JSON truncation bugs
Ensure max_tokens leaves ample margin over expected JSON payload sizes to avoid broken brackets.
- 04
Monitor token distribution p99
Review p95 and p99 token histograms to tighten over-generous ceilings safely.
How it works in practice
Token Guardrail Policy
max_input_tokens: 3000 (on_exceed: "route_cheap_long_context" up to 12000, else HTTP 413) max_output_tokens: 450 max_cost_usd_per_call: 0.015
Blocked & Rerouted Examples
Call A (1,100 in): Primary model -> $0.004 OK Call B (8,400 in): Rerouted -> cheap long-context -> $0.006 OK Call C (65,000 in): Blocked at gateway -> 413 Payload Too Large ($0.00 spent)
One runaway user paste can no longer generate a $0.80 inference bill or timeout upstream models.
Common questions
How accurately are input tokens counted before calling the model?
Fast local tokenizers approximate token counts at the gateway in microseconds before sending the payload to upstream providers.
What happens when a caller needs more tokens on paid tiers?
Bind different budget ceilings to plan-tier metadata so enterprise keys enjoy higher limits than free accounts.
Does capping output tokens risk cutting off JSON?
APIkit warns you if configured max_tokens sits too close to observed output length and routes long completions cleanly.
Related tasks in this category
See all in Multi-Model Routing & Fallbacks-
Configure automatic LLM failover across providers
Keep production AI APIs online during OpenAI or Anthropic outages with sub-second health checks and instant backup model routing.
-
Route simple and complex prompts to tiered models
Send short routine requests to sub-second lightweight models while escalating nuanced or long-context inputs to frontier models.
-
Split live endpoint traffic to A/B test AI models
Send 10% canary traffic to a challenger model or prompt variant, compare latency and schema error rates, then shift 100% safely.
-
Load-balance AI traffic across multiple keys and pools
Overcome tight upstream tokens-per-minute caps by spreading bursty endpoint traffic across multiple model regions and provider accounts.
-
Minimize AI response latency with regional routing
Route user requests to the closest fast inference region and track p95 time-to-first-token across global callers automatically.
-
Pin exact model snapshots to stop silent drift
Lock production endpoints to dated model snapshots so floating aliases never alter your tone, latency, or JSON compliance unexpectedly.