Skip to content

Cap per-request token budgets inside routing rules

Prevent massive accidental documents from blowing out per-call spend by enforcing strict input token limits and max-output caps.

Copy the prompt and paste it into the AI assistant you already use.

Checklist for tight per-request token budgets

  1. 01

    Calculate real unit cost limits

    Pick max input and output token counts that keep every single call profitable under your pricing.

  2. 02

    Choose reject vs truncate policy

    Return clear HTTP 413 payload errors for API partners or smart tail-trimming for UI widgets.

  3. 03

    Prevent JSON truncation bugs

    Ensure max_tokens leaves ample margin over expected JSON payload sizes to avoid broken brackets.

  4. 04

    Monitor token distribution p99

    Review p95 and p99 token histograms to tighten over-generous ceilings safely.

How it works in practice

Token Guardrail Policy

max_input_tokens: 3000 (on_exceed: "route_cheap_long_context" up to 12000, else HTTP 413) max_output_tokens: 450 max_cost_usd_per_call: 0.015

Blocked & Rerouted Examples

Call A (1,100 in): Primary model -> $0.004 OK Call B (8,400 in): Rerouted -> cheap long-context -> $0.006 OK Call C (65,000 in): Blocked at gateway -> 413 Payload Too Large ($0.00 spent)

One runaway user paste can no longer generate a $0.80 inference bill or timeout upstream models.

Common questions

How accurately are input tokens counted before calling the model?

Fast local tokenizers approximate token counts at the gateway in microseconds before sending the payload to upstream providers.

What happens when a caller needs more tokens on paid tiers?

Bind different budget ceilings to plan-tier metadata so enterprise keys enjoy higher limits than free accounts.

Does capping output tokens risk cutting off JSON?

APIkit warns you if configured max_tokens sits too close to observed output length and routes long completions cleanly.