Skip to content

Configure automatic LLM failover across providers

Keep production AI APIs online during OpenAI or Anthropic outages with sub-second health checks and instant backup model routing.

Copy the prompt and paste it into the AI assistant you already use.

Steps to configure outage-proof model failover

  1. 01

    Pair models with matched capability

    Test your backup model on the same eval suite so fallback quality stays acceptable during outages.

  2. 02

    Set tight first-byte timeouts

    Trigger failover if upstream time-to-first-token stalls beyond your threshold instead of hanging.

  3. 03

    Normalize output schema contracts

    Verify both primary and fallback providers support your required JSON structure cleanly.

  4. 04

    Alert when fallback ratio spikes

    Notify engineers whenever fallback routes handle more than five percent of live traffic.

How it works in practice

Routing & Failover Policy

Primary: claude-3-5-sonnet (timeout: 2.5s) Fallback 1: gpt-4o (on 5xx, 429, timeout) Fallback 2: llama-3.3-70b-groq Schema: strict JSON enforced across all tiers

Runtime Trace During Outage

[00:00.0] Primary claude-3-5-sonnet -> HTTP 529 Overloaded [00:00.2] Auto-switched -> gpt-4o [00:01.1] 200 OK {"summary":"...","x-apikit-model":"gpt-4o","x-apikit-failover":"true"}

Your users get valid structured responses during upstream outages with zero custom retry spaghetti in your app.

Common questions

How quickly does failover switch when a provider stalls?

Configurable time-to-first-token and HTTP error triggers redirect the request to your backup provider within milliseconds.

Will the response JSON structure change on a fallback model?

No. APIkit enforces your identical JSON Schema contract across primary and fallback providers so clients never break.

Can I inspect which requests used a fallback model?

Yes. Response headers and observability traces tag every failover execution along with the exact upstream error code.