Skip to content

Debug slow AI calls with per-request latency and token traces

Break down every endpoint request into gateway checks, context lookups, time-to-first-token, output tokens, and exact dollar cost.

Copy the prompt and paste it into the AI assistant you already use.

Steps to diagnose slow or costly AI traces

  1. 01

    Separate TTFT from stream duration

    Check whether delays come from slow provider queuing or oversized multi-paragraph outputs.

  2. 02

    Filter traces by retry or failover

    Spot routes silently burning latency on JSON repair retries or primary model timeouts.

  3. 03

    Spot unexpected prompt token bloat

    Compare input token counts on outlier traces to catch unclipped user pastes or huge fetched rows.

  4. 04

    Check cache hit ratios per route

    Identify high-volume endpoints with low cache hits where minor key tweaks unlock big savings.

How it works in practice

Trace Filter: latency > 2500ms

trace_id=tr_9941a | endpoint=/v1/support-draft | status=200

Span Breakdown & Root Cause

1. Auth & Rate Limit: 3ms 2. Postgres Context SQL: 18ms 3. Primary Model (Timeout): 2,500ms -> 529 Overloaded 4. Failover Model TTFT: 290ms | Gen: 610ms Total: 3,421ms | Cost: $0.0024 | Tokens: 1,140 in / 148 out

Instead of guessing why p95 jumped, traces reveal the exact upstream timeout and failover path in seconds.

Common questions

Can I correlate APIkit traces with logs in my own backend?

Yes. Pass your own X-Request-Id or trace header on requests and APIkit indexes it alongside every internal span.

Do trace logs capture whether a response came from edge cache?

Every trace marks exact cache hits, semantic hits, failover switches, and schema repair attempts clearly.

How can I avoid storing sensitive user data in debug traces?

Combine trace metrics with automatic PII redaction or zero-payload logging mode on regulated endpoints.