Debug slow AI calls with per-request latency and token traces
Break down every endpoint request into gateway checks, context lookups, time-to-first-token, output tokens, and exact dollar cost.
Steps to diagnose slow or costly AI traces
- 01
Separate TTFT from stream duration
Check whether delays come from slow provider queuing or oversized multi-paragraph outputs.
- 02
Filter traces by retry or failover
Spot routes silently burning latency on JSON repair retries or primary model timeouts.
- 03
Spot unexpected prompt token bloat
Compare input token counts on outlier traces to catch unclipped user pastes or huge fetched rows.
- 04
Check cache hit ratios per route
Identify high-volume endpoints with low cache hits where minor key tweaks unlock big savings.
How it works in practice
Trace Filter: latency > 2500ms
trace_id=tr_9941a | endpoint=/v1/support-draft | status=200
Span Breakdown & Root Cause
1. Auth & Rate Limit: 3ms 2. Postgres Context SQL: 18ms 3. Primary Model (Timeout): 2,500ms -> 529 Overloaded 4. Failover Model TTFT: 290ms | Gen: 610ms Total: 3,421ms | Cost: $0.0024 | Tokens: 1,140 in / 148 out
Instead of guessing why p95 jumped, traces reveal the exact upstream timeout and failover path in seconds.
Common questions
Can I correlate APIkit traces with logs in my own backend?
Yes. Pass your own X-Request-Id or trace header on requests and APIkit indexes it alongside every internal span.
Do trace logs capture whether a response came from edge cache?
Every trace marks exact cache hits, semantic hits, failover switches, and schema repair attempts clearly.
How can I avoid storing sensitive user data in debug traces?
Combine trace metrics with automatic PII redaction or zero-payload logging mode on regulated endpoints.
Related tasks in this category
See all in Quality Evals, Logs & Versioning-
Gate AI endpoint deployments behind automated quality evals
Block buggy prompt edits from reaching production by automatically running deterministic and LLM-graded test suites on every draft revision.
-
Compare prompt versions side by side on golden datasets
Run v1 and v2 endpoint drafts against identical ground-truth examples and inspect accuracy, token cost, and latency deltas side by side.
-
Roll back faulty endpoint releases in a single click
Keep immutable release snapshots of every published prompt, schema, and routing rule so bad edits revert instantly without git redeploys.
-
Promote AI endpoints safely from staging to production
Validate prompt edits, webhook connections, and schema changes against staging API keys before promoting the exact tested snapshot live.
-
Replay failed production payloads against draft fixes
Grab any real failing trace from production, tweak your prompt or JSON schema in a draft, and re-run the exact payload in one click.
-
Score response faithfulness to curb AI hallucinations
Run automated faithfulness evals checking every generated claim against retrieved context docs before responses ship to users.