Compare prompt versions side by side on golden datasets
Run v1 and v2 endpoint drafts against identical ground-truth examples and inspect accuracy, token cost, and latency deltas side by side.
Checklist for rigorous golden dataset comparisons
- 01
Freeze ground-truth labels cleanly
Verify human-reviewed expected outputs in your dataset so noisy labels do not skew scores.
- 02
Compare cost and latency together
Evaluate accuracy per dollar and p95 latency alongside raw correctness to spot hidden tradeoffs.
- 03
Focus diffs on changed cases
Filter comparison tables to rows where v1 and v2 disagreed so review takes minutes.
- 04
Keep datasets representative
Include common easy inputs plus rare messy edge cases matching production traffic distribution.
How it works in practice
Side-by-Side Eval Matrix (80 Golden Cases)
Baseline: v3 (Frontier Model + 600-token prompt) Candidate: v4 (Compact Model + Few-Shot v2 prompt)
Comparison Scorecard
Accuracy: v3 93.8% -> v4 95.0% (+1.2%) Avg Cost: v3 $0.0042 -> v4 $0.0007 (-83%) p95 Time: v3 1,420ms -> v4 410ms (-71%) Verdict: v4 strictly dominates across all metrics
Hard side-by-side numbers replace gut feelings whenever you tune prompts or test cheaper models.
Common questions
How do I build a golden dataset quickly without starting from scratch?
Select real production inputs from your APIkit request logs, verify or edit their expected JSON outputs, and save them straight to the dataset.
Can I compare two different model providers in the same matrix?
Yes. Compare any combination of prompt text, temperature settings, and underlying model providers in one run.
How does side-by-side diffing handle nested JSON objects?
Field-level JSON diffing highlights exact changed keys, added tags, or mismatched enums across every test case.
Related tasks in this category
See all in Quality Evals, Logs & Versioning-
Gate AI endpoint deployments behind automated quality evals
Block buggy prompt edits from reaching production by automatically running deterministic and LLM-graded test suites on every draft revision.
-
Roll back faulty endpoint releases in a single click
Keep immutable release snapshots of every published prompt, schema, and routing rule so bad edits revert instantly without git redeploys.
-
Promote AI endpoints safely from staging to production
Validate prompt edits, webhook connections, and schema changes against staging API keys before promoting the exact tested snapshot live.
-
Debug slow AI calls with per-request latency and token traces
Break down every endpoint request into gateway checks, context lookups, time-to-first-token, output tokens, and exact dollar cost.
-
Replay failed production payloads against draft fixes
Grab any real failing trace from production, tweak your prompt or JSON schema in a draft, and re-run the exact payload in one click.
-
Score response faithfulness to curb AI hallucinations
Run automated faithfulness evals checking every generated claim against retrieved context docs before responses ship to users.