Skip to content

Compare prompt versions side by side on golden datasets

Run v1 and v2 endpoint drafts against identical ground-truth examples and inspect accuracy, token cost, and latency deltas side by side.

Copy the prompt and paste it into the AI assistant you already use.

Checklist for rigorous golden dataset comparisons

  1. 01

    Freeze ground-truth labels cleanly

    Verify human-reviewed expected outputs in your dataset so noisy labels do not skew scores.

  2. 02

    Compare cost and latency together

    Evaluate accuracy per dollar and p95 latency alongside raw correctness to spot hidden tradeoffs.

  3. 03

    Focus diffs on changed cases

    Filter comparison tables to rows where v1 and v2 disagreed so review takes minutes.

  4. 04

    Keep datasets representative

    Include common easy inputs plus rare messy edge cases matching production traffic distribution.

How it works in practice

Side-by-Side Eval Matrix (80 Golden Cases)

Baseline: v3 (Frontier Model + 600-token prompt) Candidate: v4 (Compact Model + Few-Shot v2 prompt)

Comparison Scorecard

Accuracy: v3 93.8% -> v4 95.0% (+1.2%) Avg Cost: v3 $0.0042 -> v4 $0.0007 (-83%) p95 Time: v3 1,420ms -> v4 410ms (-71%) Verdict: v4 strictly dominates across all metrics

Hard side-by-side numbers replace gut feelings whenever you tune prompts or test cheaper models.

Common questions

How do I build a golden dataset quickly without starting from scratch?

Select real production inputs from your APIkit request logs, verify or edit their expected JSON outputs, and save them straight to the dataset.

Can I compare two different model providers in the same matrix?

Yes. Compare any combination of prompt text, temperature settings, and underlying model providers in one run.

How does side-by-side diffing handle nested JSON objects?

Field-level JSON diffing highlights exact changed keys, added tags, or mismatched enums across every test case.