Skip to content

Cut LLM costs with semantic similarity response caching

Match paraphrased user questions like 'reset password' and 'forgot login' using vector similarity thresholds to reuse verified answers.

Copy the prompt and paste it into the AI assistant you already use.

Steps to configure safe semantic caching

  1. 01

    Keep similarity thresholds strict

    Start at 0.93 or higher so subtly different questions with opposing intent never collide.

  2. 02

    Test tricky negative pairs

    Verify queries like 'upgrade plan' vs 'cancel plan' stay safely below your match threshold.

  3. 03

    Partition by user context

    Never share semantic cache buckets across requests that inject different database rows.

  4. 04

    Audit semantic hit samples

    Spot-check matched query pairs in logs during week one to fine-tune threshold accuracy.

How it works in practice

Two Paraphrased Requests (Threshold 0.93)

Req 1: "How do I add a custom domain to my workspace?" Req 2: "Steps to connect my own domain name in workspace settings?"

Semantic Cache Resolution

Req 1 -> MISS (computed & embedded, 710ms) Req 2 -> SEMANTIC HIT (similarity: 0.954, 42ms, $0.0000 LLM cost)

Near-duplicate customer questions resolve 15x faster while avoiding redundant frontier model generation.

Common questions

How do you stop 'how to delete account' matching 'how to create account'?

Setting high similarity thresholds (0.93–0.96) plus lightweight keyword guardrails separates opposing action verbs cleanly.

Is semantic caching safe for personalized account queries?

Use semantic caching only on shared knowledge or generic queries—or partition cache buckets strictly per user or context hash.

How much faster is a semantic hit than calling a frontier LLM?

Embedding lookup returns cached answers in roughly 30–60ms compared to seconds for full multi-paragraph generation.