---
title: "5 Examples of Semantic Caching In Practice"
description: "Early benchmarks and case studies show the cost savings and performance benefits of LLM semantic caching. Five real examples, plus the tradeoffs."
canonicalUrl: "https://zuplo.com/blog/2026/10/16/semantic-caching-examples"
pageType: "blog"
date: "2026-10-16"
authors: "billDoerrfeld"
tags: "AI Gateway, API Best Practices, ai-agents"
image: "https://zuplo.com/og?text=5%20Examples%20of%20Semantic%20Caching%20In%20Practice"
---
More and more enterprises are becoming reliant upon large language models (LLMs)
within their generative AI applications. However, they're quickly learning that
making calls to frontier AI model providers
[can be pricey](/blog/control-ai-costs-api-gateway). It's also often redundant,
especially when many end users send semantically similar prompts.

This is where [semantic caching](/blog/what-is-semantic-caching), or LLM
caching, comes in. Semantic caching can help avoid making calls to LLM APIs
altogether. It's similar to your typical web caching, but reimagined for AI
server requests. The system stores vector representations of queries and their
associated responses, and when a query is similar enough to a stored query, it
sends the associated stored response back to the user.

Semantic caching can significantly reduce AI costs and token usage, since you
skip calls to external LLM APIs altogether. It also has the side benefits of
improved speed, since you avoid having to process similar response logic over
and over again, and improved consistency, since serving more consistent
responses can make applications more deterministic.

<CalloutAudience
  variant="bestFor"
  items={[
    `Teams running high-volume chatbots, support assistants, or RAG apps on paid LLM APIs`,
    `Platform teams weighing whether semantic caching is worth the added infrastructure`,
    `Anyone looking for real-world cache hit rates and cost savings before committing`,
  ]}
/>

Although semantic caching is a relatively new concept, it's already being
implemented in user-facing AI assistants and search across healthcare,
e-commerce, and SaaS environments.

Below, we'll look at a handful of case studies and reports that demonstrate the
positive effects of introducing semantic caching in practice. We'll also look at
the possible downsides of semantic caching, and what tools it'll require to
actually get these business benefits in practice.

## 1. AWS

_An AWS test found semantic caching reduced LLM costs by 86% while maintaining
91% accuracy._

AWS set up
[a controlled test](https://aws.amazon.com/blogs/database/lower-cost-and-latency-for-ai-using-amazon-elasticache-as-a-semantic-cache-with-amazon-bedrock/)
of over 63,000 real user chatbot queries and their paraphrased variants, drawn
from a public benchmark dataset, to simulate random incoming traffic to LLMs.
Their architecture used ElastiCache as a semantic cache store for Amazon
Bedrock, with Claude 3 Haiku, a smaller, lower-cost model, handling inference.
AWS notes that semantic caching can deliver larger benefits with bigger, pricier
models.

The study found that at a 0.75 similarity threshold, the prompts hit the cache
90.3% of the time, which created an 86.3% savings in daily cost. In this
situation, total cost combined LLM inference, embedding generation, and
ElastiCache instance costs.

The AWS engineers tracked the accuracy too, which in this scenario was a measure
of how often the semantic cache returned a match for the intent of the user's
query. They found their approach was able to maintain 91% accuracy at the 0.75
threshold.

## 2. Mangoes.ai

_Mangoes.ai reported a 70% cache hit rate, 70% reduction in LLM spend, and 4x
faster responses._

This example is more than a benchmark or test — it's actually working in
production. Mangoes.ai is a behavioral health voice assistant for caregivers and
clinicians, and
[they're using semantic caching](https://redis.io/customers/mangoes-ai/) to
ensure more predictable AI answers and combat costs.

Using Redis LangCache, Mangoes.ai now avoids sending every request directly to
an LLM. Instead, it first analyzes the query and performs a cache lookup. The
founder and CEO of Mangoes.ai, Amit Lamba, is quoted as saying "we're getting a
70% cache hit rate, which saves 70% of our LLM spend." He adds that it's four
times faster, too.

The case study demonstrates that semantic caching can play a role even in
critical areas like healthcare, where the responses from generative AI can have
meaningful consequences. Notably, Mangoes.ai uses guardrails to keep
patient-specific and PII-heavy content out of the cache.

## 3. CodexiLab

_CodexiLab used semantic caching to decrease OpenAI spending by over $3,500 per
month._

CodexiLab, a product engineering consultancy, shared one of its customer
journeys involving a
[customer-support SaaS startup](https://codexilab.com/blog/llm-semantic-caching)
(which remains nameless in the article). Nonetheless, the story is impressive.

The startup built an AI agent that handles 150,000 customer support inquiries
per month. It was fully based on OpenAI's API, and that AI spend had become
quite high for a small startup, peaking at $4,500 per month.

According to CodexiLab, implementing a Redis-based LLM semantic caching layer
gave the team a 42% cache hit rate, cutting their monthly OpenAI API costs by
78% and decreasing their spend to below $900 per month. It also reduced the
average response latency for cached responses down to 15 milliseconds, compared
to 2.2 seconds for non-cached responses.

## 4. Oracle

_An Oracle benchmark found semantic caching cut estimated LLM cost by about 60%
across a 100-request workload._

Oracle architect Mark Nelson
[published benchmarking](https://blogs.oracle.com/developers/benchmarking-semantic-caching-with-oracle-ai-database-26ai-and-oracle-true-cache)
that ran tests across five different cache modes, which combined a mix of
controls, exact-match lookups, vector search, and database routes. The test ran
100 requests per mode using OpenAI's gpt-4o-mini.

In the test, both semantic cache modes were able to avoid calling the LLM for 65
of the 100 requests. The semantic-cache-true-cache mode averaged 546
milliseconds request latency, including application, database, cache, and LLM
provider.

But what was perhaps most interesting was the cost reduction. Sending all 100
calls to the LLM directly totaled 8,585 tokens and cost an estimated $0.001964
in total. That's compared to $0.00078 for the same workload with semantic
caching. By simple math, that's a 60.3% cost savings. Nelson notes these are
estimates for this specific workload, not a universal cost-saving figure.

## 5. Walmart

_Walmart reports a close to 50% cache hit rate for tail queries._

Rohit Chatter, chief software architect at Walmart,
[shared in early 2024](https://portkey.ai/blog/transforming-e-commerce-search-with-generative-ai-insights-from-walmarts-journey/)
how they're leveraging semantic caching in practice within their e-commerce
search.

He shares that they've been able to obtain a close to 50% caching rate for tail
queries. At the same time, he acknowledges that there are tradeoffs involving
latency and costs.

## The downsides of semantic caching

As the emerging case studies suggest, semantic caching clearly can reduce costs
by forgoing LLM API calls, and can improve latency by returning quick responses.
This can be used to scale an AI system and bring more consistency. However,
semantic caching does come with some downsides.

**Added latency**: Semantic caching requires additional compute power to
determine similarity, compared with
[traditional lookups](/learning-center/api-gateway-caching), which could at
times incur additional latency, especially if the architecture isn't optimized.

**Smaller savings for better answers**: Correctly pairing a prompt with the most
appropriate cached response is a science, and it's not always perfect. A
stricter similarity threshold produces closer matches, but fewer prompts hit the
cache, which means more LLM calls and smaller savings. There's also the cost of
performing the semantic cache lookups themselves.

**Accuracy problems**:
[Setting a low similarity threshold](https://arxiv.org/abs/2502.03771) for
semantic matches can grant additional savings, yet accuracy can suffer. An
accuracy rate of 90% isn't actually that great, as it means about 1 in 10 cached
responses don't match the intent of the user's prompt. This can cause poor
experiences.

**Updating the system**:
[Keeping answers in the cache relevant](https://aws.amazon.com/blogs/database/lower-cost-and-latency-for-ai-using-amazon-elasticache-as-a-semantic-cache-with-amazon-bedrock/)
is another hurdle, given information and guidance change over time. Standards
and best practices here are still evolving, and continuous feedback solutions
will likely be ad hoc.

**Balancing financials**: Determining when semantic caching is financially worth
it is another struggle, too.

## When to use semantic caching

Given there are tradeoffs with semantic caching, it's worth being clear about
when it makes sense to introduce semantic caching, and when it doesn't.

### When semantic caching makes sense

- When you have a high volume of users making the same requests
- For simple generative AI applications, like customer support bots and
  Retrieval Augmented Generation (RAG)-based chat applications
- For standalone, single-turn queries
- When the data is static, such as product information

### When semantic caching doesn't make sense

- For workloads where most queries are unique
- Multi-turn conversations that must check context memory first
- For real-time or fast-moving data, such as order status or pricing
  information
- Complex prompts that involve state changes and actions beyond basic lookup
- When the answer is specific to a single user
- Cases where small wording differences change the right answer

## What semantic caching requires

By introducing
[semantic caching](https://thenewstack.io/what-is-semantic-caching/), you can go
beyond traditional keyword matching and avoid costly LLM calls. Yet, it'll take
some additional infrastructure to properly set up. For instance, you'll need a
system of record to store historical requests and responses. This is typically
a vector database.

You'll also need an analysis engine to perform semantic lookups to determine if
equivalent responses exist in the cache. Serving LLM responses based on vector
similarity rather than exact text matches will require fine-tuning and
[continual monitoring](/blog/attribute-ai-spend-teams-apps) to determine
cost-effectiveness as well.

By using an [AI gateway](/blog/what-developers-want-from-an-ai-gateway) like
Zuplo, you can implement semantic caching for inbound requests by configuring a
policy and adjusting settings like the similarity tolerance. With this setup,
generative AI applications can rely on the gateway to cache natural language
queries.

<CalloutDoc
  title="Semantic Cache policy"
  description="Cache responses by prompt similarity instead of an exact key, and tune the tolerance for what counts as a match."
  href="https://zuplo.com/docs/policies/semantic-cache-inbound"
  icon="book"
/>

The benefit is that you don't need to stand up a separate caching service for
AI calls: the gateway that already caches your
[regular API traffic](/learning-center/how-developers-can-use-caching-to-improve-api-performance)
can cache prompts too.