AI Gateway

5 Examples of Semantic Caching In Practice

Bill DoerrfeldBill Doerrfeld
October 16, 2026
7 min read

Early benchmarks and case studies show the cost savings and performance benefits of LLM semantic caching. Five real examples, plus the tradeoffs.

More and more enterprises are becoming reliant upon large language models (LLMs) within their generative AI applications. However, they’re quickly learning that making calls to frontier AI model providers can be pricey. It’s also often redundant, especially when many end users send semantically similar prompts.

This is where semantic caching, or LLM caching, comes in. Semantic caching can help avoid making calls to LLM APIs altogether. It’s similar to your typical web caching, but reimagined for AI server requests. The system stores vector representations of queries and their associated responses, and when a query is similar enough to a stored query, it sends the associated stored response back to the user.

Semantic caching can significantly reduce AI costs and token usage, since you skip calls to external LLM APIs altogether. It also has the side benefits of improved speed, since you avoid having to process similar response logic over and over again, and improved consistency, since serving more consistent responses can make applications more deterministic.

Best for:
  • Teams running high-volume chatbots, support assistants, or RAG apps on paid LLM APIs
  • Platform teams weighing whether semantic caching is worth the added infrastructure
  • Anyone looking for real-world cache hit rates and cost savings before committing

Although semantic caching is a relatively new concept, it’s already being implemented in user-facing AI assistants and search across healthcare, e-commerce, and SaaS environments.

Below, we’ll look at a handful of case studies and reports that demonstrate the positive effects of introducing semantic caching in practice. We’ll also look at the possible downsides of semantic caching, and what tools it’ll require to actually get these business benefits in practice.

1. AWS

An AWS test found semantic caching reduced LLM costs by 86% while maintaining 91% accuracy.

AWS set up a controlled test of over 63,000 real user chatbot queries and their paraphrased variants, drawn from a public benchmark dataset, to simulate random incoming traffic to LLMs. Their architecture used ElastiCache as a semantic cache store for Amazon Bedrock, with Claude 3 Haiku, a smaller, lower-cost model, handling inference. AWS notes that semantic caching can deliver larger benefits with bigger, pricier models.

The study found that at a 0.75 similarity threshold, the prompts hit the cache 90.3% of the time, which created an 86.3% savings in daily cost. In this situation, total cost combined LLM inference, embedding generation, and ElastiCache instance costs.

The AWS engineers tracked the accuracy too, which in this scenario was a measure of how often the semantic cache returned a match for the intent of the user’s query. They found their approach was able to maintain 91% accuracy at the 0.75 threshold.

2. Mangoes.ai

Mangoes.ai reported a 70% cache hit rate, 70% reduction in LLM spend, and 4x faster responses.

This example is more than a benchmark or test — it’s actually working in production. Mangoes.ai is a behavioral health voice assistant for caregivers and clinicians, and they’re using semantic caching to ensure more predictable AI answers and combat costs.

Using Redis LangCache, Mangoes.ai now avoids sending every request directly to an LLM. Instead, it first analyzes the query and performs a cache lookup. The founder and CEO of Mangoes.ai, Amit Lamba, is quoted as saying “we’re getting a 70% cache hit rate, which saves 70% of our LLM spend.” He adds that it’s four times faster, too.

The case study demonstrates that semantic caching can play a role even in critical areas like healthcare, where the responses from generative AI can have meaningful consequences. Notably, Mangoes.ai uses guardrails to keep patient-specific and PII-heavy content out of the cache.

3. CodexiLab

CodexiLab used semantic caching to decrease OpenAI spending by over $3,500 per month.

CodexiLab, a product engineering consultancy, shared one of its customer journeys involving a customer-support SaaS startup (which remains nameless in the article). Nonetheless, the story is impressive.

The startup built an AI agent that handles 150,000 customer support inquiries per month. It was fully based on OpenAI’s API, and that AI spend had become quite high for a small startup, peaking at $4,500 per month.

According to CodexiLab, implementing a Redis-based LLM semantic caching layer gave the team a 42% cache hit rate, cutting their monthly OpenAI API costs by 78% and decreasing their spend to below $900 per month. It also reduced the average response latency for cached responses down to 15 milliseconds, compared to 2.2 seconds for non-cached responses.

4. Oracle

An Oracle benchmark found semantic caching cut estimated LLM cost by about 60% across a 100-request workload.

Oracle architect Mark Nelson published benchmarking that ran tests across five different cache modes, which combined a mix of controls, exact-match lookups, vector search, and database routes. The test ran 100 requests per mode using OpenAI’s gpt-4o-mini.

In the test, both semantic cache modes were able to avoid calling the LLM for 65 of the 100 requests. The semantic-cache-true-cache mode averaged 546 milliseconds request latency, including application, database, cache, and LLM provider.

But what was perhaps most interesting was the cost reduction. Sending all 100 calls to the LLM directly totaled 8,585 tokens and cost an estimated $0.001964 in total. That’s compared to $0.00078 for the same workload with semantic caching. By simple math, that’s a 60.3% cost savings. Nelson notes these are estimates for this specific workload, not a universal cost-saving figure.

5. Walmart

Walmart reports a close to 50% cache hit rate for tail queries.

Rohit Chatter, chief software architect at Walmart, shared in early 2024 how they’re leveraging semantic caching in practice within their e-commerce search.

He shares that they’ve been able to obtain a close to 50% caching rate for tail queries. At the same time, he acknowledges that there are tradeoffs involving latency and costs.

The downsides of semantic caching

As the emerging case studies suggest, semantic caching clearly can reduce costs by forgoing LLM API calls, and can improve latency by returning quick responses. This can be used to scale an AI system and bring more consistency. However, semantic caching does come with some downsides.

Added latency: Semantic caching requires additional compute power to determine similarity, compared with traditional lookups, which could at times incur additional latency, especially if the architecture isn’t optimized.

Smaller savings for better answers: Correctly pairing a prompt with the most appropriate cached response is a science, and it’s not always perfect. A stricter similarity threshold produces closer matches, but fewer prompts hit the cache, which means more LLM calls and smaller savings. There’s also the cost of performing the semantic cache lookups themselves.

Accuracy problems: Setting a low similarity threshold for semantic matches can grant additional savings, yet accuracy can suffer. An accuracy rate of 90% isn’t actually that great, as it means about 1 in 10 cached responses don’t match the intent of the user’s prompt. This can cause poor experiences.

Updating the system: Keeping answers in the cache relevant is another hurdle, given information and guidance change over time. Standards and best practices here are still evolving, and continuous feedback solutions will likely be ad hoc.

Balancing financials: Determining when semantic caching is financially worth it is another struggle, too.

When to use semantic caching

Given there are tradeoffs with semantic caching, it’s worth being clear about when it makes sense to introduce semantic caching, and when it doesn’t.

When semantic caching makes sense

  • When you have a high volume of users making the same requests
  • For simple generative AI applications, like customer support bots and Retrieval Augmented Generation (RAG)-based chat applications
  • For standalone, single-turn queries
  • When the data is static, such as product information

When semantic caching doesn’t make sense

  • For workloads where most queries are unique
  • Multi-turn conversations that must check context memory first
  • For real-time or fast-moving data, such as order status or pricing information
  • Complex prompts that involve state changes and actions beyond basic lookup
  • When the answer is specific to a single user
  • Cases where small wording differences change the right answer

What semantic caching requires

By introducing semantic caching, you can go beyond traditional keyword matching and avoid costly LLM calls. Yet, it’ll take some additional infrastructure to properly set up. For instance, you’ll need a system of record to store historical requests and responses. This is typically a vector database.

You’ll also need an analysis engine to perform semantic lookups to determine if equivalent responses exist in the cache. Serving LLM responses based on vector similarity rather than exact text matches will require fine-tuning and continual monitoring to determine cost-effectiveness as well.

By using an AI gateway like Zuplo, you can implement semantic caching for inbound requests by configuring a policy and adjusting settings like the similarity tolerance. With this setup, generative AI applications can rely on the gateway to cache natural language queries.

Semantic Cache policy

Cache responses by prompt similarity instead of an exact key, and tune the tolerance for what counts as a match.

The benefit is that you don’t need to stand up a separate caching service for AI calls: the gateway that already caches your regular API traffic can cache prompts too.