# Cache part of a response

A cinema-ticketing API serves `GET /v1/showtimes/board`. Every response contains
two very different kinds of data:

- **Citywide state.** Every showtime for the next seven days across 300 screens,
  plus the cinema and movie metadata each one points at. Roughly 180 KB, and it
  takes the backend about 340 ms to assemble. Every caller receives exactly the
  same bytes.
- **Caller-specific state.** The requesting account's seat holds and its loyalty
  balance. Roughly 4 KB, and it takes the backend about 35 ms.

The shared part is 90% of the backend's work and 98% of the bytes, and it's
identical for every one of the thousand callers who hit the endpoint each
minute. The backend builds it a thousand times a minute anyway, because it's
glued to a 4 KB slice that differs per caller.

Caching the shared fragment for 10 seconds fixes that. A 10-second freshness
window means the snapshot is fetched at most six times a minute, no matter
whether a thousand or a hundred thousand requests arrive. Backend calls for the
shared fragment stop scaling with traffic and start scaling with the clock.

| Per minute, at 1,000 requests    | Before  | After   |
| -------------------------------- | ------- | ------- |
| Backend calls for the snapshot   | 1,000   | 6       |
| Backend calls for account data   | 1,000   | 1,000   |
| Bytes read from the backend      | ~184 MB | ~5.1 MB |
| Backend time on the request path | ~375 ms | ~35 ms  |

At that rate the backend sends 265 GB a day before the change and 7.3 GB a day
after it: a 97% cut in egress, and a 99.4% cut in the expensive call. The
gateway still makes a thousand backend calls a minute, but they're the cheap
ones.

## Why the coarser cache layers can't help

A [CDN](./cdn-caching.mdx) can't cache this response. Caching at the edge
requires that two callers holding the same cache key receive identical bytes,
and no two callers ever do, because one account's seat holds are in the body.
Adding the API key to the cache key technically works, and it's the wrong shape:
it stores a thousand near-duplicate 184 KB objects instead of one shared 180 KB
object, each expires on its own schedule, and each miss pays the full backend
cost. For the long tail of callers who show up once every few minutes, the hit
rate rounds to zero.

The [gateway response cache](./gateway-caching.mdx) has the same problem, plus a
sharper edge. If the cache key omits the caller's identity, one account receives
another account's seat holds. That's a correctness bug, not a tuning problem.

Both layers cache **responses**. The unit actually cacheable here is a
**fragment** of a response, and nothing outside the request can see fragments.
Splitting a payload into a shared part and a personal part requires code that
understands what the payload means, which means code running inside the request,
with a cache it can address directly.

## The two request paths

Before, every request pulls the entire payload from the backend:

<Diagram height="h-48">
  <DiagramNode id="clients">1,000 requests/min</DiagramNode>
  <DiagramNode id="gateway" variant="zuplo">
    Zuplo gateway
  </DiagramNode>
  <DiagramNode id="backend" variant="orange">
    Backend
  </DiagramNode>
  <DiagramEdge from="clients" to="gateway" />
  <DiagramEdge
    from="gateway"
    to="backend"
    label="1,000 full payloads/min"
    variant="orange"
  />
</Diagram>

After, the shared fragment comes from the cache and only the small per-caller
call reaches the backend:

<Diagram height="h-72">
  <DiagramNode id="clients">1,000 requests/min</DiagramNode>
  <DiagramNode id="handler" variant="zuplo">
    Custom handler
  </DiagramNode>
  <DiagramNode id="cache" variant="green">
    ZoneCache
  </DiagramNode>
  <DiagramNode id="backend" variant="orange">
    Backend
  </DiagramNode>
  <DiagramEdge from="clients" to="handler" />
  <DiagramEdge
    from="handler"
    to="cache"
    label="snapshot: 1,000 hits/min"
    variant="green"
  />
  <DiagramEdge
    from="cache"
    to="backend"
    label="refresh: 6/min"
    lineStyle="dashed"
    variant="orange"
  />
  <DiagramEdge
    from="handler"
    to="backend"
    label="account: 1,000/min"
    variant="orange"
    fromSide="bottom"
    toSide="bottom"
  />
</Diagram>

## Split the backend call in two

Start from the handler that doesn't split anything. It forwards one request and
returns one response, and there is nothing in it a cache can grab hold of.

```ts title="modules/showtime-board.ts"
import { environment, ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function handler(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const accountId = request.user?.data.accountId;

  return fetch(
    `${environment.TICKETING_API_URL}/v1/showtimes/board/${accountId}`,
    {
      headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
    },
  );
}
```

The backend needs to expose the two halves separately before anything can be
cached: one endpoint for the citywide snapshot, one for a single account. Most
APIs shaped like this already have both, because the composite endpoint was
built on top of them.

```ts title="modules/showtime-board.ts"
import { environment } from "@zuplo/runtime";

interface ShowtimeSnapshot {
  asOf: string;
  cinemas: { id: string; name: string; screens: number }[];
  showtimes: {
    id: string;
    cinemaId: string;
    movieId: string;
    startsAt: string;
    seatsAvailable: number;
  }[];
}

interface AccountState {
  holds: { showtimeId: string; seats: string[]; expiresAt: string }[];
  loyalty: { points: number; tier: string };
}

async function fetchJson<T>(path: string): Promise<T> {
  const response = await fetch(`${environment.TICKETING_API_URL}${path}`, {
    headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
  });
  if (!response.ok) {
    throw new Error(`Backend request to ${path} failed: ${response.status}`);
  }
  return (await response.json()) as T;
}

const fetchShowtimeSnapshot = () =>
  fetchJson<ShowtimeSnapshot>("/v1/showtimes/snapshot");

const fetchAccountState = (accountId: string) =>
  fetchJson<AccountState>(`/v1/accounts/${accountId}`);
```

Two calls where there was one looks like a step backwards, and on a cache miss
it is: the handler pays both round trips. The point is that one of the two is
now addressable on its own.

`fetchAccountState` never gets cached. It runs on every request, for every
caller, and it is the reason the composed response carries
`Cache-Control: no-store` at the end.

## Cache the shared fragment

[ZoneCache](../programmable-api/zone-cache.mdx) stores JSON-serializable values
under a string key with a per-item TTL, shared across the isolates in one zone.
The snapshot is JSON and every caller wants the same copy, so it fits exactly.

```ts title="modules/showtime-board.ts"
import { ZoneCache, ZuploContext } from "@zuplo/runtime";

const SNAPSHOT_KEY = "showtime-snapshot:v1";
const FRESH_FOR_SECONDS = 10;

async function getShowtimeSnapshot(
  context: ZuploContext,
): Promise<ShowtimeSnapshot> {
  const cache = new ZoneCache<ShowtimeSnapshot>("showtimes", context);

  const cached = await cache.get(SNAPSHOT_KEY);
  if (cached) {
    return cached;
  }

  const snapshot = await fetchShowtimeSnapshot();
  await cache.put(SNAPSHOT_KEY, snapshot, FRESH_FOR_SECONDS);
  return snapshot;
}
```

Ten seconds is a freshness budget, not a round number. Work it out from the
product, in this order:

1. **How fast does the source change?** The seat-inventory feed publishes every
   two seconds. Caching for less than that buys nothing, because the backend
   returns the same numbers.
2. **What has the API promised?** If the documented contract says seat counts
   are no more than 15 seconds old, that is the hard ceiling.
3. **Leave room for the refresh itself.** A refresh that takes 340 ms against a
   15-second ceiling wants a TTL comfortably under it. Ten seconds leaves five
   seconds of margin.

The `:v1` suffix on the key is worth the four characters. When
`ShowtimeSnapshot` gains or renames a field, bump it to `:v2` and the deploy
reads a cold key instead of deserializing old-shaped objects into new-shaped
types.

## Compose the response

The handler fetches both halves in parallel and merges them into the body the
client already expects. The response shape does not change, so no consumer has
to be told about any of this.

```ts title="modules/showtime-board.ts"
import { HttpProblems, ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function handler(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const accountId = request.user?.data.accountId;
  if (!accountId) {
    return HttpProblems.unauthorized(request, context);
  }

  const [snapshot, account] = await Promise.all([
    getShowtimeSnapshot(context),
    fetchAccountState(accountId),
  ]);

  return new Response(
    JSON.stringify({
      asOf: snapshot.asOf,
      cinemas: snapshot.cinemas,
      showtimes: snapshot.showtimes,
      account,
    }),
    {
      status: 200,
      headers: {
        "content-type": "application/json",
        "cache-control": "no-store",
      },
    },
  );
}
```

`Promise.all` matters here. On a cache miss the handler pays
`max(340 ms, 35 ms)` instead of `340 ms + 35 ms`, and on a hit the account call
is the only thing left on the critical path.

:::caution{title="The composed response is not cacheable"}

`Cache-Control: no-store` is deliberate. The body now contains one account's
seat holds, so it must not land in a CDN, a browser, or the gateway response
cache. Caching happens strictly inside the handler, at the fragment level.

:::

## Collapse concurrent misses

The snapshot expires ten seconds after it is written. At a thousand requests a
minute (call it 17 a second), every request in flight at that instant sees an
empty cache and calls the backend. The backend gets a burst of simultaneous
snapshot requests, each one taking 340 ms, and during those 340 ms more requests
arrive and pile onto the same burst. That is a cache stampede, and it lands
hardest on exactly the backend the cache was supposed to protect.

A module-scoped promise map fixes it. The first miss starts the fetch and stores
the promise; every other miss for the same key awaits that promise instead of
starting its own.

```ts title="modules/showtime-board.ts"
const inFlight = new Map<string, Promise<ShowtimeSnapshot>>();

function loadOnce(
  key: string,
  load: () => Promise<ShowtimeSnapshot>,
): Promise<ShowtimeSnapshot> {
  const existing = inFlight.get(key);
  if (existing) {
    return existing;
  }

  const pending = load().finally(() => inFlight.delete(key));
  inFlight.set(key, pending);
  return pending;
}
```

Module scope in a Zuplo project means isolate scope. The map lives as long as
the isolate that loaded the module, and every request served by that isolate
shares it.

:::note

This guard collapses concurrent misses **within one isolate**, not globally. Two
isolates that miss at the same moment still make two backend calls, and a zone
running twenty isolates can make twenty. That's the honest ceiling, and it's a
fixed, small number set by your isolate count, rather than a number that grows
with request volume.

:::

## Serve stale on backend failure

Once a shared fragment is cached, a failed refresh does not have to become a
failed request. Keep the cached copy alive well past its freshness window and
fall back to it when the backend is down.

Store the snapshot with a timestamp under a long TTL, and treat freshness as
something the code decides rather than something the cache enforces. The cache
now holds an entry rather than a bare snapshot, and it holds that entry far
longer than the snapshot stays fresh:

```ts title="modules/showtime-board.ts"
interface SnapshotEntry {
  snapshot: ShowtimeSnapshot;
  storedAt: number;
}

const FRESH_FOR_SECONDS = 10;
const SERVE_STALE_FOR_SECONDS = 300;
```

`getShowtimeSnapshot` now opens the cache as `ZoneCache<SnapshotEntry>`,
compares `storedAt` against `FRESH_FOR_SECONDS`, returns the age alongside the
snapshot, and logs a warning when it falls back so that stale serves show up in
your logs. The full function is in [The complete handler](#the-complete-handler)
below; these are the lines that carry the behavior:

```ts title="modules/showtime-board.ts"
const entry = await cache.get(SNAPSHOT_KEY);
const ageSeconds = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;

if (entry && ageSeconds < FRESH_FOR_SECONDS) {
  return { snapshot: entry.snapshot, ageSeconds };
}

try {
  const snapshot = await loadOnce(SNAPSHOT_KEY, async () => {
    const fresh = await fetchShowtimeSnapshot();
    await cache.put(
      SNAPSHOT_KEY,
      { snapshot: fresh, storedAt: Date.now() },
      SERVE_STALE_FOR_SECONDS,
    );
    return fresh;
  });
  return { snapshot, ageSeconds: 0 };
} catch (error) {
  if (entry) {
    return { snapshot: entry.snapshot, ageSeconds };
  }
  throw error;
}
```

The cache TTL is now 300 seconds and the freshness window is 10. Between those
two numbers sits a copy nobody is served under normal conditions and everybody
is served during an outage. A five-minute backend failure degrades the endpoint
from "seat counts at most 10 seconds old" to "seat counts up to 5 minutes old"
instead of returning `502` to every caller.

For a lot of APIs that availability win is worth more than the cost saving that
motivated the change. Surface the age so clients can decide for themselves:
`x-snapshot-age` in the complete handler below is an integer count of seconds.

## The complete handler

Everything above, in one file you can drop into `modules/` and adapt.

```ts title="modules/showtime-board.ts"
import {
  environment,
  HttpProblems,
  ZoneCache,
  ZuploContext,
  ZuploRequest,
} from "@zuplo/runtime";

interface ShowtimeSnapshot {
  asOf: string;
  cinemas: { id: string; name: string; screens: number }[];
  showtimes: {
    id: string;
    cinemaId: string;
    movieId: string;
    startsAt: string;
    seatsAvailable: number;
  }[];
}

interface AccountState {
  holds: { showtimeId: string; seats: string[]; expiresAt: string }[];
  loyalty: { points: number; tier: string };
}

interface SnapshotEntry {
  snapshot: ShowtimeSnapshot;
  storedAt: number;
}

const SNAPSHOT_KEY = "showtime-snapshot:v1";
const FRESH_FOR_SECONDS = 10;
const SERVE_STALE_FOR_SECONDS = 300;

const inFlight = new Map<string, Promise<ShowtimeSnapshot>>();

function loadOnce(
  key: string,
  load: () => Promise<ShowtimeSnapshot>,
): Promise<ShowtimeSnapshot> {
  const existing = inFlight.get(key);
  if (existing) {
    return existing;
  }

  const pending = load().finally(() => inFlight.delete(key));
  inFlight.set(key, pending);
  return pending;
}

async function fetchJson<T>(path: string): Promise<T> {
  const response = await fetch(`${environment.TICKETING_API_URL}${path}`, {
    headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
  });
  if (!response.ok) {
    throw new Error(`Backend request to ${path} failed: ${response.status}`);
  }
  return (await response.json()) as T;
}

const fetchShowtimeSnapshot = () =>
  fetchJson<ShowtimeSnapshot>("/v1/showtimes/snapshot");

const fetchAccountState = (accountId: string) =>
  fetchJson<AccountState>(`/v1/accounts/${accountId}`);

async function getShowtimeSnapshot(
  context: ZuploContext,
): Promise<{ snapshot: ShowtimeSnapshot; ageSeconds: number }> {
  const cache = new ZoneCache<SnapshotEntry>("showtimes", context);
  const entry = await cache.get(SNAPSHOT_KEY);
  const ageSeconds = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;

  if (entry && ageSeconds < FRESH_FOR_SECONDS) {
    return { snapshot: entry.snapshot, ageSeconds };
  }

  try {
    const snapshot = await loadOnce(SNAPSHOT_KEY, async () => {
      const fresh = await fetchShowtimeSnapshot();
      await cache.put(
        SNAPSHOT_KEY,
        { snapshot: fresh, storedAt: Date.now() },
        SERVE_STALE_FOR_SECONDS,
      );
      return fresh;
    });
    return { snapshot, ageSeconds: 0 };
  } catch (error) {
    if (entry) {
      context.log.warn("Snapshot refresh failed, serving stale copy", {
        ageSeconds,
        error: String(error),
      });
      return { snapshot: entry.snapshot, ageSeconds };
    }
    throw error;
  }
}

export default async function handler(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const accountId = request.user?.data.accountId;
  if (!accountId) {
    return HttpProblems.unauthorized(request, context);
  }

  try {
    const [shared, account] = await Promise.all([
      getShowtimeSnapshot(context),
      fetchAccountState(accountId),
    ]);

    return new Response(
      JSON.stringify({
        asOf: shared.snapshot.asOf,
        cinemas: shared.snapshot.cinemas,
        showtimes: shared.snapshot.showtimes,
        account,
      }),
      {
        status: 200,
        headers: {
          "content-type": "application/json",
          "cache-control": "no-store",
          "x-snapshot-age": String(Math.round(shared.ageSeconds)),
        },
      },
    );
  } catch (error) {
    context.log.error("Showtime board composition failed", {
      error: String(error),
    });
    return HttpProblems.badGateway(request, context, {
      detail: "The showtimes service is unavailable.",
    });
  }
}
```

Point a route at it. The handler reads `request.user`, so keep an authentication
policy ahead of it in the inbound pipeline.

```json title="config/routes.oas.json"
{
  "paths": {
    "/v1/showtimes/board": {
      "get": {
        "summary": "Showtime board",
        "x-zuplo-route": {
          "corsPolicy": "none",
          "handler": {
            "export": "default",
            "module": "$import(./modules/showtime-board)"
          },
          "policies": {
            "inbound": ["api-key-inbound"]
          }
        }
      }
    }
  }
}
```

## Choose the cache primitive

Three primitives can hold a fragment. They differ in scope, in what they can
store, and in how long a value survives.

| Primitive                                                                            | Scope                                                            | Stores                                            | Reach for it when                                                                                                               |
| ------------------------------------------------------------------------------------ | ---------------------------------------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| [ZoneCache](../programmable-api/zone-cache.mdx)                                      | Shared across the isolates in a zone; survives isolate recycling | JSON-serializable values, up to 512 MB per object | The shared fragment. This is the default for the pattern on this page.                                                          |
| [MemoryZoneReadThroughCache](../programmable-api/memory-zone-read-through-cache.mdx) | One isolate; dies with it                                        | Any in-memory value                               | A very hot, very small value where a zone round trip is measurable, such as a feature flag map or a decoded configuration blob. |
| [Cache API](../programmable-api/cache.mdx) (`caches.open`)                           | Long-lived memory, keyed by `Request`                            | Full `Request`/`Response` pairs                   | The fragment is literally an upstream HTTP response you want to keep verbatim, with its `Cache-Control` and `ETag` respected.   |

When the fragment is a parsed object that code composes into a larger body,
ZoneCache wins: a recycled isolate warms from the zone rather than from the
backend. Reach for the Cache API instead when the fragment is a response you
want kept verbatim (an upstream document, an image, a signed payload) and
re-parsing it on every request would be wasted work.

## Measure the result

Measure before and after, over the same window and the same routes. Expect tail
latency to move more than the median, because a hit removes the slower of the
two backend calls.

1. **Fragment hit rate.** The cache itself does not report this, so count it:
   log a structured field on hit and on miss, or increment a counter. A healthy
   hit rate for a 10-second TTL at a thousand requests a minute is above 99%. A
   number far below that means traffic is spread thinner across zones than
   expected, or the TTL is shorter than the gap between requests.
2. **Backend call volume.** Watch the request count against the snapshot
   endpoint in the **Origins** section of Analytics, which breaks out volume,
   error rate, and latency per upstream host. See
   [Origins](../analytics/tabs/origins.md).
3. **p95 on the route.** Compare against the same window a week earlier, not
   against the ten minutes before the deploy. See
   [Metrics glossary](../analytics/reference/metrics-glossary.md) for how p95 is
   computed.

## Limitations you own

- **Hit rate depends on traffic distribution.** ZoneCache is zone-local, and a
  write in one data center does nothing for any other. Spread the same thousand
  requests a minute across six zones and the backend sees up to 36 snapshot
  refreshes a minute rather than six. That is still a 96% reduction, but not the
  99.4% a single-zone calculation predicts. Low-volume routes with globally
  scattered traffic may miss more often than they hit.
- **A stale fragment is stale for everybody at once.** With a per-caller cache,
  one unlucky client sees old data. With a shared fragment, every caller sees
  the same old data simultaneously, and they can compare notes. The TTL is a
  product decision about what the API promises, not a knob to tune for hit rate.
- **You own invalidation.** ZoneCache has no global purge and no tag-based
  eviction. An entry disappears when its TTL expires or when code calls `delete`
  in the zone it lives in. If a fragment must drop within a bounded time of an
  upstream change, the TTL is the only mechanism that guarantees it. Choose it
  accordingly, and version the key so a schema change does not depend on
  invalidation at all.
- **Two calls cost more on a miss.** A cold cache pays for both round trips.
  `Promise.all` keeps that concurrent, but the first request after a deploy into
  a fresh zone is slower than the unsplit version was. Weigh that against how
  often it happens.

## Related

- [Caching in Zuplo](./overview.mdx) — which layer to reach for, and why.
- [Cache at the gateway](./gateway-caching.mdx) — whole-response caching when
  responses really are identical across callers.
- [Build a custom caching policy](./custom-caching-policy.mdx) — the same
  primitives packaged as a reusable policy instead of a handler.
- [Function Handler](../handlers/custom-handler.mdx) — the handler contract used
  throughout this page.