Cache part of a response

A cinema-ticketing API serves GET /v1/showtimes/board. Every response contains two very different kinds of data:

• Citywide state. Every showtime for the next seven days across 300 screens, plus the cinema and movie metadata each one points at. Roughly 180 KB, and it takes the backend about 340 ms to assemble. Every caller receives exactly the same bytes.
• Caller-specific state. The requesting account's seat holds and its loyalty balance. Roughly 4 KB, and it takes the backend about 35 ms.

The shared part is 90% of the backend's work and 98% of the bytes, and it's identical for every one of the thousand callers who hit the endpoint each minute. The backend builds it a thousand times a minute anyway, because it's glued to a 4 KB slice that differs per caller.

Caching the shared fragment for 10 seconds fixes that. A 10-second freshness window means the snapshot is fetched at most six times a minute, no matter whether a thousand or a hundred thousand requests arrive. Backend calls for the shared fragment stop scaling with traffic and start scaling with the clock.

At that rate the backend sends 265 GB a day before the change and 7.3 GB a day after it: a 97% cut in egress, and a 99.4% cut in the expensive call. The gateway still makes a thousand backend calls a minute, but they're the cheap ones.

Why the coarser cache layers can't help

A CDN can't cache this response. Caching at the edge requires that two callers holding the same cache key receive identical bytes, and no two callers ever do, because one account's seat holds are in the body. Adding the API key to the cache key technically works, and it's the wrong shape: it stores a thousand near-duplicate 184 KB objects instead of one shared 180 KB object, each expires on its own schedule, and each miss pays the full backend cost. For the long tail of callers who show up once every few minutes, the hit rate rounds to zero.

The gateway response cache has the same problem, plus a sharper edge. If the cache key omits the caller's identity, one account receives another account's seat holds. That's a correctness bug, not a tuning problem.

Both layers cache responses. The unit actually cacheable here is a fragment of a response, and nothing outside the request can see fragments. Splitting a payload into a shared part and a personal part requires code that understands what the payload means, which means code running inside the request, with a cache it can address directly.

The two request paths

Before, every request pulls the entire payload from the backend:

After, the shared fragment comes from the cache and only the small per-caller call reaches the backend:

Split the backend call in two

Start from the handler that doesn't split anything. It forwards one request and returns one response, and there is nothing in it a cache can grab hold of.

Code
import { environment, ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function handler(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const accountId = request.user?.data.accountId;

  return fetch(
    `${environment.TICKETING_API_URL}/v1/showtimes/board/${accountId}`,
    {
      headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
    },
  );
}

The backend needs to expose the two halves separately before anything can be cached: one endpoint for the citywide snapshot, one for a single account. Most APIs shaped like this already have both, because the composite endpoint was built on top of them.

Code
import { environment } from "@zuplo/runtime";

interface ShowtimeSnapshot {
  asOf: string;
  cinemas: { id: string; name: string; screens: number }[];
  showtimes: {
    id: string;
    cinemaId: string;
    movieId: string;
    startsAt: string;
    seatsAvailable: number;
  }[];
}

interface AccountState {
  holds: { showtimeId: string; seats: string[]; expiresAt: string }[];
  loyalty: { points: number; tier: string };
}

async function fetchJson<T>(path: string): Promise<T> {
  const response = await fetch(`${environment.TICKETING_API_URL}${path}`, {
    headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
  });
  if (!response.ok) {
    throw new Error(`Backend request to ${path} failed: ${response.status}`);
  }
  return (await response.json()) as T;
}

const fetchShowtimeSnapshot = () =>
  fetchJson<ShowtimeSnapshot>("/v1/showtimes/snapshot");

const fetchAccountState = (accountId: string) =>
  fetchJson<AccountState>(`/v1/accounts/${accountId}`);

Two calls where there was one looks like a step backwards, and on a cache miss it is: the handler pays both round trips. The point is that one of the two is now addressable on its own.

fetchAccountState never gets cached. It runs on every request, for every caller, and it is the reason the composed response carries Cache-Control: no-store at the end.

Cache the shared fragment

ZoneCache stores JSON-serializable values under a string key with a per-item TTL, shared across the isolates in one zone. The snapshot is JSON and every caller wants the same copy, so it fits exactly.

Code
import { ZoneCache, ZuploContext } from "@zuplo/runtime";

const SNAPSHOT_KEY = "showtime-snapshot:v1";
const FRESH_FOR_SECONDS = 10;

async function getShowtimeSnapshot(
  context: ZuploContext,
): Promise<ShowtimeSnapshot> {
  const cache = new ZoneCache<ShowtimeSnapshot>("showtimes", context);

  const cached = await cache.get(SNAPSHOT_KEY);
  if (cached) {
    return cached;
  }

  const snapshot = await fetchShowtimeSnapshot();
  await cache.put(SNAPSHOT_KEY, snapshot, FRESH_FOR_SECONDS);
  return snapshot;
}

Ten seconds is a freshness budget, not a round number. Work it out from the product, in this order:

1. How fast does the source change? The seat-inventory feed publishes every two seconds. Caching for less than that buys nothing, because the backend returns the same numbers.
2. What has the API promised? If the documented contract says seat counts are no more than 15 seconds old, that is the hard ceiling.
3. Leave room for the refresh itself. A refresh that takes 340 ms against a 15-second ceiling wants a TTL comfortably under it. Ten seconds leaves five seconds of margin.

The :v1 suffix on the key is worth the four characters. When ShowtimeSnapshot gains or renames a field, bump it to :v2 and the deploy reads a cold key instead of deserializing old-shaped objects into new-shaped types.

Compose the response

The handler fetches both halves in parallel and merges them into the body the client already expects. The response shape does not change, so no consumer has to be told about any of this.

Code
import { HttpProblems, ZuploContext, ZuploRequest } from "@zuplo/runtime";

export default async function handler(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const accountId = request.user?.data.accountId;
  if (!accountId) {
    return HttpProblems.unauthorized(request, context);
  }

  const [snapshot, account] = await Promise.all([
    getShowtimeSnapshot(context),
    fetchAccountState(accountId),
  ]);

  return new Response(
    JSON.stringify({
      asOf: snapshot.asOf,
      cinemas: snapshot.cinemas,
      showtimes: snapshot.showtimes,
      account,
    }),
    {
      status: 200,
      headers: {
        "content-type": "application/json",
        "cache-control": "no-store",
      },
    },
  );
}

Promise.all matters here. On a cache miss the handler pays max(340 ms, 35 ms) instead of 340 ms + 35 ms, and on a hit the account call is the only thing left on the critical path.

Collapse concurrent misses

The snapshot expires ten seconds after it is written. At a thousand requests a minute (call it 17 a second), every request in flight at that instant sees an empty cache and calls the backend. The backend gets a burst of simultaneous snapshot requests, each one taking 340 ms, and during those 340 ms more requests arrive and pile onto the same burst. That is a cache stampede, and it lands hardest on exactly the backend the cache was supposed to protect.

A module-scoped promise map fixes it. The first miss starts the fetch and stores the promise; every other miss for the same key awaits that promise instead of starting its own.

Code
const inFlight = new Map<string, Promise<ShowtimeSnapshot>>();

function loadOnce(
  key: string,
  load: () => Promise<ShowtimeSnapshot>,
): Promise<ShowtimeSnapshot> {
  const existing = inFlight.get(key);
  if (existing) {
    return existing;
  }

  const pending = load().finally(() => inFlight.delete(key));
  inFlight.set(key, pending);
  return pending;
}

Module scope in a Zuplo project means isolate scope. The map lives as long as the isolate that loaded the module, and every request served by that isolate shares it.

Serve stale on backend failure

Once a shared fragment is cached, a failed refresh does not have to become a failed request. Keep the cached copy alive well past its freshness window and fall back to it when the backend is down.

Store the snapshot with a timestamp under a long TTL, and treat freshness as something the code decides rather than something the cache enforces. The cache now holds an entry rather than a bare snapshot, and it holds that entry far longer than the snapshot stays fresh:

Code
interface SnapshotEntry {
  snapshot: ShowtimeSnapshot;
  storedAt: number;
}

const FRESH_FOR_SECONDS = 10;
const SERVE_STALE_FOR_SECONDS = 300;

getShowtimeSnapshot now opens the cache as ZoneCache<SnapshotEntry>, compares storedAt against FRESH_FOR_SECONDS, returns the age alongside the snapshot, and logs a warning when it falls back so that stale serves show up in your logs. The full function is in The complete handler below; these are the lines that carry the behavior:

Code
const entry = await cache.get(SNAPSHOT_KEY);
const ageSeconds = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;

if (entry && ageSeconds < FRESH_FOR_SECONDS) {
  return { snapshot: entry.snapshot, ageSeconds };
}

try {
  const snapshot = await loadOnce(SNAPSHOT_KEY, async () => {
    const fresh = await fetchShowtimeSnapshot();
    await cache.put(
      SNAPSHOT_KEY,
      { snapshot: fresh, storedAt: Date.now() },
      SERVE_STALE_FOR_SECONDS,
    );
    return fresh;
  });
  return { snapshot, ageSeconds: 0 };
} catch (error) {
  if (entry) {
    return { snapshot: entry.snapshot, ageSeconds };
  }
  throw error;
}

The cache TTL is now 300 seconds and the freshness window is 10. Between those two numbers sits a copy nobody is served under normal conditions and everybody is served during an outage. A five-minute backend failure degrades the endpoint from "seat counts at most 10 seconds old" to "seat counts up to 5 minutes old" instead of returning 502 to every caller.

For a lot of APIs that availability win is worth more than the cost saving that motivated the change. Surface the age so clients can decide for themselves: x-snapshot-age in the complete handler below is an integer count of seconds.

The complete handler

Everything above, in one file you can drop into modules/ and adapt.

Code
import {
  environment,
  HttpProblems,
  ZoneCache,
  ZuploContext,
  ZuploRequest,
} from "@zuplo/runtime";

interface ShowtimeSnapshot {
  asOf: string;
  cinemas: { id: string; name: string; screens: number }[];
  showtimes: {
    id: string;
    cinemaId: string;
    movieId: string;
    startsAt: string;
    seatsAvailable: number;
  }[];
}

interface AccountState {
  holds: { showtimeId: string; seats: string[]; expiresAt: string }[];
  loyalty: { points: number; tier: string };
}

interface SnapshotEntry {
  snapshot: ShowtimeSnapshot;
  storedAt: number;
}

const SNAPSHOT_KEY = "showtime-snapshot:v1";
const FRESH_FOR_SECONDS = 10;
const SERVE_STALE_FOR_SECONDS = 300;

const inFlight = new Map<string, Promise<ShowtimeSnapshot>>();

function loadOnce(
  key: string,
  load: () => Promise<ShowtimeSnapshot>,
): Promise<ShowtimeSnapshot> {
  const existing = inFlight.get(key);
  if (existing) {
    return existing;
  }

  const pending = load().finally(() => inFlight.delete(key));
  inFlight.set(key, pending);
  return pending;
}

async function fetchJson<T>(path: string): Promise<T> {
  const response = await fetch(`${environment.TICKETING_API_URL}${path}`, {
    headers: { authorization: `Bearer ${environment.TICKETING_API_KEY}` },
  });
  if (!response.ok) {
    throw new Error(`Backend request to ${path} failed: ${response.status}`);
  }
  return (await response.json()) as T;
}

const fetchShowtimeSnapshot = () =>
  fetchJson<ShowtimeSnapshot>("/v1/showtimes/snapshot");

const fetchAccountState = (accountId: string) =>
  fetchJson<AccountState>(`/v1/accounts/${accountId}`);

async function getShowtimeSnapshot(
  context: ZuploContext,
): Promise<{ snapshot: ShowtimeSnapshot; ageSeconds: number }> {
  const cache = new ZoneCache<SnapshotEntry>("showtimes", context);
  const entry = await cache.get(SNAPSHOT_KEY);
  const ageSeconds = entry ? (Date.now() - entry.storedAt) / 1000 : Infinity;

  if (entry && ageSeconds < FRESH_FOR_SECONDS) {
    return { snapshot: entry.snapshot, ageSeconds };
  }

  try {
    const snapshot = await loadOnce(SNAPSHOT_KEY, async () => {
      const fresh = await fetchShowtimeSnapshot();
      await cache.put(
        SNAPSHOT_KEY,
        { snapshot: fresh, storedAt: Date.now() },
        SERVE_STALE_FOR_SECONDS,
      );
      return fresh;
    });
    return { snapshot, ageSeconds: 0 };
  } catch (error) {
    if (entry) {
      context.log.warn("Snapshot refresh failed, serving stale copy", {
        ageSeconds,
        error: String(error),
      });
      return { snapshot: entry.snapshot, ageSeconds };
    }
    throw error;
  }
}

export default async function handler(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const accountId = request.user?.data.accountId;
  if (!accountId) {
    return HttpProblems.unauthorized(request, context);
  }

  try {
    const [shared, account] = await Promise.all([
      getShowtimeSnapshot(context),
      fetchAccountState(accountId),
    ]);

    return new Response(
      JSON.stringify({
        asOf: shared.snapshot.asOf,
        cinemas: shared.snapshot.cinemas,
        showtimes: shared.snapshot.showtimes,
        account,
      }),
      {
        status: 200,
        headers: {
          "content-type": "application/json",
          "cache-control": "no-store",
          "x-snapshot-age": String(Math.round(shared.ageSeconds)),
        },
      },
    );
  } catch (error) {
    context.log.error("Showtime board composition failed", {
      error: String(error),
    });
    return HttpProblems.badGateway(request, context, {
      detail: "The showtimes service is unavailable.",
    });
  }
}

Point a route at it. The handler reads request.user, so keep an authentication policy ahead of it in the inbound pipeline.

Code
{
  "paths": {
    "/v1/showtimes/board": {
      "get": {
        "summary": "Showtime board",
        "x-zuplo-route": {
          "corsPolicy": "none",
          "handler": {
            "export": "default",
            "module": "$import(./modules/showtime-board)"
          },
          "policies": {
            "inbound": ["api-key-inbound"]
          }
        }
      }
    }
  }
}

Choose the cache primitive

Three primitives can hold a fragment. They differ in scope, in what they can store, and in how long a value survives.

When the fragment is a parsed object that code composes into a larger body, ZoneCache wins: a recycled isolate warms from the zone rather than from the backend. Reach for the Cache API instead when the fragment is a response you want kept verbatim (an upstream document, an image, a signed payload) and re-parsing it on every request would be wasted work.

Measure the result

Measure before and after, over the same window and the same routes. Expect tail latency to move more than the median, because a hit removes the slower of the two backend calls.

1. Fragment hit rate. The cache itself does not report this, so count it: log a structured field on hit and on miss, or increment a counter. A healthy hit rate for a 10-second TTL at a thousand requests a minute is above 99%. A number far below that means traffic is spread thinner across zones than expected, or the TTL is shorter than the gap between requests.
2. Backend call volume. Watch the request count against the snapshot endpoint in the Origins section of Analytics, which breaks out volume, error rate, and latency per upstream host. See Origins.
3. p95 on the route. Compare against the same window a week earlier, not against the ten minutes before the deploy. See Metrics glossary for how p95 is computed.

Limitations you own

• Hit rate depends on traffic distribution. ZoneCache is zone-local, and a write in one data center does nothing for any other. Spread the same thousand requests a minute across six zones and the backend sees up to 36 snapshot refreshes a minute rather than six. That is still a 96% reduction, but not the 99.4% a single-zone calculation predicts. Low-volume routes with globally scattered traffic may miss more often than they hit.
• A stale fragment is stale for everybody at once. With a per-caller cache, one unlucky client sees old data. With a shared fragment, every caller sees the same old data simultaneously, and they can compare notes. The TTL is a product decision about what the API promises, not a knob to tune for hit rate.
• You own invalidation. ZoneCache has no global purge and no tag-based eviction. An entry disappears when its TTL expires or when code calls delete in the zone it lives in. If a fragment must drop within a bounded time of an upstream change, the TTL is the only mechanism that guarantees it. Choose it accordingly, and version the key so a schema change does not depend on invalidation at all.
• Two calls cost more on a miss. A cold cache pays for both round trips. Promise.all keeps that concurrent, but the first request after a deploy into a fresh zone is slower than the unsplit version was. Weigh that against how often it happens.

Related

• Caching in Zuplo — which layer to reach for, and why.
• Cache at the gateway — whole-response caching when responses really are identical across callers.
• Build a custom caching policy — the same primitives packaged as a reusable policy instead of a handler.
• Function Handler — the handler contract used throughout this page.