AI Gateway Data Handling
- Applies to
- Gatewayruntime for API, AI, and MCP gateways
- Hosting
- Managed EdgeManaged DedicatedSelf-Hosted
By default, the AI Gateway records usage metadata for each request (who called, which model answered, how many tokens it used, what it cost, and how long it took) and doesn't keep prompt or completion text. Prompt and completion text leaves the request path only through features you add to an app's policy chain or to the gateway's code.
What Zuplo records by default
Each request to an app produces usage records that power AI Gateway analytics and budgets. They hold these fields:
| Field | What it holds |
|---|---|
| App and pool | The app that served the request and the pool it belongs to. |
| Caller | The authenticated caller's identifier (request.user.sub), and the person for a request made with a personal key on the User App. |
| Model and provider | The model that answered and the provider that served it. |
| Tokens | Prompt, completion, and embedding token counts. |
| Cost | The estimated cost, and whether it came from the provider's reported price or a rate card. |
| Latency | How long the request took. See How latency is measured. |
| Cache outcome | Whether the semantic cache served the request, its similarity score, and the tokens a hit saved. |
| Blocks and warnings | The type of block or warning, such as a guardrail match or an exhausted budget, without the matched text. |
| Budget values | For a By metadata budget rule, the value its expression read, such as a header value. See Budget expressions. |
| Request ID, account, project | Identifiers that tie the records to the request's logs. |
The gateway's request logs also hold the method, the full URL including its query string, the route, the status code, the duration, and the client's IP address and location. See Logging.
The gateway's log messages describe what happened to a request, such as which provider served it, a fallback to another model, or a policy that blocked it. When a provider rejects a request, the log message carries the provider's error message.
Neither the usage records nor the request logs include request or response bodies. How long Zuplo keeps them depends on your plan. See Data retention.
Features that store or send content
These features handle prompt or completion text when you turn them on. Each one is off until you add it.
| Feature | Content it handles | Where the content goes |
|---|---|---|
| Semantic Cache | The system prompt and the most recent messages, which form the match key, and the provider's response. | Zuplo's semantic cache, which keeps each app's entries apart. Entries expire after expirationSecondsTtl seconds: one hour by default, up to 30 days. |
| Comet Opik Tracing | Prompt messages or input, completion text, model, token counts, temperature and maximum tokens, request ID, and route. | Your Comet Opik workspace and project. |
| Galileo Tracing | Prompt and completion text, model parameters, finish reasons, token counts, duration, request ID, and route. | Your Galileo project and log stream. |
| Akamai AI Firewall | Prompts and completions, for inspection. | Akamai Firewall for AI. |
| Prompt Injection Protection | The newest message of each request, for classification. | The classifier you configure: an OpenAI-compatible API, or another app on the same gateway and its provider. |
| Smart Router | The last user prompt, for classification. | The classifier you configure, such as another app on the same gateway and its provider. |
| Log plugins | The request logs described earlier and messages that policies write with context.log. Content appears only if a custom policy logs it. | Your logging service, such as Datadog, Splunk, or AWS CloudWatch. |
| OpenTelemetry plugin | Span timings, HTTP method, URL, status code, request ID, model, token usage, and cost. Prompt and completion text only when you turn on content capture. | Zuplo's trace storage or your OpenTelemetry backend. |
The Data Loss Prevention policy scans content inside the gateway and doesn't send it to another service.
Capture prompts and completions for an app
To record the content of an app's requests for debugging or evaluation, add a tracing policy to that app. Other apps on the gateway stay unchanged.
-
Open the app's policy chain
Open the Apps tab, select the app, and then open its Policies tab.
-
Add a tracing policy
Click Add policy and choose Comet Opik tracing or Galileo tracing.
-
Enter the connection settings
For Comet Opik, enter the
apiKey,projectName, andworkspace. For Galileo, enter theapiKey,projectId, andlogStreamId. Reference the API key as an environment variable, such as$env(COMET_OPIK_API_KEY), rather than pasting it. See Options and secrets. -
Move the policy to the end of the chain
Drag the entry to the end of the chain so the trace records the request that earlier policies produced.
The change applies to new requests after the app's cached configuration refreshes, with no deploy. To stop capturing content, remove the entry or disable it.
To capture content for every new app in a pool, add the tracing policy to the pool's policy template.
Keep sensitive data out of stored content
Policies run in chain order, and each one sees the request as the earlier policies left it. Put Data Loss Prevention before the other policies in the table above, so the cache key, the traces, and the classifiers receive the masked prompt instead of the original text.