# AI Gateway OpenTelemetry Tracing
> **Scope of this page**
>
> This page documents the Zuplo **gateway** runtime (API, AI, and MCP gateways), not the Developer Portal.
>
> It applies to every Zuplo hosting option: Managed Edge, Managed Dedicated, and Self-Hosted. See [Hosting options](https://zuplo.com/docs/articles/hosting-options).
An AI Gateway runs the same
[OpenTelemetry plugin](../../articles/opentelemetry.mdx) as any Zuplo project.
With the plugin on, each request to an app becomes a trace with a span for every
policy in the app's chain, the AI Gateway handler, and every call to an LLM
provider. Provider calls are recorded as OpenTelemetry GenAI spans that carry
the model, token usage, cost, and outcome of each attempt. The gateway reads W3C
trace context (`traceparent`) from the caller and sends it on to the provider,
so an agent's model calls join the agent's own trace.
Agent
AI Gateway
MCP Gateway
LLM provider
MCP server
Trace backend
## Prerequisites
- Connect the gateway to a Git repository so you can change its code. See
[AI Gateway source control](../source-control.mdx).
- Tracing to Zuplo works on every plan. Sending traces to your own OpenTelemetry
backend requires an Enterprise plan.
## Enable tracing
1. **Register the plugin**
In `modules/zuplo.runtime.ts`, add the plugin in `runtimeInit`. With no
`exporter`, the plugin sends traces to Zuplo:
```ts title="modules/zuplo.runtime.ts"
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { environment, RuntimeExtensions } from "@zuplo/runtime";
export function runtimeInit(runtime: RuntimeExtensions) {
const isWorkingCopy =
environment.ZUPLO_ENVIRONMENT_STAGE === "working-copy";
runtime.addPlugin(
new OpenTelemetryPlugin({
sampling: {
headSampler: { ratio: isWorkingCopy ? 1 : 0.1 },
},
}),
);
}
```
To send traces to your own backend instead, add an `exporter` with your
backend's URL and authentication header, and set `OTEL_TRACES_URL` and
`OTEL_API_KEY` as
[environment variables](../../articles/environment-variables.mdx):
```ts
new OpenTelemetryPlugin({
exporter: {
url: environment.OTEL_TRACES_URL,
headers: { "api-key": environment.OTEL_API_KEY },
},
sampling: {
headSampler: { ratio: isWorkingCopy ? 1 : 0.1 },
},
});
```
See
[Export to your own backend](../../articles/opentelemetry.mdx#export-to-your-own-backend).
2. **Deploy**
Commit the change and push it to your default branch. The gateway deploys,
and new requests start producing traces.
The `ratio` keeps every trace from your working copy and 10% of traces
elsewhere. See
[Sample a percentage of traces](../../articles/opentelemetry.mdx#sample-a-percentage-of-traces)
to change it.
## What a trace contains
A request to an app produces a trace like the following. The app's policy chain
appears inside the Configuration Executor span, one span per entry. Each call to
an LLM provider gets a GenAI inference span, with the HTTP call to the
provider's host nested under it:
```text
POST gen_ai.request.model, zuplo.ai_gateway.*
├── policies:inbound
│ ├── policy:ai-gateway-configuration-loader-inbound
│ └── policy:ai-gateway-configuration-executor-inbound
│ └── policies:inbound
│ ├── policy:ai-gateway-model-filtering-inbound
│ ├── policy:ai-gateway-metering-inbound
│ └── policy:ai-gateway-semantic-cache-inbound
└── handler:aiGatewayHandlerV2
└── chat gpt-5 gen_ai.*, zuplo.ai_gateway.cost.*
└── POST: api.openai.com
```
Every span carries timings and the request ID (`zp-rid`) that also appears in
the gateway's logs. HTTP spans add the method, URL, and status code. Prompt and
completion text is recorded only when you
[turn on content capture](#capture-prompts-and-completions).
## GenAI inference spans
The gateway records one inference span for each attempt to call a provider: the
primary call, each [fallback](../fallback.mdx), a
[quota fallback](../usage-limits.mdx#when-a-limit-is-exceeded) after a usage
limit is exceeded, and each retry after a provider returns `429`. The spans
follow the
[OpenTelemetry semantic conventions for generative AI](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/),
so backends that understand GenAI telemetry, such as Langfuse, Datadog LLM
Observability, Honeycomb, and Grafana, show model, token, and cost views without
any mapping.
There's nothing extra to enable. With the OpenTelemetry plugin registered,
inference spans appear for every sampled request. All of them use the
instrumentation scope `ai-gateway`.
### Span names and kinds
Each inference span is a `CLIENT` span named `{operation} {model}`, for example
`chat gpt-5` or `embeddings text-embedding-3-small`. The operation comes from
the endpoint the client called:
| Endpoint | `gen_ai.operation.name` |
| ------------------------------------------------- | ----------------------- |
| `/v1/chat/completions`, `/v1/messages` | `chat` |
| `POST /v1/responses` | `chat` |
| `GET /v1/responses/{id}` | `fetch_response` |
| `DELETE /v1/responses/{id}` | `delete_response` |
| `/v1/messages/count_tokens` | `count_tokens` |
| `/v1/embeddings` | `embeddings` |
| Bedrock Runtime `converse` and `invoke` endpoints | `chat` |
A request that never reaches a provider, such as one a policy rejects or one
that fails validation, has no inference span. A semantic cache hit has none
either, because the gateway answers from the cache.
### Inference span attributes
| Attribute | Description |
| -------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `gen_ai.operation.name` | The operation, from the table above. |
| `gen_ai.provider.name` | The provider, for example `openai`, `anthropic`, `gcp.vertex_ai`, `aws.bedrock`, or `azure.ai.openai`. |
| `zuplo.ai_gateway.provider.label` | The name of the provider in your gateway's **AI Providers** settings. |
| `gen_ai.request.model` | The model sent to the provider. |
| `gen_ai.request.stream` | `true` for a streamed request, otherwise `false`. |
| `gen_ai.request.max_tokens`, `temperature`, `top_p`, `top_k`, `seed`, and others | Sampling parameters, when the request sets them. |
| `gen_ai.response.id` | The provider's response ID. |
| `gen_ai.response.model` | The model that answered, as the provider reported it. |
| `gen_ai.response.finish_reasons` | The provider's own finish reasons, for example `["stop"]` or `["end_turn"]`. |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Token usage. Matches the usage that [AI Gateway analytics](./analytics.mdx) and usage limits record. |
| `gen_ai.usage.cache_read.input_tokens`, `gen_ai.usage.cache_write.input_tokens` | Prompt cache tokens, when the provider reports them. |
| `gen_ai.usage.reasoning.output_tokens` | Reasoning tokens, when the provider reports them. |
| `zuplo.ai_gateway.cost.usd` | The cost of the attempt in US dollars. |
| `zuplo.ai_gateway.cost.source` | Where the cost came from: `provider`, `catalog`, or `unpriced`. |
| `zuplo.ai_gateway.cost.accuracy` | `exact`, `estimated`, or `unpriced`. |
| `zuplo.ai_gateway.attempt` | `primary`, `fallback`, `quota_fallback`, or `retry` (a retry after a `429`). |
| `server.address`, `server.port` | The provider host. |
| `error.type` | Set on a failed attempt. See [Errors and fallbacks](#errors-and-fallbacks). |
Cost uses the same prices as budgets and spend analytics, including any
[custom pricing](../custom-pricing.mdx) you set.
### Request span attributes
The request's root `SERVER` span summarizes what the gateway did with the
request:
| Attribute | Description |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `gen_ai.request.model` | The model the client asked for, before any routing or fallback policy changed it. |
| `zuplo.ai_gateway.cache.status` | `hit` or `miss`, when the app uses the [semantic cache](../../policies/ai-gateway-semantic-cache-inbound.mdx). |
| `zuplo.ai_gateway.refused_by` | The name of the policy that rejected the request, or `handler` when the gateway's handler rejected it. |
| `zuplo.ai_gateway.quota_fallback` | `true` when a usage limit sent the request to its quota fallback model. |
### Errors and fallbacks
A failed attempt sets the span status to error and records a low-cardinality
`error.type`:
- The provider's HTTP status code, for example `503` or `429`.
- `timeout` when the attempt hit the fallback timeout.
- The error name, for example `TypeError`, when the call threw before a response
arrived.
The provider's error message isn't recorded, because it can quote the request.
Attempts for one request are siblings under the handler span. A request whose
primary model returns `503` and whose fallback succeeds looks like this:
```text
handler:aiGatewayHandlerV2
├── chat gpt-5 error.type=503, zuplo.ai_gateway.attempt=primary
│ └── POST: api.openai.com
└── chat claude-sonnet-4-5 zuplo.ai_gateway.attempt=fallback, usage, cost
└── POST: api.anthropic.com
```
### Streaming responses
For a streamed response, the HTTP span ends when the provider starts streaming,
and the inference span stays open until the stream finishes. The inference span
records:
- `gen_ai.response.time_to_first_chunk`: seconds from sending the request to the
first chunk of the stream.
- `zuplo.ai_gateway.stream.end`: `complete`, `error` (including an error event
inside the stream), or `client_cancelled` when the client disconnected first.
- Token usage and cost, read from the finished stream.
The plugin waits for an open inference span before it exports the trace, up to
five minutes after the response starts, so long streams export complete.
## Capture prompts and completions
Inference spans don't record prompt or completion text by default. To record it,
set `instrumentation.genAi.captureContent` to `true`:
```ts title="modules/zuplo.runtime.ts"
runtime.addPlugin(
new OpenTelemetryPlugin({
exporter: {
url: environment.OTEL_TRACES_URL,
headers: { "api-key": environment.OTEL_API_KEY },
},
instrumentation: {
genAi: { captureContent: true },
},
}),
);
```
With capture on, each inference span records these attributes as JSON in the
OpenTelemetry GenAI message format:
- `gen_ai.system_instructions`: the system prompt.
- `gen_ai.input.messages`: the messages sent to the provider. Embeddings
requests don't record input.
- `gen_ai.output.messages`: the response the client received, including a
streamed response.
Each attribute is capped at 8 KiB and stays valid JSON when it's truncated. The
plugin turns capture on only for the boolean `true`.
Input is recorded as sent to the provider, after policies such as
[Data Loss Prevention](../../policies/ai-gateway-dlp-inbound.mdx) changed it.
Output is recorded as returned to the client, after every policy and hook that
changes the response. Well-known credential formats, such as API keys, bearer
tokens, JWTs, private keys, and cloud and GitHub tokens, are replaced with
placeholders such as `[REDACTED:api_key]`.
:::warning
Prompts and completions often contain personal or otherwise sensitive data.
Turning on content capture sends that content to Zuplo's trace storage and to
any backend you export to. Content capture doesn't remove personal data, so mask
it with the Data Loss Prevention policy, which runs before the content is
recorded.
:::
To capture content for a single app instead of the whole gateway, use a tracing
policy. See
[AI Gateway data handling](./data-handling.mdx#capture-prompts-and-completions-for-an-app).
## Query GenAI spans in your backend
Because the attributes follow the semantic conventions, you can query model
activity directly in your backend. For example:
- Compare models by latency: filter on `gen_ai.operation.name = chat` and group
span duration by `gen_ai.response.model`.
- Find spend by provider: sum `zuplo.ai_gateway.cost.usd` grouped by
`zuplo.ai_gateway.provider.label`.
- Find failing providers: filter on `error.type` and group by
`gen_ai.provider.name`.
- See how often fallbacks serve traffic: count spans where
`zuplo.ai_gateway.attempt` is `fallback` or `quota_fallback`.
- Measure time to first token: chart `gen_ai.response.time_to_first_chunk` for
streamed requests.
- Find rejected requests: filter request spans on `zuplo.ai_gateway.refused_by`.
:::note
The OpenTelemetry plugin exports traces and logs, not metrics. Use span
durations and attributes to measure latency and usage, or see
[AI Gateway analytics](./analytics.mdx) for built-in dashboards.
:::
## Follow an agent run across gateways
An agent run usually mixes model calls and tool calls. To see the whole run as
one trace, start a span in the agent and send its `traceparent` header on every
request it makes:
- **Model calls** go to the AI Gateway, which continues the trace and passes
`traceparent` on to the LLM provider.
- **Tool calls** go to the MCP Gateway, which continues the same trace. See
[MCP Gateway OpenTelemetry](../../mcp-gateway/observability/logging.mdx#opentelemetry).
Add the OpenTelemetry plugin to both projects and point them at the same
backend. The backend then shows the agent's span with the AI Gateway and MCP
Gateway spans nested under it.
Most OpenTelemetry HTTP instrumentations add `traceparent` to outgoing requests
for you. To add it yourself, inject the active context into the request headers.
This example uses the OpenAI SDK with an AI Gateway app:
```ts
import { context, propagation, trace } from "@opentelemetry/api";
import OpenAI from "openai";
const client = new OpenAI({
apiKey: process.env.ZUPLO_AI_GATEWAY_API_KEY,
baseURL: "https://GATEWAY_HOST/APP_ID/v1",
});
const tracer = trace.getTracer("my-agent");
await tracer.startActiveSpan("agent-run", async (span) => {
const headers: Record = {};
propagation.inject(context.active(), headers);
const response = await client.chat.completions.create(
{
model: "openai/gpt-6-luna",
messages: [{ role: "user", content: "Summarize today's tickets." }],
},
{ headers },
);
// Send the same headers on the agent's MCP tool calls.
span.end();
return response;
});
```
Replace `GATEWAY_HOST/APP_ID` with the app's API URL from the Zuplo Portal.
By default, a gateway samples a request that arrives with a `traceparent`
exactly when the caller sampled it, so the agent's sampling decision covers
every span in the run. The `ratio` applies only to requests that arrive without
trace context.
## View and export traces
Where you view traces depends on the plugin configuration:
- **Zuplo**: with no `exporter`, open
[**Observability → Traces**](https://portal.zuplo.com/+/account/observability/traces)
at the account level and filter by **service**, which defaults to the project
name.
- **Your own backend**: traces arrive at the `exporter` URL. Open them in that
tool, for example Honeycomb, Grafana Tempo, Jaeger, or Dynatrace.
To send traces to Zuplo and your own backend at once, or to export logs as well,
see
[OpenTelemetry](../../articles/opentelemetry.mdx#send-traces-to-zuplo-and-your-own-backend).