AI Gateway OpenTelemetry Tracing

An AI Gateway runs the same OpenTelemetry plugin as any Zuplo project. With the plugin on, each request to an app becomes a trace with a span for every policy in the app's chain, the AI Gateway handler, and every call to an LLM provider. Provider calls are recorded as OpenTelemetry GenAI spans that carry the model, token usage, cost, and outcome of each attempt. The gateway reads W3C trace context (traceparent) from the caller and sends it on to the provider, so an agent's model calls join the agent's own trace.

Prerequisites

• Connect the gateway to a Git repository so you can change its code. See AI Gateway source control.
• Tracing to Zuplo works on every plan. Sending traces to your own OpenTelemetry backend requires an Enterprise plan.

Enable tracing

The ratio keeps every trace from your working copy and 10% of traces elsewhere. See Sample a percentage of traces to change it.

What a trace contains

A request to an app produces a trace like the following. The app's policy chain appears inside the Configuration Executor span, one span per entry. Each call to an LLM provider gets a GenAI inference span, with the HTTP call to the provider's host nested under it:

Code
POST <route>                                   gen_ai.request.model, zuplo.ai_gateway.*
├── policies:inbound
│   ├── policy:ai-gateway-configuration-loader-inbound
│   └── policy:ai-gateway-configuration-executor-inbound
│       └── policies:inbound
│           ├── policy:ai-gateway-model-filtering-inbound
│           ├── policy:ai-gateway-metering-inbound
│           └── policy:ai-gateway-semantic-cache-inbound
└── handler:aiGatewayHandlerV2
    └── chat gpt-5                             gen_ai.*, zuplo.ai_gateway.cost.*
        └── POST: api.openai.com

Every span carries timings and the request ID (zp-rid) that also appears in the gateway's logs. HTTP spans add the method, URL, and status code. Prompt and completion text is recorded only when you turn on content capture.

GenAI inference spans

The gateway records one inference span for each attempt to call a provider: the primary call, each fallback, a quota fallback after a usage limit is exceeded, and each retry after a provider returns 429. The spans follow the OpenTelemetry semantic conventions for generative AI, so backends that understand GenAI telemetry, such as Langfuse, Datadog LLM Observability, Honeycomb, and Grafana, show model, token, and cost views without any mapping.

There's nothing extra to enable. With the OpenTelemetry plugin registered, inference spans appear for every sampled request. All of them use the instrumentation scope ai-gateway.

Span names and kinds

Each inference span is a CLIENT span named {operation} {model}, for example chat gpt-5 or embeddings text-embedding-3-small. The operation comes from the endpoint the client called:

A request that never reaches a provider, such as one a policy rejects or one that fails validation, has no inference span. A semantic cache hit has none either, because the gateway answers from the cache.

Inference span attributes

Cost uses the same prices as budgets and spend analytics, including any custom pricing you set.

Request span attributes

The request's root SERVER span summarizes what the gateway did with the request:

Errors and fallbacks

A failed attempt sets the span status to error and records a low-cardinality error.type:

• The provider's HTTP status code, for example 503 or 429.
• timeout when the attempt hit the fallback timeout.
• The error name, for example TypeError, when the call threw before a response arrived.

The provider's error message isn't recorded, because it can quote the request.

Attempts for one request are siblings under the handler span. A request whose primary model returns 503 and whose fallback succeeds looks like this:

Code
handler:aiGatewayHandlerV2
├── chat gpt-5              error.type=503, zuplo.ai_gateway.attempt=primary
│   └── POST: api.openai.com
└── chat claude-sonnet-4-5  zuplo.ai_gateway.attempt=fallback, usage, cost
    └── POST: api.anthropic.com

Streaming responses

For a streamed response, the HTTP span ends when the provider starts streaming, and the inference span stays open until the stream finishes. The inference span records:

• gen_ai.response.time_to_first_chunk: seconds from sending the request to the first chunk of the stream.
• zuplo.ai_gateway.stream.end: complete, error (including an error event inside the stream), or client_cancelled when the client disconnected first.
• Token usage and cost, read from the finished stream.

The plugin waits for an open inference span before it exports the trace, up to five minutes after the response starts, so long streams export complete.

Capture prompts and completions

Inference spans don't record prompt or completion text by default. To record it, set instrumentation.genAi.captureContent to true:

modules/zuplo.runtime.ts
runtime.addPlugin(
  new OpenTelemetryPlugin({
    exporter: {
      url: environment.OTEL_TRACES_URL,
      headers: { "api-key": environment.OTEL_API_KEY },
    },
    instrumentation: {
      genAi: { captureContent: true },
    },
  }),
);

With capture on, each inference span records these attributes as JSON in the OpenTelemetry GenAI message format:

• gen_ai.system_instructions: the system prompt.
• gen_ai.input.messages: the messages sent to the provider. Embeddings requests don't record input.
• gen_ai.output.messages: the response the client received, including a streamed response.

Each attribute is capped at 8 KiB and stays valid JSON when it's truncated. The plugin turns capture on only for the boolean true.

Input is recorded as sent to the provider, after policies such as Data Loss Prevention changed it. Output is recorded as returned to the client, after every policy and hook that changes the response. Well-known credential formats, such as API keys, bearer tokens, JWTs, private keys, and cloud and GitHub tokens, are replaced with placeholders such as [REDACTED:api_key].

To capture content for a single app instead of the whole gateway, use a tracing policy. See AI Gateway data handling.

Query GenAI spans in your backend

Because the attributes follow the semantic conventions, you can query model activity directly in your backend. For example:

• Compare models by latency: filter on gen_ai.operation.name = chat and group span duration by gen_ai.response.model.
• Find spend by provider: sum zuplo.ai_gateway.cost.usd grouped by zuplo.ai_gateway.provider.label.
• Find failing providers: filter on error.type and group by gen_ai.provider.name.
• See how often fallbacks serve traffic: count spans where zuplo.ai_gateway.attempt is fallback or quota_fallback.
• Measure time to first token: chart gen_ai.response.time_to_first_chunk for streamed requests.
• Find rejected requests: filter request spans on zuplo.ai_gateway.refused_by.

Follow an agent run across gateways

An agent run usually mixes model calls and tool calls. To see the whole run as one trace, start a span in the agent and send its traceparent header on every request it makes:

• Model calls go to the AI Gateway, which continues the trace and passes traceparent on to the LLM provider.
• Tool calls go to the MCP Gateway, which continues the same trace. See MCP Gateway OpenTelemetry.

Add the OpenTelemetry plugin to both projects and point them at the same backend. The backend then shows the agent's span with the AI Gateway and MCP Gateway spans nested under it.

Most OpenTelemetry HTTP instrumentations add traceparent to outgoing requests for you. To add it yourself, inject the active context into the request headers. This example uses the OpenAI SDK with an AI Gateway app:

Code
import { context, propagation, trace } from "@opentelemetry/api";
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ZUPLO_AI_GATEWAY_API_KEY,
  baseURL: "https://GATEWAY_HOST/APP_ID/v1",
});

const tracer = trace.getTracer("my-agent");

await tracer.startActiveSpan("agent-run", async (span) => {
  const headers: Record<string, string> = {};
  propagation.inject(context.active(), headers);

  const response = await client.chat.completions.create(
    {
      model: "openai/gpt-6-luna",
      messages: [{ role: "user", content: "Summarize today's tickets." }],
    },
    { headers },
  );

  // Send the same headers on the agent's MCP tool calls.
  span.end();
  return response;
});

Replace GATEWAY_HOST/APP_ID with the app's API URL from the Zuplo Portal.

By default, a gateway samples a request that arrives with a traceparent exactly when the caller sampled it, so the agent's sampling decision covers every span in the run. The ratio applies only to requests that arrive without trace context.

View and export traces

Where you view traces depends on the plugin configuration:

• Zuplo: with no exporter, open Observability → Traces at the account level and filter by service, which defaults to the project name.
• Your own backend: traces arrive at the exporter URL. Open them in that tool, for example Honeycomb, Grafana Tempo, Jaeger, or Dynatrace.

To send traces to Zuplo and your own backend at once, or to export logs as well, see OpenTelemetry.