---
title: "Reading Load Test Results with Distributed Tracing"
description: "A load test shows that latency increased or requests failed. Distributed tracing shows where. Learn how to attribute errors, measure hops with one clock, propagate trace context, and account for tracing overhead."
canonicalUrl: "https://zuplo.com/blog/2026/08/18/load-testing-with-tracing"
pageType: "blog"
date: "2026-08-18"
authors: "nate"
tags: "API Gateway, API Best Practices"
image: "https://zuplo.com/og?text=Reading%20Load%20Test%20Results%20with%20Distributed%20Tracing"
---
A load generator can tell you that p99 increased or that 500 responses appeared.
It cannot tell you which component caused either result. We have seen teams lose
hours arguing between generator output and a clean APM dashboard when the
failure sat in the infrastructure between them.
The request may cross DNS, a CDN, a WAF, a load balancer, a gateway, an origin
platform, and application code. To identify the failing segment, correlate the
load-test result with logs and traces from inside that path.
## Add observation points between the generator and application
The generator observes the final response. Application telemetry observes only
requests that reached application code. Failures in a CDN, load balancer,
gateway, or origin platform can exist in one dataset and not the other.
Collect request logs from at least one intermediary. A gateway is useful because
it can record both the upstream response and the downstream response, but the
same method works with CDN and load-balancer access logs.
Use a shared request ID or trace ID whenever possible. Time-window correlation
alone becomes unreliable at load-test volumes.
## Attribute failures one segment at a time
Compare adjacent observation points from the generator toward the application:
- If an intermediary records a failed upstream response, the failure came from
that intermediary or something behind it.
- If the generator records a failure that the first intermediary never saw, the
failure occurred before that intermediary.
- If one layer records the request and the next does not, investigate the
segment between them.
- If a layer returns its own timeout or rejection, distinguish that generated
status from a status passed through from upstream.
This works only if results are already
[segmented by status code](/blog/how-not-to-load-test-an-api). An aggregate
failure rate does not tell you whether you are tracing timeouts, overload
responses, or application errors.
## Measure each hop with one clock
Do not subtract timestamps from different machines to calculate transit time.
Clock offset, synchronization drift, scheduler delay, and uninstrumented work
all end up in the result.
For example, this expression looks reasonable but is not a reliable network
measurement:
```text
transit = (gateway timestamp + gateway duration) - origin timestamp
```
The result combines two clocks and two different instrumentation boundaries. A
sudden increase might be network latency, clock drift, queueing before the
origin timestamp, or work omitted from the gateway duration.
Measure the start and end of a hop from one process's monotonic clock. Compare
supporting measurements such as connection-establishment time, TLS time, and
upstream response time from the same observation point.
Edge runtimes need one more caveat. Some isolate-based runtimes advance their
high-resolution clocks only during I/O. Cloudflare Workers
[documents this behavior](https://developers.cloudflare.com/workers/runtime-apis/performance/),
and Zuplo's runtime follows the same model. Pure CPU work between I/O operations
can therefore appear to take little or no time. Treat self-reported duration as
I/O time unless the runtime documents otherwise.
## Use span waterfalls to find the expensive operation
Logs identify the failing layer. Distributed tracing breaks the work inside that
layer into timed spans.
A trace waterfall should cover the full request lifecycle: inbound policies, the
handler, upstream calls, outbound policies, and custom code. Compare traces at a
healthy rate and at the rate where latency starts to increase. The span that
grows is a much better lead than the end-to-end p99.
Zuplo's [OpenTelemetry tracing views](/changelog/2026/06/17/otel-tracing-views)
provide these spans in the portal. Add the OpenTelemetry plugin to
`zuplo.runtime.ts` in your `modules` directory:
```ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions } from "@zuplo/runtime";
export function runtimeInit(runtime: RuntimeExtensions) {
runtime.addPlugin(new OpenTelemetryPlugin());
}
```
`runtimeInit` runs when a gateway instance starts. Requests then produce spans
for inbound policies, the handler, outbound policies, and `fetch` calls from
custom code. See the [OpenTelemetry documentation](/docs/articles/opentelemetry)
for configuration and availability.
The useful workflow is simple, and much less glamorous than the phrase
"distributed tracing" makes it sound:
1. Filter traces to the load-test window.
2. Filter again to failed requests or the slow latency band.
3. Compare span durations with traces from a healthy rate.
4. Identify the first span whose latency or status changes.
Grafana's k6 team has a
[worked example](https://grafana.com/blog/troubleshoot-failed-performance-tests-faster-with-distributed-tracing-in-grafana-cloud-k6/)
that follows a failed performance test to a slow database query.
The Zuplo trace above is a good example of what you actually get. The request
took 116ms across 17 spans. Inbound policies used 68ms, including 41ms in rate
limiting and 26ms in authorization. The handler's upstream call took 45ms. That
is enough detail to compare the same path at 50 RPS and 2,000 RPS and see which
operation moved.
## Propagate trace context from the load generator
The [W3C Trace Context standard](https://www.w3.org/TR/trace-context/) defines
the `traceparent` header used to carry a trace ID across services. If the load
generator creates that header and every layer propagates it, the client result
and server trace become one dataset.
The k6
[`http-instrumentation-tempo` library](https://grafana.com/docs/k6/latest/javascript-api/jslib/http-instrumentation-tempo/)
can instrument requests with W3C trace context:
```javascript
import http from "k6/http";
import tempo from "https://jslib.k6.io/http-instrumentation-tempo/1.0.1/index.js";
tempo.instrumentHTTP({ propagator: "w3c" });
const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";
export const options = {
discardResponseBodies: true,
scenarios: {
steady: {
executor: "constant-arrival-rate",
rate: 200,
timeUnit: "1s",
duration: "10m",
preAllocatedVUs: 100,
maxVUs: 300,
},
},
thresholds: {
http_req_failed: ["rate<0.01"],
http_req_duration: ["p(99)<800"],
},
};
export default function () {
http.get(`${BASE_URL}/items`);
}
```
The library wraps k6's HTTP methods and creates trace context for each request.
This is not Zuplo-specific. Any component that accepts and propagates W3C trace
context can join the trace.
## Add custom spans around meaningful operations
An automatically generated span for a custom policy may still be too broad. Add
child spans around external calls or other operations you need to compare.
This policy uses `startActiveSpan` so child `fetch` spans inherit the correct
parent. The `finally` block guarantees that the span closes when the policy
throws:
```ts
import { ZuploContext, ZuploRequest } from "@zuplo/runtime";
import { trace } from "@opentelemetry/api";
export default async function policy(
request: ZuploRequest,
context: ZuploContext,
) {
const tracer = trace.getTracer("my-tracer");
return tracer.startActiveSpan("my-custom-operation", async (span) => {
span.setAttribute("endpoint", request.url);
try {
// Run policy logic and external calls here.
return request;
} finally {
span.end();
}
});
}
```
In an I/O-clock runtime, a span around `fetch` is meaningful. A span around a
tight parsing loop can appear nearly instant even when it consumes CPU. Keep
that runtime behavior in mind when reading short spans.
## Measure the cost of tracing during the load test
Tracing adds allocation, buffering, processing, export, and storage work. We
learned how much that matters during an internal test at about 8,000 RPS. The
gateway appeared to have less capacity than expected because we had left tracing
at 100% sampling.
The extra allocations and trace buffers increased memory pressure, instances
recycled earlier, and their replacements repeatedly paid startup cost. We were
measuring our telemetry almost as much as the gateway.
That investigation also found real bugs in our tracing plugin: spans left open
on error paths and per-trace buffers that leaked. We fixed them. A sustained
load test reaches error-path code millions of times, which is exactly where
telemetry bugs stop being theoretical.
Run at least two passes when capacity is the result you plan to publish:
1. Use the sampling ratio configured in production.
2. Disable tracing to measure its overhead.
Do not describe a 100%-sampled result as raw API or gateway capacity unless 100%
sampling is the production configuration.
Zuplo supports head sampling in the plugin configuration:
```ts
import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions } from "@zuplo/runtime";
export function runtimeInit(runtime: RuntimeExtensions) {
runtime.addPlugin(
new OpenTelemetryPlugin({
sampling: {
headSampler: {
ratio: 0.05,
},
},
}),
);
}
```
Head and tail sampling have different trade-offs:
| | Head sampling | Tail sampling |
| -------------- | --------------------------------- | -------------------------------------------------- |
| Decision time | At the start of the trace | After the trace completes |
| Retention | A fixed ratio of all traces | Traces selected by latency, status, or other rules |
| Infrastructure | No stateful trace buffer required | Collector holds spans until the trace is complete |
| Load-test risk | Slow traces may not be selected | The collector can become the bottleneck |
Sampling reduces export and storage work, but instrumentation still runs for
every request. A sampled pass is cheaper than full sampling, not free. Zuplo
exports traces whose root span ends in an error regardless of the head-sampling
ratio, which keeps failures available during heavily sampled runs.
## A tracing checklist for load tests
Before the run:
- Enable request IDs or trace IDs across the path.
- Confirm each intermediary records upstream and downstream status separately.
- Propagate `traceparent` from the generator.
- Choose the sampling strategy and estimate collector capacity.
- Record the production sampling ratio in the test plan.
During analysis:
- Attribute errors between adjacent observation points.
- Measure each hop from one clock.
- Compare slow traces with healthy traces at a lower request rate.
- Rerun with tracing disabled before publishing a capacity number.
Tracing turns "p99 increased at 2,000 RPS" into a specific operation that got
slower. The final article in this series covers the operational work around a
large test: provider coordination, budgets, soak tests, and abort criteria in
the [API load-testing playbook](/blog/api-load-testing-playbook).