Reading Load Test Results with Distributed Tracing

A load generator can tell you that p99 increased or that 500 responses appeared. It cannot tell you which component caused either result. We have seen teams lose hours arguing between generator output and a clean APM dashboard when the failure sat in the infrastructure between them.

The request may cross DNS, a CDN, a WAF, a load balancer, a gateway, an origin platform, and application code. To identify the failing segment, correlate the load-test result with logs and traces from inside that path.

Add observation points between the generator and application

The generator observes the final response. Application telemetry observes only requests that reached application code. Failures in a CDN, load balancer, gateway, or origin platform can exist in one dataset and not the other.

Collect request logs from at least one intermediary. A gateway is useful because it can record both the upstream response and the downstream response, but the same method works with CDN and load-balancer access logs.

Use a shared request ID or trace ID whenever possible. Time-window correlation alone becomes unreliable at load-test volumes.

Attribute failures one segment at a time

Compare adjacent observation points from the generator toward the application:

• If an intermediary records a failed upstream response, the failure came from that intermediary or something behind it.
• If the generator records a failure that the first intermediary never saw, the failure occurred before that intermediary.
• If one layer records the request and the next does not, investigate the segment between them.
• If a layer returns its own timeout or rejection, distinguish that generated status from a status passed through from upstream.

This works only if results are already segmented by status code. An aggregate failure rate does not tell you whether you are tracing timeouts, overload responses, or application errors.

Measure each hop with one clock

Do not subtract timestamps from different machines to calculate transit time. Clock offset, synchronization drift, scheduler delay, and uninstrumented work all end up in the result.

For example, this expression looks reasonable but is not a reliable network measurement:

transit = (gateway timestamp + gateway duration) - origin timestamp

The result combines two clocks and two different instrumentation boundaries. A sudden increase might be network latency, clock drift, queueing before the origin timestamp, or work omitted from the gateway duration.

Measure the start and end of a hop from one process’s monotonic clock. Compare supporting measurements such as connection-establishment time, TLS time, and upstream response time from the same observation point.

Edge runtimes need one more caveat. Some isolate-based runtimes advance their high-resolution clocks only during I/O. Cloudflare Workers documents this behavior, and Zuplo’s runtime follows the same model. Pure CPU work between I/O operations can therefore appear to take little or no time. Treat self-reported duration as I/O time unless the runtime documents otherwise.

Use span waterfalls to find the expensive operation

Logs identify the failing layer. Distributed tracing breaks the work inside that layer into timed spans.

A trace waterfall should cover the full request lifecycle: inbound policies, the handler, upstream calls, outbound policies, and custom code. Compare traces at a healthy rate and at the rate where latency starts to increase. The span that grows is a much better lead than the end-to-end p99.

Zuplo’s OpenTelemetry tracing views provide these spans in the portal. Add the OpenTelemetry plugin to zuplo.runtime.ts in your modules directory:

import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(new OpenTelemetryPlugin());
}

runtimeInit runs when a gateway instance starts. Requests then produce spans for inbound policies, the handler, outbound policies, and fetch calls from custom code. See the OpenTelemetry documentation for configuration and availability.

The useful workflow is simple, and much less glamorous than the phrase “distributed tracing” makes it sound:

1. Filter traces to the load-test window.
2. Filter again to failed requests or the slow latency band.
3. Compare span durations with traces from a healthy rate.
4. Identify the first span whose latency or status changes.

Grafana’s k6 team has a worked example that follows a failed performance test to a slow database query.

The Zuplo trace above is a good example of what you actually get. The request took 116ms across 17 spans. Inbound policies used 68ms, including 41ms in rate limiting and 26ms in authorization. The handler’s upstream call took 45ms. That is enough detail to compare the same path at 50 RPS and 2,000 RPS and see which operation moved.

Propagate trace context from the load generator

The W3C Trace Context standard defines the traceparent header used to carry a trace ID across services. If the load generator creates that header and every layer propagates it, the client result and server trace become one dataset.

The k6 http-instrumentation-tempo library can instrument requests with W3C trace context:

import http from "k6/http";
import tempo from "https://jslib.k6.io/http-instrumentation-tempo/1.0.1/index.js";

tempo.instrumentHTTP({ propagator: "w3c" });

const BASE_URL = __ENV.TARGET_URL || "https://api.example.com";

export const options = {
  discardResponseBodies: true,
  scenarios: {
    steady: {
      executor: "constant-arrival-rate",
      rate: 200,
      timeUnit: "1s",
      duration: "10m",
      preAllocatedVUs: 100,
      maxVUs: 300,
    },
  },
  thresholds: {
    http_req_failed: ["rate<0.01"],
    http_req_duration: ["p(99)<800"],
  },
};

export default function () {
  http.get(`${BASE_URL}/items`);
}

The library wraps k6’s HTTP methods and creates trace context for each request. This is not Zuplo-specific. Any component that accepts and propagates W3C trace context can join the trace.

Add custom spans around meaningful operations

An automatically generated span for a custom policy may still be too broad. Add child spans around external calls or other operations you need to compare.

This policy uses startActiveSpan so child fetch spans inherit the correct parent. The finally block guarantees that the span closes when the policy throws:

import { ZuploContext, ZuploRequest } from "@zuplo/runtime";
import { trace } from "@opentelemetry/api";

export default async function policy(
  request: ZuploRequest,
  context: ZuploContext,
) {
  const tracer = trace.getTracer("my-tracer");

  return tracer.startActiveSpan("my-custom-operation", async (span) => {
    span.setAttribute("endpoint", request.url);

    try {
      // Run policy logic and external calls here.
      return request;
    } finally {
      span.end();
    }
  });
}

In an I/O-clock runtime, a span around fetch is meaningful. A span around a tight parsing loop can appear nearly instant even when it consumes CPU. Keep that runtime behavior in mind when reading short spans.

Measure the cost of tracing during the load test

Tracing adds allocation, buffering, processing, export, and storage work. We learned how much that matters during an internal test at about 8,000 RPS. The gateway appeared to have less capacity than expected because we had left tracing at 100% sampling.

The extra allocations and trace buffers increased memory pressure, instances recycled earlier, and their replacements repeatedly paid startup cost. We were measuring our telemetry almost as much as the gateway.

That investigation also found real bugs in our tracing plugin: spans left open on error paths and per-trace buffers that leaked. We fixed them. A sustained load test reaches error-path code millions of times, which is exactly where telemetry bugs stop being theoretical.

Run at least two passes when capacity is the result you plan to publish:

1. Use the sampling ratio configured in production.
2. Disable tracing to measure its overhead.

Do not describe a 100%-sampled result as raw API or gateway capacity unless 100% sampling is the production configuration.

Zuplo supports head sampling in the plugin configuration:

import { OpenTelemetryPlugin } from "@zuplo/otel";
import { RuntimeExtensions } from "@zuplo/runtime";

export function runtimeInit(runtime: RuntimeExtensions) {
  runtime.addPlugin(
    new OpenTelemetryPlugin({
      sampling: {
        headSampler: {
          ratio: 0.05,
        },
      },
    }),
  );
}

Head and tail sampling have different trade-offs:

Sampling reduces export and storage work, but instrumentation still runs for every request. A sampled pass is cheaper than full sampling, not free. Zuplo exports traces whose root span ends in an error regardless of the head-sampling ratio, which keeps failures available during heavily sampled runs.

A tracing checklist for load tests

Before the run:

• Enable request IDs or trace IDs across the path.
• Confirm each intermediary records upstream and downstream status separately.
• Propagate traceparent from the generator.
• Choose the sampling strategy and estimate collector capacity.
• Record the production sampling ratio in the test plan.

During analysis:

• Attribute errors between adjacent observation points.
• Measure each hop from one clock.
• Compare slow traces with healthy traces at a lower request rate.
• Rerun with tracing disabled before publishing a capacity number.

Tracing turns “p99 increased at 2,000 RPS” into a specific operation that got slower. The final article in this series covers the operational work around a large test: provider coordination, budgets, soak tests, and abort criteria in the API load-testing playbook.