Dapr uses the Open Telemetry protocol, which in turn uses the W3C trace context for distributed tracing for both service invocation and pub/sub messaging. Dapr generates and propagates the trace context information, which can be sent to observability tools for visualization and querying.
Distributed tracing is a methodology implemented by tracing tools to follow, analyze, and debug a transaction across multiple software components.
Typically, a distributed trace traverses more than one service, which requires it to be uniquely identifiable. Trace context propagation passes along this unique identification.
In the past, trace context propagation was implemented individually by each different tracing vendor. In multi-vendor environments, this causes interoperability problems, such as:
Previously, most applications were monitored by a single tracing vendor and stayed within the boundaries of a single platform provider, so these problems didn’t have a significant impact.
Today, an increasing number of applications are distributed and leverage multiple middleware services and cloud platforms. This transformation of modern applications requires a distributed tracing context propagation standard.
The W3C trace context specification defines a universally agreed-upon format for the exchange of trace context propagation data (referred to as trace context). Trace context solves the above problems by providing:
This unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis.
Dapr uses the standard W3C trace context headers.
traceparent
header.grpc-trace-bin
header.When a request arrives without a trace ID, Dapr creates a new one. Otherwise, it passes the trace ID along the call chain.
These are the specific trace context headers that are generated and propagated by Dapr for HTTP and gRPC.
Copy these headers when propagating a trace context header from an HTTP response to an HTTP request:
Traceparent header
The traceparent header represents the incoming request in a tracing system in a common format, understood by all vendors:
traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
Learn more about the traceparent fields details.
Tracestate header
The tracestate header includes the parent in a potentially vendor-specific format:
tracestate: congo=t61rcWkgMzE
Learn more about the tracestate fields details.
Baggage Support
Dapr supports W3C Baggage for propagating key-value pairs alongside trace context through two distinct mechanisms:
Context Baggage (OpenTelemetry)
baggage: userId=cassie,serverNode=DF 28,isVIP=true
HTTP Header Baggage
%20
for spaces, %2F
for slashes) when setting header baggageotelbaggage.Parse()
will decode the values%20
):curl -X POST http://localhost:3500/v1.0/invoke/serviceB/method/hello \
-H "Content-Type: application/json" \
-H "baggage: userId=cassie,serverNode=DF%2028,isVIP=true" \
-d '{"message": "Hello service B"}'
For security purposes, context baggage and header baggage are strictly separated and never merged between domains. This ensures that baggage values maintain their intended format and security properties in each domain.
Multiple baggage headers are supported and will be combined according to the W3C specification. Dapr automatically propagates baggage across service calls while maintaining the appropriate encoding for each domain.
In the gRPC API calls, trace context is passed through grpc-trace-bin
header.
Baggage Support
Dapr supports W3C Baggage for propagating key-value pairs alongside trace context through two distinct mechanisms:
Context Baggage (OpenTelemetry)
baggage: userId=cassie,serverNode=DF 28,isVIP=true
gRPC Metadata Baggage
%20
for spaces, %2F
for slashes) when setting metadata baggage%20
):baggage: userId=cassie,serverNode=DF%2028,isVIP=true
For security purposes, context baggage and metadata baggage are strictly separated and never merged between domains. This ensures that baggage values maintain their intended format and security properties in each domain.
Multiple baggage metadata entries are supported and will be combined according to the W3C specification. Dapr automatically propagates baggage across service calls while maintaining the appropriate encoding for each domain.