Tracing

• 1: Distributed tracing overview
• 2: W3C trace context overview
• 3: Configure Dapr to send distributed tracing data
• 4: Open Telemetry Collector
• 4.1: Using OpenTelemetry Collector to collect traces
• 4.2: Using OpenTelemetry Collector to collect traces to send to App Insights
• 4.3: Using OpenTelemetry Collector to collect traces to send to Jaeger
• 5: How-To: Set-up New Relic for distributed tracing
• 6: How-To: Set up Zipkin for distributed tracing
• 7: How-To: Set up Datadog for distributed tracing

1 - Distributed tracing overview

Dapr uses the Open Telemetry (OTEL) and Zipkin protocols for distributed traces. OTEL is the industry standard and is the recommended trace protocol to use.

Most observability tools support OTEL, including:

• Google Cloud Operations
• AWS X-ray
• New Relic
• Azure Monitor
• Datadog
• Zipkin
• Jaeger
• SignalFX
• Dash0

The following diagram demonstrates how Dapr (using OTEL and Zipkin protocols) integrates with multiple observability tools.

Scenarios

Tracing is used with service invocaton and pub/sub APIs. You can flow trace context between services that uses these APIs. There are two scenarios for how tracing is used:

1. Dapr generates the trace context and you propagate the trace context to another service.
2. You generate the trace context and Dapr propagates the trace context to a service.

Scenario 1: Dapr generates trace context headers

Propagating sequential service calls

Dapr takes care of creating the trace headers. However, when there are more than two services, you’re responsible for propagating the trace headers between them. Let’s go through the scenarios with examples:

Single service invocation call

For example, service A -> service B.

Dapr generates the trace headers in service A, which are then propagated from service A to service B. No further propagation is needed.

Multiple sequential service invocation calls

For example, service A -> service B -> propagate trace headers to -> service C and so on to further Dapr-enabled services.

Dapr generates the trace headers at the beginning of the request in service A, which are then propagated to service B. You are now responsible for taking the headers and propagating them to service C, since this is specific to your application.

In other words, if the app is calling to Dapr and wants to trace with an existing trace header (span), it must always propagate to Dapr (from service B to service C, in this example). Dapr always propagates trace spans to an application.

Request is from external endpoint

For example, from a gateway service to a Dapr-enabled service A.

An external gateway ingress calls Dapr, which generates the trace headers and calls service A. Service A then calls service B and further Dapr-enabled services.

You must propagate the headers from service A to service B. For example: Ingress -> service A -> propagate trace headers -> service B. This is similar to case 2.

Pub/sub messages

Dapr generates the trace headers in the published message topic. For rawPayload messages, it is possible to specify the traceparent header to propagate the tracing information. These trace headers are propagated to any services listening on that topic.

Propagating multiple different service calls

In the following scenarios, Dapr does some of the work for you, with you then creating or propagating trace headers.

Multiple service calls to different services from single service

When you are calling multiple services from a single service, you need to propagate the trace headers. For example:

service A -> service B
[ .. some code logic ..]
service A -> service C
[ .. some code logic ..]
service A -> service D
[ .. some code logic ..]

In this case:

1. When service A first calls service B, Dapr generates the trace headers in service A.
2. The trace headers in service A are propagated to service B.
3. These trace headers are returned in the response from service B as part of response headers.
4. You then need to propagate the returned trace context to the next services, like service C and service D, as Dapr does not know you want to reuse the same header.

Scenario 2: You generate your own trace context headers from non-Daprized applications

Generating your own trace context headers is more unusual and typically not required when calling Dapr.

However, there are scenarios where you could specifically choose to add W3C trace headers into a service call. For example, you have an existing application that does not use Dapr. In this case, Dapr still propagates the trace context headers for you.

If you decide to generate trace headers yourself, there are three ways this can be done:

1. Standard OpenTelemetry SDK

   You can use the industry standard OpenTelemetry SDKs to generate trace headers and pass these trace headers to a Dapr-enabled service. This is the preferred method.

2. Vendor SDK

   You can use a vendor SDK that provides a way to generate W3C trace headers and pass them to a Dapr-enabled service.

3. W3C trace context

   You can handcraft a trace context following W3C trace context specifications and pass them to a Dapr-enabled service.

   Read the trace context overview for more background and examples on W3C trace context and headers.

Related Links

• Observability concepts
• W3C Trace Context for distributed tracing
• W3C Trace Context specification
• Observability quickstart

2 - W3C trace context overview

Dapr uses the Open Telemetry protocol, which in turn uses the W3C trace context for distributed tracing for both service invocation and pub/sub messaging. Dapr generates and propagates the trace context information, which can be sent to observability tools for visualization and querying.

Background

Distributed tracing is a methodology implemented by tracing tools to follow, analyze, and debug a transaction across multiple software components.

Typically, a distributed trace traverses more than one service, which requires it to be uniquely identifiable. Trace context propagation passes along this unique identification.

In the past, trace context propagation was implemented individually by each different tracing vendor. In multi-vendor environments, this causes interoperability problems, such as:

• Traces collected by different tracing vendors can’t be correlated, as there is no shared unique identifier.
• Traces crossing boundaries between different tracing vendors can’t be propagated, as there is no forwarded, uniformly agreed set of identification.
• Vendor-specific metadata might be dropped by intermediaries.
• Cloud platform vendors, intermediaries, and service providers cannot guarantee to support trace context propagation, as there is no standard to follow.

Previously, most applications were monitored by a single tracing vendor and stayed within the boundaries of a single platform provider, so these problems didn’t have a significant impact.

Today, an increasing number of applications are distributed and leverage multiple middleware services and cloud platforms. This transformation of modern applications requires a distributed tracing context propagation standard.

The W3C trace context specification defines a universally agreed-upon format for the exchange of trace context propagation data (referred to as trace context). Trace context solves the above problems by providing:

• A unique identifier for individual traces and requests, allowing trace data of multiple providers to be linked together.
• An agreed-upon mechanism to forward vendor-specific trace data and avoid broken traces when multiple tracing tools participate in a single transaction.
• An industry standard that intermediaries, platforms, and hardware providers can support.

This unified approach for propagating trace data improves visibility into the behavior of distributed applications, facilitating problem and performance analysis.

W3C trace context and headers format

W3C trace context

Dapr uses the standard W3C trace context headers.

• For HTTP requests, Dapr uses traceparent header.
• For gRPC requests, Dapr uses grpc-trace-bin header.

When a request arrives without a trace ID, Dapr creates a new one. Otherwise, it passes the trace ID along the call chain.

W3C trace headers

These are the specific trace context headers that are generated and propagated by Dapr for HTTP and gRPC.

• HTTP
• gRPC

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01

tracestate: congo=t61rcWkgMzE

Related Links

• Learn more about distributed tracing in Dapr
• W3C Trace Context specification

3 - Configure Dapr to send distributed tracing data

Configuration

The tracing section under the Configuration spec contains the following properties:

spec:
  tracing:
    samplingRate: "1"
    otel: 
      endpointAddress: "myendpoint.cluster.local:4317"
    zipkin:
      endpointAddress: "https://..."
    

The following table lists the properties for tracing:

To enable tracing, use a configuration file (in self hosted mode) or a Kubernetes configuration object (in Kubernetes mode). For example, the following configuration object changes the sample rate to 1 (every span is sampled), and sends trace using OTEL protocol to the OTEL server at localhost:4317

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing
spec:
  tracing:
    samplingRate: "1"
    otel:
      endpointAddress: "localhost:4317"
      isSecure: false
      protocol: grpc 

Sampling rate

Dapr uses probabilistic sampling. The sample rate defines the probability a tracing span will be sampled and can have a value between 0 and 1 (inclusive). The default sample rate is 0.0001 (i.e. 1 in 10,000 spans is sampled).

Changing samplingRate to 0 disables tracing altogether.

Environment variables

The OpenTelemetry (otel) endpoint can also be configured via an environment variables. The presence of the OTEL_EXPORTER_OTLP_ENDPOINT environment variable turns on tracing for the sidecar.

Next steps

Learn how to set up tracing with one of the following tools:

• OTEL Collector
• New Relic
• Jaeger
• Zipkin
• Datadog

4 - Open Telemetry Collector

4.1 - Using OpenTelemetry Collector to collect traces

Dapr directly writes traces using the OpenTelemetry (OTLP) protocol as the recommended method. For observability tools that support the OTLP directly, it is recommended to use the OpenTelemetry Collector, as it allows your application to quickly offload data and includes features, such as retries, batching, and encryption. For more information, read the Open Telemetry Collector documentation.

Dapr can also write traces using the Zipkin protocol. Prior to supporting the OTLP protocol, the Zipkin protocol was used with the OpenTelemetry Collector to send traces to observability tools such as AWS X-Ray, Google Cloud Operations Suite, and Azure Monitor. Both protocol approaches are valid, however the OpenTelemetry protocol is the recommended choice.

Prerequisites

• Install Dapr on Kubernetes
• Verify your trace backends are already set up to receive traces
• Review your OTEL Collector exporter’s required parameters:
  • opentelemetry-collector-contrib/exporter
  • opentelemetry-collector/exporter

Set up OTEL Collector to push to your trace backend

1. Check out the open-telemetry-collector-generic.yaml.

2. Replace the <your-exporter-here> section with the correct settings for your trace exporter.

   • Refer to the OTEL Collector links in the prerequisites section to determine the correct settings.

3. Apply the configuration with:

   kubectl apply -f open-telemetry-collector-generic.yaml
   

Set up Dapr to send traces to OTEL Collector

Set up a Dapr configuration file to turn on tracing and deploy a tracing exporter component that uses the OpenTelemetry Collector.

1. Use this collector-config.yaml file to create your own configuration.

2. Apply the configuration with:

   kubectl apply -f collector-config.yaml
   

Deploy your app with tracing

Apply the appconfig configuration by adding a dapr.io/config annotation to the container that you want to participate in the distributed tracing, as shown in the following example:

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "MyApp"
        dapr.io/app-port: "8080"
        dapr.io/config: "appconfig"

You can register multiple tracing exporters at the same time, and the tracing logs are forwarded to all registered exporters.

That’s it! There’s no need to include any SDKs or instrument your application code. Dapr automatically handles the distributed tracing for you.

View traces

Deploy and run some applications. Wait for the trace to propagate to your tracing backend and view them there.

Related links

• Try out the observability quickstart
• Learn how to set tracing configuration options

4.2 - Using OpenTelemetry Collector to collect traces to send to App Insights

Dapr integrates with OpenTelemetry (OTEL) Collector using the OpenTelemetry protocol (OTLP). This guide walks through an example using Dapr to push traces to Azure Application Insights, using the OpenTelemetry Collector.

Prerequisites

• Install Dapr on Kubernetes
• Create an Application Insights resource and make note of your Application Insights connection string.

Set up OTEL Collector to push to your App Insights instance

To push traces to your Application Insights instance, install the OpenTelemetry Collector on your Kubernetes cluster.

1. Download and inspect the open-telemetry-collector-appinsights.yaml file.

2. Replace the <CONNECTION_STRING> placeholder with your App Insights connection string.

3. Deploy the OpenTelemetry Collector into the same namespace where your Dapr-enabled applications are running:

   kubectl apply -f open-telemetry-collector-appinsights.yaml
   

Set up Dapr to send traces to the OpenTelemetry Collector

Create a Dapr configuration file to enable tracing and send traces to the OpenTelemetry Collector via OTLP.

1. Download and inspect the collector-config-otel.yaml. Update the namespace and otel.endpointAddress values to align with the namespace where your Dapr-enabled applications and OpenTelemetry Collector are deployed.

2. Apply the configuration with:

   kubectl apply -f collector-config-otel.yaml
   

Deploy your app with tracing

Apply the tracing configuration by adding a dapr.io/config annotation to the Dapr applications that you want to include in distributed tracing, as shown in the following example:

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "MyApp"
        dapr.io/app-port: "8080"
        dapr.io/config: "tracing"

You can register multiple tracing exporters at the same time, and the tracing logs are forwarded to all registered exporters.

That’s it! There’s no need to include any SDKs or instrument your application code. Dapr automatically handles the distributed tracing for you.

View traces

Deploy and run some applications. After a few minutes, you should see tracing logs appearing in your App Insights resource. You can also use the Application Map to examine the topology of your services, as shown below:

Related links

• Try out the observability quickstart
• Learn how to set tracing configuration options

4.3 - Using OpenTelemetry Collector to collect traces to send to Jaeger

While Dapr supports writing traces using OpenTelemetry (OTLP) and Zipkin protocols, Zipkin support for Jaeger has been deprecated in favor of OTLP. Although Jaeger supports OTLP directly, the recommended approach for production is to use the OpenTelemetry Collector to collect traces from Dapr and send them to Jaeger, allowing your application to quickly offload data and take advantage of features like retries, batching, and encryption. For more information, read the Open Telemetry Collector documentation.

• Self-hosted
• Kubernetes

docker run -d --name jaeger \
  -p 4317:4317  \
  -p 16686:16686 \
  jaegertracing/all-in-one:1.49

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    stdout: true
    otel:
      endpointAddress: "localhost:4317"
      isSecure: false
      protocol: grpc 

dapr run --app-id myapp --app-port 3000 node app.js --config config.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  ...
spec:
  ...
  template:
    metadata:
      ...
      annotations:
        dapr.io/enabled: "true"
        dapr.io/app-id: "MyApp"
        dapr.io/app-port: "8080"
        dapr.io/config: "tracing"

kubectl port-forward svc/jaeger-query 16686 -n observability

References

• Jaeger Getting Started
• Jaeger Kubernetes Operator
• OpenTelemetry Collector Exporters

5 - How-To: Set-up New Relic for distributed tracing

Prerequisites

• Perpetually free New Relic account, 100 GB/month of free data ingest, 1 free full access user, unlimited free basic users

Configure Dapr tracing

Dapr natively captures metrics and traces that can be send directly to New Relic. The easiest way to export these is by configuring Dapr to send the traces to New Relic’s Trace API using the Zipkin trace format.

In order for the integration to send data to New Relic Telemetry Data Platform, you need a New Relic Insights Insert API key.

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: appconfig
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    zipkin:
      endpointAddress: "https://trace-api.newrelic.com/trace/v1?Api-Key=<NR-INSIGHTS-INSERT-API-KEY>&Data-Format=zipkin&Data-Format-Version=2"

Viewing Traces

New Relic Distributed Tracing overview

New Relic Distributed Tracing details

(optional) New Relic Instrumentation

In order for the integrations to send data to New Relic Telemetry Data Platform, you either need a New Relic license key or New Relic Insights Insert API key.

OpenTelemetry instrumentation

Leverage the different language specific OpenTelemetry implementations, for example New Relic Telemetry SDK and OpenTelemetry support for .NET. In this case, use the OpenTelemetry Trace Exporter. See example here.

New Relic Language agent

Similarly to the OpenTelemetry instrumentation, you can also leverage a New Relic language agent. As an example, the New Relic agent instrumentation for .NET Core is part of the Dockerfile. See example here.

(optional) Enable New Relic Kubernetes integration

In case Dapr and your applications run in the context of a Kubernetes environment, you can enable additional metrics and logs.

The easiest way to install the New Relic Kubernetes integration is to use the automated installer to generate a manifest. It bundles not just the integration DaemonSets, but also other New Relic Kubernetes configurations, like Kubernetes events, Prometheus OpenMetrics, and New Relic log monitoring.

New Relic Kubernetes Cluster Explorer

The New Relic Kubernetes Cluster Explorer provides a unique visualization of the entire data and deployments of the data collected by the Kubernetes integration.

It is a good starting point to observe all your data and dig deeper into any performance issues or incidents happening inside of the application or microservices.

Automated correlation is part of the visualization capabilities of New Relic.

Pod-level details

Logs in Context

New Relic Dashboards

Kubernetes Overview

Dapr System Services

Dapr Metrics

New Relic Grafana integration

New Relic teamed up with Grafana Labs so you can use the Telemetry Data Platform as a data source for Prometheus metrics and see them in your existing dashboards, seamlessly tapping into the reliability, scale, and security provided by New Relic.

Grafana dashboard templates to monitor Dapr system services and sidecars can easily be used without any changes. New Relic provides a native endpoint for Prometheus metrics into Grafana. A datasource can easily be set-up:

And the exact same dashboard templates from Dapr can be imported to visualize Dapr system services and sidecars.

New Relic Alerts

All the data that is collected from Dapr, Kubernetes or any services that run on top of can be used to set-up alerts and notifications into the preferred channel of your choice. See Alerts and Applied Intelligence.

Related Links/References

• New Relic Account Signup
• Telemetry Data Platform
• Distributed Tracing
• New Relic Trace API
• Types of New Relic API keys
• New Relic OpenTelemetry User Experience
• Alerts and Applied Intelligence

6 - How-To: Set up Zipkin for distributed tracing

Configure self hosted mode

For self hosted mode, on running dapr init:

1. The following YAML file is created by default in $HOME/.dapr/config.yaml (on Linux/Mac) or %USERPROFILE%\.dapr\config.yaml (on Windows) and it is referenced by default on dapr run calls unless otherwise overridden `:

• config.yaml

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: daprConfig
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    zipkin:
      endpointAddress: "http://localhost:9411/api/v2/spans"

2. The openzipkin/zipkin docker container is launched on running dapr init or it can be launched with the following code.

Launch Zipkin using Docker:

docker run -d -p 9411:9411 openzipkin/zipkin

3. The applications launched with dapr run by default reference the config file in $HOME/.dapr/config.yaml or %USERPROFILE%\.dapr\config.yaml and can be overridden with the Dapr CLI using the --config param:

dapr run --app-id mynode --app-port 3000 node app.js

Viewing Traces

To view traces, in your browser go to http://localhost:9411 and you will see the Zipkin UI.

Configure Kubernetes

The following steps shows you how to configure Dapr to send distributed tracing data to Zipkin running as a container in your Kubernetes cluster, and how to view them.

Setup

First, deploy Zipkin:

kubectl create deployment zipkin --image openzipkin/zipkin

Create a Kubernetes service for the Zipkin pod:

kubectl expose deployment zipkin --type ClusterIP --port 9411

Next, create the following YAML file locally:

• tracing.yaml configuration

apiVersion: dapr.io/v1alpha1
kind: Configuration
metadata:
  name: tracing
  namespace: default
spec:
  tracing:
    samplingRate: "1"
    zipkin:
      endpointAddress: "http://zipkin.default.svc.cluster.local:9411/api/v2/spans"

Now, deploy the the Dapr configuration file:

kubectl apply -f tracing.yaml

In order to enable this configuration for your Dapr sidecar, add the following annotation to your pod spec template:

annotations:
  dapr.io/config: "tracing"

That’s it! Your sidecar is now configured to send traces to Zipkin.

Viewing Tracing Data

To view traces, connect to the Zipkin service and open the UI:

kubectl port-forward svc/zipkin 9411:9411

In your browser, go to http://localhost:9411 and you will see the Zipkin UI.

References

• Zipkin for distributed tracing

7 - How-To: Set up Datadog for distributed tracing

Dapr captures metrics and traces that can be sent directly to Datadog through the OpenTelemetry Collector Datadog exporter.

Configure Dapr tracing with the OpenTelemetry Collector and Datadog

Using the OpenTelemetry Collector Datadog exporter, you can configure Dapr to create traces for each application in your Kubernetes cluster and collect them in Datadog.

Before you begin, set up the OpenTelemetry Collector.

1. Add your Datadog API key to the ./deploy/opentelemetry-collector-generic-datadog.yaml file in the datadog exporter configuration section:

   data:
     otel-collector-config:
       ...
       exporters:
         ...
         datadog:
           api:
             key: <YOUR_API_KEY>
   

2. Apply the opentelemetry-collector configuration by running the following command.

   kubectl apply -f ./deploy/open-telemetry-collector-generic-datadog.yaml
   

3. Set up a Dapr configuration file that will turn on tracing and deploy a tracing exporter component that uses the OpenTelemetry Collector.

   kubectl apply -f ./deploy/collector-config.yaml
   

4. Apply the appconfig configuration by adding a dapr.io/config annotation to the container that you want to participate in the distributed tracing.

   annotations:
      dapr.io/config: "appconfig"
   

5. Create and configure the application. Once running, telemetry data is sent to Datadog and visible in Datadog APM.

Related Links/References

• Complete example of setting up Dapr on a Kubernetes cluster
• Datadog documentation about OpenTelemetry support
• Datadog Application Performance Monitoring