This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Debugging Dapr applications and the Dapr control plane

Guides on how to debug Dapr applications and the Dapr control plane

1 - Debug Dapr in Kubernetes mode

How to debug Dapr on your Kubernetes cluster

1.1 - Debug Dapr control plane on Kubernetes

How to debug Dapr control plane on your Kubernetes cluster

Overview

Sometimes it is necessary to understand what’s going on in Dapr control plane (aka, Kubernetes services), including dapr-sidecar-injector, dapr-operator, dapr-placement, and dapr-sentry, especially when you diagnose your Dapr application and wonder if there’s something wrong in Dapr itself. Additionally, you may be developing a new feature for Dapr on Kubernetes and want to debug your code.

This guide will cover how to use Dapr debugging binaries to debug the Dapr services on your Kubernetes cluster.

Debugging Dapr Kubernetes services

Pre-requisites

1. Build Dapr debugging binaries

In order to debug Dapr Kubernetes services, it’s required to rebuild all Dapr binaries and Docker images to disable compiler optimization. To do this, execute the following commands:

git clone https://github.com/dapr/dapr.git
cd dapr
make release GOOS=linux GOARCH=amd64 DEBUG=1

On Windows download MingGW and use ming32-make.exe instead of make.

In the above command, ‘DEBUG’ is specified to ‘1’ to disable compiler optimization. ‘GOOS=linux’ and ‘GOARCH=amd64’ are also necessary since the binaries will be packaged into Linux-based Docker image in the next step.

The binaries could be found under ‘dist/linux_amd64/debug’ sub-directory under the ‘dapr’ directory.

2. Build Dapr debugging Docker images

Use the following commands to package the debugging binaries into Docker images. Before this, you need to login your docker.io account, and if you don’t have it yet, you may need to consider registering one from “https://hub.docker.com/".

export DAPR_TAG=dev
export DAPR_REGISTRY=<your docker.io id>
docker login
make docker-push DEBUG=1

Once the Dapr Docker images are built and pushed onto Docker hub, then you are ready to re-install Dapr in your Kubernetes cluster.

3. Install Dapr debugging binaries

If Dapr has already been installed in your Kubernetes cluster, uninstall it first:

dapr uninstall -k

We will use ‘helm’ to install Dapr debugging binaries. In the following sections, we will use Dapr operator as an example to demonstrate how to configure, install, and debug Dapr services in a Kubernetes environment.

First configure a values file with these options:

global:
   registry: docker.io/<your docker.io id>
   tag: "dev-linux-amd64"
dapr_operator:
  debug:
    enabled: true
    initialDelaySeconds: 3000

Then step into ‘dapr’ directory which’s cloned from GitHub in the beginning of this guide if you haven’t, and execute the following command:

helm install dapr charts/dapr --namespace dapr-system --values values.yml --wait

4. Forward debugging port

To debug the target Dapr service (Dapr operator in this case), its pre-configured debug port needs to be visible to your IDE. In order to achieve this, we need to find the target Dapr service’s pod first:

$ kubectl get pods -n dapr-system -o wide

NAME                                     READY   STATUS    RESTARTS   AGE   IP            NODE       NOMINATED NODE   READINESS GATES
dapr-dashboard-64b46f98b6-dl2n9          1/1     Running   0          61s   172.17.0.9    minikube   <none>           <none>
dapr-operator-7878f94fcd-6bfx9           1/1     Running   1          61s   172.17.0.7    minikube   <none>           <none>
dapr-placement-server-0                  1/1     Running   1          61s   172.17.0.8    minikube   <none>           <none>
dapr-sentry-68c7d4c7df-sc47x             1/1     Running   0          61s   172.17.0.6    minikube   <none>           <none>
dapr-sidecar-injector-56c8f489bb-t2st9   1/1     Running   0          61s   172.17.0.10   minikube   <none>           <none>

Then use kubectl’s port-forward command to expose the internal debug port to the external IDE:

$ kubectl port-forward dapr-operator-7878f94fcd-6bfx9 40000:40000 -n dapr-system

Forwarding from 127.0.0.1:40000 -> 40000
Forwarding from [::1]:40000 -> 40000

All done. Now you can point to port 40000 and start a remote debug session from your favorite IDE.

1.2 - Debug daprd on Kubernetes

How to debug the Dapr sidecar (daprd) on your Kubernetes cluster

Overview

Sometimes it is necessary to understand what’s going on in the Dapr sidecar (daprd), which runs as a sidecar next to your application, especially when you diagnose your Dapr application and wonder if there’s something wrong in Dapr itself. Additionally, you may be developing a new feature for Dapr on Kubernetes and want to debug your code.

This guide covers how to use built-in Dapr debugging to debug the Dapr sidecar in your Kubernetes pods. To learn how to view logs and troubleshoot Dapr in Kubernetes, see the Configure and view Dapr logs guide

Pre-requisites

  • Refer to this guide to learn how to deploy Dapr to your Kubernetes cluster.
  • Follow this guide to build the Dapr debugging binaries you will be deploying in the next step.

Initialize Dapr in debug mode

If Dapr has already been installed in your Kubernetes cluster, uninstall it first:

dapr uninstall -k

We will use ‘helm’ to install Dapr debugging binaries. For more information refer to Install with Helm.

First configure a values file named values.yml with these options:

global:
   registry: docker.io/<your docker.io id>
   tag: "dev-linux-amd64"

Then step into ‘dapr’ directory from your cloned dapr/dapr repository and execute the following command:

helm install dapr charts/dapr --namespace dapr-system --values values.yml --wait

To enable debug mode for daprd, you need to put an extra annotation dapr.io/enable-debug in your application’s deployment file. Let’s use quickstarts/hello-kubernetes as an example. Modify ‘deploy/node.yaml’ like below:

diff --git a/hello-kubernetes/deploy/node.yaml b/hello-kubernetes/deploy/node.yaml
index 23185a6..6cdb0ae 100644
--- a/hello-kubernetes/deploy/node.yaml
+++ b/hello-kubernetes/deploy/node.yaml
@@ -33,6 +33,7 @@ spec:
         dapr.io/enabled: "true"
         dapr.io/app-id: "nodeapp"
         dapr.io/app-port: "3000"
+        dapr.io/enable-debug: "true"
     spec:
       containers:
       - name: node

The annotation dapr.io/enable-debug will hint Dapr injector to inject Dapr sidecar into the debug mode. You can also specify the debug port with annotation dapr.io/debug-port, otherwise the default port will be “40000”.

Deploy the application with the following command. For the complete guide refer to the Dapr Kubernetes Quickstart:

kubectl apply -f ./deploy/node.yaml

Figure out the target application’s pod name with the following command:

$ kubectl get pods

NAME                       READY   STATUS        RESTARTS   AGE
nodeapp-78866448f5-pqdtr   1/2     Running       0          14s

Then use kubectl’s port-forward command to expose the internal debug port to the external IDE:

$ kubectl port-forward nodeapp-78866448f5-pqdtr 40000:40000

Forwarding from 127.0.0.1:40000 -> 40000
Forwarding from [::1]:40000 -> 40000

All done. Now you can point to port 40000 and start a remote debug session to daprd from your favorite IDE.

Commonly used kubectl commands

Use the following common kubectl commands when debugging daprd and applications running on Kubernetes.

Get all pods, events, and services:

kubectl get all
kubectl get all --n <namespace>
kubectl get all --all-namespaces

Get each specifically:

kubectl get pods

kubectl get events --n <namespace>
kubectl get events --sort-by=.metadata.creationTimestamp --n <namespace>

kubectl get services

Check logs:

kubectl logs <podId> daprd
kubectl logs <podId> <myAppContainerName>
kuebctl logs <deploymentId> daprd
kubectl logs <deploymentId> <myAppContainerName>

kubectl describe pod <podId>
kubectl describe deploy <deployId>
kubectl describe replicaset <replicasetId>

Restart a pod by running the following command:

kubectl delete pod <podId>

This causes the replicaset controller to restart the pod after the delete.

Watch the demo

See the presentation on troubleshooting Dapr on Kubernetes in the Dapr Community Call #36.

2 - Debugging Dapr Apps running in Docker Compose

Debug Dapr apps locally which are part of a Docker Compose deployment

The goal of this article is to demonstrate a way to debug one or more daprised applications (via your IDE, locally) while remaining integrated with the other applications that have deployed in the docker compose environment.

Let’s take the minimal example of a docker compose file which contains just two services :

  • nodeapp - your app
  • nodeapp-dapr - the dapr sidecar process to your nodeapp service

compose.yml

services:
  nodeapp:
    build: ./node
    ports:
      - "50001:50001"
    networks:
      - hello-dapr
  nodeapp-dapr:
    image: "daprio/daprd:edge"
    command: [
      "./daprd",
     "--app-id", "nodeapp",
     "--app-port", "3000",
     "--resources-path", "./components"
     ]
    volumes:
        - "./components/:/components"
    depends_on:
      - nodeapp
    network_mode: "service:nodeapp"
networks:
  hello-dapr

When you run this docker file with docker compose -f compose.yml up this will deploy to Docker and run as normal.

But how do we debug the nodeapp while still integrated to the running dapr sidecar process, and anything else that you may have deployed via the Docker compose file?

Lets start by introducing a second docker compose file called compose.debug.yml. This second compose file will augment with the first compose file when the up command is ran.

compose.debug.yml

services:
  nodeapp: # Isolate the nodeapp by removing its ports and taking it off the network
    ports: !reset []
    networks: !reset
      - ""
  nodeapp-dapr:
    command: ["./daprd",
     "--app-id", "nodeapp",
     "--app-port", "8080", # This must match the port that your app is exposed on when debugging in the IDE
     "--resources-path", "./components",
     "--app-channel-address", "host.docker.internal"] # Make the sidecar look on the host for the App Channel
    network_mode: !reset "" # Reset the network_mode...
    networks: # ... so that the sidecar can go into the normal network
      - hello-dapr
    ports:
      - "3500:3500" # Expose the HTTP port to the host
      - "50001:50001" # Expose the GRPC port to the host (Dapr Worfklows depends upon the GRPC channel)

Next, ensure that your nodeapp is running/debugging in your IDE of choice, and is exposed on the same port that you specifed above in the compose.debug.yml - In the example above this is set to port 8080.

Next, stop any existing compose sessions you may have started, and run the following command to run both docker compose files combined together :

docker compose -f compose.yml -f compose.debug.yml up

You should now find that the dapr sidecar and your debugging app will have bi-directional communication with each other as if they were running together as normal in the Docker compose environment.

Note : It’s important to highlight that the nodeapp service in the docker compose environment is actually still running, however it has been removed from the docker network so it is effectively orphaned as nothing can communicate to it.

Demo : Watch this video on how to debug local Dapr apps with Docker Compose