Setting up Redpanda observability in Datadog

In our previous blog, we covered how to set up observability for Redpanda using Prometheus and Grafana. In this post, we discuss observability with Datadog and show you how to set up Datadog to monitor Redpanda clusters.

Datadog is a comprehensive monitoring and analytics platform that can be integrated with Redpanda to provide valuable insights into the performance and health of your data streaming stack. With this handy integration, you can collect, visualize, and analyze key metrics from your Redpanda clusters — all within the Datadog interface.  

Integrating Redpanda and Datadog 

To start, let’s take a look at what you get when pairing Datadog with Redpanda.

1. Data collection

• Prometheus endpoint:  Redpanda exposes a Prometheus-compatible metrics endpoint. Datadog can scrape metrics from this endpoint. This is the primary method for collecting Redpanda metrics. 

• Datadog Agent:  You'll need to deploy the Datadog Agent on the hosts running your Redpanda brokers or within your containerized environment. The Agent is responsible for collecting metrics and forwarding them to Datadog.

• Integration configuration:  You configure the Datadog Agent to connect to Redpanda's Prometheus endpoint. This typically involves specifying the endpoint URL and any necessary authentication details in the Datadog Agent's configuration file (redpanda.d/conf.yaml).  Autodiscovery in containerized environments will be available in the next major integration release.

2. Collected metrics 

Datadog collects a wide range of Redpanda metrics, providing visibility into various aspects of your cluster's performance and health. Some key metrics are cluster-related (partitions, latency, throughput requests, error rates). Some are related to resource utilization (CPU and memory), others are application-related (like inter-broker communication), and IO-related. You can even get low-level Raft metrics.

3. Dashboards and visualization

Redpanda and Datadog teams have worked together to provide pre-built dashboards for Redpanda, offering immediate visibility into key metrics. You can also create custom dashboards tailored to your specific monitoring needs. These dashboards allow you to visualize trends, identify bottlenecks, and correlate Redpanda performance with other parts of your infrastructure.

4. Alerting

You can set up alerts in Datadog based on Redpanda metrics. This automatically notifies you of potential issues, such as high latency, low disk space, or unhealthy brokers. Datadog's anomaly detection features can also help identify unusual behavior in your Redpanda cluster.  

5. Log management

Datadog can also collect and analyze logs from your Redpanda brokers, providing further context for troubleshooting performance issues or errors. You can configure log collection within the Datadog Agent.  

How to integrate Redpanda and Datadog

To get started, you need the following:

1. Install a Redpanda cluster.
2. Set up a Datadog account with your own Datadog API Key and the name of your Datadog site.
3. Install the Datadog Agent with the Redpanda integration.
4. Configure the Redpanda integration in the Datadog Agent by pointing it to Redpanda's Prometheus endpoint.
5. Restart the Datadog Agent.
6. Explore the pre-built Redpanda dashboards in Datadog or create your own.
7. Set up alerts based on relevant Redpanda metrics.

Next, we walk you through setting up a single-node Redpanda cluster and the Datadog Agent in Docker.  Then, we’ll see some metrics in the Datadog dashboard. 

Step 1. Install Redpanda 

Create a self-managed Redpanda cluster or a Redpanda cloud cluster, which can be dedicated or a BYOC/BPYOVPC cluster.

Here’s an example Docker Compose file to bring up a single-node Redpanda cluster, if you want to try this locally on your laptop or in your dev environment. 

name: redpanda-docker
networks:
  redpanda_network:
    driver: bridge
volumes:
  redpanda: null
services:
  redpanda:
    image: docker.redpanda.com/redpandadata/redpanda:v25.1.7
    command:
      - redpanda start
      # Mode dev-container uses well-known configuration properties for development in containers.
      - --mode dev-container
      # Tells Seastar (the framework Redpanda uses under the hood) to use 1 core on the system.
      - --smp 1
      - --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092
      # Address the broker advertises to clients that connect to the Kafka API.
      # Use the internal addresses to connect to the Redpanda brokers
      # from inside the same Docker network.
      # Use the external addresses to connect to the Redpanda brokers
      # from outside the Docker network.
      - --advertise-kafka-addr internal://redpanda:9092,external://localhost:19092
      - --pandaproxy-addr internal://0.0.0.0:8082,external://0.0.0.0:18082
      # Address the broker advertises to clients that connect to the HTTP Proxy.
      - --advertise-pandaproxy-addr internal://redpanda:8082,external://localhost:18082
      - --schema-registry-addr internal://0.0.0.0:8081,external://0.0.0.0:18081
      # Redpanda brokers use the RPC API to communicate with each other internally.
      - --rpc-addr redpanda:33145
      - --advertise-rpc-addr redpanda:33145
    ports:
      - 18081:18081
      - 18082:18082
      - 19092:19092
      - 9644:9644
    volumes:
      - redpanda:/var/lib/redpanda/data
    networks:
      - redpanda_network

‍

Start the container by using this command:

docker compose -f single-node-rp-cluster.yaml up -d

Step 2. Install Datadog Agent

Manually install the Redpanda integration using the following Docker file and build an image locally. You can optionally push the image to your Docker registry so that other developers in your organization can directly pull this image instead of having to build this integration every time.

Here’s the Dockerfile with the Redpanda integration:

FROM gcr.io/datadoghq/agent:latest

ARG INTEGRATION_VERSION=2.1.0

RUN agent integration install -r -t datadog-redpanda==${INTEGRATION_VERSION}

Next, build an image locally with this command: 

docker build -t my-dd-rpd-agent:2.1.0 .

To run this in Docker in a development environment, use the image from the previous step and start the Datadog Agent container by running the command below: 

docker run -d --name dd-agent \
-e DD_API_KEY=<your datadog api key> \
-e DD_SITE="<your datadog site>" \
-e DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true \
-v /var/run/docker.sock:/var/run/docker.sock:ro Take Redpanda for [a test drive here](https://redpanda.com/try-redpanda). Check out [our documentation](https://docs.redpanda.com/docs/home) to understand the nuts and bolts of how the platform works, or [read our blogs](https://redpanda.com/blog/categories/tutorial) to see the plethora of ways to integrate with Redpanda. To ask our Solution Architects and Core Engineers questions and interact with other Redpanda users, join the [Redpanda Community](https://redpanda.com/slack) on Slack. 
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-v /var/lib/docker/containers:/var/lib/docker/containers:ro \
my-dd-rpd-agent:2.1.0

For a production environment, you might want to deploy the Datadog Agent in a Kubernetes cluster via Helm chart or the Datadog Operator. You can use the Agent image with the Redpanda integration that was pushed to your organization's Docker Hub registry. You’ll need to override the image in Helm or Operator install by creating a values.yaml file to use the right image. Then you can use the custom values.yaml file in your Helm install.

An example datadog-values.yaml file for Helm install with overrides will look like this:

datadog:
  apiKeyExistingSecret: datadog-secret
  clusterName: dd-cluster
  site: datadoghq.com
agents:
  image:
    repository: docker.io/<dockerhub-username>/<image-name>
    tag: "<tag>"
    pullPolicy: IfNotPresent

Once the Datadog Agent is deployed, you can confirm its installation.

The Redpanda integration in the Datadog Agent includes a sample conf.yaml.example file that contains all the config needed for the integration. You’ll need to copy this sample file as conf.yaml file and modify it in the Redpanda integration folder to specify the Prometheus endpoint from which metrics will be scraped. 

In the context of this post, if you’re running both the Datadog Agent and Redpanda cluster as Docker containers, you can modify the openmetrics_endpoint value like this:

- openmetrics_endpoint: http://host.docker.internal:9644/public_metrics

For a Repanda Cloud cluster, you’ll need a Prometheus endpoint URL for this configuration. For these changes to be effective, restart the Datadog Agent’s Docker container.

Step 3. View the Datadog dashboard 

Redpanda helpfully provides prebuilt dashboards to view the metrics, or you can build your own. Once you log in to Datadog, navigate to the dashboard, and you’ll see the prebuilt Redpanda Overview dashboard.

Produce some messages on a topic to see metrics on the Datadog dashboard. You can either use the Redpanda CLI (rpk) or your custom producer to produce messages to any topic.

In the Datadog integrations GitHub repository, you’ll see a list of all the metrics that get scraped and mapped within Datadog. You can also read more about Redpanda metrics in our Docs. 

Wrapping up

Integrating the Datadog Agent with your Redpanda cluster empowers you to monitor critical Redpanda metrics, define alerts, detect anomalies, and ensure the overall health and performance of your data streaming infrastructure.

Note that the Redpanda integration folder in the Datadog integrations repository is under active development. Similarly, as Redpanda evolves, we’ll add more metrics that will be available in Datadog.

In the meantime, if you have questions about this integration, jump into the Redpanda Community on Slack and ask away.