Introducing multi-language dynamic plugins for Redpanda Connect

Today, we're excited to introduce dynamic plugins for Redpanda Connect — a new Apache 2.0 licensed feature that makes our data streaming platform even more flexible and interoperable. This plugin framework allows you to create and load plugins at runtime, opening up a world of new integration possibilities beyond Go. The dynamic plugins framework is available in Beta as part of Redpanda Connect v4.56.0.

What are dynamic plugins and why should you care?

Redpanda Connect has always supported plugins, but they've been "compiled plugins," meaning they needed to be built in Go and compiled directly into the Redpanda Connect binary. While this approach works well for many use cases, it creates limitations when you want to:

1. Use existing libraries not available in Go
2. Write plugins in your language of choice (e.g., Python)
3. Add plugins without rebuilding the entire Redpanda Connect binary
4. Deploy plugins independently of your main Redpanda Connect deployment

Dynamic plugins solve these challenges by allowing Redpanda Connect to load plugins at runtime from external executables. These executables communicate with the main Redpanda Connect process via gRPC, creating a flexible, language-agnostic plugin system.

How it works

Under the hood, Redpanda Connect orchestrates different plugins as subprocesses and uses gRPC over a Unix socket to pass data back and forth between processes. The protocol definition for these plugins closely mirrors the existing interfaces defined for plugins within Redpanda Connect's core engine, Benthos.

The architecture is elegant in its simplicity:

• Redpanda Connect includes three new "compiled plugins," one for each component type (BatchInput, BatchProcessor, and BatchOutput)
• These components act as wrappers that can load and communicate with external plugin executables
• The external plugins and Redpanda Connect communicate over gRPC, with well-defined service interfaces
• Each plugin maps to a single subprocess, keeping things modular and isolated

Why use gRPC for plugins?

Using gRPC for our plugin system offers several advantages:

• Language agnosticism: Write plugins in virtually any language that supports gRPC
• Process isolation: Plugins run in separate processes, so crashes won't take down the main Redpanda Connect engine
• Modern serialization: Efficient data transfer with Protocol Buffers
• Battle-tested technology: Used by major tech companies for critical systems

Creating dynamic plugins

Out of the gate, we have support for creating plugins in both Golang and Python.

# Initialize the environment
uv init yell_plugin
cd yell_plugin
uv add redpanda_connect

# Create a plugin
cat <<EOF > main.py
import asyncio
import logging
import redpanda_connect

@redpanda_connect.processor
def yell(msg: redpanda_connect.Message) -> redpanda_connect.Message:
    msg.payload = msg.payload.upper()
    return msg

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    asyncio.run(redpanda_connect.processor_main(yell))
EOF

# Create the plugin configuration
cat <<EOF > plugin.yaml
name: yell
summary: Just the simplest example
command: ["uv", "run", "main.py"]
type: processor
fields: []
EOF

# Create the pipeline
cat <<EOF > connect.yaml
pipeline:
  processors:
    - yell: {}
EOF

# Run the pipeline
rpk connect run --rpc-plugins=plugin.yaml connect.yaml

The configuration follows the same pattern for input, processor, and output plugins.

Performance considerations

While introducing gRPC does add some serialization and IPC overhead, we've designed the system to minimize performance impact:

1. We use batch components exclusively to amortize the cost of cross-process communication
2. Each plugin runs as a separate process, keeping the architecture clean and resource usage predictable

That said, for performance-critical workloads where every microsecond counts, the best approach remains using native Go plugins compiled directly into the Redpanda Connect binary. Dynamic plugins shine for flexibility and language choice, while compiled plugins offer maximum performance.

Language SDKs for easier plugin development

To make plugin development as smooth as possible, we're launching with official SDKs for both Go and Python:

• The Go SDK provides a familiar environment for existing Redpanda Connect developers, with type-safe interfaces that mirror the core Redpanda Connect components
• The Python SDK opens up Redpanda Connect to one of the most popular languages for data processing and AI/ML workloads

We're particularly excited about the Python SDK as it bridges the gap between the streaming data world and the rich Python ecosystem. This includes access to:

• Deep learning frameworks like PyTorch, TensorFlow, and Hugging Face Transformers
• LLM orchestration tools like LangChain and LlamaIndex for building RAG applications
• Data processing libraries such as Pandas, NumPy, and SciPy

Python plugins create a seamless bridge between streaming data pipelines and AI/ML capabilities, all without needing to leave your Redpanda Connect environment.

Real-world use cases

Dynamic plugins open up exciting new possibilities for Redpanda Connect:

• Real-time ML inference: Imagine a Python plugin that performs sentiment analysis on customer feedback streams using a pre-trained BERT model from Hugging Face. The plugin could enrich messages with sentiment scores and emotion classifications in real-time, enabling immediate actionable insights.
• Integration with non-Go libraries: Wrap essential libraries that don't have Go equivalents.
• Complex data transformations: Use Python's data science stack for statistical anomaly detection, time series forecasting, or natural language processing directly within your streaming pipeline.
• Lower barrier to entry: Allow data scientists and ML engineers to contribute plugins in Python without learning Go.
• Simplified deployment: Update plugins independently of the core Redpanda Connect system.

For example, a media company could implement a Python processor plugin that analyzes the sentiment of social media posts about their content. Using PyTorch and a pre-trained transformer model, the plugin can classify posts in real-time, tag content with emotions (such as joy, anger, and surprise), and route high-impact messages to the appropriate teams—all while maintaining the performance and reliability benefits of Redpanda's streaming infrastructure.

When to use dynamic vs. compiled plugins

While dynamic plugins are powerful, they're not always the right choice. We recommend:

• Use compiled plugins for most standard use cases and best performance
• Use dynamic plugins when you need language flexibility or to wrap existing libraries not available in Go

Getting started

Let's walk through creating and running your first dynamic plugin using the fizzbuzz_processor.py example from our GitHub repository.

Prerequisites

1. Install or update to the latest Redpanda Connect version:

rpk --version
rpk version (Redpanda CLI): v25.1.4 (rev 6cd1d4ef6b)
rpk connect upgrade
Redpanda Connect successfully upgraded from 4.55.1 to the latest version (4.56.0).

2. Set up a Python environment

# Create a new directory for your plugin
mkdir fizzbuzz_plugin
cd fizzbuzz_plugin

# Initialize with uv (or use pip/conda)
uv init .
uv add redpanda_connect

Using the FizzBuzz processor example

Step 1: Get the example files

Download the fizzbuzz_processor.py example from our repository:

# Download the example processor
curl -o fizzbuzz_processor.py https://raw.githubusercontent.com/redpanda-data/connect/main/public/plugin/python/examples/fizzbuzz_processor.py

The FizzBuzz processor transforms numeric messages according to the classic FizzBuzz rules, demonstrating how to:

• Use the @redpanda_connect.processor decorator
• Handle message payloads and transformations
• Implement conditional logic within a processor
• Follow Python SDK conventions

Step 2: Create the plugin configuration

Create plugin.yaml:

name: fizzbuzz
summary: Transforms numbers according to FizzBuzz rules (3=Fizz, 5=Buzz, 15=FizzBuzz)  
command: ["uv", "run", "fizzbuzz_processor.py"]
type: processor
fields: []

Step 3: Create the pipeline configuration

Create connect.yaml:

input:
  generate:
    interval: 1s
    mapping: |
      root = counter() % 20 + 1  # Generate numbers 1-20

pipeline:
  processors:
    - fizzbuzz: {}

output:
  stdout:
    codec: lines

Step 4: Run your FizzBuzz plugin

Execute the pipeline with your dynamic plugin:

rpk connect run --rpc-plugins=plugin.yaml connect.yaml

You should see output like:

1
2
Fizz
4
Buzz
Fizz
7
8
Fizz
Buzz
11
Fizz
13
14
FizzBuzz
16
17
Fizz
19
Buzz

Understanding the pattern

The configuration follows the same pattern for input, processor, and output plugins:

• Plugin definition (plugin.yaml): Defines the plugin name, command to execute, type, and any configurable fields
• Python implementation: Uses the @redpanda_connect.processor decorator and implements the transformation logic
• ‍Pipeline configuration (connect.yaml): Uses your custom plugin just like any built-in Redpanda Connect component

Conclusion

Dynamic plugins represent a significant step forward for Redpanda Connect, breaking down language barriers and enabling more flexible integrations. By leveraging the proven Go-plugin framework and gRPC, we've created a robust system that maintains the performance and reliability you expect from Redpanda while dramatically expanding what's possible.

This is just the beginning. We're excited to see how our community uses dynamic plugins to solve real-world streaming data challenges. We'd love to hear your feedback on this new feature and learn what use cases you're building. Your input helps us prioritize future enhancements and keeps us focused on solving the right problems. So let us know what you think in the Redpanda Community Slack!

Want early access to upcoming features? Be the first to try out new Redpanda Connect capabilities and beta features. Just send us your cloud username on Slack and we'll happily add you to our early access program.