Introducing Redpanda Connect

Announcing Redpanda Connect and the acquisition of benthos.dev

By
on
May 30, 2024

Building an experience that wows users instantly is directly proportional to the effort placed by the builders at every turn, every commit. Not to mention hundreds of hours wasted using bad products to form strong opinions.

The time it takes to create your first, globally-distributed cluster from sign up in Serverless is ~100 milliseconds with Redpanda. 90 seconds to deploy a production cluster of Kubernetes on EKS. And of course, less than 1 second to bootstrap a 3-node cluster on your laptop. Sheer terror for anyone else.

Today, you can connect anything to anywhere in 140 milliseconds with Redpanda Connect.

time `rpk connect run`
rpk connect run  0.14s user 0.06s system 17% cpu 1.148 total

Any form factor. Any scale. Our largest Redpanda Connect user is pushing 50Gbps via 2X 25Gbps NICs to consume market data. Our smallest Redpanda Connect user is using 100 millicores and 10 MiB of memory, processing hundreds of messages a second deployed on a heavily resource-constrained environment.

The largest – 225 to be exact – connectivity platform for the Kafka ecosystem. There simply does not exist a more complete, end-to-end platform for streaming your data that spans every single form factor from embedded IoT devices to the largest cloud workloads.

Redpanda Connect comes via the acquisition of the popular benthos.dev stream processing framework. Ash, Founder and CEO, has been tirelessly working on Benthos for the past seven years, evolving the platform with some of the world’s largest companies spanning finance, oil & gas, healthcare, SasS, Adtech, and social media, etc.

We’ve always grown Redpanda with three core cultural values: Customer obsession, Be a good human, and Win as a team. While the traction of the product speaks for itself with almost 8,000 GitHub stars and users in every imaginable industry, the company bet was on Ash – the human and brains behind benthos.dev – and team. Thank you: we couldn’t have found a better partner, and welcome to the Redpanda family!

For Redpanda, this represents a meaningful transition to the most complete, end-to-end streaming platform. Whether you’re squeezing microsecond-level latency for the most demanding workloads, or you’re building one of the world's largest social networks like ShareChat with 390 million monthly active users, or you’re building the most reliable banking products like Zafin IO for the world's largest commercial banks, you can now build and scale your business without the need to stitch multiple projects together. All under one roof. I couldn’t be more excited about the future of Redpanda.

For users, this means continued support of 223 Apache 2 Licensed, open source connectors with the large financial backing of Redpanda and customer footprint. It means increased focus on reliability, day 2 operations and security. We’ll be bringing the known and loved Benthos Studio into Redpanda Console (UI) for a visual representation of the connectivity in your business.

Lastly, we know firsthand how difficult it is to build a framework that supports intelligent retries, routing, supervision, and built-in telemetry. We know many of you bet your business on the core engine of [Redpanda] Benthos, and we couldn’t be more proud to continue to support you by keeping the core engine MIT with all the freedom and protections it provides. We can’t wait to continue building together. Please see the core engine repo here.

Redpanda Connect: open source connectors

We couldn’t be happier keeping 99%+ of the connectors with an Apache 2, Open Source license. For the connectors below, you can purchase support from the Redpanda team for “Certified” connectors. These connectors have very large production users. For the databases, below we note that they tend to be in the regular test matrix. “Community” connectors are just that – Slack and GitHub support on a best-effort basis.

Every connector is in production today, but we want to ensure that your experience is curated and that Redpanda can stand behind the stability and scalability of the Certified group first.

Connector Type Support License
amqp_0_9 Input Certified Apache 2
amqp_0_9 Output Certified Apache 2
archive Processor Certified Apache 2
aws_dynamodb_partiql Processor Certified Apache 2
aws_kinesis Input Certified Apache 2
aws_kinesis Output Certified Apache 2
aws_kinesis_firehose Input Certified Apache 2
aws_lambda Processor Certified Apache 2
aws_s3 Input Certified Apache 2
aws_s3 Output Certified Apache 2
aws_sqs Input Certified Apache 2
aws_sqs Output Certified Apache 2
bloblang Processor Certified Apache 2
bounds_check Processor Certified Apache 2
cache aws_dynamodb Processor Certified Apache 2
cache aws_dynamodb Output Certified Apache 2
cache aws_s3 Processor Certified Apache 2
cache aws_s3 Output Certified Apache 2
cache file Processor Certified Apache 2
cache file Output Certified Apache 2
cache memcached Processor Certified Apache 2
cache memcached Output Certified Apache 2
cache memory Processor Certified Apache 2
cache memory Output Certified Apache 2
cache nats_kv Processor Certified Apache 2
cache nats_kv Output Certified Apache 2
cache redis Processor Certified Apache 2
cache redis Output Certified Apache 2
cache ristretto Processor Certified Apache 2
cache ristretto Output Certified Apache 2
cached Processor Certified Apache 2
compress Processor Certified Apache 2
csv Input Certified Apache 2
decompress Processor Certified Apache 2
dedupe Processor Certified Apache 2
generate Input Certified Apache 2
group_by Processor Certified Apache 2
group_by_value Processor Certified Apache 2
http Processor Certified Apache 2
http_client Input Certified Apache 2
http_client Output Certified Apache 2
jmespath Processor Certified Apache 2
jq Processor Certified Apache 2
json_schema Processor Certified Apache 2
kafka Input Certified Apache 2
kafka Output Certified Apache 2
kafka_franz Input Certified Apache 2
kafka_franz Output Certified Apache 2
log Processor Certified Apache 2
mapping Processor Certified Apache 2
metric Processor Certified Apache 2
mutation Processor Certified Apache 2
nats Input Certified Apache 2
nats Output Certified Apache 2
nats_jetstream Input Certified Apache 2
nats_jetstream Output Certified Apache 2
nats_kv Input Certified Apache 2
nats_kv Output Certified Apache 2
nats_kv Processor Certified Apache 2
nats_request_reply Processor Certified Apache 2
opensearch Output Certified Apache 2
parquet_decode Processor Certified Apache 2
parquet_encode Processor Certified Apache 2
protobuf Processor Certified Apache 2
rate_limit Processor Certified Apache 2
redis Processor Certified Apache 2
redis_hash Output Certified Apache 2
redis_list Input Certified Apache 2
redis_list Output Certified Apache 2
redis_pubsub Input Certified Apache 2
redis_pubsub Output Certified Apache 2
redis_script Processor Certified Apache 2
redis_streams Input Certified Apache 2
redis_streams Output Certified Apache 2
schema_registry_decode Processor Certified Apache 2
schema_registry_encode Processor Certified Apache 2
select_parts Processor Certified Apache 2
sftp Output Certified Apache 2
sleep Processor Certified Apache 2
socket Input Certified Apache 2
socket Output Certified Apache 2
socket_server Input Certified Apache 2
socket_server Output Certified Apache 2
sql_insert mysql Output Certified Apache 2
sql_insert mysql Processor Certified Apache 2
sql_insert oracle Output Certified Apache 2
sql_insert oracle Processor Certified Apache 2
sql_insert postgres Output Certified Apache 2
sql_insert postgres Processor Certified Apache 2
sql_insert sqlite Output Certified Apache 2
sql_insert sqlite Processor Certified Apache 2
sql_raw mysql Input Certified Apache 2
sql_raw mysql Output Certified Apache 2
sql_raw mysql Processor Certified Apache 2
sql_raw oracle Input Certified Apache 2
sql_raw oracle Output Certified Apache 2
sql_raw oracle Processor Certified Apache 2
sql_raw postgres Input Certified Apache 2
sql_raw postgres Output Certified Apache 2
sql_raw postgres Processor Certified Apache 2
sql_raw sqlite Input Certified Apache 2
sql_raw sqlite Output Certified Apache 2
sql_raw sqlite Processor Certified Apache 2
sql_select mysql Input Certified Apache 2
sql_select mysql Processor Certified Apache 2
sql_select oracle Input Certified Apache 2
sql_select oracle Processor Certified Apache 2
sql_select postgres Input Certified Apache 2
sql_select postgres Processor Certified Apache 2
sql_select sqlite Input Certified Apache 2
sql_select sqlite Processor Certified Apache 2
unarchive Processor Certified Apache 2
websocket Input Certified Apache 2
websocket Output Certified Apache 2
workflow Processor Certified Apache 2
amqp_1 Input Community Apache 2
amqp_1 Output Community Apache 2
avro Processor Community Apache 2
awk Processor Community Apache 2
azure_blob_storage Input Community Apache 2
azure_blob_storage Output Community Apache 2
azure_cosmosdb Processor Community Apache 2
azure_cosmosdb Input Community Apache 2
azure_cosmosdb Output Community Apache 2
azure_queue_storage Input Community Apache 2
azure_queue_storage Output Community Apache 2
azure_table_storage Input Community Apache 2
azure_table_storage Output Community Apache 2
beanstalkd Input Community Apache 2
beanstalkd Output Community Apache 2
cache Processor Community Apache 2
cache couchbase Processor Community Apache 2
cache couchbase Output Community Apache 2
cache gcp_cloud_storage Processor Community Apache 2
cache gcp_cloud_storage Output Community Apache 2
cache lru Processor Community Apache 2
cache lru Output Community Apache 2
cache mongodb Processor Community Apache 2
cache mongodb Output Community Apache 2
cache ttlru Processor Community Apache 2
cache ttlru Output Community Apache 2
cassandra Input Community Apache 2
cassandra Output Community Apache 2
cockroachdb_changefeed Input Community Apache 2
command Processor Community Apache 2
couchbase Processor Community Apache 2
discord Input Community Apache 2
discord Output Community Apache 2
elasticsearch Output Community Apache 2
file Input Community Apache 2
file Output Community Apache 2
gcp_bigquery Input Community Apache 2
gcp_bigquery Output Community Apache 2
gcp_bigquery_select Processor Community Apache 2
gcp_cloud_storage Input Community Apache 2
gcp_cloud_storage Output Community Apache 2
gcp_pubsub Input Community Apache 2
gcp_pubsub Output Community Apache 2
grok Processor Community Apache 2
hdfs Input Community Apache 2
hdfs Output Community Apache 2
http_server Input Community Apache 2
http_server Output Community Apache 2
javascript Processor Community Apache 2
mongodb Input Community Apache 2
mongodb Output Community Apache 2
mongodb Processor Community Apache 2
mqtt Input Community Apache 2
mqtt Output Community Apache 2
msgpack Processor Community Apache 2
nanomsg Input Community Apache 2
nanomsg Output Community Apache 2
nats_stream Input Community Apache 2
nats_stream Output Community Apache 2
nsq Input Community Apache 2
nsq Output Community Apache 2
parquet Input Community Apache 2
parse_log Processor Community Apache 2
pulsar Input Community Apache 2
pulsar Output Community Apache 2
redis_scan Input Community Apache 2
sentry_capture Processor Community Apache 2
sftp Input Community Apache 2
sql_insert clickhouse Output Community Apache 2
sql_insert clickhouse Processor Community Apache 2
sql_insert cosmodb Processor Community Apache 2
sql_insert cosmosdb Input Community Apache 2
sql_insert mssql Output Community Apache 2
sql_insert mssql Processor Community Apache 2
sql_insert snowflake Output Community Apache 2
sql_insert snowflake Processor Community Apache 2
sql_insert trino Output Community Apache 2
sql_insert trino Processor Community Apache 2
sql_raw clickhouse Input Community Apache 2
sql_raw clickhouse Output Community Apache 2
sql_raw clickhouse Processor Community Apache 2
sql_raw cosmodb Processor Community Apache 2
sql_raw cosmosdb Input Community Apache 2
sql_raw cosmosdb Output Community Apache 2
sql_raw mssql Input Community Apache 2
sql_raw mssql Output Community Apache 2
sql_raw mssql Processor Community Apache 2
sql_raw snowflake Input Community Apache 2
sql_raw snowflake Output Community Apache 2
sql_raw snowflake Processor Community Apache 2
sql_raw trino Input Community Apache 2
sql_raw trino Output Community Apache 2
sql_raw trino Processor Community Apache 2
sql_select clickhouse Input Community Apache 2
sql_select clickhouse Processor Community Apache 2
sql_select cosmodb Processor Community Apache 2
sql_select mssql Input Community Apache 2
sql_select mssql Processor Community Apache 2
sql_select snowflake Input Community Apache 2
sql_select snowflake Processor Community Apache 2
sql_select trino Input Community Apache 2
sql_select trino Processor Community Apache 2
subprocess Input Community Apache 2
subprocess Output Community Apache 2
subprocess Processor Community Apache 2
twitter_search Input Community Apache 2
wasm Processor Community Apache 2
xml Processor Community Apache 2

See our Connector Catalog for the latest list.

Redpanda Connect: Enterprise connectors

Two connectors require a paid subscription to Redpanda: snowflake_put and splunk. All future improvements, security vulnerability patches, and builds require a subscription to Redpanda. While we worked hard to keep most of the connectors permissively licensed, which may include competitors, I want to be transparent about our monetization model so there is no ambiguity or uncertainty.

Connector Type Support
License
snowflake_put Output Certified Enterprise [RCL]
splunk [hec] Output Certified  Enterprise [RCL]

Redpanda Benthos core engine

For developers and engineers who built their entire business on [Redpanda] Benthos, you can continue to do that, as we’re keeping the core engine MIT Licensed. All those hard-to-build, hard-to-test features that made [Redpanda] Benthos so reliable over the years are free and open source for you to use. This is a win for the community that built their business on top of [Redpanda] Benthos – we are so sure of it, we bet the future of Redpanda connectivity on this engine itself. I can’t wait to see how we evolve it together.

More to come, but don’t hesitate to continue collaborating with us on github.com/redpanda-data/benthos

Get in touch

It would be my and the team’s privilege to have the opportunity to partner with you. Whether you are using Redpanda Connect, or the Redpanda Console for the best UI in the streaming world or our storage engine Redpanda. Please reach out!

.alex founder & ceo

Graphic for downloading streaming data report
Save Your Spot
Redpanda Connect for Cloud
Christina Lin
&
&
&
September 12, 2024
Text Link
New AI connectors and GPU runtime support for AI models
Tyler Rockwood
&
&
&
September 12, 2024
Text Link
Cloud Topics: Efficiently stream data through object storage
Noah Watkins
&
Matt Schumpert
&
&
September 12, 2024
Text Link