Introducing the Agentic Data Plane

Today marks a singular moment in time for me and Redpanda. I wasn’t part of the internet’s birth, but every generation has a chance at defining the next 100 years of human progress. The shift of our generation is that AI agents now define how work gets done — logic that once lived in code. The existential threat and opportunity for enterprises is that agents collapse execution time. Agent autonomy requires a continuous feedback loop, re-delineating the boundaries of security, data, and infrastructure. Every system and interaction is being reinvented end-to-end.

You cannot tame chaos with a bolt-on feature. Over the past year, we’ve been quietly building what we now call the Agentic Data Plane (ADP) — a unified, governed access layer that connects all your data systems and mediates every agentic interaction. Redpanda already powers Tier 0, mission-critical systems, and so we extended the same engineering philosophy when building for agents. Our Agentic Data Plane gives you the connectivity, context, and governance you need to deploy AI agents across your entire data infrastructure, safely. 

We are not new to reinventing the wheel when the road changes. What is different today from 2019 is that we are co-designing with the world’s most demanding workloads from the Global Fortune 2000, how AI, data, and infrastructure intersect to ship agents to production.

Redpanda’s real-time streaming engine gives us a foundational layer for Human-in-the-Loop (HITL), async mailboxes, durable model replay, and observability. The next era of agents demands more: context management, deep connectivity, governance, and querying capabilities that only a holistic platform can deliver.

That’s why we built Redpanda ADP — with agents at the center of it all.

What is the Agentic Data Plane?

The ADP is a unified runtime and control plane that safely exposes enterprise data to AI agents. It combines: (A) a low-latency streaming layer for events and HITL workflows, (B) a distributed, Iceberg-native query engine for real-time context, (C) 300+ high-quality connectors to bring context to the models, and (D) a fully managed, global policy and observability layer that enforces access, records intent, and enables replayable audits.

In practice, ADP gives agents a single, governed API to discover, query, act on, and revoke access to data in production across BYOC, Self Managed, and Cloud.

Things we’ve built: 

• Remote MCP + authentication + authorization for OBO (on-behalf-of) workloads with IdP integration. 
• A set of knowledge-based agent templates for common data sources, including Git for code repos, Jira, GDrive, etc.
• A declarative Agent Runtime.
• Redpanda Streaming for observability and HITL architecture, in addition to classic Apache Kafka^® workloads for mission-critical systems. 

Things we’ve bought, and you should expect a rolling integration:

• A disaggregated compute-storage query engine focused on Apache Iceberg™+Streaming analytical workloads and agent context management. 
• Early preview starts mid-December.

Things we’re doubling down on: 

• Governance. We are focusing on the two core areas of governance for Redpanda: Access Controls and Observability. From OBO to task-based authentication, DLP hooks, per-agent consent workflows, and immutable audit trails with configurable retention.
  

Oxla acquisition

Firmware reboot! In addition to the ADP, today we are incredibly excited to announce that Redpanda has acquired Oxla: a team obsessed with performance, correctness, and data catalogs. 

Oxla (rpk oxla)  is a distributed query engine, single binary, written in C++, built for demanding Iceberg queries and merging real-time context with historical data. It is a PostgreSQL wire protocol engine with separated compute-storage, oriented to bring low-latency context management for agents looking to merge streams or large data sets, search, or simply filter aggregations in real time. As agents merge unbounded historical queries, possibly accessing petabytes of data, SQL is the best mechanism to filter and aggregate while the model summarizes.

The categorical shifts in data started with moving structured, self-hosted data to the cloud. Separating compute and storage, bringing an order of magnitude of savings in operations, and most importantly, in scalability. 

The second shift was the idea that you can do analytics on structured and unstructured data with the same tools, which gave rise to the lakehouse — a shapeless dumping ground, leaving the complexities of managing the data to the query engines. 

The third shift is now here with AI Agents. AI has outgrown the static nature of warehouses and lakehouses into live operational and analytical data. Agents don’t just need to watch data flow by. They need to reach into it, interact with it, and act on it safely. And by safely, I mean in a controlled, governed, observable, auditable way. If agents are given free rein over sensitive, regulated data, with no ability to audit their work and no global governance to enforce boundaries, we are headed for disaster.

Why ADP matters to the enterprise 

It is not simply that applications are code-generated. The structure of these applications is fundamentally different from the ones we grew up building. The driving force behind agents is the outsourcing of the business logic. There is no git-checkout behind a failed credit card application to understand if the model has a gender bias.

The fear from CIOs is not the code of the agent itself, it is governance. In simple terms, it is access controls: can I trust that data is accessed by the right things? And observability: when things go wrong, can I understand what happened? 

The friction sits between the need to accelerate agentic adoption and the safety of your data. We want the promised land, but not at the expense of our customers, trust, or compliance. Worse, the new digital workforce often interacts with systems created in the API era of root-token permissions, with all-or-nothing as the norm. Agents need centralized governance, enforceable guardrails, and the ability to explain when things go wrong.  

At its core, the Agentic Data Plane’s mission is to carefully unlock the enterprise's confidential data for agentic access. To bring all your agents from all vendors in a governed, audited, secure way.

If you’ve been part of a large, successful business, you are always working with systems of record that made sense to someone in 1970. Today, engineers need an extensible framework that leverages standards like MCP, with 300+ built-in connectors to internal databases, caches, queues, files, and protocols you didn’t know existed, so you can actually bring in agents to perform useful work without the boilerplate.  

The commitment is: open protocols, zero lock-in, because you can’t afford it; MCP, A2A, for agents; PostgreSQL to filter in SQL, summarize in model; durable log for HITL; and Iceberg for long-term data state. The best tool for the job, either self-hosted or in any cloud. 

The future of enterprise AI isn’t about more models — it’s about giving agents governed, contextual access to the right data.

For builders, by builders

Sign up for Redpanda Cloud today and give Redpanda Streaming, Redpanda Connect, and Remote MCP a run. You can also request early access to Agents, Oxla, and modular integration using the contact form below. If you are not part of the community and have questions, the engineering team tends to hang out in our Redpanda Community Slack. Lastly, for those who want to see it all working end-to-end, join our upcoming virtual conference, Streamfest, happening on November 5th.

I couldn’t be prouder to be building together. Thank you to everyone at Redpanda, our partners, and customers who continue to work with us. Let’s go!

– 

This note was handcrafted by a hooman.

.alex

‍