R2: The engine of engines

Redpanda has never shied away from reinventing the wheel when the road changes and we can do 10× better. The last time we did it, with the Apache Kafka® API, it ended up powering core banking in the US, AI labs, top retailers, chip manufacturing, and autonomous vehicles. For the engineer in me, this was a dream come true.

Today we're introducing R2: a new engine of engines, and the foundation every Redpanda data product will run on from here forward.

Starting with the upcoming 26.3.1, Redpanda Streaming, Redpanda SQL and upcoming products will integrate R2 as the core engine. As a result, Redpanda Streaming/Operator will no longer be source available starting with 26.3.1. We will continue to maintain a Redpanda Community Edition free to use up to 50 cores for companies of any size.

Community discussions moved to github.com/redpanda-data/community. 

R2: the engine of engines

R2 is the new engine that powers Redpanda's native products. Every design decision traces back to something production taught us over seven years of running Redpanda at scale for the most demanding workloads in the world, from space exploration and low-latency payments to embedded environments and high-volume write-through object storage.

R2 is also a new programming model for us, designed for the age of agentic coding. Code generation is now cheap, but building an ultra-reliable product remains hard. Many of our users bet their business on Redpanda because we are often the ultimate system of record. A failure means data loss and a hit to their bottom line. The challenge is to harness advances in AI to accelerate innovation while strengthening the correctness, safety, and reliability that our customers' businesses depend on.

R2 rests on four layers:

1. Built from the ground up with full Deterministic Simulation Testing (DST), including virtual clocks and thread pacing of coroutine scheduling for many-core systems.
2. Proof-based code evolution for the core libraries: TLA+ checks for all concurrency and Lean proofs for all arithmetic. Combined with DST, this has prevented ~40 non-trivial faults from making it to mainline. 
3. Explicit error algebra (std::expected<T, E>) so that all callers crash or handle 100% of the errors. All code is compiled with -fno-exceptions, so an unhandled exception cannot be lost.
4. First-party hardware acceleration across GPUs (CUDA and ROCm) and Apple Metal.

Most importantly, it lands us on one shared foundation after hundreds of years of collective engineering effort across Redpanda data products. It's the foundation we're building the company on.

R2 is built from scratch using C++23 coroutines. It gives us one async abstraction over I/O, GPUs/accelerators, and files. Functionally, there's one local dev event loop across Mac and Linux, which is a much-improved developer experience. All products now share a single codebase.

Foundations: DST, TLA+, and Lean

Each core runs 1 reactor on 1 OS thread, plus N alien threads (N = 1 by default). Each reactor tick runs four phases: reap completions into work-class queues, run them with deficit-weighted round robin (DWRR, up to 64 resumes per tick), flush the staged ops, then park until the next completion. 

The completion source is pluggable. On Linux it's io_uring, on macOS it's kqueue. For testing, R2 uses a simulation source: a virtual clock, a seeded PRNG, virtualized I/O and seeded ordering, so that bugs can be replayed from seed.

• Deterministic simulation. Accelerates testing time from years to minutes. 
• TLA+ specs for cross-core communication, message passing, locks, DWRR scheduling, the GPU admission controller, backpressure, etc.  
• Lean proofs for algorithmic completeness. Think proving that inline bitsets wrap INT64 correctly, or that a Deadline operator behaves deterministically on time wrap.

Some of the most important bugs were found in new code that hadn't been hardened in production yet, because the state space can be explored during code generation on CI instead of on live production systems. This is the first time we've made formal proofs part of every core code check, and the compounding effect is that all the code we've ported over has largely "just worked" and, in a few cases, even highlighted latent bugs in the original implementation. 

R2 uses a DWRR scheduler, so when we introduced quota management for GPUs and accelerator memory allocation, the TLA+ and Lean checks found a gnarly edge-case deadlock caused by priority inversion during an oversubscribed capacity run, even though the scheduler was correct. It took years of hardening to find these kinds of bugs inside R1, the current Redpanda Streaming engine, and this foundation has changed how we develop better software across all upcoming products.

Test cycles from years to minutes

Coroutine suspension points are explicit, so under the simulation source every interleaving is enumerable and replayable from a seed. Paired with TLA+ specs and deterministic clocks, a timing bug that would take years of production traffic to hit shows up in minutes of simulated time.

‍

Small computers: memory and threads

One of the most challenging parts of a thread-per-core architecture for new programmers is memory pressure. A seemingly large machine looks like a bunch of smaller computers with relatively limited memory. 

Our previous future scheduler split memory evenly across cores at boot and gave each shard/logical core its own allocator, with every cross-core free sent back to its owner. The fast path was cheap, but it made the small-computer problem literal. When a shard runs out of memory and cannot borrow from an idle neighbor, it forces the engineer to build sophisticated machinery for backpressure, memory reclamation, moving workloads dynamically across cores, etc. 

mimalloc, TCMalloc, and other general-purpose allocators have since caught up and, with much less co-design, perform well across a wide range of workloads, including slow-drip requests, ultra-low latency, and slow object storage writes with large arenas. 

R2 keeps thread-per-core for execution, but stops carving up memory by hand. We still do message passing from source to destination to enforce core-local execution and a single concurrency and parallelism model across the codebase. Every R2 binary links mimalloc: each thread allocates from its own heap with no lock, a block freed on another core goes onto its page's free list and returns when the owner next collects, and freed pages go back to the kernel after a short purge delay. We also ran TCMalloc in per-CPU mode. It's a good allocator, but mimalloc beat it on our workloads, so we removed it.

Background alien threads for things like compaction or blocking syscalls are not pinned to a core, so the Linux CFS scheduler can move them. They leverage simpler reclamation and larger-than-logical-core allocations to simplify workload movement across CPUs and ship that logic to the Linux CFS scheduler, lowering latency for pathological workloads where most work moves to a single core and increasing system throughput. 

What R2 means for Redpanda Streaming

R1 is, to the best of my knowledge, the first Kafka-API distributed log that delivered an ever-changing workload shape dynamically on a single cluster, with no detached consensus (no ZooKeeper, no KRaft) and every acknowledged write durable on disk by default. Safety, latency, throughput, consistency, availability, and networking costs are all tunable at the topic level. Nested control loops push the bottleneck to the next piece of hardware depending on the workload. It is the best engineering work any team I've been part of has ever shipped, and our customers' speed and scale test it every day.

R1 stays on GitHub as is. R2 carries all of that forward starting with 26.3.1, and the new Redpanda Community Edition usage limits apply from that version on.

What R2 means for Redpanda SQL

Redpanda SQL (formerly Oxla) is our Postgres-compatible analytical database. We acquired it to deliver a unified abstraction for streams and tables, grounded in the stream-table duality that Redpanda CTO, Tyler Akidau, helped formalize in One SQL to Rule Them All (SIGMOD 2019). No separate ETL pipelines, just SQL to consume, transform, and deliver data wherever it's needed.

Since 2010, cores per socket grew 32× (8 to 256) while memory bandwidth per core grew only 2.5×, and fell to 3.2 GB/s on 2024's densest part. For analytical SQL, that bandwidth is now the cost-dominant factor, and we built Redpanda SQL around it.

Threads vs. coroutines

In a blocking design, a thread is two things at once: the unit of I/O concurrency and a single execution on a core. Those want opposite values. We rebuilt Redpanda SQL's S3 filesystem cache on R2's one-reactor-per-core runtime, then ported the whole query engine (benchmarks below).

We ported the whole query engine to R2 and ran it against Redpanda SQL today (Oxla 1.4.0, default config) on S3: same data, same EC2 m7a.8xlarge nodes, one engine at a time, on the Star Schema Benchmark and ClickBench, with every answer checksummed against the original.

‍

‍

At 64 clients, the threaded version grew to 20,000–23,000 threads per node. The R2 port ran 35 threads per node at every level, 1 per physical core. Cold scans win the biggest because each core keeps many parked reads in flight, versus roughly 1 per core before.

Not everything wins yet: warm ClickBench is level (1.02× excluding 1 regex GROUP BY query), and the port uses 20–30% more peak memory. Both are next on the profiler.

R2 unifies all our native products: Redpanda Streaming, Redpanda SQL, and upcoming releases. It's also a new system to us, designed for the age of agentic coding, and it has already stopped non-trivial ~40 faults from making it to mainline.
‍

.alex