How Jump Trading drives faster insights at scale with Redpanda

Jump Trading is a proprietary trading firm with a focus on algorithmic and high-frequency trading strategies. To be successful we tap into real-time insights—so streaming data infrastructure is critical to our business.

Over the past 20 years, Jump Trading iterated through several messaging and streaming data platforms. Our primary use cases are telemetry and log files. We have evaluated and deployed most of the best-of-breed on-premises and cloud messaging systems. In doing so, we found all of them lacking due to some combination of:

• Consistency of performance (tail latency, tolerance of slow consumers, etc.)
• Access to low-level metrics to debug performance issues (especially in cloud products)
• Reliability of message delivery
• Cost-effectiveness
• Compatibility with other products

As Jump Trading grew, our workloads required us to process millions of messages per second, and it became clear that we needed a new streaming data platform that could support this mission-critical aspect of our infrastructure.

The search for a simplified streaming data platform

When looking for a new solution, our starting point was Apache Kafka®. We really like the Kafka protocol because it is an industry standard — lots of data tools speak Kafka, making it much easier to engineer pipelines.

What we did not like was the idea of a Java application in such a mission-critical role. We did not want memory allocation controlled by a Java Virtual Machine (JVM), we wanted memory tailored to the application itself and we found that Redpanda satisfied that requirement by its implementation in C++. Plus, Jump Trading is more focused on C++ software architecture, so the Redpanda code base was in-line with our core software competencies.

With Redpanda, we could get Kafka compatibility along with stability, ease of maintenance, and platform simplicity. Plus, there were no ZooKeepers to manage. By eliminating the Java dependencies in Kafka, Redpanda gave us less complexity and superior performance on like-for-like hardware.

Furthermore, the simple architecture of Redpanda as a single binary and no JVM dependencies created a better experience for our engineers. And all this plus no data loss due to the Raft-native design. The choice became a no brainer.

Today we have implementations of Redpanda on bare metal, in containerized environments like Podman and Kubernetes, and the Redpanda fully-managed cloud service. The communication and interactions with Redpanda’s support and engineering team was a key factor that kept us on board, particularly in the early days of product adoption. We’d push the envelope in earlier iterations of the product, trigger bugs, and the Redpanda engineers would fix them seemingly overnight.

The keys to Redpanda’s success

It’s worth spending a little more time delving into the three key benefits that have made Redpanda so popular within Jump Trading: simplicity, performance and open source compatibility.

Simplicity

We really appreciate Redpanda’s single binary installation, upgrading with no downtime, and the maintenance cycle. The lean architecture is less complex than Kafka to deploy and scale because we do not need to install, monitor and maintain ZooKeeper, nor do we need to deal with JVM compatibility issues or JVM upgrades.

We also like the built-in Prometheus exporter that gives us visibility into performance internals, helping us steer operations and system load configuration. And, from a getting started perspective, we appreciate the documentation and blog posts available on the website, as well as the free courses at Redpanda University.

It is essential for our engineers to have solutions with open versus proprietary standards, because it translates to less time learning new skills. That is why we were drawn to Redpanda’s Kafka-compatible API. It gives Jump Trading access to the entire existing open-source Kafka ecosystem. When companies build on open-source standards, they are leveraging the foundational body of content publicly available, accelerating engineering time to productivity.

And because we are not locked into proprietary tech, we have the flexibility to work across different environments and platforms without fear of vendor lock-in. In theory, we could swap Redpanda for any other compatible Kafka platform if our needs changed. In the same manner, we can also use Redpanda with any Kafka-compatible tools, like the Amazon Kinesis Client Library.

In addition to Kafka compliance, it is great to see Redpanda’s S3-compatible storage feature for cost-effective data retention. This is a huge win because there is an entire ecosystem of tooling out there already for S3.

Performance

One of Jump Trading’s largest workloads is our mission-critical telemetry pipeline. It is complex, data-intensive and very producer-heavy. We have tens of thousands of nodes publishing data, such as system measurements, market information, and other varied bits of data. Additionally, we have high-performance nodes consuming and processing this data in real time. Our many-to-few data pipeline is fully supported and highly efficient with Redpanda.

Because of Redpanda’s performance-engineered architecture, we see little jitter in terms of our p95 and p99 latencies. We have very few fat or angry tails. Redpanda’s C++ codebase means we can load it up to line rate data loads and know that all the packets and messages will get delivered within an optimal distribution of latencies. That is huge, especially for a financial company that is latency-sensitive.

In an industry where you tend to have bursty traffic, it is invaluable to have a message bus that doesn’t have fat tails. And with Redpanda, we are seeing really low latencies: 50 milliseconds for our p95 and 150 milliseconds for p99.

Open-source compatibility

Redpanda is a source-available platform, which aligns with our values. We believe in open-source software development and actively support many open-source projects. Having the code base available for inspection allows us to understand it in depth, assist in troubleshooting when possible, and be a partner in the growth of the product.

Going above and beyond with Redpanda

We started with a self-hosted deployment of Redpanda on bare metal, then added containerized deployments, and are now expanding into fully-managed, dedicated cloud instances. As of now, we have been running critical workloads on Redpanda for two years, reliably shipping billions of messages daily.

Jump Trading is a leader in global financial markets and our streaming data infrastructure can make or break our business. With Redpanda, it’s not just meeting our requirements, but taking us to the next level.

— Alex Davies, CTO, Jump Trading. Originally posted on The New Stack.

For more stories about how Redpanda is helping companies tap into simple, powerful, and cost-efficient data streaming, check out the customers page and browse the Redpanda Blog for examples and tutorials. You can also try Redpanda to see it in action for yourself. If you have questions, ask away in the Redpanda Community on Slack.

‍