New York Stock Exchange powers smarter trading decisions with real-time market data

Challenge 

Adding real-time data offerings drove up cost and complexity

The New York Stock Exchange (NYSE), which dates back to 1792, is the largest stock exchange in the world, trading on average over 2.5 billion shares and processing over 500 billion messages across all of its markets. A significant part of NYSE’s business involves selling market data to provide hedge funds, high-frequency traders, and retail and individual investors with accurate, real-time information to inform strategic trades. 

“To put things into perspective, Google averages about 16 billion queries a day. We handle nearly 1.1 trillion records daily, and 99% of the time, we make a decision in less than 20 microseconds. So scale and latency are absolutely critical to our success,” said Vinil Bhandari, Head of Cloud & Full Stack Engineering. 

Continuing its multi-year cloud journey, the NYSE Group was looking to expand its data offerings to include real-time data, in addition to historical trade and reference data available on AWS Cloud. However, consuming real-time data directly from NYSE required its customers to build, manage, and maintain on-premises environments or work with a third-party provider — a time-consuming and difficult process.

“We needed a platform that could deliver stock exchange data with high throughput and very low latency across the financial market almost instantaneously, while allowing for easy integration directly into customer applications.” Vinil explained.

Why Redpanda

Low-latency architecture and trusted engineering support 

NYSE Cloud Streaming is a data distribution platform that enables customers to access real-time market data with sub-hundred-millisecond latency in the cloud. 

“All the flavors of Kafka we evaluated didn’t fare well with bursty traffic since most are built on Java, which impacts the latency profiles,” Vinil said. “Redpanda, in contrast, has rewritten the Kafka protocol in C++, making it ideally suited for unpredictable workloads and easily scales to meet changing needs.”

Besides Redpanda’s innovative architecture, NYSE also highly values the Redpanda user experience and documentation, allowing the organization to self-serve on a regular basis. When higher levels of support are needed, Redpanda is always there to lend a helping hand—engineers collaborating with engineers.  

“We found Redpanda’s team very easy to work with. My staff appreciate teaming up with Redpanda engineers. They are not support engineers. They are not documentation engineers. They're engineers,” said Vinil.

“We come from a legacy company—a 230-year-old organization where relationships matter a lot. Redpanda is a trusted partner.”
— Vinil Bhandari, Head of Cloud & Full Stack Engineering, NYSE

Results 

Empowering customers to make smarter investment decisions

Redpanda serves as the pipeline for NYSE Cloud Streaming, reducing cost and complexity while providing significantly lower latency than any alternative for consuming data feeds in AWS Cloud. 

“Redpanda is the only Kafka-compatible data streaming platform that can cost-effectively and reliably manage the massive scale of the NYSE,” said Vinil.

With Redpanda in its corner, NYSE can now count on:

• Better control over memory and CPU usage 
• Lower latency, no JVM, no garbage collection
• Major resource savings with Redpanda Console, skipping the need to build a monitoring front-end
• Tiered storage to allow keeping data in blob storage (e.g., S3)
• Access to the Redpanda Connect ecosystem of 300+ connectors
• A RESTful interface for CRUD operations on the cluster
• rpk command line utility to help with cluster management

“Now, whether you're sitting in Timbuktu or Hong Kong, we can easily open up a data access point via the power of the cloud and immediately provide customers with data wherever they reside across the world,” said Vinil.

As a result, NYSE trusts Redpanda as a strategic partner to deliver real value in actual real time, enabling NYSE’s consumers to get crucial data when millions of dollars are at stake with high-volume trades.