Stay informed with our weekly newsletters focused on enterprise AI, data, and security trends. Subscribe Now
Real-time streaming data is essential across various industries. At the New York Stock Exchange (NYSE), it represents a valuable asset.
As a leading financial exchange with a rich history of data sharing, NYSE initially used telegraph ticker tape a century ago. It now utilizes advanced low-latency technologies on-premises for data connectivity.
Currently, NYSE is transitioning to open-source Apache Kafka technology, delivering NYSE Best Quote and Trades (BQT) data to the AWS cloud.
AI Scaling Hits Its Limits
Power constraints, rising costs, and delays are impacting enterprise AI. Join our discussion to learn how top teams are:
- Turning energy into a strategic advantage
- Architecting efficient inference for real throughput gains
- Unlocking competitive ROI with sustainable AI systems
Secure your spot to stay ahead: https://bit.ly/4mwGngO
NYSE partnered with data platform vendor Redpanda, using a C++ implementation of Kafka.
NYSE’s adoption of Redpanda’s C++ platform outperformed traditional Kafka by 4-5x, showcasing limitations in handling bursty data workloads.
This performance is crucial for AI applications needing consistent low-latency data. Kafka streaming can enable agent communications, competing with methods such as Google’s A2A and extend to Model Context Protocol (MCP).
“Large models have indexed public data sets; the next step is private data sets. Redpanda unlocks access to these,” said Alex Gallego, CEO of Redpanda.
NYSE’s Cloud Initiative
NYSE developed a cloud streaming platform for customers unable to access its data centers. It serves fintech and retail brokers needing AWS-based market data access.
“Not all consumers can access our data center, but small firms in Hong Kong can have their AWS account,” said Vinil Bhandari, NYSE’s head of cloud engineering.
NYSE streams its BQT feed from seven exchanges, requiring new infrastructure development.
NYSE’s Choice of Redpanda and Language Importance
NYSE manages over 500 billion messages daily across exchanges. Market volatility can spike messages 1,000x in microseconds.
Java implementations struggle due to latency issues caused by garbage collection.
“Java’s classic Kafka doesn’t handle burst traffic due to garbage collection. Redpanda’s C++ implementation helps manage volatility better,” Bhandari noted.
Programming language choice led NYSE to Redpanda over Confluent or Amazon MSK.
Technical decisions led to marked improvements.
“We’re 4-5 times faster with Redpanda compared to rivals using Kafka,” Bhandari said.
For streaming platform considerations, Java-based solutions may falter during traffic spikes, while C++ can maintain performance.
Leave a Reply