The New York Stock Exchange manages multiple terabytes worth of data, and the job of doing so is getting more complicated as the organization expands globally through mergers, acquisitions and partnerships.

The NYSE merged with Euronext of Paris and the electronic stock exchange ArcaEx in 2006, and has forged agreements to share technology resources and data with stock exchanges in India, Tokyo, Malaysia and Brazil. The number of data centers in the United States and Europe is being consolidated from 10 to four.

"We're building a common platform for all of our trading technology to live on and [plan to] consolidate data centers globally," says Mark Schaedel, vice president of data products for NYSE Euronext. "We've gone through a rapid expansion process over the past few years. We have a plan in place to make use of this global community and allow customers to reach in and access these data pools."

The NYSE is moving US and European platforms to a "common customer gateway" this year, and consolidating all networks into an electronic trading system it calls the Secure Financial Transaction Infrastructure (SFTI), according to a presentation during an earnings conference call. Consolidation of European data centers began last year. and the global consolidation project is scheduled to be done by 2010.

When it comes to delivering market data to customers, the NYSE relies on the business intelligence vendor 1010data to store information and provide the analytical engine that makes it possible for customers to make queries about historical trends. 1010data, which has extensive experience on Wall Street, provides the back-end servers and storage along with a core database engine that lets customers "dig into the data and pose flexible questions of it," says David Frankel, the company's vice president of business development.

The analytical tools are instrumental in the NYSE's various Web-based market data products, the NYSE's own internal market research data analysis, and the "back-end service for its NYXdata Web site which gives subscribers access to a vast amount of NYSE data," 1010 states.

Whereas a customer might use an Oracle database to make "tens of millions of minuscule updates," 1010's system is designed for those times "when you need to move a mountain to answer one question" and is thus ideal for historical analysis, Frankel says.

The system "allows customers to build trading models and strategies around historical events and test those strategies," Schaedel says.

The information for the NYSE's data products is stored in six Intel quad-core servers with "multiple terabytes" of direct-attached storage at the NYSE's Manhattan data center. The NYSE chose to have 1010 manage the data remotely, freeing up the time of its own employees to focus on the NYSE's areas of expertise. "The nice thing is we don't have to manage it. We can focus on how we find value in the data," Schaedel says.

1010 has been on the scene for NYSE since 2001 and will be instrumental in rolling out the exchange's market data products to a wider audience as part of its global expansion strategy.

The NYSE handles billions of rows of data and has to make it available to various commercial trading firms, and vendors that have licensing agreements allowing them to distribute the data to their own customers.

"This is really what our customers are facing, [the data amounts] become so large it's a job in itself to manage that repository," Schaedel says.