Fly-by analysis

While most of these "real time" products achieve their scorching performance by moving data into memory, one high-performance product -- the StreamBase "stream-processing engine" from StreamBase Systems -- just grabs incoming data and analyzes it as it flies by.

StreamBase applications use an "inbound" query-processing model, in which records are processed before they're indexed and stored. The records flow through the query, which can also transform the data while it's moving.

Vision Systems & Technology (VSTI) is helping several defense and intelligence agencies evaluate StreamBase prototypes. StreamBase can filter torrents of incoming data -- structured or unstructured -- and decide on the fly which should be presented to an analyst at once, which can be stored for later queries and which can be discarded, says Carol Lundquist, an IT consultant at VSTI.

The technology can generate alerts when a passing record contains, for example, a certain name or phone number. "You can put keywords in an Oracle table, and anytime a keyword is added, it gets dumped down to StreamBase immediately," Lundquist says.

"Some government systems are being flooded with data," she explains. "The Oracle systems are having trouble keeping up, and you get data falling on the floor." One government system Lundquist worked on loaded 1 billion records in a day, she says.

Filtering can be the salvation for some of those systems, says Bryan Harris, CTO of VSTI. "The idea is to load the needles, not the haystack," he says.

Harris says streaming technologies may complement rather than compete with traditional back-end RDBMSs. And they are not necessarily an alternative to the in-memory products, either, he says. "If you are doing queries across many different IT systems, that introduces a lot of processing across the entire network," Harris says. "In-memory data caching, if done right, can greatly reduce the amount of system resources used in total. But it doesn't really address the streaming issue." Depending on its mix of applications, a company could benefit from a combination of back-end databases, in-memory databases and streaming technology, he says.

Another variation on the in-memory database theme comes from Ants Software in California. Because Ants Data Server is a SQL-compliant relational database, the company says, it is readily compatible with the major back-end databases, such as Microsoft's SQL Server, IBM 's DB2 and Informix, and products from Oracle, Sybase and MySQL AB. And because it can reside on disk, in memory or both, there's no need to build an interface between different front- and back-end databases. The combination of these characteristics makes Ants easily scalable, the company says.

But Ants' major claim to fame is that, although it uses relational technology, it avoids almost all of the table row locking that can slow a traditional RDBMS to a crawl under heavy loads. According to Ants CEO Boyd Pearce, the reason traditional RDBMSs fail under load is because they aren't very clever at detecting when a real conflict is occurring and a lock is needed. "There are very few cases where you really need locks," he says. Bellevue, Wash.-based Wireless Services Corp., which provides hosted data services to wireless carriers, chose Ants primarily because it provides an easy upgrade path from SQL Server. Small customers may reside entirely on a single SQL server, says CTO Curt Miller. When that system grows, it migrates to two or more SQL Server boxes. And when it reaches a certain point, Miller adds Ants servers for high-performance front-end message processing. Without Ants, the table-locking function at volumes much above 1,000 messages per minute "kills me," he says.

Because Ants supports the Open Database Connectivity standard, the migration is a snap. "Now if I want to use Ants, I just change the driver parameters to say, 'Talk to Ants,' " Miller says. While the traditional back-end database will continue to be the choice of most users with big data repositories, the rise of multitier systems, as well as an increasing number of applications that process torrents of data, seem likely to pull these newer technologies into the mainstream.