Real-time business intelligence you can act on
- 05 November, 2008 11:03
Sometimes changing the name of a product isn't just marketing sleight of hand, rather it actually succeeds in giving potential customers insight into the true purpose of the product at hand.
And so we find Coral8 CEO Terry Cunningham, whose company sells what is traditionally called CEP (complex event processing) technology, telling me that he would like to see that technology called "continuous intelligence," or CI, instead.
What CI attempts to do is to remove the divide between the transaction and query systems.
When you eliminate lag time between OLTP (online transaction processing), event analysis, and the automated execution of a subsequent event, you end up with a very powerful tool.
Complex event processing: Ticks, blips and clicks
Whether you call it CI or CEP, this tool isn't new. It has found a home for many years in the financial services industry, where CEP enables trading houses to track and store in real time the universe of equities, including derivatives at the rate of 50,000 events per second during peak trading hours. With opportunities lasting only seconds, the systems are used for programmatic trading, which allows portfolio managers to set up complex rules for determining when to buy or sell and then executes those orders automatically based on real-time data analysis of the prices flying by.
Cunningham calls this CEP for "ticks."
There are other industries where CI would be critical to improving services, such as energy grid management. Or, think logistics. Knowing where all your trucks, how fast they are traveling, and whether they are likely to maintain their delivery schedule would both improve customer service and decrease overhead.
This growing industry, in which sensors are being placed to monitor devices in the field on a continuous basis, is what Cunningham calls monitoring the "blips."
Contrast this with the way business intelligence usually works. Data is put into the bucket, and a tool then asks questions about that data.
Numerous BI companies say they offer real-time reporting. For the most part, this amounts to pushing the refresh button more often, Cunningham says.
"It is physically impossible to do real-time data processing in a traditional data warehouse. By the time you collect the millions of bits of data, minutes or even hours have gone by, while your business may require answers in milliseconds," says Cunningham.
Page BreakCI is not necessarily a replacement for a data warehouse. Some CI applications may still require that the data be put in context, and that happens when it is merged with the information from a data warehouse.
"If you calculate the volume weighted average price of a stock at $26.50, you won't know if that is good or bad unless you look up the history," Cunningham says.
Then, of course, there's the clicks: anything that happens online.
Automating ticket pricing by watching airline ticket sales, destinations, and plane capacity has been around for quite awhile, and it has helped push the market for CI into the broader context of all e-commerce.
Continuous intelligence: Old concept, new possibilities
The concept isn't new even in retailing. The change now is that the concept can be actualized.
Way back, Compaq coined the phrase "ZLE," for "zero latency enterprise." The classic ZLE example was of a shopper using a store credit card at checkout and while the product was being scanned, the technology would match the purchase -- in nanoseconds -- with past customer purchases, current store-inventory levels, margin, sales forecast for the item, and likelihood to buy again. It would take the results of all that and make a new offer before the receipt would be printed out.
A simpler example is when CI monitors customer behavior and uncovers that the potential buyer has come back three times to check the price. On the fourth time, they are hit with a special offer.
Sallie Mae uses Coral8 to track online loan applicants in order to better understand abandonment rates, improve performance, and do what those brick-and-mortar retailers do: make a personalized offer.
But while the idea was very attractive in the mid-'90s, the technology is only now catching up. Multicore processors and huge amounts of cache memory are enabling practical applications of this new kind of database to store information in memory, analyze it, and then dump the data or push it off to the standard database when done.
Page BreakSo what needs to be done to turn the potential of CEP/CI into reality? Business rules must be created and defined in an SQL-like language.
The same guys that have been buying BI for years will understand and buy these kinds of solutions, says Cunningham.
Continuous intelligence will allow companies to process data that has been around for decades but has been ignored. From shop-floor maintenance, where sensors monitor machine vibrations and send alerts when thresholds suggest something is out of whack, to airport baggage conveyor belts that send alerts the moment a bag falls off rather than relying on a Crystal Report to match scans before and after bags pass through the system, CI will improve the way we do business.
Business problems are getting more complex, and applications need to keep up, notes Cunningham.
When I asked Cunningham what's next, his response encapsulated for me what I like best about technology and its possibilities.
He said that he didn't really know, only that there's someone out there who will take this technology and say, "Wow, you know what I can build with this!"
Ingenuity and possibility. That's what keeps me excited about this industry.