Real-time Web data analysis is gaining acceptance for business decision-making, but the complexity of creating real-time applications still keeps most projects small.
In an era of instant gratification, it's only natural that IT managers are finding themselves pressured to deploy real-time Web analytics systems. But beware: Any trek into the Web's real-time world will be one populated by legacy, batched data, as well as rapid-fire, up-to-the-click information.
There are compelling reasons for using real-time Web information to make business decisions. Many of them are related to e-commerce markets, while others are emerging in product development and supply chain areas. But the complex nature of the applications involved means that most companies are only dipping their toes into real-time Web analytics waters.
"It's the land of pilot projects," quips Louis Columbus, a senior analyst at AMR Research Inc. in Boston.
That's because any IT department that dives in "must assume integration and interoperability as core competencies," he says.
A few companies, however, are wading more deeply into the technology.
Reaching Critical Mass
Company: NetFlix.com Inc.
Real-Time Challenge: Give customers instant movie recommendations based on current Web site activity and historical user dataYour critiques of films starring Jack Nicholson or Nicole Kidman get thrown back at you in real time if you're one of the 500,000 subscribers to NetFlix.com's Web site. A pan, a thumbs up, past rentals, your profile and current Web activity all contribute to the pages served up by the "recommendation engine" from the Los Gatos, Calif.-based online movie services company.
Initially, the IT department started with a commercial application. That work stopped, says Neil Hunt, the company's vice president of e-commerce, because the canned software was ill-suited for evaluating on the fly the subjective world of tens of thousands of movie critics.
Yet, the capability was critical in order for NetFlix to distinguish itself online. "We felt we needed to own the technology," says Hunt.
So he hired mathematicians with C++ experience to write the algorithms and code to define movie clusters, relate opinions to the clusters, evaluate thousands of ratings per second and factor in current Web site behavior to deliver a specially configured Web page before a site visitor can click again.
Hunt says NetFlix's "significant" investment in proprietary technology was only made after it was deemed critical to the company's added value to the market.
He says the real-time analytics can also tell marketing managers what Web page design is working best for a given promotion and allow them to make changes immediately based on dynamic feedback.
"It's a good tool for effective design," he says.
Analytics as a Web Service
Company: Infinity Pharmaceuticals Inc.
Real-Time challenge: Speed drug development by facilitating the process of evaluating new chemicalsAlthough real-time Web analytics are mostly associated with consumer site recommendation engines like the one at NetFlix, some rapid-response software is showing up in critical applications for use inside companies.
Andy Palmer, CIO at Infinity Pharmaceuticals in Boston, says his drug discovery company's business is dependent on the underlying real-time analytics application that its researchers use when generating statistical models of how compounds will behave in a given chemical assay. With so many different models, he says, "the more real time and interactive you can be, the more effective you will be."
The toughest nut to crack for real-time Web analytics is data integration, Palmer says. This is where real-time information regularly meets batched data. For example, the company's scientists constantly need to integrate their database of chemical models with outside sources such as databases of chemical compounds.
Infinity's IT staff solved the problem by building a real-time system using Web services as its application model, so XML interfaces are coded into every program. The design also includes a standardized metadata model, and Infinity maps its data dictionaries to that.
"That way, data integration is done upfront instead of after the fact," Palmer says.
User response time is the next big problem. "We want to reduce latency to real time," he says.
To do so, Infinity has made a commitment to visualization software. "Without visualization, the number of factors presented would be too large to make real decisions," Palmer says.
The visualization software, a package from Somerville, Mass.-based Spotfire Inc., renders the analytic output from millions of compounds. "As you apply the different predictive models, you change them and view them dynamically" in Spotfire, says Palmer. "This is something that used to take weeks, but now it's just a few seconds.
Four Steps To Project Success
Before trodding down a real-time path, IT managers might do well to heed the following advice from Louis Columbus, an analyst at AMR Research:
1. Plan for two to six months of implementation time.
2. Expect substantial investments for data integration work, high-performance servers and storage subsystems to meet insatiable data storage requirements.
3. Visualization tools are critical to make massive data dumps understandable.
4. Try before you buy. Ask for references and a free evaluation period before committing to an analytics tool.
Knowing When Rules Apply
In the rules-based world of real-time analytics, some Web applications benefit more than others.
Real-time applications are showing up everywhere inside companies, but not all applications lend themselves to the rules-based world of real-time analysis.
Nonetheless, "people are thinking of more clever ways to do real-time work," says Richard Lowery, national director of the customer relationship management practice at Born, a Minnetonka, Minn.-based consultancy.
Clearing the Pipes
Robb Thomas, communications manager at Cincinnati-based Roto-Rooter Services Co., used real-time tools to off-set an anticipated increase in wide-area network (WAN) costs.
The company's 130 franchises run Internet and internal application traffic on the corporate WAN. Thomas set up a proxy server last year to manage all of the Web page requests so he could run company policy Web address filters centrally for remote offices.
Even with the filtering, adding the Web traffic to the WAN bogged it down. But today's budget realities nixed the idea of upgrading the network backbone.
Instead, Thomas opted for a rules-based bandwidth allocation system from NetReality Inc. in Santa Clara, Calif., that can change bandwidth levels in real time. The rules for who gets what bandwidth when can be as complex or as simple as you want, says Thomas. But each one should correspond to a specific business criterion, not an item on an IT wish list, he says.
Sounding the Alarms
Linda Belanger's company has been edging toward real-time alerts for its sales and customer data. "On our radar is to do 'alarm' reports so we can give instant information to our sales force about a customer in his territory," says Belanger, senior manager of decision support at Office Depot Inc. in Delray Beach, Fla.
She envisions the appropriate pagers beeping with information and instructions as soon as the system reaches a rules-defined threshold. In this way, Belanger says, real-time tools will make her company's sales force more effective.
But she adds that while real-time updates to the sales force are a plus, staffers must still evaluate the information before responding to customer issues. And the quicker the sales staff gets customer data, Belanger says, the more effective it is at generating revenue.
Real-Time Account Management
Company: Cisco Systems Inc.
Real-Time Challenge: Provide real-time account order activity feedback to sales departmentIt's not just movie buffs and scientists who need to see real-time information in a meaningful way. At Cisco, company executives says that ideally, everyone in the business should have access to real-time information.
"The whole corporation is moving to real time," says Mike Zill, director sales finance IT at Cisco. "It's difficult to have the applications stay in batch when the architecture is message-based."
According to Zill, channel account managers in the sales department use a Web-based "dashboard," or graphical user interface-based view, from OneChannel Inc. in Mountain View, Calif., that gives them real-time views of their accounts' activities. When a business condition hits a predetermined threshold, the OneChannel software triggers an alert, sending a message or warning to the user's dashboard.
For example, if Cisco's sales department has a top-10 list of new products it wants sold through, say, Ingram Micro, the application will let the Cisco channel manager know the instant the distributor's sales fall outside target levels.
To achieve this, Cisco had to build deep hooks into its supply chain, which wasn't easy, acknowledges Zill. The firm has established agreements with its partners to receive point-of-sale data via the Internet or, in some cases, through electronic data interchange.
Most of the data is batched. Few partners will feed real-time point-of-sale data to the company, Zill says.
Once it receives the data, Cisco couples it with real-time, Web-based inventory information and processes it using analytics software from Hyperion Solutions Corp. in Sunnyvale, Calif. Channel managers can then query the Hyperion software in detail through the OneChannel dashboard to find the underlying causes of any distribution problem.
"The response time is fast enough so you're not waiting," Zill says.
And that's the essence of real time for any user.