The leaders of big organizations, especially businesses, can pay a terrible price for driving forward while gazing in the rear-view mirror. Many who did this at the turn of the millennium raced headlong into the Permafrost Economy on the fuel of wishful thinking and soothing official lullabies. Meanwhile, data in their own systems could have shown them both the coming chill and actions likely to buffer it. That is, the data could have if these leaders had invested in a software category called BA (business analytics).

A term that first appeared in 1997, BA was quickly muddied by overzealous marketers and clueless pundits, usually by confusing BA with BI (business intelligence).

From BI to BA

BA software expresses a broad range of applications and designs. Typically it sucks up vast quantities of data stored in data warehouses and complementary sources, then runs advanced math and statistical operations on it in search of relationships. Unlike BI, BA software can examine every possible interaction, sometimes finding relationships analysts wouldn't have seen if the software was limiting the fields it imported. Most clients deliver a mixture of graphic and numeric windows for visualization, support the graphics to spot and select attractive exceptions, and then iteratively run tests on the selected items.

Too many otherwise intelligent business managers get distracted by BI when they should be paying attention to what BA can tell them, says Herb Edelstein, founder of the Data Warehouse Institute and president of Two Crows, a consultancy.

"BI practice is a global name for all kinds of backwards-looking querying and reporting. Sometimes the tools are very intelligent, and usually they're attached to a data warehouse," Edelstein says. "The past is useful and necessary, but too many people use reporting to look backwards, not to ask the question, How do we handle our business in the future?

BI's binding to the past and BA's focus on the future points out the essential difference between the tools, explains Anne Milley, the SAS Institute's director of analytical intelligence.

"It doesn't matter how many different reports you have if they are pre-built," Milley says. "Good analysis requires follow-up ... not just 'how much' and 'how many,' but an attempt to understand 'why.' Analytics tools are iterative and interactive. It's not just presented data. BA tools exist for exploring directions and finding answers. It's a platform for continuous learning."

In traditional shops with BI, the responsibility for analysis falls on experts armed with the delivery of whatever reports and OLAP manipulations they can execute. The business analysis goes on in an expert's cranium. The questions that experts trigger are based on their presumptions, making it unlikely they will discover a relationship or trend they hadn't previously considered.

"With BI, you see one report, or one graph, at a time. With BA you get every possible report evaluated, and then it delivers you the most relevant ones," says Joerg Rathenberg, senior director of marketing at BA software vendor Kxen.

The mechanics of OLAP cubes looking into hundreds of field's worth of data usually overwhelms researchers' systems, impelling them to exclude fields. This radically reduces the computing requirement but also radically reduces the possibility of uncovering subtle interactions among dozens of factors. The human-only process of business analytics will miss some real trends.

Therefore, it usually falls upon a line of business experts to interpret BA software answers and sort out the real from the accidental statistical artifacts.

Given the need for forward-looking tools that aim to answer the whys embedded in the tidal waves of data stored in transactional and other enterprise systems, it's not surprising a wide range of BA tools is aimed at answering specific organizational problems. Existing BA solutions solve a range of problems.

Why BA?

Obvious BA applications have already erupted from the fecund, data-rich environments proffered by enterprise CRM and ERP systems. The most rapid adoption of BA seems to come when both the size of the data and the number of options are intimidating.

For example, telecom marketing groups have used BA software to attack the problem of customer churn. Michael Berry, a founder of Boston's Data Miners, consults with those groups by dragging actionable information out of the their huge volumes of transactional data.

Berry gets answers to the right questions. "Who are the customers at risk to cancel their subscription?" he asks. "What are the drivers? What can we forecast, not just about who, but about when, because time is a factor we can assess, too."

Berry's consultancy helped T-Mobile USA set up SAS Institute's BA tools such as Enterprise Miner and Quadstone's eponymous Quadstone System. These tools applied sophisticated calculations and predictive statistical routines to question such things as what kinds of interactions with customer support increased or decreased chances of renewal, and how time between the interaction and the renewal window affected rates.

Many pharmaceutical formulators adopted BA quickly. When performing experiments to find all possible complex compounds that might work for a certain task, scientists want early testing indications of what compounds may work so they can focus resources. At the same time, researchers want to visualize how it's playing out statistically so they can imbue decisions with their own wisdom.

Gabriel Fuchs, senior manager at Switzerland's La Suisse Assurances, brought in Spotfire's DecisionSite to analyze the data behind its insurance renewal risk-management process. "We have 60 big accounts we have to follow, and there's too much information for any team to see the sweep of it," Fuchs says. "It's so easy to iterate and analyze now; we use alert systems with color coding and we get to visualize trends, spot potential risk. And it saves us vast amounts of paper, too -- one application saved 14,000 pages of printed reports."

BA Work Cycles

There are organizational and job-role differences between the way enterprises use BI products and how they use BA.

A predominant workflow for traditional BI follows a typical IT line-of-business labor division. During the construction of a data warehouse or data mart, IT analysts look over the canned reports included with their products. Analysts take prototype reports to the line-of-business domain experts and ask them which of these serve the domain experts' work. That set of reports will constitute the core of what line-of-business folks get.

Industrious IT analysts will then attempt to build some custom reports, asking the domain experts what else they would like to know based on the available data fields, intrinsic arithmetic, and functional operators. As domain experts deal with their BI tools, they uncover the kinds of new relationships these tools can expose and, depending on the tool set and the organization's structure, either the expert will construct new queries or put requests into IT's queue to do that.

BA work cycles aim to present structured, unpruned data to the domain experts. Some products try to remove the query creation burden from the IT group altogether with an interface designed to be friendly enough that people with statistical or scientific -- or even marketing -- background can build and store procedures.

The BA products aren't so much aimed at generating the same routine repeatedly as they are at mining the data to catch the new relationships and trends any organizational system is facing. So BA client software usually is designed for a domain expert to troll, ask a question, get "visualizations" (graphic and numeric results) and ask the follow-up questions inspired by the answers, iteratively.

Like a well-equipped handyman's tool kit, BA software comes equipped with a bunch of techniques and tools you can throw at data to try to reveal relationships. More and more, BA vendors are presenting preformed templates for attacking specific business problems, such as marketing segmentation, pharmaceutical trials, or risk management. In general, BA clients are for professionals; those without statistical background, at least in the domain the data comes from, likely will not take full advantage of the systems.

How much responsibility should the BA software take for interpretation and choosing options? How much should the software operator take on? The answers dictate who in the organization will run the software.

Some vendors, such as Kxen, aim to ease the burden by attempting to automate wisdom. This opens up the tool's use to significantly wider audiences. But SAS Institute, for example, builds its software for a range of statistically aware departments.

IDC's Henry Morris, group vice president, who formulated the term "business analytics," says the next logical designs are "policy hubs," constellations of related BA and BI systems that trigger more, increasingly intelligent analysis.

But that concept is far off for most organizations -- they can get plenty of mileage out of BA now. "Today," Edelstein says, "our data warehouse and analysis software is way ahead of our ability to use it organizationally."

Food for Thought

Any effort to introduce a new category of software more complex than a desktop notepad faces vectors of organizational resistance and technical hurdles. BA is no different. Although the returns can be astronomical, the barriers are just as high as any other project. The most common barriers include the following.

Legacy Orientation

Upper management has a proclivity for seat-of-the-pants decision-making and a reliance on the legacy approach of eyeballing and historically based guesses. In the classic business dichotomy of science vs. seat-of-the-pants, seat-of-the-pants usually wins.

Fear of the unknown BA calculations and processes aren't "transparent" and some fear what they don't understand. The usual resistance-to-change factors will appear: fear of losing legacy reports, not wanting to learn new programs, fear about losing jobs.

Data Preparation

Preparing data for most BA clients requires some conversion, such as from text fields to numeric fields, and some of this requires manual triggering of search-and-replace routines. Some products can take a stream of data directly from a data warehouse, but many others require a proprietary data store for data prepared for the BA work.

Data Sanitation

Data cleanliness is always a serious issue, but it's much more critical when you're making decisions that can appear to be not "transparent" based on the data.

IT resources Many organizations don't have staff with statistical/domain expertise. Hard-math analysts and pattern-recognition statisticians have always been relatively sparse, but five years of layoff trends have exacerbated the imbalance.