The ideal data mining experts are technologically savvy and understand databases. But they're also comfortable and familiar with statistics and business and have a much broader business perspective than most IT positions allow, said Howard Dresner, a research director at international consulting firm GartnerGroup Inc. in Stamford, Connecticut, who specializes in helping clients set up their own data mining systems.
Currently, most information technology jobs in this field are supporting rather than starring roles. But that means there are more opportunities to break in. Todd Swift, a partner at consulting giant Ernst & Young LLP and its national director of data warehousing solutions for SAP AG, sorts those supporting players into the following four key roles:
If you've built a complex Web site or put together a graphical user interface, you can move into data warehousing and data mining by building front-end interfaces that allow end users -- executives and managers -- to access warehoused data in a meaningful way. These jobs call for IT pros with strong Visual Basic, PowerBuilder, client/server and Web-based programming skills, as well as the ability to figure out what the end user wants and match it with tools available on the market, says Dave Buch, IT director at Capital One Bank in Falls Church, Virginia.
Buch builds a technical infrastructure that allows high-volume access to warehoused data focusing on analysis and reporting, not transaction processing. Those jobs tend to be at outside contractors rather than end-user companies, although they may be in-house at businesses where data warehousing is central to the business. Because Capital One has a huge data warehouses and many sophisticated users accessing it, Buch looks for employees who have experience with databases holding as much as 1T byte of raw data but who also have a mainframe or Unix background. They may be required to extract data from legacy systems and apply new technology to it.
IT professionals with these skills design and implement the process of standardizing and conforming information from disparate systems. This category includes the administrator who maintains the data warehouse, figures out how and how much to index the data, summarizes the data and builds aggregates. "The data mining software isn't smart enough to determine which variables are important, so part of the job is to take the stored data and add derived data based on what's there. Then you start being able to see interesting patterns," said Michael Berry, co-author of Data Mining Techniques: For Marketing, Sales, and Customer Support (John Wiley & Sons Inc., New York, 1997). Once you learn to use the application software involved, expect to demand a high salary and develop a certain amount of expertise in the data itself. The next step is project management.
Typically, this is a job for a project manager who combines business and technical savvy to help end users get the most out of the data. The project manager builds specific data models -- especially for specific business or performance management reports -- and may be called upon to suggest ways to act on the results. Project managers can command salaries that reach into the low six figures; independent consultants report charging US$1,000 per day and up.
Because this position offers an overview of both technical and business issues, it can lead to the executive level or consulting.
(Fitter is a freelance writer in Arlington, Massachusetts.)