Few IT projects are more frightening than data integration and reconciliation. Actually, let us rephrase that. One thing is more frightening -- when data integration goes bad.
Sometimes it's a problem of starting out with bad data, through user error or even deliberate sabotage. Sometimes the data starts out good but gets lost, truncated, or altered when it moves from one system or database to another. Your data may go stale, or it may become collateral damage in a turf war inside your organization -- everyone clinging to their own little piece of the data store, nobody willing to share. The task certainly isn't helped by the overwhelming volume of data companies generate each day.
Data projects can go bad in many ways. Here are five of the most common: what went wrong, what happened as a result, and what you can do to avoid having the same thing happen to you. The names of the companies involved have been obscured to protect the guilty. Don't let your own project become someone else's horror story.
1. The "Dear Idiot" letter
Be careful where you get your data -- it may come back to haunt you. This tale of terror comes from the customer call center of a large financial services institution. As in nearly all help desks, service reps take calls and enter customer information into a shared database.
This particular database had a salutation field that was editable. Instead of being constrained to Mr., Ms., Dr., etc., the field could accept 20 or 30 characters of whatever the rep typed. As service reps listened to the complaints of angry customers, some of them began adding their own, not entirely kind, notes to each record, like, "what an idiot this customer is."
This went on for years. No one noticed because no other system in the organization pulled data from that salutation field. Then, one day, the marketing department decided to launch a direct mail campaign to promote a new product. They came up with a brilliant idea. Instead of purchasing a list, why not use the service desk database?
So the letters went out: "Dear Idiot Customer John Smith."
Strangely, no customers signed up for the new service. It wasn't until the organization began examining its outgoing mail that it figured out why. The moral of this story?
"We don't own our data any more," says Arvind Parthasarathi, vice president of product management and data quality for data integration specialists Informatica. "The world is so interconnected that it's likely someone will pick up your information and use it in a way you never anticipated. Because you're pulling data from everywhere, you need to make sure you have the right level of data quality management before you use it for anything new."
What constitutes the "right level" will vary depending on how you use the data. "In the direct mail industry, getting 70 to 80 percent of your data correct is probably good enough," he adds. "In the pharmaceutical industry, you want to be at 99 percent or better. But no company really wants, needs, or will pay for perfect data; it's just too expensive. The issue always is, how will it be used and at what point is it good enough?"
2. Dead men cast no votes
Data cleansing can be a matter of life and death -- literally. PR specialist Nancy Kirk was volunteering in the congressional elections of 2006, calling registered voters to get them to the polls, when she noticed something odd: Three out of ten voters she dialed were deceased and thus ineligible to vote (except in certain precincts in Chicago).
The problem of having data that is literally dead is not uncommon in the commercial world, and it has real consequences for the living.
Jim Keyster, president of The Keane Organization's investor retention and communication services division, has spent the past year rolling out an investor data quality program for Keane's clients, which include major insurance companies, mutual funds, and Fortune 500 firms.
On average, Keyster says, 8 to 15 percent of clients' data records contain anomalies such as mistyped Social Security numbers or outdated addresses. But about one in five of those anomalies is a shareholder who's been dead for more than ten years. In one case, a client had an "active" account for a shareholder who last drew breath more than 72 years ago.
"This isn't client negligence, it's just a naturally occurring problem," Keyster says. Private companies go public, change names, get acquired, or spun off, and their shareholder data follows along, often for decades.
Read up on the latest ideas and technologies from companies that sell hardware, software and services. Gaining Competitive Advantage Through Enterprise Planning
Strategies for Eliminating .PST Files
Business Intelligence and Enterprise Performance Management: Trends for Emerging Businesses
Email Archiving 101—Customer Case Study
Email Archiving Implementation: Five Costly Mistakes to Avoid
Controlling storage costs with Oracle database 11g
Best Practice in Building an Integrated Information Management Strategy
Mimosa™ NearPoint™ for Microsoft® Exchange Server: Email Archiving 101
Zones provide focussed content from Computerworld and leading technology partners.Discover how SOA can create smarter outcomes for your business.
Attend and learn:
- How SOA is helping leading companies to become more agile
- Where you should be applying SOA processes in your company
- The top SOA implementation mistakes to avoid
Click here for more information.
- +
Computerworld Live Podcast #97: The Future of Enterprise Networking 25/07/2008 09:45:36
This week CW Live chats with Mark Thompson, global sales and marketing manager for HP ProCurve, on the future of the enterprise networking. Mark discusses the trends we can expect to see in the near future and how the right infrastructure can ensure your enterprise network is secure. - +
Computerworld Live Podcast #96: Security at the Edge 11/06/2008 09:22:22
CW Live speaks with Amol Mitra, HP ProCurve Director of Marketing for Asia Pacific and Japan. Today's topic: how enterprises are starting to shift away from simply controlling security via server logins, firewalls and moving to more adaptive security frameworks. - +
Data Management Edition #10: Multi-Petascale Systems 02/05/2008 09:12:33
This week we look at sustainability and the development of multicore technologies to build multi-petascale systems. - +
IT Security Edition #11: How to poison the Storm botnet 01/05/2008 08:51:55
This week CW Live presents a case study on how to poison the notorious Storm botnet . Plus we take a look at Cisco's plans for Ironport. - +
IT Security Edition #10: Cyber-battles fought and won 24/04/2008 11:09:47
Vendors bow to end user pressure to improve product security, and we take a look at the latest concepts shaping the cyber-battlefield of the future.
Fortinet November Threatscape Report Shows Calm Before Holiday Storm 2008-12-05 16:00:00+11
Epicor® Cited as an Order Management Solutions Leader by Independent Research Firm 2008-12-05 15:52:00+11
F-Secure: Growth In Internet Crime Calls For Growth In Punishment 2008-12-05 13:00:00+11
International researchers gather in Sydney to preview the clever web 2008-12-05 09:48:00+11
Borderless corporate networks to shift focus to secure content management in Australia in 2009 2008-12-04 16:06:00+11
Data grids and service-oriented architecture
When choosing an SOA strategy, corporations must ensure data availability, reliability, performance and scalability. A data grid infrastructure, built with clustered caching provides a framework for improved data access that can create a competitive edge and sustain customer loyalty. Read on to discover how this can be created within your organisation.












