The perils of dirty data
- 31 October, 2007 10:00
Few IT projects are more frightening than data integration and reconciliation. Actually, let us rephrase that. One thing is more frightening -- when data integration goes bad.
Sometimes it's a problem of starting out with bad data, through user error or even deliberate sabotage. Sometimes the data starts out good but gets lost, truncated, or altered when it moves from one system or database to another. Your data may go stale, or it may become collateral damage in a turf war inside your organization -- everyone clinging to their own little piece of the data store, nobody willing to share. The task certainly isn't helped by the overwhelming volume of data companies generate each day.
Data projects can go bad in many ways. Here are five of the most common: what went wrong, what happened as a result, and what you can do to avoid having the same thing happen to you. The names of the companies involved have been obscured to protect the guilty. Don't let your own project become someone else's horror story.
1. The "Dear Idiot" letter
Be careful where you get your data -- it may come back to haunt you. This tale of terror comes from the customer call center of a large financial services institution. As in nearly all help desks, service reps take calls and enter customer information into a shared database.
This particular database had a salutation field that was editable. Instead of being constrained to Mr., Ms., Dr., etc., the field could accept 20 or 30 characters of whatever the rep typed. As service reps listened to the complaints of angry customers, some of them began adding their own, not entirely kind, notes to each record, like, "what an idiot this customer is."
This went on for years. No one noticed because no other system in the organization pulled data from that salutation field. Then, one day, the marketing department decided to launch a direct mail campaign to promote a new product. They came up with a brilliant idea. Instead of purchasing a list, why not use the service desk database?
So the letters went out: "Dear Idiot Customer John Smith."
Strangely, no customers signed up for the new service. It wasn't until the organization began examining its outgoing mail that it figured out why. The moral of this story?
"We don't own our data any more," says Arvind Parthasarathi, vice president of product management and data quality for data integration specialists Informatica. "The world is so interconnected that it's likely someone will pick up your information and use it in a way you never anticipated. Because you're pulling data from everywhere, you need to make sure you have the right level of data quality management before you use it for anything new."
What constitutes the "right level" will vary depending on how you use the data. "In the direct mail industry, getting 70 to 80 percent of your data correct is probably good enough," he adds. "In the pharmaceutical industry, you want to be at 99 percent or better. But no company really wants, needs, or will pay for perfect data; it's just too expensive. The issue always is, how will it be used and at what point is it good enough?"
2. Dead men cast no votes
Data cleansing can be a matter of life and death -- literally. PR specialist Nancy Kirk was volunteering in the congressional elections of 2006, calling registered voters to get them to the polls, when she noticed something odd: Three out of ten voters she dialed were deceased and thus ineligible to vote (except in certain precincts in Chicago).
The problem of having data that is literally dead is not uncommon in the commercial world, and it has real consequences for the living.
Jim Keyster, president of The Keane Organization's investor retention and communication services division, has spent the past year rolling out an investor data quality program for Keane's clients, which include major insurance companies, mutual funds, and Fortune 500 firms.
On average, Keyster says, 8 to 15 percent of clients' data records contain anomalies such as mistyped Social Security numbers or outdated addresses. But about one in five of those anomalies is a shareholder who's been dead for more than ten years. In one case, a client had an "active" account for a shareholder who last drew breath more than 72 years ago.
"This isn't client negligence, it's just a naturally occurring problem," Keyster says. Private companies go public, change names, get acquired, or spun off, and their shareholder data follows along, often for decades.
Join the Computerworld Australia group on Linkedin. The group is open to IT Directors, IT Managers, Infrastructure Managers, Network Managers, Security Managers, Communications Managers.
Thanks a million, Drupal
Optus goes over the top with VoIP service
Turnbull asks how the NBN got that way
U.S. retailers insist on PIN requirement in smartcard rules
Yelp speeds database access with flash storage