Friday | 21 November, 2008
More data? Get used to it
Metadata and other content is gobbling up precious disk space
Mark Hall (Computerworld (US)) 11/04/2007 12:23:04

As part of the IT team at the Virginia State Police department, Lt. Pete Fagan's job is to ensure that criminal investigators, police officers in the field and other authorities get the most accurate, timely and detailed crime-related information possible. Crime never stops, so the data volumes are huge, dynamic and getting bigger every day. Criminal records often stretch back decades and include source material from countless systems in myriad formats. Multimedia content is a growing necessity. Metadata, a new necessity, is gobbling up precious disk space. Storage capacity, as you might imagine, is an ongoing problem for the department.

But that's not the half of it.

You see, Fagan's goal is to record every request for information on the police department's storage-area network. Not just a log notation of the request, but the entire transaction as it happened. According to the lieutenant, the system will store everything the requestor sees at the moment the request is fulfilled. If the requestor gets to view dozens of JPEGs, MP3 files, videos and other capacity-hogging content, Fagan wants all that information stored separately from the original data store.

Fagan isn't a data pack rat. He's simply doing his job with the tools at hand. For him, information in the moment, knowing exactly what was known and when, could prove crucial to the criminal justice process. Just having the most recent files won't do, because some criminal cases can hinge on when something was known and by whom. Because those criminal files are always in flux, Fagan needs a storage system that can function as a time machine to retrieve not just a facsimile of the past, but the actual files and data from another time. And he needs to store those exact past experiences for more than 300 million requests per year.

Fagan's time machine is emblematic of an impending storage crisis facing IT. More broadly, will it be a harbinger of loftier roles for CIOs? Or will it -- and projects like it -- ultimately condemn IT executives to being perceived as technologists with tactical, not strategic, value to the business?

Content explosion

The Virginia State Police department is in the midst of updating its 1TB storage infrastructure with new gear from Fujitsu to handle the estimated 11TB of data capacity that will be needed in the near future. While 11TB pales in comparison with the petabyte of data stored on average by Fortune 1,000 companies, the department's growth rate is much faster than larger organizations'. That puts it in the middle of the inexorable march to the global zettabyte storage requirement. According to an IDC study, 161 exabytes of digital information were created and stored worldwide last year. (An exabyte, you'll recall, is 1 billion gigabytes.) IDC projects that by 2010, global data creation and storage will reach 988 exabytes, a mere 12 billion gigabytes shy of a zettabyte.

When Computerworld.com's Lucas Mearian reported on the IDC study, he wrote, "The data explosion means the role of IT managers will expand considerably." The IDC report happened to be funded by EMC , and when I chatted about it with EMC executive Chuck Hollis, he speculated that a savvy CIO can take this growth as an opportunity to become "less of a technologist and more of an informationist." He argued that IT leaders can emerge as the primary arbiters of information standards and processes for companies -- much like CFOs raised themselves from lowly accountants to powerful executives sitting at the right hand of the CEO.

Wish it were true.

Even if IDC is right about storage demands skyrocketing in the next three years, you won't see CIOs casting a strategic eye on their companies' information policies or dictating data-retention policies to their business units. Instead, they'll be running around with their hair on fire trying to keep up with the ever-increasing amounts of information pouring into their shrinking corporate SANs. Oh sure, the CIO can outlaw certain file types or act as a trusted adviser who shows business leaders how to approach information life-cycle management. But a storage capacity crisis won't enhance IT's reputation. It will undermine it.

The only way CIOs will improve their standing during a global storage shortage is to ensure that their businesses don't suffer from it. I don't care how clever a CIO thinks a new information management scheme might be or how powerful he thinks it will make him. Such an approach will be meaningless to people like Lt. Fagan, who need data to do their jobs. And I wouldn't want to be the IT executive who tries to force a new storage policy on Fagan. After all, the lieutenant does carry a gun.

Mark Hall is a Computerworld U.S. editor at large. Contact him at mark_hall@computerworld.com.

Related Features
  • +

    9 Paths to Higher Performance 10/12/2007 14:09:23

    When an organization brings together talented people in a creative, collaborative environment it fosters a culture of high performance, which in turn leads to superior business results
    Like high-achieving individuals, some organizations seem to have the Midas touch. Virtually every initiative they touch earns them gold and even those that fail never seem to cost them much of anything at all
  • +

    Ticked Off at Tick the Box Mentality 04/02/2008 13:01:15

    Does your executive search firm know the difference between an MIS manager and a CIO, and if it does, can it explain that difference to its corporate clients?
    Does your executive search firm know its MIS managers from its elbow? Does it even know the difference between an MIS manager and a CIO, and if it does, can it explain that difference to its corporate clients?
  • +

    Strategies for Dealing With IT Complexity 24/12/2007 10:30:47

    Every innovation, every business process improvement, comes with an IT complexity tax that must be paid by CIOs in time, money and sweat. Here are strategies to mitigate the increasing complexity of IT as it enables new business.
    Every innovation, every business process improvement, comes with an IT complexity tax that must be paid by CIOs in time, money and sweat. Here are strategies to mitigate the increasing complexity of IT as it enables new business.
Additional Resources
Executive Guides
Whitepapers
Zones
Zone logoZones provide focussed content from Computerworld and leading technology partners.
Newsletter Subscription
Sign up for our Computerworld newsletters!
RSS Feeds
Market Place

 

Smart SOA World Tour

Discover how SOA can create smarter outcomes for your business.

Attend and learn:

  • How SOA is helping leading companies to become more agile
  • Where you should be applying SOA processes in your company
  • The top SOA implementation mistakes to avoid

Click here for more information.
Whitepaper

Everything you need to know about email and web security (but were afraid to ask)

What you don’t know can destroy your business. It’s hard to imagine modern business without the internet but in the last few years it has become fraught with danger. Read on to discover how internet security can give your business a competitive advantage.

Enterprise IT Buyer's Guide
Find Technology Vendors Fast
 
Find vendors by name | Find by category
Sponsored Links