The data deluge: The growth of unstructured data

Business is drowning in unstructured data, argues Kevin McIsaac

The volume of digital data created and storage by organisations continues to grow exponentially, typically anywhere from 30% -- 60% per annum. For most organisations this level of digital data growth is not new, however what is different is that growth is now being driven by unstructured data. Over the last 20 years organisations have made significant investments to deal with structured data, resulting in well managed structured information that supports and drives the business. On the other hand few organisations have invested similarly in unstructured data (e.g., e-mail, faxes, and documents) and many organisations are now finding the growth in unstructured data is a significant business problem.

Various industry sources estimate that structured data is growing at 30-40% per annum and unstructured data is growing at 65%-200% per annum. Traditionally, structured data has been the largest class of digital data in an organisation, however in recent years the extensive use of e-mail, word processing, PowerPoint and audio/video have shifted this balance to unstructured data in many organisations.

The volume of unstructured data in organisations has recently been estimated to be as high as 85% of all data. Even if this is not yet the norm for most organisations, unstructured data is typically over 50% of their storage capacity. Despite this shift, more than half of all organisations lack commercial content/document management systems for their unstructured data. This is like trying to work with structured data without using a database management system or an enterprise application such an ERP or a SCM.

Even though most organisation's storage capacities are growing at a ferocious rate this typically has not translated into massive storage hardware budget blowouts. This is due to "Shugart's Law" which states that "The price per bit of magnetic storage devices halves every 18 months."

Using Shugart's Law, and the assumption that the storage is replaced every 4 years, a storage capacity growth of 63% per annum can be accommodated with a fixed budget for storage hardware. Hence the problem with high rates of data growth is not the cost of storage hardware, rather it is the cost of managing that hardware and the data

The use of IT to support the collection, organisation and processing of structured data has driven significant improvements in workforce productivity. That is, the implementation of enterprise applications has enabled organisations to deal effectively with massive volumes of transactions using relatively small numbers of back office staff.

Unfortunately business is drowning in unstructured data and does not yet have the applications to transform that data into information and knowledge. As a result staff productivity around unstructured data is still relatively low. For example:

  • "Office workers spend an average of 9.5 hr/wk searching, gathering and analysing information, with 60% of that on the Internet." - Outsell.
  • "Creative professionals spend an average of 1 out of every 10 hrs on file management. Searching accounts for a 3rd of that time." - GISTICS
  • "A typical business user sends and receives around 600 e-mails per week", Ferris Research.
  • "The average office worker spends 49 min/day managing e-mail. Upper level managers spend up to 4hrs/day." - ePolicy Institute
All that searching and filing, responding and deleting takes an enormous toll on workplace productivity.

Editors Note:

Kevin McIsaac is researching unstructured data growth in Australia. To take part and receive a copy of the report click here.

Kevin will be presenting the survey findings during his presentation at Computerworld's Data Management Forum in Sydney 24/10, Melbourne 25/10 and Brisbane 26/10. For more information or to register click here

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

More about CreativeFerris ResearchHISOutsell

Show Comments