It isn't just the massive growth of digital data, which is escalating at a rate of 30 to 60 percent per annum, that is keeping IT managers awake at night. It is the exponential growth of unstructured data that is emerging as the real source of enterprise mayhem.
Intelligent Business Research Services (IBRS) analyst Kevin McIsaac said for most organizations this high rate of growth is not new.
"What is different is that the growth is being driven by unstructured data," he said.
In the last 20 years, McIsaac said organisations have already made significant investments to deal with structured data.
He said this has led to well-managed, structured information that supports and drives the business.
"On the other hand few organisations have invested similarly in unstructured data (e.g., e-mail, faxes, and documents) and many organisations are now finding the growth in unstructured data is a serious business problem," McIsaac said.
"While structured data is growing anywhere between 30-40 percent per annum, unstructured data is growing at 65 to 200 percent per annum."
Traditionally, structured data has been the largest class of digital data in an organisation.
However in recent years, McIsaac said extensive use of e-mail, word processing, PowerPoint and audio/video has shifted this balance.
He estimates unstructured data is as high as 85 percent in some organizations.
"Even if this is not yet the norm for most organisations, unstructured data is typically 50 percent plus of storage capacity," McIsaac said.
"Despite this shift, more than half of all organisations lack commercial content/document management systems for their unstructured data. This is like trying to work with structured data without using a database management system or an enterprise application such an ERP or a SCM."
Even though most organisation's storage capacities are growing at a ferocious rate, McIsaac said this hasn't translated into massive budget blowouts for storage hardware.
He said this is due to "Shugart's Law" which states that "The price per bit of magnetic storage devices halves every 18 months."
Using Shugart's Law, and the assumption that storage is replaced every four years, a storage capacity growth of 63 percent per annum can be accommodated with a fixed budget for storage hardware.
"Hence the problem with high rates of data growth is not the cost of storage hardware, rather it is the cost of managing that hardware and the data," McIsaac explained.
"The use of IT to support the collection, organisation and processing of structured data has driven significant improvements in workforce productivity.
"That is, the implementation of enterprise applications has enabled organisations to deal effectively with massive volumes of transactions using relatively small numbers of back office staff.
"Unfortunately business is drowning in unstructured data and does not yet have the applications to transform that data into information and knowledge. As a result staff productivity around unstructured data is still relatively low."