I often liken the information management evolution to mathematics. You have to understand addition before moving on to multiplication and then to algebra, etc. Similarly, you have to be comfortable with your storage management to feel confident in your data management and to understand your requirements for information management. I see three trends in the market that are converging to move information management to the next level.
The first trend is the pain many customers are experiencing in managing their application data lifecycle today. A couple of weeks ago, I wrote about the legal aspects of data retention and the problems associated with communication data such as e-mail, voice-mail and instant messaging. This pain is not new, by any means, but the scale of the problem is growing with the amount of data being placed into digital formats.
While I was working in marketing for a storage vendor in the mid-1990s, a customer came to the company with a problem. He had just been put in charge of open systems storage at his company's data center after spending some time in the IBM 390 environment. He told me he needed an automated policy-based data lifecycle management system that would automatically move data off his company's expensive primary storage systems to less expensive storage solutions. He told me that thehierarchical storage management (HSM)solutions available at that time for Unix and Windows systems were too file-based, i.e., they had no awareness of data context or of the application utilizing the data. This pain is being felt even more strongly in markets where compliance to federal regulations is currently required or eminent.
The second trend is the availability of very inexpensive disk storage targeted at the nearline or secondary storage operations. Large and small vendors alike, such as NetworkAppliance, EMC, and Scale8, have announced products in this space. These products are being positioned as the first line of abstraction from primary application data stored in online disk subsystems rather than traditional tape or optical storage subsystems. Using an inexpensive disk for nearline storage is ideal for applications that require fast random access to data, for example file-level restores from backups. However, the lack of total integration between storage media, including disk-to-disk and further to tertiary storage disk-to-tape or disk-to-optical has limited the growth in this market.
The third trend is that database performance is tracking neither the significant increase in processor speeds or storage capacities. Customers migrate their database applications to run on higher performing processors only to gain minimal increases in application performance, even for the most "tuned" database. Depending on the database architecture, the ever-increasing size of the data being managed becomes a proverbial ball and chain to performance.
These three trends are converging to necessitate the emergence of application data lifecycle management companies such as OuterBay. The discipline of application data management automates the storage relationships (primary, secondary, tertiary, etc.,) and the application data relationships.
For some application environments a simple HSM capability, with migration policies based on size and time last accessed may work well. However, typical HSM solutions do not work well with databases or other applications that use several "files" to create a data object. HSM has no understanding of these relationships, or application data context. The ability to understand these higher-level abstractions is imperative to maintaining application data integrity when moving or archiving data to less expensive storage at any time but especially in environments where compliance reigns.