Data storage has always been a challenge but in recent times it has become harder to manage, purely because of the sheer amount of information organisations are dealing with. This includes structured data from enterprise systems and unstructured data from social networks – all accessed using connected and increasingly, mobile devices.
These trends have raised significant issues for storage managers around how to best manage capacity to cope with the constant influx of data, while optimising performance, managing disaster recovery activities and controlling costs. At the same time, IT managers and other technologists have more choice than ever when it comes to controlling storage infrastructure – including managing all or parts of their storage onsite or in the cloud.
Throughout May and June, IT leaders discussed these issues and more at a series of Computerworld roundtable events in Brisbane, Melbourne, Perth and Sydney, sponsored by IBM.
All attendees agree information is growing at a rapid rate inside their organisations. Some say their storage capacity is growing by upwards of 50 per cent each year, forcing them to rethink how they classify information and provision systems to keep a lid on storage costs.
Robby Castillo, IT infrastructure manager at the Royal Australasian College of Surgeons, says his organisation is running out of storage capacity due to the growth in development teams and their work.
“We need more space and to leverage new technology but there’s a price tag attached to that and convincing management we need to do it in a certain way has been a challenge,” he says. The organisation has allocated budget for storage refreshes this year and Castillo says there are “plenty of fancy solutions out there” but all must be considered carefully in terms of outlay on servicing, as well as future costs.
“It’s not just about provisioning space, it’s about provisioning solutions,” he says. “You have to think about your retention, your backup, your disaster recovery infrastructure – there has to be space for that too.”
One roundtable attendee from a large council says his organisation is experiencing a huge explosion in storage and now manages 1.8 petabytes of data. “One of the biggest drivers is we are focusing on data replication across two storage facilities.”
Another senior infrastructure architect at a large logistics company has disparate storage in 100 data centres across 52 countries. This has left him with the significant task of consolidating and classifying information.
“Storage growth has put more of an emphasis on finding out which data is relevant and which is not; and exactly the best place to store this data,” he says.
Demand for storage has grown by about 40 per cent annually since 2004, says another infrastructure manager, this time at a major Australian law firm. “The challenge is the integration of data, given the different apps and ways things have been stored now and in the past,” he says. “What lawyers want is to have a simple, single pane of glass. They want to be able to drill into various pockets of storage, yet have it all displayed in front of them.”
Bernard Ching, national IT manager at Greencap, believes some of the real challenges are backup and data retention. “We have an extremely mobile workforce and the challenge is to enable user access on the road and protect data with our backup and data retention policy,” he explains.
“We write so much data that we have to run 24 hours per day to backup the terabytes of data being generated daily. Data is written in a certain way by an application, and this may not be supported in the future, or you might not have the licence or a password to access it.
“We can stick our heads in the sand and say ‘we won’t be around in 100 years’ time’ but ultimately, we need to give these issues more focus.”
Discipline in storage management
With an explosion in structured and unstructured data, organisations need to take a closer look at classifying their information. Many attendees across various state capitals highlighted a lack of discipline around data storage requirements and classifying information five years ago.
“The hardest part is getting discipline in data management around what users and the organisation want to do with this old information now, and how valid or invalid it is,” says Grant Cresswell, IT manager at Opera Australia.
“By the time you get everyone together and make some decisions on it, you could have stored it, saved it and backed it up or archived it several times.”
At Opera Australia, video is driving massive storage growth and triggering a rethink of how to record, manage and store information both for short-term and long-term use.
“It [video] has become a significant part of our world – everything we do is recorded and the video may be used for a bunch of different purposes. When they start out, people may not know all the possible uses for the material,” Cresswell says. “So we start with the big picture – high-definition, no loss video – and you’re supposed to find a place to store it all.”
One attendee from a large government department is also finding his storage needs are being driven by video-based content. “We store a couple of hundred terabytes of storage and the growth for us is in the areas of audio and video – we do a lot of surveillance as part of our investigations into crimes,” he says.
Superchoice CIO, Ian Gibson, points out the explosion in marketing and customer data and the increasing need to store this data is another trend to watch. “How does your company’s offer engage with young audiences who engage with social media and how do you capture that huge amount of unstructured data and do something useful with it? This is where the interesting challenge is going to be,” he says.
Consolidating storage to reduce costs
Energy company, SP AusNet, is undergoing a refresh of its storage environment, consolidating systems, vendors and management tools to improve service, reduce costs and reduce corporate risk.
“We are improving our [storage] offering to our business consumers and expected to reduce the overall unit costs when we finish,” explains Chris Howard, the company’s ICT business office manager. “The unit of storage capacity we provide to the business is going to cost much less, but will deliver an overall better service. Lower cost and better service – that is ultimately what we are trying to achieve.”
The refresh was prompted by SP AusNet’s ageing storage environment, which contained a number of different technologies and multiple vendor platforms. Additionally, infrastructure usage was very constrained and the organisation frequently ran out of capacity. “The storage refresh takes a strategic approach to storage acquisition and greatly reduces the likelihood of shortages of capacity issues,” Howard says.
“Because SP AusNet is deploying smart meters and collecting large amounts of data, we now consider ourselves to be on the big data journey. While we have not fully matured on this yet, we are starting to understand what analytics can do for us and how this can be used to deliver a better service to our end customers: Power consumers.
“Consequently, we are aiming to create a [storage] framework where we are not constrained; for us storage management is not about being logically ordered but being able to create a capability for our business that enables us to move forward.”
At SP AusNet, a range of applications require quick responsiveness and sit on high-speed storage. Yet there are other systems where end users wait for longer response times. “In the new environment, these continue to be better than the legacy storage environment,” Howard says.
The focus is now on maximising the use of management tools, as well as applying business rules and allowing these to manage the usage and storage of data. Reducing the storage management overhead, applying simple business rules to archive data, and improving the service overall was very important during the refresh, says Peter Owens, infrastructure project manager at SP AusNet. Owens is in charge of implementing the new storage environment.
“We don’t have the issue of storing [data] for 100 years like some institutions, but we have 45-year-old data sets to deal with,” he continues. “We have transformers and substations that have historic plans that have been digitised plus other bits of data that we don’t need to keep that long.
“We are also doing a backup technology refresh and creating a new archive strategy, which is a component of the overall storage environment refresh. We will no longer be storing data that hasn’t been accessed for long periods such as 12 months on expensive disk; the management tools will automatically move it off to lower tiers of storage.”
The simplification of SP AusNet’s storage environment means IT staff will no longer be spending hours provisioning, allocating and managing a complex storage environment. “This all comes back to data classification and understanding how data is actually used and what recovery times are appropriate,” Owens says. “We had an example a few months back where we archived old information that no one was aware a small business group was using at the same time.
“It all went off to tape, which was great, but then all of a sudden, we had contention issues when five people were accessing the system and the tape library was going berserk.
“It comes back to business processes; part of the archiving is looking at how often the data is used, how often it needs to be accessed and making recommendations. For example, we don’t need 6MB photos because that’s what the camera takes when 200KB is fine.”
Still life left in tape
Most attendees admit they continue to rely on tape as part of their storage environment, with several claiming the technology has been hard to get rid of, even though it has been relegated to a tertiary storage tier. “Tape is not dead and it’s becoming a cheap medium,” says Joe Screnci, A/NZ storage business unit executive at IBM. “And with linear tapes, that’s giving us really easy ways to leverage that cheaper data but making it appear like a simple USB device. This life of tape is also a consideration – 20 years’ shelf life.”
According to Dan Schubert, general manager of IT at workforce accommodation services group, The MAC, tape is no longer core to business and is simply a tertiary backup. He points out organisations can legislate and put policies in place to manage tape but when “someone jams a tape sideways, or leaves it on top of the UPS, there’s the backup cycle gone”.
“Getting away from having to rely on people that aren’t trained and pulling backups back to a central location is important; but I do think that there will be always a place for tape as that final backup,” Schubert says. “Has anyone tried to recover from a defunct solid state when it has failed? You can send tapes off to an engineer who can read the 1s and 0s but on solid state, it’s gone for good, you’re back to sticking it in the freezer and hoping it comes good.”
Jason Glasby, group technology manager at building products manufacturer Brickworks Limited, recalls a situation where his organisation couldn’t retrieve tapes for a week-and-a-half from the biggest tape suppliers in Western Australia due to industrial action.
“For us, tape is still very critical in backup,” he says. “However in theory, there was no real risk to the business, other than that I ran out of tapes and didn’t have a full back up going off to the site. Our systems are replicated between the east and west coasts [of Australia] to prevent this issue.
“What is the shelf-life of flash storage? Tape was good as we still wanted a portable medium.”
The consumerisation of storage
As if the challenges of data weren’t big enough, end users are also increasingly using inexpensive public cloud-based storage services like Dropbox, Google Drive and Skydrive, to store personal files and in some cases, confidential work documents. Although IT departments can put controls around the use of these public storage offerings, several roundtable attendees claim organisations need to come up with a better alternative for their users. “These cloud storage offerings are attractive because they are easy to use,” says one attendee. “IT departments can create as many policies around their use as they want, but people are still going to use these public cloud services. IT departments need to provide a viable alternative.”
Another attendee adds that with the proliferation of smart devices, there’s nothing stopping an end user from downloading confidential documents between their smartphones and a service like Dropbox. One of the challenges is that many users don’t think too much, if at all, about security, particularly when they demand having information at their fingertips.
“A lot of people focusing on security base it on securing back-end data,” says one attendee. “But they should also be focusing on tracking how a person is accessing data and having the smarts to raise a red flag indicating that people are accessing something they shouldn’t be accessing.”
Whatever the storage, platform or governance approach IT managers take, Jean-Michel Carre, midrange storage technical sales leader at IBM’s Systems and Technology Group, says the ultimate aim should be freeing up more time to explore innovative technology solutions that will drive business growth and revenue.
He also points out the storage technology smarts are available today to help drive a new age in information management, and will continue to evolve to provide more intelligent ways to manage real-time environments.
“If I were the IT manager today, I’d like to spend most of my time running new applications and new services for my customers,” Carre says. “What hardware my data sits on isn’t what I care about. Our customers worldwide say they spend 70 per cent of the time just getting things working and maintaining them. Only 30 per cent of the budget is dedicated to developing and deploying new applications.
“In this very competitive world, if we can shift 20 per cent from daily storage management to new applications, that’s going to be very important for your business.”