More mission-critical IP telephony and business computer systems are being hosted in data centres but two recent outages at Equinix and Primus raise new questions about their inherent reliability.
Members of the Australian Network Operators Group began reporting Primus in Melbourne suffered a power loss at around 4pm on Sunday, August 9.
“I just spoke to the datacenter manager who said he was having trouble getting through to them,” one member wrote on the mailing list.
Primus managing director, Ravi Bhatia, cited a fault with a sub-station as the cause of the outage.
According to Primus, the sub-station fault prevented a generator from starting but that’s what generators are there for – to provide power independent of the public grid in the event of an outage.
Primus is working to upgrade its power management infrastructure.
The outage comes one month after Equinix’s Sydney data centre suffered a power outage causing downtime for hosted VoIP services.
“Pretty poor, ~2.5 hrs and counting, this joint has worse uptime than my house... hard to believe we pay for this,” AusNOG member Damien Coxall wrote about the Primus outage.
Carlo Minassian, managing director of Sydney-based managed network security services provider earthwave, said such outages highlight the need for regular testing of power generation equipment.
“It is unfortunate this has happened to tier-1 facilities, but it shows they don’t have the right test procedures and have not validated their infrastructure to work in the event of a disaster,” Minassian said.
One data centre earthwave uses is owned by Fujitsu and located in Sydney. The data centre has multiple generators on site which are kept on standby and tested every quarter.
“Over the past nine years I have been using our facility we have had a number of outages, one lasting about five hours, and the generators worked during that time,” Minassian said.
“I would be definitely asking questions about the business continuity and DR plans and looking for the test results.”
Despite the outages, Minassian said data centres are always going to be more reliable than in-house server rooms as they are designed from the ground up for high-availability.
“Computer rooms in buildings are generally not built to that standard,” he said. “It will be difficult for a corporate or government organisation to invest in excess of $50 million to build a data centre.”
James Braunegg from the Micron21 Melbourne Outer East data centre said the important question will be "is this going to happen again next weekend?" when the power in Primus is upgraded.