The fallout from the 2016 Census continues, with IBM pointing fingers at Nextgen and Vocus over the debacle — while at the same time acknowledging a significant technical misstep of its own.
IBM was contracted by the Australian Bureau of Statistics to provide the online portion of the Census. In its submission to the inquiry examining the circumstances that saw the Census website rendered inaccessible after a series of distributed denial of service (DDoS) attacks, IBM said it “deeply regrets the inconvenience that has been caused to the Australian public and the Government by reason of the eCensus site not being accessible during the period it was temporarily unavailable”.
The Census site was subject to a series of DDoS attacks on 9 August (Census night).
IBM says it “anticipated and planned for the risk of DDoS attacks” with a geoblocking strategy, dubbed Island Australia. “This method was chosen because the primary risk of DDoS attacks of sufficient size to disrupt site availability was considered to be from foreign sources.”
The ABS and the Australian Signals Directorate were both aware that IBM intended to use geoblocking to mitigate DDoS attacks, IBM said. (The ABS has previously said it accepted assurances from IBM that its measures to deal with DDoS attacks were adequate.)
Public access to the Census website was via links provided by Nextgen and Telstra, with the network providers directed by IBM to activate Island Australia shortly after the second DDoS attack was launched against the site at 11.45am.
The fourth DDoS attack on Census night — which took place at 7.27pm and lasted 14 minutes — “was of significant size and had the effect of causing the site to become unresponsive and unavailable to the public” IBM claimed.
“The attack was foreign-sourced and hit the eCensus site via the Nextgen link at a time when IBM had already directed Nextgen (and Telstra) that Island Australia was to be in place and in circumstances where Nextgen had provided repeated assurances to IBM prior to the attack that it had done so.”
“In fact, the assurances were incorrect,” IBM claimed. “IBM was informed — later that day after the attack had passed — that a Singapore link operated by one of Nextgen’s upstream suppliers [Vocus] had not been closed off and this was the route through which the attack traffic had entered the Nextgen link to the eCensus site.”
Vocus “admitted the error in a teleconference with IBM, Nextgen and Telstra,” IBM said.
“Had Nextgen (and through it Vocus) properly implemented Island Australia, it would have been effective to prevent this DDoS attack and the effects it had on the eCensus site. As a result, the eCensus site would not have become unavailable to the public during the peak period on 9 August 2016.”
However, IBM’s submission noted that the 7.27pm attack also caused a system monitoring dashboard showed “what appeared to be” a spike in outbound traffic.
“As a result, some IBM employees who were observing the monitor mistakenly formed the view that there was a risk that data was being exfiltrated from the website and that the risk needed to be further investigated,” the submission stated. “Out of an abundance of caution, IBM shut down access to the site and assessed the situation. The cause of the problem was identified. No data exfiltration occurred.”
After shutting down public access to the Census site, the routers at IBM’s end of the Nextgen and Telstra links were rebooted. The Nextgen link remained down, IBM said. The router at the end of the Telstra link “failed to reload its configuration”. “This meant that both links were unavailable,” IBM said.
Resolving the issue with that router took an hour and 20 minutes. “IBM emphasises that the router on the Telstra link was owned by IBM and it takes responsibility for the configuration error with this piece of equipment,” IBM said.
Vocus and Nextgen responded in their own submissions.
Vocus said it “does not agree that the fourth DDoS attack was the cause of the site becoming unresponsive” and that the size of the attack — 563Mbps — “is not considered significant in the industry”.
“Such attacks would not usually bring down the census website which should have had relevant preparations in place to enable it to cater for the expected traffic from users as well as high likelihood of DDoS attacks,” Vocus said.
The cause of the extended outage was the “false positive” by IBM’s monitoring system that identified harmless traffic as data exfiltration and the subsequent decision by IBM to shut down its Internet links.
Nextgen for its part confirmed previous reports that IBM had rejected its offer of DDoS protection ahead of the Census. The company said it had “provided all possible assistance” to implement Island Australia.
The company’s submission stated: “IBM instructed Nextgen to activate ‘Island Australia’ for testing by IBM on 5 August 2016 at 6am, and IBM advised Nextgen that the testing was successful soon after. The same configuration used on 5 August 2016 was to be used if IBM was ever to instruct Nextgen to activate ‘Island Australia’. Nextgen cannot comment on the testing conducted by IBM as it was not involved with it.”
The fourth DDoS attack affected both the Nextgen and Telstra links, Nextgen said, with IBM keeping the Nextgen link down until it was comfortable that there was no data breach.
“There were a number of routes without geoblocking during the fourth DDoS attack, and which were not identified during testing,” Nextgen said.
Following the fourth attack, the network provider implemented DDoS protection on the link. “This was provided at Nextgen’s cost and continued to provide full support to IBM on the service,” Nextgen said.