So you’ve got approval for a big data project and addressed the top four challenges. Now the final hurdle is deploying the big data network.
Speaking at the IDC Asia-Pacific Business Analytics conference in Sydney recently, Arista Networks Australia principal engineer Lincoln Dale shared his tips with delegates for creating and deploying big data networks.
“If we look at what the network looks like behind big data, what we tend to have is racks and racks of servers with built-in storage,” he said.
“The key here is that big data networks are not built like enterprise IT networks."
According to Dale, with a big data network CIOs and network engineers need to automate the infrastructure as much as possible.
“For example, adding a new rack or server should be a case of someone stacking it and turning it on,” he said.
“You don’t want a system administrator spending a couple of days putting an image [of a server] on the network. If something fails, you don’t want it to take out the entire [big data] cluster.”
Dale added that technology such as open source distributed computing platform Hadoop can handle big data projects and potential system failures.
“A key point of big data is that performance matters. You can do unstructured queries and data mining in ways to find out things,” he said.
Turning to the future of big data, Dale cited Moore’s Law — a law which was created in 1970 and predicts the amount of transistors on integrated circuits doubles every two years.
According to Dale, Moore's Law also applies to digital data. For example, IDC’s <i>Worldwide Storage Services Opportunity for Big Data 2012-2016</i> global report found that digital data is doubling approximately every two years, with 1.8 trillion gigabytes of data created in 2011 alone.
“The challenge is that a lot of this data loses value over time so the sooner you can reach an outcome from a given piece of data it has more value,” he said.
For example, if a consumer is on an online shopping website and navigates away from the site before coming back, this shows they are on other e-commerce sites looking for better prices.
“You can potentially make use of big data analytics to offer the consumer a better price sooner which could result in a sale,” he said.
Follow Hamish Barwick on Twitter: @HamishBarwick