Drowning in the data lake: Why a simple data analytics strategy is better than none at all

Many Australian businesses now fit into one of two categories: Those still yet to tap their big data reserves or those drowning in data overload

Australian businesses have proven to be big adopters of data analytics practices over the last couple of years. More than 50 per cent of Australian organisations are now equipped with big data solutions and spending on data analytics is forecast to reach $US114 billion globally by 2018. In fact, 90 per cent of all data ever produced has been made in the last two years. But are these companies just moving with the times or is it the end game of actionable insights that’s driving them?

When put to good use, data can provide endless opportunities for innovation and growth, save money and time and expedite services. Despite the opportunities, many Australian businesses now fit into one of two categories: Those still yet to tap their big data reserves or those drowning in data overload.

One of the major reasons is that big data is proving difficult to manage. Terms like ‘data lake’ have been coined to describe the wasteland of data housed by some companies, often without the faintest idea of how to use it.

Research conducted by TNS Marketing Monitor across the Asia Pacific region in July 2015 reveals that at a time when businesses have never had so much data at their disposal, the sheer volume and variety of it is making it difficult to analyse and identify valuable insights.

Seventy per cent of the marketing professionals surveyed say they find it difficult to integrate data from different sources because there is so much available. While they know they should be able to make decisions in real-time, many are struggling to integrate traditional and digital methods.

The problem is many CIOs and marketing teams have taken the approach of collecting data in the hope that it will make it easier to gather analysis in the future. This approach to big data is not only inefficient and expensive, it can often kill a data analytics project in its tracks. Instead, businesses should start these initiatives the other way around.

Before collecting data, businesses should determine what insights they want and need from customers, competitors and allies, prioritising high business value. This means that instead of trawling through data looking for common links or themes, businesses can be prepared with a strategy to discern the information that is most relevant to them, to effectively maximise their time and money. By applying methodologies like Design Thinking and Agile Development, businesses can ensure that they focus on the right problems and develop highly viable and feasible solutions.

The best way to analyse the information you gather is to build a custom analytics solution, designed to deliver the insights required. While a bespoke platform has historically required significant time and financial investment, the use of open source technology is changing the analytics landscape. Open source big data platforms offer incredible capabilities to build not only insights solutions but also forecasting and predictive analytics solutions through mathematical and statistical models that have the ability to crunch through large volumes of data.

Open source opens up a world of access to the latest technology that would otherwise take time and resources to develop. Where we used to rely on human intervention to process and compute data, open source means companies can effectively codify and automate significant pieces of their operations to make better use of their time and money while extracting better insights. The technology is perfect for enterprises that have a growing expectation of flexibility and faster results. There’s no vendor lock-in and the associated costs are much lower than proprietary solutions.

Through its investment in open source technology with the recently launched Infosys Information Platform (IIP), Infosys is both an advocate and active member of the open source community, helping enhance and improve the quality of software freely available around the world.

Open source software was used to help a major mining company process streaming data to enable predictive maintenance on its fleet of vehicles. The solution was delivered at an astounding one/one-thousandth of the cost of the existing install base. It was only by approaching the analytics solution focused on the outcomes and with the flexibility of open source technology, that these results were so rapidly and effectively achieved.

The growth of data analysis is also having an impact on consumers who are now becoming more open to sharing their personal data by way of mobile phones, wearables and other connected devices in the IoT world. For example, a 2013 Infosys survey of 5,000 consumers in five countries, including Australia, found:

• 87 per cent wanted their bank to mine personal data to protect against fraud

• 88 per cent of patients wanted doctors to have digital access to their medical history

• 62 per cent of consumers were happy to share online and mobile shopping behaviour for more targeted ads, and;

• 53 per cent would opt-in for retailers to track their smartphone location for targeted offers

From my experience, businesses are mindful of the value that can be extracted from their customers’ insights, but are also keenly aware that they could quickly lose the trust of their customers if it’s not safeguarded.

Read more: Moving from data to analytics and insights

Untangling the web of big data can seem a daunting task, but by determining which insights you want to extract at the beginning of the process and quickly building an analytics platform with the flexibility of open source technology, businesses can now access actionable insights and foresights faster, easier and more cost-effectively.

Thanks to open source technology, you can have your data and analyse it too.


Dr Wasim Sadiq is adjunct professor at the University of Queensland and GM of research at Infosys.

Join the Computerworld newsletter!

Error: Please check your email address.

Tags data analyticsbig dataData Lakedata lakes

More about AgileInfosysTNSUniversity of Queensland

Show Comments