Businesses are always looking for better ways to sell their products. If there was a way for a business to gain a competitive advantage over their competitors, why wouldn’t they take it? Goldman Sachs has replaced 600 of their traders with 200 engineers over the last two decades, seeing a need for (and the importance of) technology. Trades are now executed through algorithms designed to emulate what a human trader would do. And what are they using? Machine learning through big data. That’s a competitive advantage.

All our mobile phones are tracking sensory data – our movements via GPS, who we’ve been in contact with ­– via Bluetooth. Seeing where populations are congested or knowing people’s movements could give us better insights on how to better improve transport links. Vast quantities of sensory data – or big data – were used successfully to track the 2010 Haiti cholera outbreak by using Twitter feeds. If you’ve used Amazon, then you’ll know that other retailers are now feeling the pressure to, as Harvard Business Review phrases it, “dynamically recommend products and set prices that appeal to individual consumers.”

It’s no wonder that everyone wants to take advantage of this new tool. This new technology that is giving us the insights that we just didn’t have access to before. The problem with that statement is that big data isn’t new and it isn’t a technology. Big data is large amounts of data, which is simply information. “Big data” has been in our libraries for hundreds of years, but when we reached the point where we started storing information electronically, that data became more easily available – but still impossible to digest without innovative technologies to do it for us.

Now readily available to help us take advantage of big data are technologies like Apache Hadoop, Amazon Web Services, Apache Spark and the abundance of NoSQL data stores. Businesses are seeing the value. Giving businesses options like the ability to foresee what their customers might want to buy in addition to what they’ve already bought, or to know where to put their business based on human traffic – why wouldn’t they want to take advantage? However, much like all tools and technologies, use them incorrectly and you’ll get the incorrect results.

Does that mean that companies are merely implementing the above-mentioned technologies incorrectly? Yes and no. The technology stack to use big data is a multi-layered architecture of complexity. 

Sameet Agarwal, who previously led the data infrastructure team at Facebook said: “big data has become too attached to the technology[…] many big data projects fail because they outlay massive, upfront resources and deploy rigid architectures that don’t promote flexibility once a project is in flight.” Having a shortage of skills and reliance on the technology to access big data is proving difficult for companies to take advantage of its capabilities.

The other complexity isn’t only around technology, though. If you don’t know how to analyse your data going in and you don’t know how to treat the data coming out (your insights), then storage and technology are just another one of the concerns. For example, a store certainly wouldn’t recommend a low-fat diet food alternative to a person who has an eating disorder. They wouldn’t offer a pack of cigarettes to an ex-smoker. And yet, there are many examples of companies where predictive analytics have gone horribly wrong and done just that.

Gartner predicted that, through 2017, 60 percent of big data projects will fail to go beyond piloting and experimentation, and will be abandoned. An incompetent workman blames his tools, but if you have no work to do, then why are you searching for new tools? Big data is here as a means to show insights that simply haven’t been available to us in the past. It isn’t a trend. It isn’t going away. But if we don’t implement its use correctly, the trend of failed use cases will merely increase.

Choose the right team of IT professionals to analyse your business problem, that have the right technical data skills, and understand data science analytics, and you’ll be on the right path to a successful implementation.