Opinions and Musings: The Future of Big Data

While people are moving to cities today more than ever before, those moves are not always accompanied by the appropriate infrastructure to accommodate the population rise. While new roads and public transport systems can be built in time, they usually follow within the next decade after the influx of people. 

The same is true today for how we collect data. Tech companies have developed the capability to collect consumer information faster than they’ve been able to build the infrastructure to manage that data and act on it. 

So, while over 90% of the world’s data has been collected in the last two years, about that much is also sitting in data storage facilities dormant. 

In an interview with Forbes, CMO of AdTech firm BazaarVoice Sara Spivey reflects on what that means further. She explains that while it would be much more efficient for marketers and developers to identify specific objectives and then get the matching data, most often the reverse process happens, leaving companies with mounds of data that’s nearly impossible to mine. It takes hours of analysis and algorithm building to be able to identify a tenable, “aha” insight, she says. 

What’s more, CEO of Barrett Digital Jeff Barrett explains that there are tremendous costs associated with dormant data. Investing resources to build collection methods then not using the data effectively is like buying a new home exercise machine that no one ever uses, explains Barrett. There’s also the problem of lost revenue opportunities because too much data is so difficult to mine through, teams often don’t even fully understand what they have. That also can lead to lower efficiency and overall productivity, and quality issues. 

There’s no solution yet other than relying on the exponential advancement of artificial intelligence, which can hopefully sort through some of it and identify meaningful, actionable patterns faster than teams can.  (Side note: Check out our tutorial on using Keen with Google’s Natural Language Processing API to see how you can start making strides easily today. )

However, AI is only as good as the human-led efforts to set it up correctly. In his TedTalk, founder of the Institute and leading researcher at MIT Max Tegmark outlines two key ways he believes we can ensure a better future with this tech, and what to avoid. He explains that researchers should very carefully and consciously work every day to ensure that an AI-powered device’s goals are aligned with ours. He uses the example of when humans led the black rhinoceros to extinction, it wasn’t because they hated rhinos for the sake of it, but because their goals were misaligned.

Using predictive data can also be helpful, but isn’t always a perfect representation of the future. In our interview with Silvio DaSilva, he mentions, “Innovation is always going to be coming and will continue to become cheaper and easier. But, it’s tricky. You can predict ideas but not always know how it’s going to be.”

Other than AI, where else should we expect advancements in data interpretation? DaSilva also expects great innovations in voice and audio formats of data viz. Voice “audiolization” is on the horizon. News organizations are starting to explore alternatives because traditional data visualizations are time-consuming, difficult, and expensive to create. They can also put undue pressure on the audience to interpret the data correctly. But if you create a story for that audience, you can go a lot further with the data.

Ready to start putting your data to work today? Sign up for a demo with Keen to learn how you can get up and running fast.