The ever increasing availability of big data to organisations is producing endless sources of insight and possibility, which promise greater business impact and industry disruption. As big data initiatives mature, organisations are combining the power of big data processes with the smarts of artificial intelligence (AI) to speed up the delivery of business value.
Computers have developed to a point where it’s fair to say they can now see, hear, speak, read, write and remember. But traditionally what they haven’t done very well is think.
Recent developments in the sophistication of AI has created the opportunity to replicate human thinking skills with machines that follow rules. The rules that implement instructions are known as algorithms. Algorithms have been engineered to a point which now allows computers to learn, so it is possible for the “thinking skills” of modern systems to depend on data, not humans but for this computer “thinking” to be accurate and produce insights of value big data is critical.
The author of “Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are” Seth Stephens-Davidowitz writes that big data offers four superpowers; see, new, things and act.
When given a limited amount of data, an analyst can select and aggregate to create insights but when provided with enough data, the possibility is created to zoom in on unique individuals or special groups which can yield valid patterns. Such patterns would be unlikely to show up with the “small” data, either because the unique individuals would not have been observed at all or not observed enough to stand out in context of the whole population. But when provided big data the population of individuals can be split into groups along more dimensions such as sex, age, and income making it possible to identify clusters within a population with distinct characteristics.
Thus, big data’s super power number one: it allows us to see both the long tail and the big picture.
Big data allows an analyst to create insights that were previously impossible.
Google Maps allows you to see where you are and how to get to your destination. What Google also does is keep track of your location and allow you to see where you went on your Google Maps Timeline. By aggregating this data and counting up the visits over time to each restaurant, Google can offer new insights such as the times when the restaurant is busy and how long people typically stay.
So, big data’s super power number two: it yields valuable insight which can help inform strategic business decisions.
Theorists believe Google searches tell us more about human interests than any traditional survey method. Why? Because people are often dishonest or distort the truth.
Big data measurements, such as the count of visitors to a restaurant at different times of day, allow analysts to derive insights based on the actual behaviour of people, not merely the what they say they do.
Hence, big data’s super power number three: the insights can inform what humans actually do, rather than what they say they do.
With enough people interacting with the internet and with the internet overlaid with a measurement system, analysts can conduct experiments to determine how people use the internet. These are known as A/B tests. One group gets treatment A. Another group gets treatment B. The behaviour of the two groups is compared. Based on the comparison, the process is optimised.
As more and more human activity is mediated via internet interactions, this optimization process expands into more domains.
Ultimately, big data offers super power number four: insights that provide an accurate representation of peoples’ preferences and what drives them to make decisions.
Big data’s superpowers alone previously have not been compelling enough for many businesses to restructure around optimised data-driven processes. Traditional information systems have worked well enough to remain competitive with other businesses. Critically, investments in data processing continue to be made from an accounting or tangible assets perspective.
Meanwhile, businesses such as Amazon or Uber have come in and used big data to disrupt their respective industries. The intangible algorithms these firms use for processing data keep getting better.
In a new book, “Capitalism Without Capital: The Rise of the Intangible Economy”, the authors explain how investments sunk into intangible assets can often be scalable and produce synergies out of proportion to physical assets. The futurist Ray Kurzweil calls this phenomenon the “Law of Accelerating Returns” because intangibles can have an exponential impact, multiplying one another, unlike the traditional diminishing marginal returns of physical labour and assets.
Big data and the current generation of AI software work best together, synergistically. Businesses can leverage this synergy via the insights and efficiencies generated by applying AI to big data. In particular, algorithms which learn from data have the power to help a business scale with intangible knowledge and skills which make it possible to outperform less knowledgeable and less skilful competitors.