Sign up

Data 101 – part 2

February 23, 2017

Big data won’t work without humans

In the first part of “Data 101” we shared our perspective on the interplay between exponential data growth and the equally exponential fall in the cost of storing and processing data. In this second piece we look at misconceptions about disruption in analytical methodology and question whether Stephen Hawking is right about AI.

Electric sheep and rogue computers

We begin this chapter of our story in 1968.

While in the real world, computers had only just progressed to basic pattern recognition, science fiction writers were imagining a singularity event where Artificial Intelligence was becoming a threat to humankind.

In “Do Androids Dream of Electric Sheep?” Philip K. Dick pictured a dystopian world with androids aspiring to be human, while Arthur C. Clarke’s “2001: A Space Odyssey” introduced us to HAL, the computer prepared to do anything to survive.

And, just half a century later, Stephen Hawking told the world, “The development of full artificial intelligence could spell the end of the human race.”

Artificial intelligence and Moore’s Law

So, in just 5 decades, have computers become geniuses that are going to trigger a Terminator-style Armageddon?


At the moment, you could program computers to take over the world but they wouldn’t make a conscious decision to do this of their own accord.

While there is a big buzz about ‘smart’ homes, ‘smart’ devices and world-beating chess computers, machines are actually pretty dumb when it comes to the things that define human intelligence like creative decision making.

One of the culprits for people’s misconceptions about the march towards Artificial Intelligence is Moore’s Law. Before we explain further, I would like to point out that Mr Moore himself is innocent of all charges – a scientist can’t be held responsible for people mangling a perfectly fine theory. What Gordon Moore observed in 1965 was that the number of transistors in a dense integrated circuit would double every year during the following decade. Over time Moore refined this to doubling every two years, but the base concept of rapid exponential growth was sound.

The problem came when people assumed that greater processing power would quickly lead to machines with human-like intelligence. This didn’t happen for one simple reason – the algorithms are not becoming smarter, it’s just that they have more data, and therefore the applications are becoming wider. However, the algorithms are really just as smart as they were 50 years ago.

Stuck in the 60s?

With the decreasing costs of big data storage and computing leading to an explosion in new applications in the data science field, there is one key component that hasn’t changed since the 1960s.

The underlying methods used to compute data haven’t changed in the last 20 to 30 years. While new data science applications are rapidly being created, the scientific theories used to create the models haven’t been updated or changed in any significant way for decades.

The Forbes article “A Short History of Machine Learning” essentially depicts how the first data processing algorithm that allowed computers to use very basic pattern recognition was written in 1967. “This could be used to map a route for traveling salesmen, starting at a random city but ensuring they visit all cities during a short tour.”

Since that time, the basic method for machine learning (artificial intelligence) hasn’t changed. What has changed is our development of applications to use these methods.

For example, the discovery of statistical regression used to predict the future and determine the factors that cause an outcome to happen in the early 1800s. And it’s now a central piece of modern statistics and data science.

Priceonomics state in their article “The Discovery of Statistical Regression” that:

“Even with the development of increasingly sophisticated algorithms for prediction and inference, good old least squares regression is still perhaps the crown jewel of statistical analysis.” — Dan Kopf, 11/6/15

And, if you would like to spark a war among data scientists, you could even argue we haven’t progressed much in over 2,000 years as the ancient Greeks developed an analogue computer called the Antikythera mechanism more than a hundred years before the birth of Jesus.

For clustering, the vast majority of the theory was developed in the 1960s-1970s, and for neural networks, the discovery period happened in the 1970s. Stanford made the last major improvement to neural network theory in 1986.

“Smarter” but not intelligent

These scientific theories developed long ago and were not fundamentally updated since the early 1980s. Today, they are enjoying a new lease of life thanks to the economic considerations we discussed in the first article.

Essentially, as more data is being stored and processed, today’s algorithms are becoming “smarter”, and as such, reach their own latent technical potential. This is made possible by the vast amount of data they’re able to mine to find new patterns.

Today’s applications wouldn’t have been profitable if applied in the 1980s or even early 2000s, because of the sheer cost of storing and processing the vast amounts of data required back then. They can be profitable today as the costs to store and process data go down – because of the basic economic force of supply and demand, more applications are becoming economically viable.

To recap, applications grow exponentially in volume thanks to an underlying growth in data, in turn made possible by the exponential decrease in data-related costs. But this growth is NOT due to substantial developments in the underlying theory.

As we exploit the current theories to their fullest potential, we may be able to extract the very last bit of information from our data, even the really ‘difficult’ information like voice, picture, or language recognition. But that would be the furthest limit our current theories would allow.

“Hasta la vista, Singularity!”

The real difference is that exploiting our current data models will require huge amounts of data training and increased effort to get a “probabilistic answer” that you could teach a 3-year-old child to do in a matter of minutes with a zero-error rate.

The current theories are far from being able to develop complex human characteristics like intuition, faith, innovation, and much more. This is why Evo Pricing asserts that big data won’t work without humans.

While “singularity” sells, and humans are justifiably worried, when you consider the current state of data science, we are actually living through a period of “more of the same”.

It’s just that this is exponential: we have a lot more data and applications, but none lead to true innovation that is disruptive in the sense implied by the concept of singularity.

So, with the current rate of change in theory, for the time being we can safely say “Hasta la vista, baby” to Artificial Intelligence.

About the author

Martin Luxton is a writer and content strategist who specializes in explaining how technology affects business and everyday life.

Big Data and Predictive Analytics are here to stay and we have only just begun tapping into their enormous potential.

Hey! Was this page helpful?

We’re always looking to make our docs better, please let us know if you have any
suggestions or advice about what’s working and what’s not!

Send Feedback