Agile Is the Watchword

Everyone, myself included, was so excited to be done with 2020. The Covid-19 pandemic made it a challenging year, and the arrival of vaccines gave us all hope that things would be going back to normal.

A few months in, however, we realize that it will take some time for life to return to normal. Italy has just gone back into lockdown again, and vaccine rollouts continue to lag. Things are likely to be unpredictable for some time.

Even then, how will we define normal? People remain unpredictable in the best of times. Trends come and go at an alarming rate. Even in the most reliable industries, reality often fails to fit historical patterns dependably.

For data scientists? All that unpredictability = More confusion for our models. Accuracy is an ephemeral goal, one that often leads to disappointment.

The solution? Solve for impact, not accuracy.

Essential to letting machines and data find the answers? A rapid-response approach that ensures errors self-correct to give optimal results.

Curating a more responsive approach

An effective and responsive approach requires building a prescriptive system: a system run on feedback, rather than premised on accuracy, from the start. Essentially, a prescriptive AI is geared towards continuous learning and discovery. This learning model attempts to efficiently achieve a particular objective, not accurately measure and fulfil demand.

What’s the difference? On the surface, they both lead to better results. The critical distinction lies in impact. A system that targets accuracy is just looking to closely match the predicted and observed outcome, while a prescriptive system can take any path to get the desired impact.

In effect: prescriptive models can adapt agilely to the inherent unpredictability in our world.

They are less constrained to historical data in a cause-effect relationship, which means they can respond to emerging patterns more quickly. This agility leads to more optimal outcomes. Even when dealing with high data volatility, the model can adjust through rapid iteration.

But agility has to be baked into your entire approach to data science for it to deliver optimal results. If agile is the watchword, there are three areas where agility matters most.

1. Agile models

Foremost in achieving a more responsive system? Agile models.

The more responsive your model, the better it can adapt in the face of changing trends. As agility increases, so do your returns. Since the world is full of uncertainty, that requires a new approach.

Often, data scientists are taught that effective iteration = more accuracy. That’s a problem. Unless our models have been designed to target the exact area of improvement that will have the most significant impact, you fail to solve the core problem. In fact, you fail to solve it increasingly more efficiently!

That’s the difference between predictive and prescriptive analytics. Predictive analytics tells you what is likely to happen if things continue much as they are; prescriptive analytics shows you how to achieve the desired result more efficiently. Prescriptive models are more agile to changing realities — and they, therefore, tend to have a greater impact. Experts in the field are calling it the future of data science.

In the face of data volatility, prescriptive analytics ensures that you reach your objectives more effectively. No matter how accurate your forecast, you can still maximize your impact when your model is agile.

2. Agile development and project management

No model is ever simply “done”, of course. Iteration is, to some degree, inherent in every data science project. The more the system learns, the more effective it becomes. It develops on its own over time.

As data scientists, we are actively involved in this process along the way, developing new functionalities, honing our models, and increasing its efficacy. To ensure that we are providing the most responsive models, this process itself must be agile. You need to embrace agile development and project management.

The Agile Manifesto summarizes the values of this approach best:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

At its core, Agile development is itself a prescriptive approach. We work to achieve optimal impact over perfect software, iterating to improve over time. With this framework, you can deliver results more quickly and improve rapidly in response to new needs and obstacles.

Exactly how you choose to implement agile development is up to you. There are many excellent frameworks out there. Some of the most common include Scrum and Kanban, and almost all organizations adapt these to their needs. Evo has its own approach to Agile development, which you can learn more about for free at Evo University.

The critical aspect of Agile development and project management: feedback. Like agile models, agile development is impossible without effectively integrating feedback into the system. Feedback, both positive and negative, is what moves you closer to your goal and ensures the best possible outcomes.

3. Agile mindset

Neither of those is possible, however, without one final area of agility: your mindset.

An agile mindset is a pre-requisite to both agile models and agile development processes. If you don’t consider continuous learning and adaptability core values, you’ll never implement these principles effectively in practice.

A person with an agile mindset has a growth mindset; they constantly look for ways to improve. You have to learn to regard feedback as an inherent good, something that allows you to iterate more efficiently. That means you have to be open to new ideas and prepared to make changes.

As Dr. Carole S. Dweck describes people with a growth mindset, they:

• Embrace challenges
• Persist despite setbacks
• See effort as a path to mastery
• Learn from criticism
• Find lessons and inspiration in the successes of others

The same is true of people with an agile mindset.

Both are always ready to improve.

What sets an agile mindset apart is speed. People with an agile mindset don’t wait to make their move. Yes, they think through problems logically, but they act. They are willing to take risks in order to learn rapidly. They adapt to changing conditions without feeling like they are personally attacked by a change in course. Only this mindset delivers the kind of agile models that make a more significant impact.

The faster the response, the greater the returns

The faster we can respond to changes in the data, the greater the impact. That seems evident on the face. Yet, in practice, people are often in disbelief of the results. In supply chain, prescriptive systems routinely outperform traditional predictive systems by 20–30% greater inventory efficiency for more sales with less inventory.

Such a difference can cause cognitive dissonance: How is such a dramatic difference possible?

It comes down to agility. A rapid-response approach is not just about getting answers faster. It’s about facing unpredictability with an approach that accounts for the inherent volatility of our data.

We can’t ever achieve a perfect forecast. No matter how hard we try, we’ll fall short. Our attempts simply magnify inefficiencies. Instead, data scientists have to solve for impact. A prescriptive model — an agile model — makes all the difference. We can’t eliminate data volatility, but we can minimize its reach with the right approach.