The paradox of mathematical optimization should not scare you away from its benefits
Optimization is a primary selling point of AI to businesses nowadays. Everyone wants to improve their efficiency, revenue and other critical KPIs. These are reasonable goals — and achievable ones. We can use optimization principles to improve performance significantly.
The impossibility of perfection
No matter how much you attempt to achieve optimal performance, however, full optimization is impossible. No model can account for all possibilities or achieve a perfect forecast. AI technology simply hasn’t advanced enough to make that possible for the foreseeable future.
Moreover, defining what is “optimal” provides its own challenge. Every business has unique needs and, as such, optimizes for diverse goals. A strategy that is optimal for one company is unlikely to perform as well in another situation. Perfection is a moving target, not a simple one-size-fits-all answer.
That’s why optimization is a paradox. Its very purpose renders it impossible. The more you attempt to define it, the more elusive it becomes.
Why we love a paradox
It’s not like facing down a paradox is anything new. We face down contradictions every day, and for mathematicians like me, it’s usually a lot of fun. Just consider the paradoxical problem that’s been making its way around the internet again:
If you choose an answer to this question at random, what is the probability that you will be correct?
At first glance, it seems easy enough to answer. But the more you think about it, the more possibilities seem like right answers. These possibilities then make those answers incorrect. Eventually, you realize that the problem has no answer. You would have to break mathematical principles and logic. This kind of question can do your head in, but trying to reason it through is an excellent way to sharpen critical thinking skills.
The paradox of optimization can be particularly annoying, though. While effective optimization will provide a lift, it will never achieve what it sets out to do. You will never give that optimal outcome.
I argue, however, that is what makes optimization just as fun as any other thought exercise. You get a chance to challenge assumptions about the best ways to do business. Optimization opens new possibilities, and unlike solving the problem above, you get the satisfaction of a tangible outcome. For example, Boggi Milano may not have fully perfected their supply chain through optimization, but even a few iterations of an optimization algorithm increased inventory efficiency by 18%. Rewards for working through the paradox are much greater in optimization — and you can always inch closer to perfection.
The mathematics behind optimization
Still, even without the paradox of attempting the impossibility of optimization, optimization is a challenge. The math and algorithmic design of optimization are incredibly complex. Take a look at just a tiny piece of only one section of one of the optimization formulas I use:
Many people find optimization intimidating for just this reason. If all your complicated programming and difficult calculations won’t even give you the ultimate answer, why bother?
Optimization is like onions
If that isn’t complicated enough, no optimization algorithm will involve a single optimization calculation. Like ogres and onions (and parfaits!), optimization has layers. You’ll seldom have a working model that does not contain at least a few layers of calculations that depend on each other to give a result.
Take Evo’s Replenish tool, for example. Replenish can be used to optimize the supply chain for systemwide store-level replenishment. As you can imagine, optimizing this requires optimizing a large number of individual factors. The Replenish algorithm contains 3 distinct optimization loops:
- One to determine the best assortment of products for each store based on demand
- One to fit this within the overall systemwide demand, balancing what’s on offer with what’s needed elsewhere
- One to move goods between stores and from warehouses in the most cost-efficient manner
Multiple calculations must be done within each of these optimizations, optimizing factors used within these larger optimization models. Layer upon layer of optimization is necessary for a single model. Peel them back, and you realize that optimization is even more complicated than you ever imagined. This leaves plenty of room to refine and improve results over time, a process both frustrating and exciting for data scientists like me.
There is always a margin of error, yet there are always ways to improve my results: the paradox of optimization in a nutshell.
If optimal results are impossible, is optimization worth it?
So if you can never achieve perfection, is optimization worth it? Any executive will obviously say yes. Even a marginal improvement can have a significant impact on the bottom line, something benefitting any business.
But that doesn’t make optimization any less complex. Perfection is impossible, yet we must continuously work towards achieving it. We have to embrace the paradox. In attempting the impossible, we get a little closer to the ideal every day.
If you are interested in delving deeper into the calculations or programming necessary for optimization, there’s an optimization course available at Evo University. I encourage you to check it out for a deeper dive into the topic.
Not enrolled yet? Now’s the time! Head to https://evo.ltd/join for free access.
Conclusion: worth it
The many functional and intrinsic challenges of mathematical optimization should not scare you away from capturing its benefits.
After all, it is not possible to do better than optimal! You cannot beat a well designed optimizer fed with good quality data.
As I always told my dad, who in spite of working all his life at IBM has a certain Italian pride when behind the steering wheel: Google Map navigation will beat your driving knowledge, unless you know something it doesn’t. He still doesn’t trust me today, maybe one day?
About the author
Fabrizio Fantini is the brain behind Evo. His 2009 PhD in Applied Mathematics, proving how simple algorithms can outperform even the most expensive commercial airline pricing software, is the basis for the core scientific research behind our solutions. He holds an MBA from Harvard Business School and has previously worked for 10 years at McKinsey & Company.
He is thrilled to help clients create value and loves creating powerful but simple to use solutions. His ideal software has no user manual but enables users to stand on the shoulders of giants.