KPI effectiveness in the pandemic era

How can you fairly measure different markdown strategies in times of market disruption?

When you test a new business strategy, a critical part of the process is establishing measures to gauge its success. You need to know what is working and what is not — and you need to agree with all key stakeholders how you will objectively assess the outcomes.

As a consultant who works with companies to help them transform supply chain and pricing processes, I can tell you first-hand that this process is always fraught. How do you ensure the chosen measures fairly reflect the real outcome of the test?

The answer: choosing the right KPIs.

Data collected in the test can be used to measure the objective difference between the new strategy and the old one. A well-chosen KPI will inform you of your progress: whether or not the new approach is delivering results.

The unique challenge of discount strategy KPIs

Usually the ideal KPIs seem fairly straightforward: a strategy to increase revenues should be measured by the change in revenues. Even when looking at more complex measures of success, usually there are some clear candidates to measure success. A better replenishment strategy, for example, should generally reduce both unsold inventory and stockouts while maximizing sales. While the exact KPIs always depend on the unique business objectives and circumstances, the general shape of success is usually agreed upon by everyone.

Discounts are different.

Retailers use discounts, markdowns and promotions to improve any number of their business KPIs, such as revenue, profit, margin and sell-through during a given time period. These KPIs are often either/ or decisions. Increasing sell-through often means sacrificing margin. More revenues from discounted products yields lower profits than theoretically possible from full-price sales.

Even worse, poorly designed discounts may lift specific KPIs while disguising real damage. While lowering prices can attract more customers, it will not necessarily improve a retailer’s business; on the contrary, poorly run discount campaigns can reduce revenues, destroy margins and even damage your brand and reputation.

Therefore, it is important for retailers to set up a strategy for discounting that starts from the goals that are desired (e.g., get new customers, increase revenues, maximise margins, clear out old inventory, etc.). This is not a simple decision. Retailers must take into account all the internal and external factors and constraints affecting the outcome and include business rules to govern the overall process.

The right goals — and the right KPIs to measure progress towards those goals — differ greatly depending on the type of discount. For example, markdowns are a particular kind of discount typically targeted to low-demand, end-of-lifecycle products with the aim of clearing excess inventory at the end of a selling season.

Another type of discount promotion is mid-season sales. They emerge as an opportunity for retailers to get rid of products that will soon expire and from which they probably can get improved benefits now, rather than wait until the end of season. Yet another, black days, are autumn mid-season sales extension that happens over the week of Black Friday (last week of November) that is often motivated by a fear of missing out on traditionally high-traffic shopping days.

It would be practically impossible to list all the types of promotions and the business goals that trigger them. There are several other critical promotion initiatives that can be added to the ones mentioned above, like loyalty and referral programs, bundled discounts and many others.

Complicating an already challenging measure: how to assess discount strategy amidst market disruption

No matter the type of promotion, measuring its effectiveness is at least as important as its planning and execution for retailers. In fact, measurement is even essential during execution, as prices are reviewed at least 2 or 3 times during longer promotions like markdowns, which last several weeks.

But how can you measure your discount strategies while a pandemic is still ongoing?

We ran into this problem while helping a client transitioning from a manual markdown approach to an AI-automated one. The transition itself started in the early months of the pandemic, when many retailers were slashing prices without a clear strategy in an attempt to move inventory in a stalled economy. This environment made the usually difficult decision as to objectives even more fraught — and the KPIs to measure them even more muddled. The solution to be chosen was far from a trivial decision.

One massive advantage of prescriptive AI technology: the ability to simulate multiple outcomes before choosing a solution. Another: the ability to follow multiple objectives at once, weighted in accordance with their importance. After a few simulations, we settled on a ranked strategy that considers revenues, margin, and sell-through each as critical objectives in turns. The usual KPIs were originally considered as candidates to measure the effectiveness of markdown recommendations: cumulative pieces, margin, revenues and units sold.

Measuring markdown strategies in times of market disruption: a case study

The issue with these traditional measures of success became evident while examining the outcome of an A/B test run for two of this client’s brands during the 2021 autumn mid-season sales. Stores in the test group applied the prices recommended by the Evo Markdown tool, while stores in the control group prices were formulated manually by the client’s analysts as usual.

The resulting outcomes of the two A/B tests were profoundly different. So much so that the differences cannot be explained only by the abilities of the different brand analysts or the tool itself. This happened despite the accuracy taken in choosing test and control groups that were as possibly similar in all the relevant KPIs in the previous few weeks preceding the testing period.

Why would such similar brands have such different reactions to the new recommendations? Perhaps these brands were not as similar as the chosen KPIs suggested. To dig deeper, we needed to expand our selection of KPIs.

The search for new KPIs

In order to investigate the issue, we analyzed other weekly trends like:

· Walk-ins: number of customers entering the shops (with or without purchasing)

· Conversion: Percent of walk-ins that purchase something;

· Tickets: Number of receipts issued;

· Units per tickets: Average number of items acquired per ticket.

The outcome, still measured as a % of test vs control groups for the brands, showed a different story.

We noticed that the results could be partially explained by the fact that the test group benefitted from an increased number of walk-ins during the testing period for Brand A, yet the test group was hit by a decreased number of walk-ins for Brand B. The number of tickets and units per ticket showed a similar pattern.

So clearly these additional factors should somehow be taken into account to fairly plan and assess the new strategy. But why? That is perhaps best explained by the nature of the pandemic disruptions.

Pandemics and markdowns

Before proposing a better methodology, let’s review from a broader perspective how 2021 compared with 2019 (the immediate pre-Covid period) looking at these factors the % difference between the Units, Tickets, Walk-ins, Conversion rate and Items per Tickets in 2021 compared with 2019.

The first half of 2021 was clearly impacted by the lockdowns with a huge reduction in walk-ins, tickets and units sold during March and April. During this period, conversion rate and items per tickets were at their highest, demonstrating that the few people who visited the stores had a strong motivation to buy, at least stronger than average.

Mid-Season Sales were, in turn, relatively more successful in 2021, perhaps reflecting a sort of rebound effect of the bad period immediately before. Interesting and a bit more challenging to understand were the less successful 2021 Summer promotions compared to 2019. Perhaps there were fewer people on vacation explaining poor performance in key shops close to holiday resorts.

Overall, in 2021 conversion rate and items per ticket outperformed the 2019 values. Consumers’ that make it outside to shop in person seem more motivated to buy during the pandemic. In other words, once they cross the first barrier and enter the shop, consumers are more willing to buy and in larger amounts.

These trends do not negate the importance of measuring more traditional measures like margin or overall revenues, but they do contextualize the results: more factors are at play when the market is not behaving “normally”.

A new measure of success

So coming back to the original question, how can we interpret the success of a new strategy in a fair and balanced way during times of high market disruption?

It’s all about expanding KPIs.

If markdowns are reaching fewer people, for example, maybe the context needed for margin and revenues matter more at a smaller level, i.e. per ticket and per walk-in. When we measured how the customers entering the shop behaved responded to the strategy, we can better understand the impact of the new strategy when contextualized by the lower traffic, especially in the more mass-market context of Brand B (compared to Brand A which has a more luxury niche).

Overall, the new KPIs confirmed a difference in the markdown strategy performance between Brand A and Brand B.

Nonetheless, the gap in performance is less. We can see more clearly, in fact, that Week 3 had much lower success than other for both brands, allowing us to dive deeper into the data and outside circumstances that week to learn what may have contributed to the lag in performance. Additionally, Brand B did show some significant improvement over the control when contextualized this way, providing important data for the next iteration.

In other words, we combine KPIs to provide more information NOT to ignore or minimize negative results. When the AI can take this data and drill deeper into the factors that define success, more patterns are recognized — and the next markdowns will be better calibrated for the intricacies of each Brand.

What this means for your KPIs

So what does this mean beyond this particular case? Revenues, margins and units alone are not sufficient KPIs to measure discount strategy results. It’s clear that a traditional approach overlooks important information.

While here walk-ins could be added to the equation to improve your understanding of what happened, many other factors could influence results. Perhaps particular shop-related issues that could be human (e.g. a change in management) or location related may be throwing your results and considering what should be outliers as trends. Perhaps another factor entirely is muddling the overall trend. The key is to consider all the factors that could be affecting the outcome of your test and then narrow them to the most important in your analysis.

When it comes to KPIs, it is far too easy to get lost in the analysis until the data loses all meaning; that is certainly not what I am advocating for here. Instead, I recommend you be careful in your conclusions and open to alternative explanations.

When you remain open to new possibilities from your data, traditional limitations in measuring your success disappear. This makes you more agile in difficult markets. Let’s be honest: market disruption is becoming a feature rather than an exception in these post Covid times. A re-assessment and expansion of KPIs is how you can adapt to — and thrive in — any market challenge you face moving forward.

A big thank you to Benedetto Cavicchi for assisting in the data analysis and chart creation that made this article possible.

About the author