Don’t Let Metrics Critics Undermine Your Business

The irresistible rise of data-driven, analytics-enriched digital instrumentation and decisions has provoked an intellectual backlash. Call it a counterrevolution. Those lucky employees who haven’t been automated into professional obsolescence instead find themselves enduring what economic historian Jerry Z. Muller calls the “tyranny of metrics.” Numbers rule their workplace lives, and there’s no escape.

“The problem is not measurement,” Muller declares, “but excessive measurement and inappropriate measurement — not metrics, but metric fixation.”

“Don’t Let Metrics Undermine Your Business,” warns Harvard Business Review’s September-October 2019 cover story: “Strategy is abstract by definition, but metrics give strategy form, allowing our minds to grasp it more readily. ... The mental tendency to replace strategy with metrics can destroy company value.”

Anti-metrics admonitions even come from innovators who enjoyed success in no small part due to thoughtful key performance indicators. “If we live for KPIs, and do something just for the sake of KPIs, then Alibaba is finished,” declared CEO Daniel Zhang, who took over from now-retired company founder Jack Ma.

No doubt these fears and concerns are sincerely felt. However, they are wildly and measurably misplaced. Metric fixation is neither a root cause nor a reason for unhappy outcomes or defective cultures. The opposite is true: Organizations hurt themselves and their customers precisely because they don’t value metrics enough. Instead of treating measurement seriously or strategically, management defers or defaults to KPIs that inspire neither insight nor foresight. Leadership fails to understand the power and potential of metrics as an asset.

To be sure, not everything that counts can be counted and not everything that can be counted counts. But that cliché becomes less and less true as KPIs learn how to learn from greater volumes and varieties of data. Effective digital transformation invariably means measurement gets smarter, faster, and more adaptive. This challenges leaders to become both more strategic and sophisticated in their use of metrics. The evidence suggests few legacy leaderships have the cultural or quantitative strength to overcome these challenges. They lack the self-discipline and self-knowledge essential to making data-driven metrics work. Leadership, not measurement, is the problem.

Serious leaders from the most effective digital disruptors — Google, Amazon, Netflix, Tencent, and, yes, Alibaba — relentlessly fixate and focus on defining metrics and KPIs that inspire strategic success. Indeed, their KPIs are their strategy. They know it and their people know it. They use metrics to lead the enterprise, not just manage it. This distinction is not subtle; our research on KPIs shows it’s the secret sauce for positive outcomes.

When Alibaba launched Taobao – the C2C online market that would become the world’s largest — the management team’s main KPI for the platform was new job creation. Ma told the startup leadership they needed to create 1 million jobs that year. “If you bring in revenues past a certain goal,” Ma warned, “I will penalize you. Because it means you are trying to make too much money from the site at this point.”

That sounds like intentional, inspirational, and strategic leadership: The KPIs — and their trade-offs — are mindful and meaningful. People understand them. Their value — and values — are clear. They work. “At Amazon,” one former executive recalls, “everything that can be measured is. Every piece of data is tested and analyzed — not just web design or product features, but finance, HR, and operations processes. ... More than anyone I’ve ever met, Bezos knew that things don’t improve unless they’re measured.” The company’s founder oversees and enforces what he calls Amazon’s “culture of metrics.”

This post has a dual purpose: first, to clarify why this metrics backlash is both unjustified and unfair, and second, to observe that the future of strategic advantage increasingly belongs to the strategic quantifiers. There truly is strength in numbers. Serious leaders invest in — and build on — that strength. Leaders who don't are unserious, innumerate, or both.

Quantification’s Critics

Quantification’s critics discount the growing wealth of KPI best practices in favor of “metrics behaving badly” narratives. That’s a huge mistake.

For organizations serious about strategic measurement, those narratives are fake news. Anti-metrics rhetoric portrays top management less as miscalculating perps than innocent victims. These claims are akin to blaming speedometers for drunk driving or faulting GPS for running red lights: Don’t let dashboards undermine your driving. Metrics are made culpable for their masters’ sins.

Serious leaders respect Goodhart’s law — best defined by Cambridge anthropologist Marilyn Strathern as “when a measure becomes a target, it ceases to be a good measure” — and take pains to avoid it. Perhaps most important, serious leaders hold themselves accountable for the metrics by which they hold their people and their processes accountable.

In their Harvard Business Review cover article, Michael Harris and Bill Tayler observe that “every day, across almost every organization, strategy is being hijacked by numbers.” Unpack that remarkable assertion: Do “thug metrics” somehow intimidate CEOs, CFOs, CMOs, and their boards? Does top management have so little agency, self-awareness, or self-control that numbers can so predictably subvert their master plans? It’s as if executives have nothing to do with the KPIs they choose to impose. Again, the opposite is true: These leaders have everything to do with them. Why don’t they take responsibility? That's the better question.

The Wells Fargo fiasco is cited by both Muller and Harvard Business Review as a canonical example of despotic metrics. “The real source of Wells Fargo’s problems was measurement,” Harris and Tayler flatly declare. “When the bank decided to actively track daily cross-sales numbers, employees rationally responded by working to maximize them. Throw in financial incentives, a permissive culture, and intense demands for performance, and they might even illegally open some unauthorized accounts, all in the name of advancing the ‘strategy’ of cross-selling.”

Is this not the very definition of Goodhart’s law? The problem is not the measure; it’s the target. More explicitly, it’s leadership’s bitter brew of carrots and sticks that toxically transformed managerial metrics into targets. Remarkable, isn’t it, what Wells Fargo executives — innocently? inadvertently? arrogantly? deliberately? — chose not to quantify in their quest for synergy? What organizations don’t — or won’t — measure is arguably every bit as revealing as what they do. Ironically, the critics minimize this.

Cross-selling can prove a tremendously healthy strategy for vendors and customers alike — just ask Bezos or Ma. Corporate corruption, not cross-selling, is the true dysfunction here. Declaring measurement the real source of Wells Fargo’s malfeasance is profoundly unserious. Scales don’t cause obesity.

Full disclosure: Before doing work on KPIs, I explored writing a book about perverse incentives for MIT Press. In my research, I was struck less by people’s transcendent willingness and ability to game their incentives than by leadership’s pathological unwillingness and inability to think through — let alone test — their incentive systems’ design assumptions. Without exception, the most perverse consequences — some amusing, others tragic — were foreseeable by anyone who took history or human nature seriously. Of course, the chosen metrics mattered; they always do. But achieving perversity requires incentive. Who defines and determines what gets recognized and rewarded in the enterprise? Top management. Leadership. You don’t get what you measure; you get what you pay for. Serious leaders know this.

Mistaking Metrics for Incentives

This is very much the theme and message of Muller’s tyranny. In domain after domain — education, public service, medicine, the military, business — the economic historian immaculately details pay-for-performance perversities. “Managers and employees learn to lie, to massage, embellish, or disguise the numbers that are used to calculate their pay,” he writes. “But since these are the very numbers that executives use to coordinate the actions of the organization and decide on the allocation of future resources, the productivity and efficiency of the organization is damaged as resources are misallocated.”

Indeed. So why conflate metrics with incentives? Muller, who explicitly references Goodhart’s law, shows how organizations of all kinds invariably screw themselves by imposing pay-for-performance schemes that invariably leave everybody worse off. His analysis ultimately betrays his anti-metrics hypothesis: Poorly designed incentives — not mistaken measurement — pervert numbers, behaviors, and outcomes. Leaders and their credentialed technocrats — who he clearly believes should know better — foolishly link performance management and compensation in ways that invite abuse. The tyranny Muller details comes not from metrics fixation but incompetent compensation. Without the consent of the governed, subversion becomes the organizational (ab)norm. Manipulative metrics signal cultural weakness. Cultural weakness denotes leadership weakness. Wells Fargo’s humiliation didn’t begin or end with numbers.

How Much Room for Judgment and Expertise?

Like his Harvard Business Review peers, Muller bemoans the decline of quality leadership as deplorable leaders succumb to pay-for-performance pathologies. “[Tyranny] is not about the evils of measuring,” he writes. “It is about the unintended consequences of trying to substitute standardized measures of performance for personal judgment based on experience.”

This sounds noble and pragmatic. But how, exactly, should the quality of “personal judgment based on experience” be fairly and accurately assessed? Asking top executives to judge their own decision-making prowess is asking for trouble. Do subjective-yet-sophisticated judgments and decisions — gut instincts — truly outperform their more objective counterparts? The evidence says no. More often than not, decisions driven by standardized models and metrics prove superior to human expertise.

How cognitive biases dramatically distort personal and professional judgment has proven a rich Nobel Prize-winning research domain for Daniel Kahneman, Richard Thaler, and other economists. (Duke’s Dan Ariely calls these systemic errors “predictably irrational.”) Experts consistently overestimate their decision-making prowess; how information (such as metrics, KPIs, and medical diagnostics) is framed often leads to significant misinterpretation. The data rudely suggests that deference to “experienced” management and “experts” is often misplaced.

As Philip Tetlock, the pioneering University of Pennsylvania researcher in expert decision-making, bluntly observes, “This is a recurring theme in the psychological literature — the tension between human-based forecasting and machine or algorithm-based forecasting. It goes back to 1954. Paul Meehl wrote on clinical versus actuarial prediction in which clinical psychologists’ and psychiatrists’ predictions were being compared with various algorithms. Over the past 58 years there have been hundreds of studies done comparing human-based prediction with algorithm- or machine-based prediction, and the track record doesn’t look good for people. People just keep getting their butts kicked over and over again.”

Tyranny’s adherents don’t reference this butt-kicking literature legacy. They should. Serious investment in metrics simply isn’t possible without explicitly identifying and confronting cognitive biases. In industries as diverse as professional sports and global investment, dispassionate data-driven analytics have disruptively transformed value creation precisely because leadership has stepped up to eliminate bias. The success of the Houston Astros in baseball or Renaissance Technologies in investment, for example, reflects a data-driven strategic commitment to minimizing cognitive illusion while optimizing metrics innovation.

Hoping that predictable human fallibility makes a better bet than predictive algorithms may be good-hearted, but it surely isn’t rational. Quality leaders have both the courage and humility to defer to quality data. To digitally update Bacon’s famous aphorism about nature, “Data, to be commanded, must be obeyed.”

Metrics Are Much More Than a Proxy for Strategy

Perhaps the weakest and least imaginative of the anti-metrics arguments is the Harris and Tayler surrogation snare: KPIs, they assert, can’t quite grasp the essence of strategic aspiration. “Surrogation is especially harmful when the metric and the strategy are poorly aligned,” the authors observe. “The greater the mismatch, the larger the potential damage.”

No doubt. But surrogation itself begs the larger leadership question: Why such poor alignment and mismatch? Metrics — and their users — deserve better. Precisely because they grasp alignment’s strategic and operational importance, serious executives pursue metrics (and metrics ensembles) that merge and marry KPI with desired strategic outcome.

In this respect, KPIs might better be described as KSIs — key strategic indicators. Eliminating misalignment is exactly what leaderships need to do. The surrogation argument blames metrics instead of mismanagement for mismatch. That’s a cop-out. Crafting KPIs/KSIs for, say, cultivating influencers or customer success or supplier innovation — or even ethical cross-selling — may be difficult, but that does not excuse leadership’s fiduciary obligation to do it.

This so-called surrogation snare really represents a surrogation surrender: leadership’s failure to commit to transformative value-added KPIs. Given ongoing revolutions in digital instrumentation, machine learning, and big data, this epistemological capitulation is unwarranted and unwise. If there was ever an opportune time for organizations to revisit their fundamentals of metrics, measures, and monitoring in the service of strategy, that time is now. The quantitative opportunities for disruptive innovation and insight have never been faster, better, or cheaper. The better Harvard Business Review cover line would be “Don’t Let Weak Metrics Undermine Your Business.”

Indeed, Harris and Tayler rightly recommend that organizations diversify their reliance on key metrics; they further encourage leaders to collaborate with employees to design better KPIs. While that is exactly the right direction to go, it affirms my essential assertion: The true underperformers in this digital disruption era are not the measures but their managers. Quantitative leadership is fast becoming indistinguishable from strategic success.

But what of Zhang’s KPI warning to Alibaba’s employees? Like Amazon, Facebook, Netflix, and Google, they’ve grown up in a successful KPI leadership culture of accountability and growth. Why would Ma’s successor bad-mouth the very metrics that made his company world-class? Based on my research, answers from middle managers and top executives who’ve worked with — and for — Ma and Zhang — are eye-openingly obvious. No one really believes Zhang will allow Alibaba’s metrics to undermine its business.

While Alibaba is demonstrably brilliant at defining and designing strategic metrics, leadership expects more from its people than KPI compliance. Zhang is correct: Cultural acquiescence and obedience from his Chinese workforce is not enough; he wants metrics innovation and challenge. Alibaba, Zhang has told these executives, must become more creative about how it measures its customers’ success and its own.

The current generation of key performance indicators and quantification, he says, is no longer good enough for the next. For how many other organizations is that now true? That’s a leadership insight and exhortation that defies any tyranny of metrics.

The author would like to thank David Kiron for his contributions to this article.

Don’t Let Metrics Critics Undermine Your Business