The Perils of Overconfidence and Benefits of Rational Thinking

Humans, as a rule, are bad at estimating and worse at predicting. This is why films such as The Big Short, which depict real life predictions that come true, are notable as exceptions that prove the rule — we are bad at estimating. Yet for all this, there is one question posed to every manager after submitting a project brief, change request, or budget impact statement: "what's your level of confidence?"

There are good reasons for the emphasis on confidence levels. The first is that in a world increasingly enthralled to 'scientism', confidence levels are sought by anyone who has done statistics 101. The second reason is that, particularly in management circles, any person who doesn't know, looks simultaneously incompetent and unhelpful. Traits that are, except for criminality, bottom of the deck in the game of management Top Trumps.

With appearances being nine tenths of repetitional success, it is impressed on anyone who wants to climb the career ladder that confidence, often despite the evidence, is essential to keeping your job and getting your pet project through the necessary approval stage gates. This is why so many bad managers are able to climb the organisational hierarchy because they are able arrange it so they only look unhelpful and incompetent alternately — using overconfidence to make up the difference.

The late great psychologist and economist Daniel Kahneman observed as much in his seminal Thinking Fast and Slow:

Experts who acknowledge the full extent of their ignorance may expect to be replaced by more confident competitors, who are better able to gain the trust of the clients.

— Kahneman, 2012.

However, if we invoke too many conditional elements into our work, and with this high levels of uncertainty, we introduce paralysis — which can be fatal.

Overconfidence Bias

More than 2,500 years ago Aesop, the Greek fabulist and story teller not the skin care company, included in his collection of fables the story of The Fox and the Cat:

The fox ran into the cat and asked, 'How many tricks and dodges do you know?' The cat replied, 'Actually, I don't know more than one.' The fox then asked the cat, 'What trick is that?' The cat said, 'When the dogs are chasing me, I know how to climb trees and escape.' The cat then asked the fox, 'And how many tricks do you know?" The fox said, I know seventeen, and that gives me a full bag of tricks! Come with me, and I'll show you my tricks so that the dogs won't be able to catch you.' The cat agreed and the two of them went off together. The hunters began to chase them with their dogs, and the cat said, I hear the dogs; I'm scared.' The fox replied, 'Don't be afraid! I will give you a good lesson in how to get away.' The dogs and the hunters drew nearer. 'Well,' said the cat, I'm going to have to leave you now; I want to do my trick.' And so the cat jumped up in the tree. The dogs let the cat go and chased the fox until they caught him: one of the dogs grabbed the fox by the leg, another grabbed his belly, another his back, another his head. The cat, who was sitting up high in the tree, shouted, 'Fox! Fox! Open up your bag of tricks! Even so, I'm afraid all of them put together are not going to save you from the hands and teeth of those demons!'

— Aesop, 2008.

This fable is timeless because it speaks to the problems of overconfidence in tricks (aka 'processes' in management circles) and failure to decide in a timely manner (analysis paralysis). Therefore beware the promotion of overconfidence over rational decisions as it will end badly when it comes to the crunch.

In the corporate world, there are seldom real hounds with sharp teeth, but there are plenty of metaphorical ones. They may take the shape of organisational restructures, failures to read the market, or simply a boss who is unable to deliver on their overconfident promises and tries to save their career by shifting blame onto their team. In the face of such hounds, admission that one is merely guessing is considered unacceptable. While I would agree that people should not guess, when organisations are devoid of psychological safety and create situations which threaten employees' livelihoods, it is no wonder that the pretense of knowledge emerges as the preferred solution for people long on confidence but short on capability.

These anecdotal observations are supported by the data in several studies, one of which was by:

... professors at Duke University [who] conducted a survey in which the chief financial officers of large corporations estimated the results of the S&P index over the following year. The Duke scholars collected 11,600 such forecasts and examined their accuracy. The conclusion was straightforward: financial officers of large corporations had no clue about the short-term future of the stock market; the correlation between their estimates and the true value was slightly less than zero! When they said the market would go down, it was slightly more likely than not that it would go up. these findings are not surprising. The truly bad news is that the CFOs did not appear to know that their forecasts were worthless.

— Kahneman, 2012.

Armed with the knowledge that even the most senior managers in an organisation are unaware their judgement is worthless, and given the slow gestation period of the average corporate program, a confident employee can eke out several years of salary before the chasm between what they claim to be able to do and what they can actually do is uncovered. By which point they have the experience on their resume and can bounce forward to their next role which is granted based on a track record of job titles rather than achievements.

Until the prevailing reality of this employment merry-go-round is broken, overconfidence will remain the go to strategy for many. Even though the evidence suggests that a pessimistic outlook would do a better job of positioning the organisation for success or simply Calling Time on Sunk Costs.

Of course, tone is set at the top, and if you feel frustrated by the incompetence of colleagues, this is invariably stimulated by what is happening at the top of the organisational hierarchy. I am neither alone, nor the first, to observe this trend. Peter Bevelin, author of several books based on key learnings from Charlie Munger and Warren Buffett, observed in Seeking wisdom: From Darwin to Munger:

What tends to inflate the price that CEOs pay for acquisitions? Studies found evidence of infection through three sources of hubris: 1) overconfidence after recent success, 2) a sense of self-importance; the belief that a high salary compared to other senior ranking executives implies skill, and 3) the CEOs belief in their own press coverage. The media tend to glorify the CEO and over-attribute business success to the role of the CEO rather than to other factors and people. This makes CEOs more likely to become both more overconfident about their abilities and more committed to the actions that made them media celebrities.

—Bevelin, 2007.

Reducing Overconfidence, Increasing Rational Thinking

Management plays a pivotal role in changing an organisation's culture by Leading Psychological Safety in Teams. This process helps to ensure that an environment is created which allows for uncertainty to be seen for what it is — a rational stance in an inherently uncertain world. When such an environment is not established, we get what currently abounds — the most overconfident decision makers are rewarded.

Returning to the CFO study outlined above, which shows even the top executives in a field cannot say they truly know what tomorrow brings — the data clearly shows they are unable to predict the future. Because of this, it is essential organisations shift the system of rewards and punishments from being centred on who is confident they can predict the future to who can position for a largely unknown tomorrow. This is important as it moves thinking from the unknown to the measurable. This happens because even if people or teams don't know how to prepare, they can at least answer the question 'are we prepared or are we not?' Examples of preparedness are:

• I do not know when a recession will hit, but I do know that at some point there will be a severe downturn in the economy. Therefore, my best strategy is to position myself for the inevitable rather than living pay cheque to pay cheque and trying to guess when I will need some savings.
• I do not know what my next role will be, but I know I will not have my current job in its present form until the end of my career. Therefore, I can position myself for the inevitable by acquiring new skills and additional knowledge for when the next opportunity comes.

But, and this is where measurement and efficiency comes in, there is a cost to being prepared. While it is a poor plan to live hand to mouth, it is also not efficient to prepare for every doomsday scenario. This enables efficiency to be measured by only preparing when it is necessary. Something which brings us to anecdotes.

If I am overconfident about my future or the future of the organisation I lead, chances are I can surface data to prove my point. Or at the very least explain away the data which shows me in error. This is where the anecdotal evidence from rational, and perhaps even slightly pessimistic, people is essential because it will provide a better insight into what is really happening than either the predictions of overconfident people or what is apparent from the data. As Jeff Bezos put it:

We have so many metrics that we monitor. The thing I've noticed is that when the anecdotes and the data disagree the anecdotes are usually right. There's something wrong with the way you're measuring it. […] You need to check that data with your intuition and instincts.

— Jeff Bezos.

For these reasons, when asked about my level of confidence I lean into the anecdotes even if the data doesn't bear them out. If anecdotally someone is a bully who harasses their colleagues, my confidence is that this is what they are, and I prepare for that behaviour even if the data indicates they are a good employee. If anecdotally the strategic plan is flawed, my confidence is that a new strategy is needed even if the data indicates the organisation should double down on the current strategy. If anecdotally a corporate culture exists which protects the incompetent while the high performers tend to leave, my confidence is that the organisation needs to address this talent management flaw even if the data indicates the company is an employer of choice.

In sum, my level of confidence in a rational decision made on the anecdotal data is high, my confidence in the 'official data' is low as almost anything can be proven with data — particularly when someone with positional authority is controlling the narrative.

Good night, and good luck.

Further Reading

Bevelin, P. (2007). Seeking wisdom: From Darwin to Munger, Malmö, Sweden: PCA Publications.

Kahneman, D. (2012). Thinking, fast and slow, London: Penguin Books.

Winter, Robert N. (2024).

• Calling Time on Sunk Costs.
• Why Positioning Eats Predicting For Breakfast.