The smartest guys in the room are usually wrong


First published in the Sunday Times on 17th January 2016.

Many of us in business spend our lives trying to predict the future, but we are mostly on a fool’s errand. We crave certainty and so we convince ourselves we have remarkable powers of foresight. But in reality we suffer from “the illusion of knowing” — the trap of thinking we can tell what is coming.

I was reminded of this failing recently, when a company I part-own demonstrated that annual budgets are no more than educated guesses — and consequently are often badly wrong. Yet once such hunches are backed up by reams of spreadsheets, supported by all sorts of sensible-sounding assumptions, and spelt out to several decimal points, we think these conjectures are the truth.

Modern science and technology, such as the concept of “big data”, are partly to blame for our present arrogance. Previous eras had more humility. As the journalist and critic HL Mencken said: “Penetrating so many secrets, we cease to believe in the unknowable. But there it sits nevertheless, calmly licking its chops.”

The utter share price collapse last year at most of the mining giants was a case in point. At least £250bn of market value was demolished across only four stocks — Rio Tinto, Glencore, Anglo American and BHP Billiton. Several are now among the walking wounded, obliged to sell off assets in a devastated sector. I imagine that between them they employ huge numbers of highly qualified analysts and vast quantities of computing power, all focused on understanding commodity cycles, and accurately judging the demand and supply of minerals. These are the world experts. But seemingly none of them saw the threats.

I was a stockbroking analyst once, and in those days there was great emphasis on the difference between a profit estimate (not very reliable) and a profit forecast (must be met). But were they that different? Ultimately, the screenwriter William Goldman’s remark about Hollywood is apt here: “Nobody knows anything.”

Today’s business plans for companies seeking capital are much more elaborate than they used to be. They usually contain reams of statistics, surveys and figures. But an awful lot of such plans are closer to fantasy novels than documents upon which investors should depend. Promoters, investors, strategists and economists too often think of finance and capitalism as disciplines like physics or engineering. They believe that if you are “smart” enough, you can measure human activity precisely and draw appropriate conclusions. But as the Motley Fool website says: “Finance is much closer to something like sociology. It’s barely a science and driven by irrational, uninformed, emotional, vengeful, gullible and hormonal human brains.”

The statistician Nate Silver’s recent book, The Signal and the Noise, makes a number of important points about forecasting. He argues that the deluge of data now available means people perceive patterns where none exist, and that apparently clever models can do more harm than good. A case in point was the gobbledegook used by banks and rating agencies to predict default rates on mortgages before the 2008 housing collapse in America.

A deadly mix of perverse incentives, inadequate regulation, group think, irrational behaviour and over-confidence created the greatest asset bubble of modern times. Some of those who refused to follow the herd are portrayed in the new movie The Big Short.

Similarly, a majority of hedge funds have suffered a weak performance over the past two years. All those overpaid geniuses, with their high IQs and sophisticated black boxes, have delivered mediocre returns. Hedge fund managers are slick at selling their expertise, and enriching themselves. But they are really no better than the rest of us in knowing what lies ahead.

I have found that experience gained in the front line can outweigh theoretical predictions cooked up in an ivory tower. Too many of those who prepare forecasts and predictions are like “eunuchs in a harem”, not wise enough to see that the world is a messy and imperfect place where mathematical constructs frequently confuse rather than illuminate. By contrast, entrepreneurs are typically flexible, adaptable and pragmatic in their outlook — all necessary attributes when planning for the future.

From the collapse of the Soviet Union to the rise of China, most of the great events in modern history caught our governing elites unaware.

Nevertheless, serious organisations cannot just leave everything to fate and gut instinct. I believe one should never place unquestioning reliance on plans, forecasts and predictions — especially if they are prepared by gurus. As the economist JK Galbraith wrote: “Very specific and personal misfortune awaits those who presume to believe that the future is revealed to them.” He should have added,
“. . . and anyone who follows their advice”.