Keeping Score

Matt Yglesias is the only one doing what needs to be done

In Superforecasting, published in 2015, Phil Tetlock and I made an argument for change which I humbly think is important and irrefutable. And it changed … nothing. It was, in fact, roundly ignored by the people who most needed to hear it. With one important exception I’ll get to later.

Think of a sport you’re familiar with. (I’m Canadian so of course I think of hockey, but I’ll keep this generic for those of you who mentally put “ice” before “hockey.”) A new match/game/bout is set to start on TV. The camera focuses on the face of one of the participants. On the screen, a statistic or two appears. A commentator cites more.

These numbers are performance measures. Every sports broadcast is riddled with them. Players, coaches, and managers know them intimately. So do serious commentators and bettors. Even casual fans who call in to talk shows can reel off numbers to make the argument that this player is magnificent while that one is a clod. It is a rare and foolish person who would ignore performance statistics altogether and judge players by that one time they saved the day or screwed up horribly.

Now think of another form of highly publicized competition — the pundits, analysts, and professors who tell us not only what is happening, and why, but what will happen next.

The bill will or won’t pass.

The politician will or won’t be elected.

The economy will or won’t slip into a recession that will or won’t be brief and shallow.

War will or will not break out.

As in sports, the participants who do well in these predictive contests — I’ll get to the definition of “doing well” in a moment — can win fame and fortune. As in sports, prediction contests entertain large audiences.

But unlike sports, prediction contests sometimes make an enormous difference in the lives of real people in the real world. To take the most obvious example, predictions that change popular expectations of the future can change popular behaviour and thereby change the future — as when a widely expected electoral rout motivates large numbers of people appalled by that prospect to vote in an election they otherwise would have ignored, or when an expectation of hard times ahead prompts consumers to pull back on spending and shove the economy closer to the cliff.

But there are more direct means by which predictions shape the future. Like the public at large, policy-makers listen to the pundits, analysts, and professors. So do markets. Investors. Corporate executives. Generals. Dictators.

And yet, when I published Future Babble more than a decade ago, one reviewer said there was no need for a skeptic’s take on predictions because no one takes predictions seriously. I was stunned. I’ve read impressively dumb things in print over the years, but that one left me well and truly gobsmacked.

That was only my first encounter with a spectacularly misinformed line of thinking. In the years since, I’ve heard many people insist the future is unpredictable so they don’t make predictions and don’t listen to people who do. I smile whenever I hear that because I know what’s coming if I continue the conversation — that same person will tell me about how the next election will play out and how she has cut back on spending because the economy is going to tank.

As English soccer star Paul Gascoigne once said, “I don’t make predictions and I never will.”

We all have expectations about how the future will unfold. We can’t help but form them because it is impossible to make decisions without them. Should I buy a house? The answer to that depends heavily on my expectations of the future. If I’m not sure my job is secure, or I think the housing market is due for a painful correction, that will influence my thinking. And how do I form my expectations of the future? I listen to the pundits, analysts, and professors. And I listen to the people around me … who listen to the pundits, analysts, and professors. The same is true of politicians, investors, executives, and all the rest. So, yes, these predictions matter. They really, really do.

Unlike sports.

I’m a giant Boston Bruins fan and the Bruins are the best team in the NHL, but if they don’t win the Stanley Cup this year it will merely bum me out for a day or two (or three). But the pundits, analysts, and professors? Their performance can make a serious difference in my life, and in the lives of millions of others.

And yet, it is sports that is scored and analyzed with impressive sophistication while no one cites the performance statistics of the pundits, analysts, and professors? Those stats don’t even exist. Despite this, people have very definite opinions about which pundits, analysts, and professors are magnificent and which are clods. How do they form these opinions if not with performance statistics? There are three principal means.

Ideological affiliation: What the pundit says makes sense to me … because it is in line with my general thinking about how the world works. This is nothing more than unvarnished confirmation bias, that old devil.

Impression: If you think appearance, status markers, glib stories, firm handshakes and confident words only influence the foolish and uneducated, con men everywhere would love to meet you.

Anecdote: He called the housing crash of 2008! He must be right that hyperinflation is coming!

I once mentioned Paul Krugman to a rich and powerful hedge fund manager. He scoffed. Krugman is clueless, he said with iron confidence. He backed that judgment by citing one forecast Krugman got very wrong. That’s the equivalent of saying a particular baseball player is a worthless bum because he struck out that one time. Dumb, dumb, dumb. And it was coming from an extremely intelligent, sophisticated thinker who would never accept such sloppy thinking in his work. Or when watching sports.

All this leads to a conclusion that still leaves me shocked and shaking my head: We can dramatically increase the intellectual rigour involved in the creation, judging, and use of important predictions that make a big difference in the world simply by applying the same standards we routinely use with inconsequential games.

That is nuts. That should not be true. But it is.

Now, there’s a lot more to this argument. Most importantly, while sports are designed to be scored and nuanced performance measures can be developed without too much difficulty, it’s a lot harder with predictions. But it’s not impossible. Phil Tetlock has spent much of his career proving that. We showed how it’s done in Superforecasting.

But the core argument is simple: Keep score. In baseball, a key performance statistic is batting average. What’s Paul Krugman’s batting average? I don’t know. That hedge fund manager doesn’t know. The New York Times doesn’t know. Not even Paul Krugman knows. And that’s ridiculous.

When we published Superforecasting in 2015, it would have been lovely if the Times and other media outlets have leapt up and said, “you’re so right!” They did not. In fact, the media ignored the book entirely. (The industry that most embraced it — finance — was the one that least needed it. Confirmation bias is an awesome force.)

Nor did the pundits, analysts, and professors rush to line up and get scored. In fact, early in the writing of Superforecasting, when Phil’s research program was still underway, we sent a letter to some 80-odd famous pundits types asking them if they’d like to join the program, make predictions, and get scored. This offer was almost universally rejected, which was not at all surprising since, for an already-famous pundit, there is a ton of downside risk in being tested this way. One of the few exceptions was Ian Bremmer, who is both a famous pundit and the head of a highly influential geopolitical risk firm, and thus had every incentive to politely ignore Tetlock and Gardner. The fact that he didn’t speaks better of him than any adjective I could write here.

Post-publication there was another high-profile exception.

You probably know Matt Yglesias. He co-founded Vox. Now he writes one of the most popular Substack newsletters, Slow Boring, and has an enormous following. Sitting pretty atop the pundit pile, he should avoid anything that could risk his reputation as a clever observer worth listening to.

But Matt read Superforecasting, found the arguments for scoring predictions compelling, and put it into practice. More remarkably, he published the results. Which he described as “terrible.”

That was last year. Now he’s done it again.

Please read his latest post. It’s a superb illustration of how expressing thoughts in terms of numerical probabilities can sharpen thinking. It also demonstrates how scoring invites post-mortem analysis that teaches the most valuable lessons.

Matt has approached the exercise as one of personal development. (I may have bumped into Matt once or twice long ago but I don’t know him. I’m calling him “Matt” because I can’t write “Iglesias” without thinking “Julio,” then realizing I can’t spell, and then realizing I didn’t think of “Enrique,” which means I’m old. So “Matt” it is.)

And that’s the main point of the exercise: not that I per se want to become a superforecaster, but that writing these things down with odds attached is a good way of trying to beat back that overconfidence. “If I grab a pair of dice, I probably won’t roll snake eyes five times in a row” and “D.C. probably won’t cancel five days of school for snow this winter” are both predictions I would stand behind, but there’s actually an incredibly large gap between the probabilities associated with these two things. Casual writing tends to elide the difference between “this would be unusual” and “this is spectacularly unlikely.” It also often struggles with efforts to express an idea like “this probably won’t happen, but the odds of it happening aren’t tiny and are rising, and the consequences would be really bad so I’d like you to worry about it.” Attempting to quantify with exact numbers even occasionally is a way of trying to break bad habits.

But to do that, it’s important to examine what went right and what went wrong.

That’s all true. If you want to see reality more clearly, and anticipate the future more accurately, numbers and scoring are indispensable.

I’ll add another benefit: As long-time readers know, I’m a little obsessed with hindsight bias, which I think is far more pernicious than even psychologically informed observers realize. What’s the best cure for hindsight bias? Record what you think will happen in future in the clearest terms possible. Keep that on file. When time passes and you recall yourself scoffing about Y2K — “I knew it would never amount to anything” — you can check the records, see that you were one of the many people who stocked up on canned food and ammunition, and know that hindsight bias is hoodwinking you. Once upon a time, lengthy diary entries played the same role. No one has time for that any more. And anyway, assigning numerical probabilities is better for this purpose than even the wordiest Victorian verbiage.

But let’s set personal improvement aside. There is a bigger reason to take note of what Matt is doing.

It is outrageous that people with no proven track record of predictive insight are paid by the biggest platforms in the world to tell us what will happen next. Or rather, it should be outrageous. Predictions matter. If The New York Times delivers badly inaccurate predictions to its readers, it is providing a flawed product. If The New York Times doesn’t even test the accuracy of the predictions it provides, it is negligent.

So how can we get to a world where The Times feels it should test its writers? Look at what Matt Yglesias has done. Ask other pundits where their results are. Expect to see them. Demand to see them. Scorn those who have nothing to show.

Only when the demand side changes will the supply change.

Only then will we get better predictions and better decisions.