It's Highly Probable You'll Read This

Why numbers are better than words in matters of probability

You have medical symptoms of something Bad. You see a doctor. The doctor agrees you may have the Bad Thing. But he’s not sure. The doctor does some tests and sends them off. It will take a week to get results back. Before sending you out of the office to wait, the doctor says, “whether you have the Bad Thing is nowhere near inevitable.”

Do you feel reassured are alarmed by the doctor’s statement?

Or to put a finer point on it, what exactly do you think the statement means? If you were to translate that into the probability that you have the Bad Thing, what percentage would it be?

Not one hundred per cent, clearly. And not zero per cent. So where is it between the two?

I ask this because people routinely use language to describe probabilities. We say things like “may” and “unlikely” and “highly improbable” and “almost certain.” That’s fine in most cases. But when you’re describing something very important — particularly when it’s something you can prepare for, alter, or even forestall — that can easily lead to trouble.

Which brings me to a headline in Foreign Policy, one of the world’s premier magazines covering foreign affairs. Have a look below.

Do you feel reassured? Or more frightened? Personally, I’m not sure.

Read as a literal statement, “nowhere near inevitable” could mean any probability from zero right up to, well, almost any probability. Ninety per cent. Ninety-five.

Of course the statement does imply that war is possible, so we’ll probably have to change that zero to at least one per cent. But a range of one per cent to ninety-five per cent doesn’t exactly narrow things down, does it?

But maybe you say, “come on, the author is clearly trying to be reassuring, so the top end of that range has to be much lower.”

OK. So let’s change that ninety-five per cent to, say, seventy-five per cent. Now we have a range from one to seventy-five per cent. Which, again, doesn’t help a lot.

And here’s where it gets tricky: People aren’t likely to be read it as carefully as I have and give it a wide range. Instead, they may casually pick a neighbourhood in that range and think, “yeah, he means that.” Or even a single number.

And if you have many people making similar decisions, you could have one person thinking the phrase “nowhere near inevitable” means seventy-five per cent and another thinking it’s fifty per cent and a third thinking it means ten per cent. And so on. Everyone would think they understood the meaning. But everyone would have taken away a different meaning.

This isn’t hypothetical. In Superforecasting, Phil Tetlock and I wrote about a 1951 US National Intelligence Estimate that judged the probability of a Soviet invasion of Yugoslavia. All the expert analysts sat around a table together and agreed that the evidence indicated an invasion was “a serious possibility.” That was the conclusion put in the Estimate. And the Estimate went to the State Department, the Pentagon, and the White House. But thanks to a remark in a casual conversation wth a State Department official, the official in charge of the Estimate realized there may be some confusion about what the term meant. So he went back to the experts who had all agreed there was “a serious possibility” of invasion and asked them to express, in numerical terms, what they thought “a serious possibility” meant. Their figures ranged from twenty per cent to eighty per cent — meaning that what looked like a consensus masked broad disagreement.

It doesn’t matter what words you use. There’s no avoiding the danger if you express probability with language. Even if you use language like “certain” and “always,” which, read literally, would seem to allow only one interpretation.

As reported in this excellent Harvard Business Review story, Andrew and Michael Mauboussin asked people in a survey to translate common words expressing probabilities into numeric terms. In every case, they found there was a range of responses. Some ranges were enormous.

Now, to be clear, I use the language of probability constantly. Because it’s usually fine for the task at hand. We don’t need to talk like Commander Data constantly. But we do need to be aware that there is a danger here. And when it comes to important decisions, particularly when there are multiple people involved, we need to seriously consider replacing words with numbers.

Yes, there is a countervailing danger of being excessively impressed by “false precision.” Eighty-three-point-seven per cent of people agree. (I just made that up.) But as Phil and I argue in Superforecasting, that danger can be caught and corrected with a spoonful of skepticism. The danger posed by vague language is far more common and more dangerous.

I should also note in passing one of the strongest reasons to use numbers, not words, when expressing probabilities: The only way forecasters can get better at forecasting is by having clear, prompt feedback, and only numbers can deliver that because only numbers can be counted and scored.

As consumers of news and analysis, we need to get in the habit of thinking critically about this stuff. When a leading magazine of foreign affairs has the audacity to use an almost comically elastic phrase like “far from inevitable” in a headline — particularly about a matter of war and peace — their readers should roll their eyes and not-so-gently suggest they do better.

So let me ask a couple of questions again.

When your doctor tells you it is “far from inevitable” that you have the Bad Thing, how would you translate that to a percentage?

What about that Foreign Policy headline? What percentage do you think it means?

Leave your answers in the comments below.

And as always, if you find this interesting, or helpful, please consider a paid subscription. The probability I will continue to publish this newsletter rises substantially every time a new subscriber signs up. And it is all but certain you will make my day.