How to judge a decision

Hint: Ignore the outcome.

How can we judge the quality of a decision?

Intuitively, it’s the result that we care about so the result settles it. If the result of the decision is what we wanted, it’s a good decision. If not, not. Simple. For most people, that intuition is enough. We know a decision is good if the outcome is good, and that’s the end of it.

We saw this playing out in the commentary this week about Donald Trump’s decision to join Israel’s air assault on Iran and drop America’s entire stockpile of its largest non-nuclear bomb on key Iranian nuclear sites.

Trump, as is his wont, immediately declared the operation a total, absolute, smashing success before the smoke had even cleared.

Trump’s critics warned the conflict could escalate.

When Iran responded with a face-saving feint of an attack, then agreed to a ceasefire, a parade of right-wing pundits like Ben Shapiro lined up to declare Trump nothing less than a genius, while Trump awarded himself a Nobel Peace Prize.

Trump’s critics tut-tutted. Blowback could come in the form of terrorism. Or other, unpredictable miseries. Remember Dubya standing in front of that Mission Accomplished banner?

Then came the damage assessment reports.

First was the leaked initial assessment of the Defense Intelligence Agency, the military’s central intelligence office. It suggested the Iranian nuclear program had been set back mere “months.” Trump’s critics crowed that this was proof Trump’s decision was a disaster. Trump exploded, saying the report insulted the brave patriots of the Air Force.

Then Trump’s CIA director, John Ratcliffe, a former Texas Congressman, along with Defense Secretary Pete Hegseth, supported Trump’s claim of success. Notably, however, both used much more cautious language than Trump.

At the time I’m writing this, the latest assessment comes from the International Atomic Energy Agency, and it’s universally respected head Rafael Grossi, who said the key attack, on Fordo, rendered the facility inoperable, and gave some more complicated details, but avoided crude language like “success” or “failure” and declined to provide a clear estimate of how deep the setback to Iran’s program is. All that complexity and ambiguity gave partisans plenty of scope to interpret that how they liked.

The state of Iran’s nuclear program is crucial, obviously.

But most of the wrangling we’re seeing? It is blatantly about making the outcome look as good or bad as possible in order to make Trump’s decision look as good or bad as possible — on the assumption that the outcome is the yardstick against which decisions should be measured.

Unfortunately, that assumption is wrong. Plain, flat-out wrong.

Consider this simple illustration.

Imagine I’m you’re friend and I tell you I have a brilliant idea.

I’m going to sell my house, empty my bank accounts, fly to Las Vegas, and bet every penny I own on a single spin of the roulette wheel. I’ve even got my slot picked out. I’m going to bet my life savings on lucky 28.

What do you think of my decision?

“You are an idiot,” you say.

And you are right, of course. The probability of winning such a bet is 2.63% in American roulette. Or to put that the other way around, my chance of losing is a little higher than 97%.

It’s not only the probabilities that suggest I’m an idiot. So do the consequences. If I do this and I lose, I would be wiped out. That’s catastrophic risk. And remember, there are no circumstances pressing me to take this a drastic step. I don’t have to come up with money for ransom. I don’t have to pay for a child’s life-saving treatment. There is nothing remotely so urgent that it would make this horrible plan one worth contemplating.

It’s a bad decision. Categorically. Objectively. If I go ahead with this, I am a truly an idiot.

But I go ahead with it anyway.

And …

What do you know?

Lucky 28 delivers.

My reward? The casino gives me 35 times my bet! I’m rich!

I charter a private jet and fly back home to talk with you again. What do you think of my decision now? I ask. “Am I still an idiot?” I say with a smirk.

If you have sufficient intelligence to read these words, you know the correct answer.

Yes, I am still an idiot. I am rich and an idiot. I got lucky, sure, but that doesn’t change the quality of my judgement a damned bit. It was a terrible mistake. And I remain an idiot.

None of this is to say outcomes never matter when judging decisions. They may contain relevant information, and that information should be considered. But the outcome should never completely settle the judgement. Often, it shouldn’t even be a significant factor, or even carry the slightest weight. After all, decisions can only be based on the information available at the time they were made and the outcome isn’t known then.

But as I said at the top, this isn’t how we intuitively think about decisions. Instead, we put heavy emphasis on the outcome, and commonly use it alone as the yardstick.

That’s bonkers. But it’s also very human. Psychologists call it “outcome bias.” It’s a big reason why we underestimate the role of luck in human affairs — because when there is a smash hit, or a disastrous flop, we almost automatically ascribe that outcome to the people involved rather than consider all the circumstances outside their knowledge or control which may have determined the outcome.

Or to put that more bluntly, you can do everything right but fail. Or do everything wrong — lucky 28! — yet succeed.

If we don’t recognize that, and judge accordingly, we can make terrible mistakes, such as concluding that wise men are fools. Or fools wise men.

So how should we judge Trump’s decision? (Note that in what follows, I am not discussing issues of legality, domestic or international, or ethics. They’re important but separate questions.)

The first thing to do is stop thinking in binary terms.

People naturally tend to think in terms of yes/no, it will/it won’t, success/failure. Reality is seldom so simple and when we try to assess it that way, we make dumb mistakes. Note that this goes both for judging the decision and the outcome. Listen to any serious expert talk about the consequences of the attack and you’ll hear a complicated assessment of “some factors on this side” and “other factors leaning the opposite way” and “remains to be seen” and “pending further investigation.” That’s the language of probability, not certainty.

It’s also the language of reality. Political speech loaded with absolutes and binaries — the stuff Trump and too many of his critics constantly spout — is the language of self-serving delusion.

The second thing to do is to separate the decision from the outcome and judge the decision and the decision-maker on the basis of what was known at the time of the decision. Only if the decision contains information relevant to that analysis should it be considered. Often, it won’t.

Now, what do we know about Trump’s decision, who he consulted, what he was told, what questions he asked, how we information together and drew a conclusion? Very little. In fact, close to nothing.

What’s more, even if that whole process were an open book today, it would take painstaking, forensic analysis to assemble what are undoubtedly complex facts, get a clear picture of all the relevant factors that should have been on the table, and draw conclusions about which decisions were best. Historians and political scientists often do this sort of analysis with major political decisions. It can take years of heavy lifting.

Given all that, why would anyone think a politician’s self-interested insta-comments, or some pundit’s hot take, are capable of providing real insight and valid judgement?

Look, I can easily imagine facts which would show Trump’s decision was exactly right. But I can just as easily imagine facts which suggest Trump’s decision-making was horrendous. And that’s true whether the attack completely obliterated Iran’s nuclear program or only broke a few teacups.

Until we know far, far more, it’s simply impossible to draw meaningful conclusions. It will take weeks, months, and years to deliver informed, meaningful judgements.

And that helps explain why outcome bias is so common.

Judging based on the outcome — or at least the apparent outcome — is usually quick, easy, and definitive. That is what we crave psychologically. That is what works in the political arena. That is what the 24/7 news cycle demands.

Consider this short post. If I had put together a plausible story for why Trump’s decision was monumentally foolish, a horde of people would rush to congratulate me. And maybe take out a paid subscription. If I had come to the opposite conclusion, a different horde would have done the same. But saying “we don’t know and can’t know for weeks, months, or years?” No one is going to congratulate me. And my cash register will not ring.

Very simply, there are overwhelming incentives to indulge outcome bias to the hilt. But there are no incentives of any kind to be more thoughtful.

So was Trump’s decision good? I have feelings about that, of course. But that’s all they are. Gut hunches. I don’t know yet. Not in any meaningful sense.

Neither do the pundits. Or the president. Or you. Welcome to our complex, ambiguous, infuriatingly uncertain reality.

Be sure to hit the like button. And take out a paid subscription.

One last point. It’s a little tangential but it’s important.

One aspect of this story can be judged, plainly and indisputably. It is Trump’s handling of intelligence following the attack, notably his savaging of the DIA’s initial damage assessment.

The best intelligence analysis — the sort we want informing top leaders’ decisions — is the product of a strict and severe focus on collecting evidence, analyzing it with extreme care, and drawing whatever conclusions follow.

Who is happy to hear those conclusions must be irrelevant. Who is angered by the conclusions is irrelevant. Whose agenda the conclusions could advance or retard is irrelevant. These sorts of political considerations should never be the analyst’s concern and good intelligence analysts work hard to keep them from contaminating their work.

But still, intelligence analysis is notoriously susceptible to political influence. Even with offices full of professionals and managers determined to give analysts the political protection they need to do professional work.

If the guy at the top of the pyramid makes it clear that he wants certain answers, does not want other answers, and will make life miserable if the intelligence agencies do not deliver, it is all but impossible to stop intelligence from becoming politically biased. And that can make intelligence worse than useless.

Recall that Vladimir Putin was confident Ukrainians wouldn’t resist his “special military action,” allowing his invasion to be swift and low-cost. Why did he think that? In part, because he lives in a bubble. In part because the prime directive of his administration — as in most dictatorships — is pleasing the boss. Tell Vladimir Putin what he wants to hear and you will be rewarded. Say something that makes him wince and you will be punished. A government like that gets very good at seeing what the boss wants to be reality, which ensures that the boss becomes increasingly deluded.

Most modern American presidents have understood this. Most have, more or less, at least tried to behave accordingly. There have been damaging exceptions. But the importance of professionalism and independence was at least understood.

Until this president. The brutal, public tongue-lashing Trump delivered to the DIA for daring to draw an unwelcome conclusion will do lasting damage to the US intelligence community. Add three more years of that sort of behaviour and we can be sure that American intelligence will grow increasingly stupid.

The consequences of that may be unpredictable, but that there will be consequences is the safest of bets.