Calibration: Knowing How Much to Trust Your Own Judgement

How often do we walk into a decision already certain of the answer? And if certainty arrived before the evidence did, what exactly is it that we are trusting, the mind or merely the feeling of having a mind?

There is a discipline, not widely taught but quietly essential, that sits beneath every act of good reasoning. It goes by the name of calibration, and it concerns the relationship between how confident we are in a belief and how likely that belief is to be correct. A well-calibrated thinker is not necessarily one who is always right. That would be an impossible and rather inhuman standard. A well-calibrated thinker is one who is right in rough proportion to their confidence, someone who, when they say they are ninety percent sure, turns out to be correct about nine times in ten, and who, when genuinely uncertain, says so without embarrassment. The discipline, in other words, is not about accuracy alone. It is about the honest alignment of belief with evidence.

This may sound technical, even bureaucratic, as though we are being asked to assign numerical probabilities to every passing thought. But the underlying idea is ancient and recognisably human. Aristotle wrote of the virtue that lies between extremes, and calibration is precisely that virtue applied to the inner life of belief. The overconfident person overshoots; the underconfident person undershoots. Both are making the same fundamental error, which is a failure to let the strength of a belief track the strength of the evidence that supports it.

Overconfidence is the more commonly diagnosed failure, and for good reason. Psychologists who study judgement have found repeatedly that people, when asked to estimate quantities or assess probabilities, tend to place far too narrow a range around their guesses. They are surprised more often than they expect to be. Experts are not immune; in some domains, expertise seems to intensify the problem, lending a feeling of authority to conclusions that the evidence only weakly supports. The practised lawyer, the seasoned clinician, the experienced investor, each can fall prey to the quiet seduction of fluency, the sense that because thinking comes easily, the thinking must be sound. But fluency is a feature of the process, not a guarantee of the product.

Yet it would be a mistake to treat underconfidence as merely a lesser vice. Someone who habitually doubts their own reasoning beyond what the evidence demands is not being humble; they are being inaccurate in a different direction. Epistemic cowardice, the tendency to retreat into vagueness so as to avoid being wrong, is still a form of miscalibration. It carries its own costs: paralysis in decision-making, the outsourcing of judgement to whoever happens to sound most assured, and a subtle corruption of the intellectual character, because a person who never commits to a belief can never really learn from being mistaken. Revision requires that there was something to revise.

So how does one go about tuning belief to evidence? The first and perhaps most unsettling step is to take seriously the idea that confidence is a quantity, something that can be too high or too low, and not simply a psychological state to be either embraced or overcome. This means asking, when you hold a view, not just whether you believe it but how much you believe it, and whether that degree of belief is proportionate to what you actually know. It means distinguishing between the vivid feeling of being right and the slower, less dramatic business of having good reasons.

One useful practice, developed and refined within forecasting research, is to make predictions explicit and then track them. When we keep our beliefs vague, we protect ourselves from the discomfort of being wrong, because vague beliefs can always be reinterpreted after the fact to have been correct. Specificity is uncomfortable but corrective. The forecaster who commits to a concrete probability, writes it down, and returns to it later is building a feedback loop that gradually pulls confidence towards accuracy. Most of us, living without such loops, simply accumulate unexamined certainties.

A second practice is to cultivate what some researchers call reference class thinking. When assessing how likely something is, we naturally focus on the features of the particular case in front of us. But this inside view, vivid and detailed as it is, tends to inflate confidence. The outside view asks instead: what usually happens in situations of this general type? How often do projects like this one come in on time? How often do predictions like this one pan out? The answer is frequently humbling, and that humility is precisely the point. It is not a counsel of despair but a corrective lens.

There is also something to be said for the deliberate practice of steel-manning the opposition. If you find yourself holding a view with great confidence, the question worth putting to yourself is not whether you can defend it but whether you genuinely understand why intelligent people disagree with it. Overconfidence often lives in the gap between knowing your own arguments and understanding the opposing ones. A belief that cannot survive a serious encounter with its best alternatives is not yet a well-founded belief.

Calibration matters, finally, because it is a form of respect, for evidence, for other people, and for the genuine difficulty of knowing things. A culture of well-calibrated thinkers would not be a culture of diffident non-committal. It would be a culture of people who hold their views firmly where the evidence is strong, lightly where it is weak, and who are prepared to say, clearly and without shame, that they do not know. The courage to be uncertain is not the absence of intellectual character; it is one of its highest expressions.

What would it mean, then, to carry calibration into the ordinary business of a thinking life? And if we began to notice the distance between how sure we feel and how sure we ought to be, would we find that distance humbling, or would we find it, strangely, liberating?