IX Coach

Calibration Training

Building the feedback loop between confidence and accuracy

How to train well-calibrated confidence

Calibration training teaches you to match your stated confidence levels to your actual accuracy: if you say "70% confident," roughly 70% of those claims should turn out to be correct. Tetlock’s forecasting research and probability training studies show calibration is trainable with practice and feedback — the evidence here is specific and replicable.

Most people are poorly calibrated in a specific direction: overconfident in domains where they feel competent, underconfident in new ones, and systematically bad at expressing rare-event probabilities. Calibration training is the deliberate practice of measuring the gap between stated confidence and actual accuracy, then closing it with feedback. It is not about being uncertain — it is about being accurately uncertain. Below are the practices that make the training work.

Practices

Practice this with IX Coach

Practice this with IX Coach

Research

Related concepts

Frequently asked questions

How to train well-calibrated confidence

Calibration training teaches you to match your stated confidence levels to your actual accuracy: if you say "70% confident," roughly 70% of those claims should turn out to be correct. Tetlock’s forecasting research and probability training studies show calibration is trainable with practice and feedback — the evidence here is specific and replicable.

How do you practice confidence interval estimation?

A 90% confidence interval should contain the true value 90% of the time. Most people’s 90% intervals are far too narrow — real values fall outside them 30–50% of the time, reflecting severe overconfidence in precision. Practicing interval estimation with immediate feedback recalibrates the internal sense of "how much I know," which transfers to more honest uncertainty expression in decisions.

How do you maintain a scored prediction log?

Calibration requires a feedback loop: you state a probability, the outcome happens, and you learn whether your stated confidence matched the base rate of your predictions. Without recording and scoring, confidence remains a feeling rather than a measurable quantity. The Brier score is the standard metric: lower is better, rewarding confident correct predictions and penalizing confident wrong ones proportionally.

How do you identify the specific domains where you are most overconfident?

People are not uniformly overconfident; they are typically more overconfident in domains where they have experience but limited feedback, where the domain uses jargon they understand, or where they have made public commitments. Diagnosing which domains produce your worst calibration focuses debiasing effort where it will have the largest return.

How do you anchor your confidence level on base rates, not gut feel?

Gut-feel confidence is contaminated by how fluently the answer came to mind, how motivated you are for the outcome, and how recently a similar event happened. None of these is a reliable proxy for accuracy. Base rates from a defined reference class provide a hard external anchor that separates the confidence estimate from the emotional and cognitive noise that gut feel mixes in.

How do you pre-commit to resolution criteria before making a prediction?

Vague predictions survive almost any outcome — "I said things might get complicated" is consistent with disaster or success. This outcome-flexibility is a form of motivated reasoning that prevents calibration feedback from ever arriving. Pre-committing to specific, verifiable resolution criteria closes the escape hatch: the prediction either resolves correctly or it doesn’t, and you can’t post-hoc reconstruct what you "really" meant.

How do you build calibration reps with low-stakes trivia and almanac questions?

Calibration skill, like any skill, requires repetition with feedback. Real-world decisions resolve slowly, have high emotional stakes that distort honest scoring, and occur too infrequently for rapid iteration. Trivia and almanac questions provide high-volume, immediate-feedback practice in a domain with no emotional interference — making them an ideal training environment for the underlying metacognitive skill that transfers to high-stakes predictions.

How do you distinguish uncertainty (quantifiable) from ignorance (unquantifiable)?

Knight’s distinction between risk (quantifiable uncertainty) and uncertainty (unquantifiable) matters practically: assigning a 60% probability to an event requires a reference class or a model. When neither exists, the number is a feeling dressed as a statistic. Distinguishing the two prevents false precision from masking genuine ignorance, and prompts the honest response: "I don’t have enough information to put a calibrated number on this."