Calibration
Calibration measures whether stated probabilities match observed frequencies over many predictions.
In plain English
A well-calibrated forecaster who says 30 percent is right about 30 percent of the time across many such calls. It is a separate question from whether those forecasts beat the market.
Example
Of one hundred predictions each given a 30 percent chance, roughly thirty should happen.
Why it matters
Calibration is how you check whether stated confidence means anything, rather than trusting that it does.