‹ Culture 3.14 · What we ask ourselves
Question · Decisions
How can uncertainty be communicated without turning it into a misleading number?
87 % confidence reads as a promise, even when the model is only saying that it has seen similar cases.
Why it matters
A number with decimal places conveys a precision the procedure that produced it rarely has. Whoever receives it compares it with other numbers and takes decisions on the basis of that comparison.
The effect is worse when the figure is well calibrated in aggregate and badly calibrated for the specific case in front of you.
What we know so far
Intervals work better than points, and categories with a defined operational meaning work better than intervals. Saying “this case bears little resemblance to what the model has seen” conveys more than any percentage.
Showing the calibration alongside the prediction, even though it takes up space, has changed whole conversations.
What remains open
It remains open how to communicate the uncertainty that comes not from the model but from the data that is missing, which is the most frequent kind and the one no interval captures.
And how to stop a cautious category from becoming, after several months of use, a synonym for “go ahead”.