Probability, uncertainty, and base rates
Interpret model confidence without mistaking it for a guarantee.
- Distinguish frequency and confidence
- Use a base rate in a calculation
- Recognize the need for calibration
Start with a population
A probability statement needs an event and a population. If a detector finds 90% of rare events, that says nothing by itself about how many alerts are correct. The frequency of the event and the false-positive rate also matter.
Imagine 1,000 messages, of which 10 truly need escalation. A system finding nine of those but incorrectly flagging 50 ordinary messages produces 59 alerts, only nine correct. Excellent recall can coexist with poor alert precision.
Confidence needs checking
A model score is not automatically a calibrated probability. Calibration asks whether events assigned roughly 0.8 confidence actually occur about 80% of the time across comparable examples. This requires held-out observations and enough data.
Uncertainty can also come from unfamiliar inputs, missing information, or a changing environment. A numeric score cannot capture all of these by itself. Use uncertainty to guide review and evidence gathering instead of disguising it with extra decimal places.
A small experiment you can run.
The numbers describe a constructed scenario, not a benchmark result. They show why a rare target changes the interpretation of an alert.
population = 1000
actual_positive = 10
true_positive = 9
false_positive = 50
precision = true_positive / (true_positive + false_positive)
recall = true_positive / actual_positive
print("Prevalence:", actual_positive / population)
print("Precision:", round(precision, 3))
print("Recall:", round(recall, 3))
Save the file, open your terminal in that folder, and run python probability-and-uncertainty.py. Use python3 or py if required by your installation. Setup guide
The initial precision is approximately 0.153 while recall is 0.9.
Calculate a precision target.
- Keep nine true positives.
- Find the largest integer number of false positives that still gives at least 75% precision.
- Explain the practical cost of reviewing the resulting alerts.
Compare with a suggested solution
At most three false positives are allowed: 9 / (9 + 3) = 0.75. A fourth drops precision below the target. Whether that trade-off is acceptable depends on the cost of missed events and unnecessary reviews.
One idea to take with you.
Make it part of your progress.
Finish the practice and answer the knowledge check to mark this lesson complete.
Go deeper with primary documentation
Optional references for further study. This lesson and its examples were written for Artificials.
Python math module