Probability Calibration Workshop - Lesson 4

This is the fourth interactive lesson of a Probability Calibration Workshop -- presented by Brian Lucena at PyData Global. Model calibration in machine learning refers to the process of ensuring that the probabilities from your model are precise. For full context, start with the Introduction and Lesson 1-3 before proceeding to this lesson. The notebooks associated with this workshop can be found in the repo https://github.com/numeristical/resources in the folder "CalibrationWorkshop". The workshop roughly covers the following topics: Why calibration? - What it means for model outputs to be *well-calibrated*. - Why and when it is important (or not). - Specific scenarios where calibration may be valuable. Assessing the model - How to determine if the model is well-calibrated. - Reliability diagrams and how to use them. - Issues with calibration for values close to 0/1. Calibrating the model - Illustration of the various techniques: - Isotonic Regression - Beta Calibration - Platt Scaling - Spline Calibration - Demonstrating their use and results on real data. - Tradeoffs between the approaches. - Calibrating multi-class models. Assessing the calibration - Did the calibration improve model performance? - Are there flaws in the calibration? - How to adjust the calibration and improve further.