11: Overfitting (75min)

Machine Learning From Data, Rensselaer Fall 2020. Professor Malik Magdon-Ismail talks about overfitting. Being able to handle overfitting is what separates the professional from the amateur. We pin down exactly what overfitting is and its cause. We find out that stochastic noise (random measurement error) and deterministic noise (overly complex target functions are the primary cause of overfitting. Complicit in this crime is the learning model/hypothesis set. The noise leads the learning astray especially when the hypothesis set is complex. This is the eleventh lecture in a "theory" course focusing on the foundations of learning, as well as some of the more advanced techniques like support vector machines and neural (deep) networks that are used in practice. Level of the course: Advanced undergraduate, beginning graduate. Knowledge of probability, linear algebra, and calculus is helpful. Material is from Chapter 4 of "Learning From Data", amlbook.com, 2012.