13: Validation and Model Selection (79min)

Machine Learning From Data, Rensselaer Fall 2020. Professor Malik Magdon-Ismail talks about validation and one of its main uses, model selection. Validation is a simple but powerful concept. We discuss how validation works as a method to sneak peak at the out-of-sample error for the deficient hypothesis. We talk about the tradeoff in choosing the validation set-size K to balance the accuracy of the validation error with the deficiency of the validated hypothesis. This leads us to cross-validation, the Queen of validation techniques. Unfortunately this queen is not so efficient to compute, so we take shortcuts in practice unless there is an analytic formula for the leave-one-out cross validation error, which there is for regularized linear regression. This is the thirteenth lecture in a "theory" course focusing on the foundations of learning, as well as some of the more advanced techniques like support vector machines and neural networks that are used in practice. Level of the course: Advanced undergraduate, beginning graduate. Knowledge of probability, linear algebra, and calculus is helpful. Material is from Chapter 4 of "Learning From Data", amlbook.com, 2012.