02 - FORMAL SETUP - INTRODUCTION TO REGRESSION AND KERNEL METHODS

************************************************************************** BECOME ONE OF THE FIRST STUDENTS OF THE NEW STANDARD MACHINE LEARNING CURRICULUM! https://forms.gle/baxK3Cwz8q2xuJEt5 ************************************************************************** Timestamps for this lecture: 0:00 Supervised Learning 3:25 Unsupervised Learning 9:14 Semi-Supervised Learning 9:35 Reinforcement Learning 11:52 Classification 15:04 Regression 15:49 Structured Prediction 18:03 Loss Functions 21:51 True Risk 24:07 Bayes Risk & Bayes Predictor 26:45 Consistency of Learning Algorithms 30:08 Empirical Risk 31:22 Proof: Empirical Risk is an unbiased Estimator of True Risk 35:42 Empirical Risk Minimization 37:25 Approximation vs. Estimation Errors 41:03 Under- and Overfitting 45:47 Bias-Variance Tradeoff 51:11 Proof: Expected Quadratic Error = Variance + Bias^2 55:02 Is there actually a Bias-Variance Tradeoff? Get the lecture slides: https://drive.google.com/file/d/1nIqISTpYxBYsl4ym2Hn95dKmVdXsX127/view?usp=sharing There is a shortage of well-presented learning resources for rigorous Machine Learning. In the FUNDAMENTALS OF MACHINE LEARNING program @ äon intelligence, laser-focused modules are provided for competent introductions, as well as in-depth discussions of advanced content. The INTRODUCTION TO REGRESSION AND KERNEL METHODS module provides the ideal introduction to the field of Machine Learning for anyone with a reasonable level of mathematical maturity. The contents of this module include: 01 - PREREQUISITES: Basic concepts of linear algebra and multivariable calculus are stated and practised with simple examples. Outlook on prerequisites of advanced content is given, including probability theory, eigenvalues and their eigenvectors, matrix decompositions, and special spaces in mathematics. 02 - FORMAL SETUP: The principle of empirical risk minimization is introduced along with famous error decompositions. Hypothesis spaces and their relation to the phenomenon of overfitting and underfitting are discussed. 03 - LINEAR REGRESSION: A vectorized hypothesis of linear relationship between features and lables is presented before formally introducing vector norms to measure the magnitude of error vectors. Via convex optimization of the mean squared error loss, we arrive at the normal equation, a closed form solution to the linear regression problem. Finally, polynomial regression as well as ridge regression are presented and solved. 04 - THE KERNEL TRICK: Using Mercer's theorem, kernels are introduced as a way to reduce the computational load of evaluating the Gram matrices present in polynomial ridge regression. It is demonstrated that the Gaussian kernel maps into an infinite dimensional feature space. Ridge regression is solved using the kernel formalism. 05 - REPRODUCING KERNEL HILBERT SPACES: Basic concepts of functional analysis are reviewed before providing a formally rigorous definition of kernels and the reproducing kernel Hilbert spaces (RKHS). Basic properties of functions in the RKHS are presented, including the representer theorem. 06 - GAUSSIAN PROCESSES: The formalism of Bayesian linear regression is introduced, leading to probability densities around our point estimates. Gaussian processes are defined and kernel matrices are used as covariance matrices. 07 - THE RELATIONSHIP BETWEEN KERNELS AND GAUSSIAN PROCESSES: Selected notions of linear algebra and measure theory are reviewed before examining the relationships between kernel methods as a frequentist idea and gaussian processes as a Bayesian idea. 08 - OUTRO: The resources used to compile the content are presented. Also, an outlook to the next module is given.