03 - LINEAR REGRESSION - INTRODUCTION TO REGRESSION AND KERNEL METHODS

************************************************************************** BECOME ONE OF THE FIRST STUDENTS OF THE NEW STANDARD MACHINE LEARNING CURRICULUM! https://forms.gle/baxK3Cwz8q2xuJEt5 ************************************************************************** Timestamps for this lecture: 00:00 Problem Statement 04:22 Vectorizing our Hypothesis (Single Observation) 06:13 Vectorizing our Hypothesis (Full Dataset) 09:23 Finding the Vector of Prediction Errors 10:24 Norm Axioms 14:14 Proof: Norms are convex in the Vector Elements 16:52 Metric Axioms 19:15 Metrics induced by Norms 20:49 Proof: Metric induced by Norm is, indeed, a Metric 25:22 The p-Norms 26:14 Proof: The p-Norms are, in fact, Norms 29:16 The L1-Norm 32:16 The L2-Norm 34:04 Proof: The infinity Norm is the maximum Element 36:43 The L0-Norm 38:07 Intuition: Unit Balls with the p-Norms 41:38 Mean Squared Error (MSE) as Empirical Risk 44:51 Intuition: Convexity 48:59 Second Derivative Test 52:26 Proof: MSE is convex in w 1:00:07 Proof: Solution to Linear Regression 1:02:28 Polynomial Regression 1:10:05 Ridge Regression 1:15:34 Proof: Solution to Ridge Regression Get the lecture slides: https://drive.google.com/file/d/1nIqISTpYxBYsl4ym2Hn95dKmVdXsX127/view?usp=sharing There is a shortage of well-presented learning resources for rigorous Machine Learning. In the FUNDAMENTALS OF MACHINE LEARNING program @ äon intelligence, laser-focused modules are provided for competent introductions, as well as in-depth discussions of advanced content. The INTRODUCTION TO REGRESSION AND KERNEL METHODS module provides the ideal introduction to the field of Machine Learning for anyone with a reasonable level of mathematical maturity. The contents of this module include: 01 - PREREQUISITES: Basic concepts of linear algebra and multivariable calculus are stated and practised with simple examples. Outlook on prerequisites of advanced content is given, including probability theory, eigenvalues and their eigenvectors, matrix decompositions, and special spaces in mathematics. 02 - FORMAL SETUP: The principle of empirical risk minimization is introduced along with famous error decompositions. Hypothesis spaces and their relation to the phenomenon of overfitting and underfitting are discussed. 03 - LINEAR REGRESSION: A vectorized hypothesis of linear relationship between features and lables is presented before formally introducing vector norms to measure the magnitude of error vectors. Via convex optimization of the mean squared error loss, we arrive at the normal equation, a closed form solution to the linear regression problem. Finally, polynomial regression as well as ridge regression are presented and solved. 04 - THE KERNEL TRICK: Using Mercer's theorem, kernels are introduced as a way to reduce the computational load of evaluating the Gram matrices present in polynomial ridge regression. It is demonstrated that the Gaussian kernel maps into an infinite dimensional feature space. Ridge regression is solved using the kernel formalism. 05 - REPRODUCING KERNEL HILBERT SPACES: Basic concepts of functional analysis are reviewed before providing a formally rigorous definition of kernels and the reproducing kernel Hilbert spaces (RKHS). Basic properties of functions in the RKHS are presented, including the representer theorem. 06 - GAUSSIAN PROCESSES: The formalism of Bayesian linear regression is introduced, leading to probability densities around our point estimates. Gaussian processes are defined and kernel matrices are used as covariance matrices. 07 - THE RELATIONSHIP BETWEEN KERNELS AND GAUSSIAN PROCESSES: Selected notions of linear algebra and measure theory are reviewed before examining the relationships between kernel methods as a frequentist idea and gaussian processes as a Bayesian idea. 08 - OUTRO: The resources used to compile the content are presented. Also, an outlook to the next module is given.