05 - REPRODUCING KERNEL HILBERT SPACES - INTRODUCTION TO REGRESSION AND KERNEL METHODS

************************************************************************** BECOME ONE OF THE FIRST STUDENTS OF THE NEW STANDARD MACHINE LEARNING CURRICULUM! https://forms.gle/baxK3Cwz8q2xuJEt5 ************************************************************************** Timestamps for this lecture: 00:00 Inner Product Axioms 03:28 Simple Consequences of these Axioms 06:58 Inner Products induce Norms 10:56 Recap: Hilbert Spaces 12:53 Recap: Cauchy Sequences Intuition 14:02 Example of a non-complete Metric Space 15:42 Recap: An important Example of a finite-dimensional Hilbert Space 19:54 Recap: Infinite-dimensional Hilbert Spaces 31:49 Formal Definition of Kernel Functions 33:33 Partial Evaluation of Kernel Functions 36:48 Constructing a Vector Space G from these Basis Functions 38:12 Defining an Inner Product on this Vector Space G 40:21 Conditions for which this Inner Product is a proper Inner Product 42:52 Turning G into a proper Hilbert Space 43:47 Definition of Reproducing Kernel Hilbert Spaces 47:12 Evaluation Functionals 49:12 An Example of a non-continuous Evaluation Functional 53:32 The Functions in RKHS are well-behaved 55:25 Some other nice properties of RKHS 57:42 The Representer Theorem 1:01:33 Proof: The Representer Theorem Get the lecture slides: https://drive.google.com/file/d/1nIqISTpYxBYsl4ym2Hn95dKmVdXsX127/view?usp=sharing There is a shortage of well-presented learning resources for rigorous Machine Learning. In the FUNDAMENTALS OF MACHINE LEARNING program @ äon intelligence, laser-focused modules are provided for competent introductions, as well as in-depth discussions of advanced content. The INTRODUCTION TO REGRESSION AND KERNEL METHODS module provides the ideal introduction to the field of Machine Learning for anyone with a reasonable level of mathematical maturity. The contents of this module include: 01 - PREREQUISITES: Basic concepts of linear algebra and multivariable calculus are stated and practised with simple examples. Outlook on prerequisites of advanced content is given, including probability theory, eigenvalues and their eigenvectors, matrix decompositions, and special spaces in mathematics. 02 - FORMAL SETUP: The principle of empirical risk minimization is introduced along with famous error decompositions. Hypothesis spaces and their relation to the phenomenon of overfitting and underfitting are discussed. 03 - LINEAR REGRESSION: A vectorized hypothesis of linear relationship between features and lables is presented before formally introducing vector norms to measure the magnitude of error vectors. Via convex optimization of the mean squared error loss, we arrive at the normal equation, a closed form solution to the linear regression problem. Finally, polynomial regression as well as ridge regression are presented and solved. 04 - THE KERNEL TRICK: Using Mercer's theorem, kernels are introduced as a way to reduce the computational load of evaluating the Gram matrices present in polynomial ridge regression. It is demonstrated that the Gaussian kernel maps into an infinite dimensional feature space. Ridge regression is solved using the kernel formalism. 05 - REPRODUCING KERNEL HILBERT SPACES: Basic concepts of functional analysis are reviewed before providing a formally rigorous definition of kernels and the reproducing kernel Hilbert spaces (RKHS). Basic properties of functions in the RKHS are presented, including the representer theorem. 06 - GAUSSIAN PROCESSES: The formalism of Bayesian linear regression is introduced, leading to probability densities around our point estimates. Gaussian processes are defined and kernel matrices are used as covariance matrices. 07 - THE RELATIONSHIP BETWEEN KERNELS AND GAUSSIAN PROCESSES: Selected notions of linear algebra and measure theory are reviewed before examining the relationships between kernel methods as a frequentist idea and gaussian processes as a Bayesian idea. 08 - OUTRO: The resources used to compile the content are presented. Also, an outlook to the next module is given.