The Problem with Gradient Descent #SoME3

Gradient Descent is the backbone of modern Machine Learning. However, it's far from perfect and has a major problem that prevents it from being used in most of the real life applications. In this video, I'll start with the very basics of Mathematical Modeling, and then use Linear Regression to explain Gradient Descent. I'll also show the problem with using Gradient Descent and then explain a quick way to fix the issue. Code Repository: https://github.com/tgautam03/Algorium/blob/master/GD_Problem/LinearRegression.ipynb I'm also building a Deep Learning Framework (inspired from the likes of PyTorch and JAX) from scratch in Julia. You can check out the development progress on this project using the following link JAC Repository: https://github.com/tgautam03/jac.jl Credits: - Music: Moonlight by Kris Keypovsky can be found at https://freemusicarchive.org/music/kris-keypovsky/single/moonlight/ - Heat animation: https://www.youtube.com/watch?v=6jQsLAqrZGQ - EQ animation: https://youtube.com/shorts/pRoA7gC--no?feature=share Timestamps: 00:00 - Introduction 00:46 - Physics Based Modeling 01:20 - Data Based Modeling 04:10 - Cost Function 05:56 - Gradient Descent 09:00 - Gradient Descent with Momentum 10:38 - Conclusion