L34: Scheduling learning rate using decay & line search

Welcome to Lecture 34 of the course "Deep Learning" by Prof. Mitesh M.Khapra Full Course: https://study.iitm.ac.in/ds/course_pages/BSCS3004.html Video Overview In this lecture we focus on practical techniques to optimize gradient descent for more efficient and effective training. We begin by identifying the limitations of simply increasing the learning rate and explore strategies to dynamically adjust both learning rate and momentum during the training process. Key techniques such as step decay and validation loss based learning rate reduction are explained along with an adaptive method to fine tune momentum. The lecture then introduces the concept of line search where multiple learning rates are tested at each iteration to identify the most effective step size. This method allows for faster convergence and better stability when compared to standard gradient descent. By the end of the session you will have a toolkit of actionable strategies to tune hyperparameters and improve the overall performance of gradient based optimization. About IIT Madras' online Bachelor of Science programme IIT Madras offers four-year BS programmes that aim to provide quality education to all, irrespective of age, educational background, or location. The BS programme has multiple levels, which provide flexibility to students to exit at any of these levels. Depending on the courses completed and credits earned, the learner can receive a Foundation Certificate from IITM CODE (Centre for Outreach and Digital Education), Diploma(s) from IIT Madras, or BSc/BS Degrees from IIT Madras. For more details, Visit: https://www.iitm.ac.in/academics/study-at-iitm/non-campus-bs-programmes #gradientdescent #momentum #Nesterov #learningrate #optimization #linesearch #machinelearning #deeplearning #Adam #AdaGrad #stochasticgradientdescent #minibatch #hyperparameters #tipsandtricks #adaptivelearning #trainingstability #lossmonitoring #lrdecay #gradientoptimization #convergencespeed #optimizers #modeltraining #batchtraining #neuralnetworktraining #dynamicupdates #learningratetuning #trainingstrategies #mlalgorithms #deeplearningtechniques