AI Evals Explained — From Basics to Advanced (Full Masterclass)

In this video, we have discussed how AI Evals ACTUALLY Work That Will Change How You Build AI Products. *************** AI PM Course + AI PM Interview Prep *************** Become AI PM ► https://topmate.io/technomanagers/new/fK374qFpvL *************** Become a Job Ready AI PM in 12 Weeks *************** Become AI PM ► https://topmate.io/technomanagers/page/IpmTa0nW5e This is a 12-week, mentor-led live cohort for PMs, engineers & builders who want to ship a portfolio-ready AI product 0:00 - Introduction to AI Evals 07:55 - Deterministic Systems 09:47 - LLMs are NON-Deterministic in Nature 14:26 - Capability vs Consistency 22:21 - What are Evals? 23:14 - Types of Evals 34:48 - How to decide which eval to use? 38:14 - Reference vs Reference Free Evals 42:18 - Trace Analysis 52:35 - Steps to do Trace Analysis 57:11 - Open Coding vs Axial Coding 01:08:26 - How many traces to read? 01:22:34 - Evals for RAG 01:24:49 - Precision, Recall and MRR 01:45:37 - Deep Dive on LLM as Judge 01:52:48 - Cohen's Kappa & Mathew's Coefficient 02:02:07 - Confusion Matrix 02:03:30 - Position Bias 02:05:35 - Verbosity Bias 02:09:04 - Final Report Card for LLM as Judge 02:20:00 - Eval Demo ***************Tech & Strategy Newsletter*************** Subscribe here ► https://shaileshsharma.substack.com/ ***************AI/ML & Tech Concepts for Product Managers*************** Tech Concepts Made Easy ► https://topmate.io/technomanagers/1084615 ***************AI Strategy for Everyone (PMs, Founders, Leaders)*************** Strategy Book ► https://topmate.io/technomanagers/1472775 *************** PM Interview Mastery Course *************** Be the Top 1% Interview Candidate ► https://topmate.io/technomanagers/1470531 *************** For Brand Collaboration*************** Email Me: [email protected] Linkedin ►https://www.linkedin.com/in/shailesh-sharma/ ***************Join this channel to get access to the perks******** https://www.youtube.com/channel/UCHKKCi2VXl878ryw_seADFg/join