Apples, Oranges, and ML Models: Model Validation vs Benchmarking

Title: Apples, Oranges, and ML Models: Model Validation vs Benchmarking- DevConf.IN 2026 Speaker(s): Gaurav Kamathe --- In the rush to operationalize machine learning, teams often celebrate “great benchmark results” while overlooking whether their model has truly been validated for its intended purpose. The result? Impressive numbers that crumble in real-world deployment — models that outperform baselines but underperform expectations. This talk explores the subtle — yet crucial — difference between model validation and model benchmarking. While both rely on similar metrics, they answer fundamentally different questions. We’ll unpack how these two processes differ in goal, methodology, and risk management, using simple mental models and relatable real-world analogies. You’ll learn how to design evaluation workflows that distinguish between proving correctness and proving competitiveness — and why this distinction is essential for reproducibility, transparency, and trust, especially in open-source and collaborative ML environments. --- Full schedule, including slides and other resources: https://pretalx.devconf.info/devconf-in-2026/schedule/