DQX is a data quality framework for Apache Spark that enables you to define, monitor, and react to data quality issues in your data pipelines.
Chapters
00:00 - Introduction to DQX
01:33 - Understanding DQX
04:44 - DQX vs Lakehouse monitoring
06:20 - Requirement
10:10 - Live Demo
22:22 - Defining and applying custom checks
27:31 - Future enhancements
Documentation
Repos: https://github.com/databrickslabs/dqx
Documentation: https://databrickslabs.github.io/dqx/
Alex Ott: https://www.linkedin.com/in/alexott/
Marcin Wojtyczka: https://www.linkedin.com/in/marcinwojtyczka/
Youssef Mrini, Databricks, NextGenLakehouse
