Monitoring Databricks with System Tables

In this video I focus on a different side of monitoring: What do the Databricks system tables offer me for monitoring? How much does this overlap with the application logs and Spark metrics? Databricks System Tables are a public preview feature that can be enabled if you have Unity Catalog on your workspace. I introduce the concept in the first 3 minutes then summarize where this is most helpful in the last 3 minutes. In between are some example queries and table explanations. * All thoughts and opinions are my own * Blog post with more detail: https://dustinvannoy.com/2024/02/22/databricks-monitoring-with-system-tables Options to enable system tables: Curl - https://learn.microsoft.com/en-us/azure/databricks/administration-guide/system-tables/#--enable-system-table-schemas dbdemos notebook - https://www.databricks.com/resources/demos/tutorials/governance/system-tables?itm_data=demo_center Useful blogs: https://medium.com/@24chynoweth/databricks-system-tables-an-introduction-e11a06872405 https://www.databricks.com/blog/improve-lakehouse-security-monitoring-using-system-tables-databricks-unity-catalog Useful docs: https://learn.microsoft.com/en-us/azure/databricks/administration-guide/system-tables/ https://learn.microsoft.com/en-us/azure/databricks/administration-guide/account-settings/audit-logs More from Dustin: Website: https://dustinvannoy.com LinkedIn: https://www.linkedin.com/in/dustinvannoy Github: https://github.com/datakickstart CHAPTERS 0:00 Intro 1:15 Enable system tables 3:28 Billing usage query 6:55 Docs highlights 7:57 Lineage query 9:02 Audit query 11:17 Additional tables 13:20 Compare to Log4J/Metrics 14:08 Summary