Azure Spot VMs are incredibly cheap CPUs that come with the risk of being evicted if enough demand for full-price CPUs occurs in the region. Luckily, Spark is a resilient distributed system that can easily handle replacing nodes, and so we're left with a very cost effective approach to provisioning lower-priority workloads!
In this video, Simon walks through the process for provisioning a cluster using Spot VM workers, how to get to the lower-level configuration and some of the gotchas to be aware of!
As mentioned in the video, we're supporting International Trans Day of Visibility, so please check out our blog for more information here: https://www.advancinganalytics.co.uk/blog/2021/3/31/international-trans-day-of-visibility-2021-and-advancing-analytics
And for further details on Spot VMs in Databricks, check out the announcements here: https://docs.microsoft.com/en-gb/azure/databricks/release-notes/product/2021/march#save-your-azure-databricks-tco-with-azure-spot-vms--public-preview
As always, don't forget to Like, Subscribe and Love one another!
