Database Internals - LSM Tree Compaction : In depth

Understanding Compaction Strategies in LSM Engines: Challenges and Optimizations In this video, we go deep into the compaction process within LSM (Log-Structured Merge) trees. Following up on the previous video, this episode explores the key challenges of space, read, and write amplification within LSM engines. It also examines various real-world compaction strategies like leveled, size-tiered, and time-window compactions, used by databases such as Cassandra, RocksDB, and InfluxDB. The video aims to provide insights into optimising LSM engines to enhance database performance, making it essential viewing for anyone interested in data storage and management. Watch the previous video on LSM Trees: https://youtu.be/CizxYuq-z6E Chapters: 00:00 Introduction and Recap 01:03 Understanding Amplification in Databases 06:47 Leveled Compaction Strategy 13:54 Size-Tiered Compaction Strategy 19:13 Time-Window Compaction Strategy 24:27 Real-World Database Implementations 27:51 Conclusion and Call to Action References: RocksDB Tuning Guide: https://github.com/facebook/rocksdb/wiki/RocksDB-Tuning-Guide Cassandra: https://www.datastax.com/blog/leveled-compaction-apache-cassandra ScyllaDB: https://www.scylladb.com/tag/incremental-compaction/ InfluxDB: https://docs.influxdata.com/influxdb/v2/reference/internals/storage-engine/ InfluxDB Compacter: https://www.influxdata.com/blog/compactor-hidden-engine-database-performance/ For memberships: join this channel as a member here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/join Don't forget to like, share, and subscribe for more insights! ============================================================================= Like building stuff? Try out CodeCrafters and build amazing real world systems like Redis, Kafka, Sqlite. Use the link below to signup and get 40% off on paid subscription. https://app.codecrafters.io/join?via=geeknarrator ============================================================================= Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #database #nosql #lsm #sql #writeaheadlog