​Simplifying Iceberg Ingestion and Table Maintenance

Meetup: Seattle (October 27, 2025) Speaker: Sida Shen LinkedIn: https://www.linkedin.com/in/sida-shen-165303193/ Slides: https://docs.google.com/presentation/d/1Je8Be7jSRAbhzjNMu2L9Mj7jTMw0iD-W/edit?usp=sharing&ouid=108557621002966898161&rtpof=true&sd=true ​Working with Apache Iceberg often means juggling extra services for ingestion pipelines and background compaction. But those workflows don’t have to be so heavy. This talk dives into practical strategies for reducing small-file problems and keeping data immediately queryable, from writing optimally sized files at ingest to triggering compaction only when it’s actually needed. We’ll share benchmark results that highlight what’s possible in open source Iceberg today: up to ~5× faster writes on highly partitioned tables, ~100× fewer small files, and stable performance with no OOMs even with thousands of partitions. We’ll look at how a modern query engine can bring these techniques together, cutting down on extra services while still keeping tables healthy." ​ Apache Iceberg, Apache, Iceberg, the Iceberg logo, and the Apache feather logo are either registered trademarks or trademarks of the Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including the Apache Software Foundation.