Iceberg

A folder of Parquet files is not a table. An interrupted write leaves a third of a month behind, two files with a different timestamp unit break the read, and a count that runs during a rewrite returns 82 million taxi trips instead of 41. In this chapter, we see how Apache Iceberg turns the same files into a table with ACID guarantees: manifests, manifest lists, metadata files, and a catalog, then hidden partitioning, atomic commits, time travel, schema evolution, and compaction, measured on New York’s taxi trips with PyIceberg, Polars, and DuckDB. We also compare Iceberg with Hive tables and Delta Lake.