Why Parquet Dominates Modern Data Engineering
·12 mins
Changing one line from CSV to Parquet makes reading the fares of ten million New York taxi trips 27 times faster, and the file is a sixth of the size. In this chapter, we measure why with Polars, DuckDB, PyArrow, and pandas: columnar storage, types in the file, the footer, encodings, pages, compression, predicate pushdown, and row group size. We also see where row storage still wins.