Parquet

Parquet (Apache Parquet) is an open file format for tables, designed for analytics. A CSV writes a table row by row; Parquet writes it column by column: all the dates together, then all the amounts, then all the countries. It is binary rather than plain text, so you cannot open it in a text editor, but almost every data tool reads it: Spark, Amazon Athena, pandas, DuckDB and the major data warehouses.

Storing by column pays off twice. First, a query that needs two columns out of fifty reads only those two and skips the rest of the file. Second, values in a column look alike (the same few countries repeated millions of times), so they compress very well, often to a fraction of the CSV size. Each file also records the minimum and maximum of every column for each block of rows, so a query for March can skip blocks that only contain January.

Parquet also keeps the schema inside the file: column names and types, so a date stays a date and nobody has to guess. That is why it became the default format of the data lake: engines like Athena charge by data scanned, and converting raw logs from CSV or JSON to Parquet can make the same query both faster and much cheaper. The trade-off is that it is made for writing once and reading many times; changing a single row means rewriting a file.