Larget datasets with pandas

What you should know

Setting up

The problem with big data

Big data is dead

Big data systems

Load only what you need

Types

Categorical data

Arrow

Challenge: Daily passengers

Solution: Daily passengers

Iterative computation

SQL

Faster calculations

Challenge: Maximum passengers

Solution: Maximum passengers

dask

polars

Beefy VM

Challenge: Median ride distance

Solution: Median ride distance

Next steps