Pandas Should Go Extinct is a bold title for a blog post making the case that Python’s beloved DataFrame library has outlived its usefulness. The argument is simple: Pandas forces developers into distributed querying systems sooner than they should go there, pushing them toward expensive tools like Spark, DataBricks, Snowflake, or Dask before their workloads actually demand it. The writer’s thesis is that most teams will never reach the scale where those systems matter, and that the gap between the “Pandas cliff” and genuine big-data territory can be bridged with modern, single-machine tools like Polars and DuckDB.
The post, which the writer delivered as a talk at Latency Conference, opens with a straightforward warning. Pandas is not the problem for the cute, cuddly bears used for international diplomacy. It is the problem for the Python DataFrame library that millions of analysts depend on daily. The cliff arrives at roughly 100 gigabytes of data, and the writer argues that crossing it is unnecessary for most teams.
Adoption Pathway
The writer sketches a rough guide of when teams typically move from one DataFrame library to another based on data size. People start with Excel, graduate to Pandas somewhere in the gigabyte range, and stay with Pandas through tens of gigabytes. Then memory issues, slow computation, or frustration with Pandas’ complex API set in. The natural next step is to move to a “real” tool — the kind priced for enterprise budgets — built for what the post calls Big Data ™️.
The cliff is the moment where Pandas stops being enough. Memory runs out, computations take forever, and the API starts feeling baroque. The traditional answer is to upgrade to a distributed system. But the writer argues that distributed systems are often sold as silver bullets, and that the threshold for genuine necessity is much higher than teams tend to believe.
The 100GB Threshold
The core claim is that the gap between the Pandas cliff and the scale where distributed systems are genuinely necessary sits around the 100GB mark. That gap, the writer argues, can be effectively filled by modern, high-performance, single-machine tools. The two tools named specifically are Polars and DuckDB.
The writer backs this up with a look at how much “Big Data” actually exists in the wild. In 2024, Amazon published a paper entitled “Why TPC is not enough: An analysis of the Amazon Redshift fleet.” The paper compared telemetry data from Amazon’s distributed analytics database, Redshift, with industry-standard database benchmarks. As part of that analysis, Amazon published fleet statistics on query run times and table sizes.
What Amazon’s Paper Shows
The paper’s findings are striking. Assuming an average row size of 1KB in a Redshift table and clusters of 10 machines each guzzling data at 8GB/s from S3, the numbers break down like this:
- 94.68% of tables in the Redshift fleet contain fewer than 100GB of data
- 86.9% of queries operate on 80GB of data or less
Those figures come from summing the first three rows of the runtime table and taking the assumption that 10 machines at 8GB/s can handle 80GB in one second. The assumption of 8GB/s is based on an admittedly outdated benchmark, and the assumption of 1KB per row is optimistic. Even at 10KB per row, the math still lands at a table size of 1TB.
Jordan Tigani of MotherDuck did a deep dive into the dataset, though the writer notes that MotherDuck is a SaaS business selling DuckDB hosting, so some scepticism is warranted.
Medium Data Problem
The takeaway is blunt. Most teams do not have Big Data, and probably never will. They have Medium Data problems, and need Medium Data solutions. The distributed systems pitched as answers to Big Data are often overkill for workloads that stay comfortably below the 100GB threshold.
The writer’s framing is that teams hit the Pandas cliff and reflexively jump to a distributed system without stopping to ask whether they need one. The gap between the cliff and genuine necessity is real, and it can be closed with tools that run on a single machine.
Polars and DuckDB
The writer proposes two alternatives to Pandas. Polars is a Rust-based DataFrame library that feels familiar to Pandas users but differs in several important ways. DuckDB is an in-memory analytics DB — essentially SQLite for analytics.
To demonstrate the difference, the writer points to the 1 Billion Row Challenge. The challenge asked for the fastest Java program to compute the min, mean, and max of a 1 billion row CSV containing weather station data. The fastest accepted implementation ran in 1.5 seconds.
The original challenge used a bare-metal Hetzner AX161 server with 32 cores and 128GB of RAM running Debian 12. For these tests, the writer used an m7a.8xlarge from AWS instead, also running Debian 12. The configuration is the same — 32 cores, 128GB of RAM, an AMD CPU — but the writer notes that not using bare-metal dedicated hardware may affect reproducibility somewhat.
The Code Comparison
The writer includes implementations of the 1 Billion Row Challenge in Pandas, Polars, and DuckDB. The Pandas version reads the CSV, groups by the weather station, and computes the aggregate min, mean, and max figures. The key point is that Pandas executes each step sequentially and eagerly — it reads in the entire dataset, groups it, and then performs the aggregation.
python
def do_1brc_pandas(file_path: str):
df = (
pd.read_csv(file_path, sep=";", names=["station", "measurement"])
.groupby("station")
.agg({"measurement": ["min", "mean", "max"]})
.round(2)
)
The output serialisation specified in the original challenge is skipped for performance tests. The implementations all include the ability to serialise the output, which was used to unit test the code. Given that the output format for the 1 Billion Row challenge is non-standard, the writer did not test serialisation across the libraries.
Verdict on Pandas
The post’s argument is that Pandas should go extinct because it forces premature adoption of distributed querying systems. The writer acknowledges that the back-of-envelope calculations are shaky ground and that the assumptions are optimistic. But the central claim stands: the gap between the Pandas cliff and the 100GB mark is real, and it is filled by tools that do not require a full distributed architecture.
The scepticism caveat applies to the MotherDuck deep dive, but the writer’s own scepticism about the assumptions is part of the story. The post is not a sales pitch for Polars or DuckDB. It is a warning about the costs of staying with Pandas past the point where it makes sense.
The gap between the Pandas cliff and the 100GB mark is real, and the tools that fill it exist today. The question is whether teams are willing to let Pandas die.
Get the Notebook.
The day's best stories and every fresh verdict, in plain English, in your inbox by seven. One email a day, no more.