Pre-Release of Polars 2.0

For me, the superpower of polars is production stability.

Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.

Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.

I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!

Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.

> We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience

I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.

I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!

Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.

Moving towards streaming and generally out-of-core is great

We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...

Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL.

But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS

I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.

The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?

Seeing "release will land in the following weeks" kind of immediately turns me off.

every major version of polars is a reminder that the API you finally memorized was always just a suggestion

What does this project have to do with Serbia? Are the developers in Belgrade?