logoalt Hacker News

trombonechamptoday at 11:31 AM5 repliesview on HN

Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.


Replies

nemothekidtoday at 2:10 PM

If the correctness of my program depends on the ordering of data (for operations that would otherwise be commutative), that seems like that should be something explicit rather than implicit.

I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.

sanderjdtoday at 1:58 PM

Is "non-deterministic" the right description for this? I read it as describing an implementation where ordering is not preserved, but deterministically. Is that a misreading?

Permiktoday at 1:13 PM

This is a tricky field, the problem is not actually the non-determinism of the processing algorithms, but implicit ordering of the data.[1] The implicit ordering of the data is a footgun that -- as seen in the paper -- has already claimed victims. Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.

[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.

show 1 reply
hopfenspergerjtoday at 12:41 PM

It's standard sql behavior, users always specify the ordering they want as part of the query.

winniewinnietoday at 12:12 PM

Perhaps to get better results on benchmarks.

show 1 reply