logoalt Hacker News

mrtimotoday at 5:40 PM3 repliesview on HN

I've moved from python/polars/pandas to DuckDB and have not looked back


Replies

viccistoday at 6:56 PM

Same. Almost every time I would use its streaming interfaces in Python, it would STILL materialize everything into memory. That was like 6 months ago. Maybe streaming interfaces actually work, but I found them to be leaky abstractions that required a ton of hand holding to make sure they didn't build a bunch of memory pressure, if you're lucky enough to even have a way to do it.

For example, last time I used it, you couldn't do NDJSON streaming scans from S3 (looks like fixed with PR #26563).

sirfztoday at 7:57 PM

Yep, and chdb as well. Said this before, chdb's DatStore is a pretty neat pandas replacement too.

ismailmajtoday at 5:43 PM

even for just in-memory quick data analysis?

show 1 reply