logoalt Hacker News

skeeter2020yesterday at 3:44 PM2 repliesview on HN

>> some performance comparisons vs sqlite.

That's not really the purpose; it's really a language-independent format so that you don't need to change it for say, a dataframe or R. It's columnar because for analytics (where you do lots of aggregations and filtering) this is way more performant; the data is intentionally stored so the target columns are continuous. You probably already know, but the analytics equivalent of SQLite is DuckDB. Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.


Replies

mandeepjyesterday at 5:26 PM

> Arrow can also eliminate the need to serialize/de-serialize data when sharing (ex: a high performance data pipeline) because different consumers / tools / operations can use the same memory representation as-is.

Not sure if I misunderstood, what are the chances those different consumers / tools / operations are running in your memory space?

show 3 replies
actionfromafaryesterday at 4:59 PM

Thanks! This is all probably me using the familiar sqlite hammer where I really shouldn't.