logoalt Hacker News

pletnes • today at 3:43 PM • 2 replies • view on HN

What is the benefit of using async io for csv and parquet files?


Replies

skeeter2020 • today at 4:32 PM

Helps use all of the available bandwidth by not blocking (otherwise idle) worker threads waiting for data to arrive. Requests and processing are managed independently, so you essentially get better utilization & improved performance.

jtbaker • today at 3:55 PM

highest impact is when there is network overhead, like accessing them over S3/HTTP/S, and you're accessing many files as part of a query, like using a glob `-*.parquet` pattern. I imagine the same applies when you're doing local I/O, but not as pronounced of an improvement due to lower latency.

➕ show 1 reply