alt Hacker News> Not just reading data from S3, but the actual computations too.
Async io generally causes a CPU bump as a side-effect, when the bottleneck moves from the disk to the CPU (more usually memory bandwidth).
My experience is from a different SQL engine, but the improvement in scan rates directly translates all other operators getting more work to do per unit time.
More CPUs pegged at 100% is always a good sign of smooth flows, but optimizing further requires data layout changes from that point. Even if it is a sub-linear speedup out of concurrency scaling - more with more is still a win, assuming you can pay for more (the Dy-skew paper has an example I can quote).
Yep that makes sense.
One thing we've seen with DuckDB, which they also mention in their docs, is that it's possible to 'over parallelize'. Beyond some number of cores, the overall workflow measured in wall time actually slows down when you add more CPUs, because of the overhead of parallelizing, despite them running at 100%.
We observe this even if we pull the data from S3 onto the local SSD before processing,