alt Hacker NewsSub-millisecond? Literally nobody in large data analysis is doing that. Can you cite some sources or show me what sort of setups (with any DB tech) that you're able to run meaningful queries on S3 sources with even sub-second latency?
Sub-millisecond? Literally nobody in large data analysis is doing that. Can you cite some sources or show me what sort of setups (with any DB tech) that you're able to run meaningful queries on S3 sources with even sub-second latency?
I'm doing this with clickhouse querying parquet files on S3 from an EC2 instance in the same region as the S3 bucket (yes DuckDB pretty similar). S3 time to first byte within AWS is 50ms and I get close to saturating an big EC2 instance's 100Gb link doing reads. For OLTP type queries fetching under 1 MB you'll see ~4 round trips + transfer time of compressed data so 150-200 ms latency.