logoalt Hacker News

anentropic11/07/20241 replyview on HN

I'm looking for low latency queries over not-very-big data (40-100M rows) in user-facing dashboards

How does the latency of Iceberg-on-S3 compare to say an EBS volume?


Replies

exAspArk11/08/2024

I'd say that querying data from S3 is not ideal when low-latency queries are required. Generally, there could be a few roundtrip requests to fetch metadata (JSON, Avro) and data (Parquet) files, which may lead to around 1s or so latency. However, we have caching on our roadmap (it could be just a simple TTL for the fetched data or some more sophisticated caching depending on the synced & queried data)

show 1 reply