logoalt Hacker News

Ozzie_osman • today at 3:00 PM • 3 replies • view on HN

  We sharded over 20 TB that we know about.
This is probably a typo, right? 20TB isn't that big. I would imagine they've sharded a lot more than that

Replies

singron • today at 4:41 PM

If your working set is 20 TB, then it's pretty big. Each database has its own mix of hot/cold data, so it's impossible to compare without more information. A better measure might be IOPS. RDS has fairly low maximum IOPS unless you spend a lot more for provisioned IOPS or use Aurora.

rbranson • today at 3:35 PM

You are correct. As a point of comparison: almost ten years ago at Segment we had a single Aurora PostgreSQL instance with ~50T of data, it was used to index potential identity data in a much larger corpus of files stored in S3.

GiorgioG • today at 3:13 PM

For a vast majority of use cases 20TB is positively enormous.

➕ show 5 replies