logoalt Hacker News

iluzoneyesterday at 8:49 PM1 replyview on HN

That's a fair point, and honestly disk cost rarely is the blocker these days. ~4GB assumes a few bytes per event, in our case it is closer to 100GB a day. That would be stored as raw text. Being able to rebuild that would require us to have some kind of pipeline that can do that effectively and that kind of defeats the point of Trifle.

That said, it really depends on the use case. For us the last 24h-48h holds the most value and anything older than that is just a reference where margin of error is acceptable. Even if we correct the historical data from 2 weeks ago, it wouldn't change the decisions we make today. For others it may be the opposite.

Tbh nothing stops you from doing both. Trifle is just a library that writes into your own database. If your data requires occasional re-analysis, write them into their own events table. Then you have the ability to rebuild the stats as you need to. I've seen both sides of the table, and as Trifle code lives with the rest of the code, it is simpler to handle this yourself than try to do this somehow via the App (which is optional in the first place) and UI.


Replies

freakynittoday at 3:29 AM

Mentioning below in case it helps someone looking into such estimates:

For a website analytics product that I was building, where every page-view/event stores many values such as browser, device, os, page-params, referrer.. and other typical stuff, it was coming at about 200GB/Billion events, after DB-level compression... the database obviously was columnar.

At these levels of storage needs, things start to cost real deal. With expected monthly intake of just around 10 Billion events, that ramps up to 2TB added every month.

Because of this, I had to move to S3-scaled/backed, nvme-cached architecture, but, that made query times fluctuate to some noticeable extent.

So, that ~4GB-only is for teeny-tiny data per metric sample.. probably 3-4 numeric values at max.