alt Hacker News> Five months ago, we rebuilt our warehouse on DuckDB instead of ClickHoue
They switched from ClickHouse to DuckDB five months ago, then switched from DuckDB to their own Trino derivative?
The article then ends before saying if the new setup solved their problems.
It’s great that we can now make these migrations with little downtime (I assume?) and try different things, but I wish we could have read more about what testing they did before jumping. The fact that they jumped to DuckDB without realizing it wasn’t going to work at their scale and spent only a few months with it before jumping again is not convincing.
I hoe there’s more to this story that just didn’t make it into the blog posts. As written it’s giving me flashbacks to some teams I worked with who changed tech stacks every six months because they liked migrating more than tuning and maintaining.
Replies
Things like this are a part of why I push back on people telling me we can do important things later. I don’t want to swap the engine out of a machine that’s moving that fast. There’s no time to learn about how it behaves with real versus synthetic traffic and what the failure modes are when you’re doing many thousands of transactions per second.
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
>The article then ends before saying if the new setup solved their problems.
This reads to me like a recruiting post; thats why they are open if it solved their problems. I'm not a fan of the way they framed it, it seems they have made their architecture decision and now need help implementing this.
I'd imagine if you are the kind of person that wants to work on large scale data query problems this is an invite for you to apply. That said, if you are that person it might not be ideal to join after all the fun decisions have been made.
Founder here -- this is about our warehouse product which is separate from our main analytics DB. Our DuckDB warehouse was used by a handful of pilot customers. Our clickhouse database is serving all of our analytical queries.
It reminds of people who use ORMs, never pop the hood to analyze the queries, indexes, etc - then decide it's the database that's the problem.
> The article then ends before saying if the new setup solved their problems.
They got to put ClickHouse, DuckDB and Trino on their resume & LinkedIn. Problem solved!