alt Hacker NewsSpeaking as long-term (>15 years) user of Postgres in science, I am getting worried about the lack of columnar type of storage in Postgresql. As the datasets become bigger and bigger, the limitations of PG's storage are becoming more and more significant. I know there are various extensions (i.e. cetus) that may offer such functionality, but then you depend on that extension being supported in the future, as well additional complexity.
Replies
You might be using the wrong database if that’s what you’re hoping for. Columnar databases are a separate category.
It’s like saying that you’re getting worried Apple doesn’t sell washing machines.
From a computer science perspective, I'm not exactly sure how a transactional database would implement a columnar type. Postgres + CDC + an actual analytical database like ClickHouse would be your strongest bet at scale.
"As the datasets become bigger and bigger,"
I think the better way is to use Postgresql for new data and routinely archive off older data to data warehouse type database, to keep the Postgres one small.
(Many companies also now use a RDBMS alongside either a KV database or document store in main app)
In this regard I do find DuckDB amazing.
Fine by me, as a 25 years user. More is not necessarily better, see redis.
I was about to post something similar.
There was a big fanfare about orioledb a while ago, and i think it got bought by people that wanted to push that into mainstream postgres?
Did it die somewhere along the road?
A bit of a shameless plug, but I've been working on this for a few months in the form of an extension: https://github.com/xataio/deltax
When I started I thought there's too much inherent overhead in using Postgres tables for storage and using the Postgres executor, so figured it would be pretty cool to match Timescale in performance. I didn't think it would be possible to get close to dedicated analytical DBs. But as the project progressed and the performance got better and better, I'm now firmly in the camp of doing analytics with Postgres + an extension.