alt Hacker NewsCool, I can definitely see this smoothing the path towards a full DW solution, assuming that is ever needed. Could you see it working with something like dbt, say doing transformations in a dedicated pg database then serving the transformed data to users via the read replica?
Out of interest, do you know any good resources covering the current state of data engineering? I find the area quite impenetrable compared to software engineering. Almost like much of it is trade secrets and passed down knowledge and none of it written down.
Our plan is to make BemiDB work with dbt by leveraging the Postgres-compatibility (supported dbt adapters https://docs.getdbt.com/docs/trusted-adapters). So it should be possible to transform data from Postgres or directly from BemiDB, which may actually perform better.
You're right, the data engineering world is complex, constantly evolving, and has many various solutions. I'd also like to know about any good resources that people use :)
For us, we mostly talked to many potential users asking about their data setups and challenges, and had many conversations with friends and experts in this field. I also read a few weekly newsletters, substracks, and follow people in this space on X (many recently started posting on Bluesky). For a deeper research, reading docs and specs, experimenting, watching talks, listening to podcasts, reading subreddits, etc.