alt Hacker NewsThe databases we use today in production have severe limitations and are not even close to what is theoretically possible. Many traditional parts of a database (indexing, caching, scheduling, et al) are AI-complete algorithm problems. Entire sub-classes of database (e.g. graph or spatial) famously have persistently poor scalability and performance because of open questions in the foundational computer science.
Just the fact that increasing the generality, scalability, and performance of databases asymptotically converges on designing AGI suggests that it is, in fact, "deep tech". And this property has to mesh with other practical constraints on database behavior. Many problems in databases are hard with little forward progress in decades.
It is true that most database research is not deep tech but there is ample room for it to be if one is sufficiently ambitious.
Replies
Could you explain what practical research there is to be done? The heavy theory I know does not seem to be very useful in practice. Optimal join algorithms, Yannakakis adjacent algorithms, tree decomposition of queries all seem to be worse than well implemented naive algorithms. But maybe the implementations of the new algorithms just are not good? I really don’t know.
I personally built quite several specialized hyper-performant DB engines, and studied most popular OSS projects, and believe most DB questions are theoretically answered long ago(decade back). The puzzle is mostly to assemble pieces together to fit specific tradeoff of performance/simplicity/functionality and not overengineer system.