alt Hacker NewsI found this post absolutely fascinating! I got thinking, is there a Turing-complete simple abstraction that is differentiable, without approximation? I have heard about Neural Turing machines (but only as a layman), but from a quick peek they are similarly hard to reason about as other NNs?
Not exactly an answer to your question, but in the ballpark and hopefully you might find interesting https://arxiv.org/pdf/2404.17625
In the section on "Automatic differentiation" he considers the question of differentiable computation graphs.