alt Hacker NewsIt may be like what happened in ResNets using blank space in the image as working memory (because they didn't have any), so they would use non-important parts as a scratchpad.
It may be like what happened in ResNets using blank space in the image as working memory (because they didn't have any), so they would use non-important parts as a scratchpad.
There's a great visualization of this at 28:45 in this video (starting at 23:45 may give good context)
https://youtu.be/QgH9sr7G13Q?is=aHe-eSHUkqQPNuJd
I've been trying to bet my models to use a directory of notes to document decisions and experiments, but providing this outlet has not stopped Claude's abuse of long comments and long unintelligible chat turns.