alt Hacker NewsI'm not the author, but I would like! Is it feasible to run on the same hardware as the 27b model itself?
Replies
jasonjmcghee • today at 2:30 PM
Fwiw you can just Google this for a model and often someone has done it
I'm not the author, but I would like! Is it feasible to run on the same hardware as the 27b model itself?
Fwiw you can just Google this for a model and often someone has done it
The using a J lens is super cheap compared to inference. You basically add a single matrix multiply per layer. You probably wouldn't even notice the overhead in a good implementation.
You should even be able to create a j lens from scratch, but it might take a while. I was able to do it in a few hours on an H100. Creating the J lens is basically the equivalent of calculating a few thousand training steps for a model (256,000 backprops in my case). I've got more details in a blog post:
https://blog.lwarfield.dev/layer-scope/
I'm currently at work and can't those matrixes up until I get home. I'll update this comment with a link later.