alt Hacker NewsWhat kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
What kind of work are you doing? For example, if I have some code in the hot path and I want to do all the usual tricks to help the compiler vectorize it, such a small model is not able to do much.
RAG is your friend (or any vector db). No model can vectorize an entire codebase in context.
Even a big mainstream product (like Gemini) cannot handle more than ~1k lines without missing details and making mistakes. And about every 1k lines, it seems to forget the previous 1k, doesn’t it? So you can never hold more than a file or 2 (or 3) in context at a time without losing details.
What you find is that the big models like Gemini are doing vector storage and retrieval too, and breaking prompts down into chunks for various models to handle to assemble a thorough response.
If you want that kind of control in your outputs, and be able to hold a lot in your inputs, I don’t see any other way regardless of which model you use.