Invariably when local AI comes up, people try to see how it can be used for frontier-level AI tasks like asking it to one shot a web app from scratch and then they get disappointed when they discover it's not performing at the same level as Opus.
But they're ignoring that a local LLM can still be very useful for any kind of repetitive task, things like categorization or generating embeddings, translation or summarization tasks. I'm self-hosting Karakeep and use a local LLM for the AI tasks like embedding.
And once you've got that local LLM running, you can start thinking about integrations with your own custom built apps.
Also, they are thinking of one single LLM model at a time. What if we have a network of dynamically loaded micro-models loaded on demand based on the user intent? Think of a global DNS of micro-LLMs. We need to decentralize AI. I started building this here https://modelxare.com