I addressed this a little bit in the comment below, but the cycles add up. I'm doing some pretty crazy things higher up in the stack, that I'm not quite ready to release yet. But even micro optimizations here add up at the scale I'm working at to allow me the headroom I need. I'm relying on a high-frequency recursive agentic loop that chokes a real-time guarantee without every optimization I can give it.

I started by removing the IPC overhead from a weaviate db connection, and doing all my vector math in house with a lightweight sqlite db. This became the next obvious target for optimization once I saw how much that saved me doing things in-house.