alt Hacker News> Compiled to native code for low-latency plugin execution – no interpreter overhead, no JIT, no process startup cost.
If you're running the compiled code in-process, how is that not JIT? And isn't that higher-latency than interpreting? Tiered-JIT (a la V8) solves exactly this problem.
Edit: Although the example programs show traditional AOT compile/execute steps, so "no process startup cost" is presumably a lie?
Replies
dana321 • today at 7:35 PM
They could have just used cranelift for jit, but no..
Mog is AOT-compiled, not JIT'd.
JIT means the code is interpreted until some condition kicks in to trigger compilation. This is obviously common and provides a number of advantages, but it has downsides too: 1) Code might run slowly at first. 2) It can be difficult to predict performance -- when will the JIT kick in? How well will it compile the code?
With Mog, you do have to pay the up-front cost of compiling the program. However, what I said about "no process startup cost" is true: there is no other OS process. The compiler runs in process, and then the compiled machine code is loaded into the process. Trying to do this safely is an unusual goal as far as I can tell. One of the consequences of this security posture is that the compiler and host become part of the trusted computing base. JITs are not the simplest things in the world, and not the easiest things to keep secure either. The Mog compiler is written entirely in safe Rust for this reason.
This up-front compilation cost is paid once, then the compiled code can be reused. If you have a pre-tool-use hook, or some extension to the agent itself, that code runs thousands of times, or more. Ahead-of-time compilation is well-suited for this task.
If this is used for writing a script that agent runs once, then JIT compilation might turn out to be faster. But those scripts are often short, and our compiler is quite fast for them as it is in the benchmarking that I've done -- there are benchmarking scripts in the repo, and it would be interesting to extend them to map out this landscape more.
Also, in my experience, in this scenario, the vast majority of the total latency of waiting for the agent to do what you asked it is due to waiting for an LLM to finish responding, not compiling or executing the script it generated. So I've prioritized the end-to-end performance of Mog code that runs many times.