The implementation of a language model is usually good enough for running _a single forward pass_ through the model, but to host it via an inference engine you typically need to convert a few operations. For instance the MLP can be easily split across GPUs (tensor parallelism) and MoEs also have a way of parallelizing.

Most of it is pretty standard, since not that many different layers and primitives are used in LLM architectures, but once in a while something new comes along that needs more effort. MoEs are one example, they are sparse and allow for completely different inference patterns, which takes a while to figure out.

Last year I started a blogpost series about this topic (that I hope to update some time). I start from a minimal gpt implementation by Karpathy and build the engine around it, you might like it: https://pieter.ai/blog/2025/nanogpt-inference/