I was asking the same question, turns out mistral.rs [0] has pretty good abstractions in order to not depend and package llama.cpp for every platform.

[0] https://github.com/EricLBuehler/mistral.rs