Why not use C++?
For the most part, these aren't security critical components.
You already have a massive amount of code you can use like say llama.cpp
You get the performance that you do with Rust.
Compared to Python, in addition to performance, you also get a much easier deployment story.
Rust is much better than C++ overall and far easier to debug (C++ is prone to very difficult to debug memory errors which don't happen in Rust).
The main reasons to use C++ these days are compatibility with existing code (C++ and Rust are a bit of a pain to mix), and if a big dependency is C++ (e.g. Qt).
Lots of reasons, but a big one is that dependency and build management in C++ is absolutely hellish unless you use stuff like Conan which nobody knows. In Rust, you use Cargo and everyone is happy.
I've worked with C++ in the past, it's subject to taste. I like how Rust's rigidness empowers rapid change _without_ breaking things.
Besides, the ML ecosystem is also very mature. llama.cpp has native bindings (which Swiftide supports), onnx bindings, ndarray (numpy in Rust) works great, Candle, lots of processing utilities. Additionally, many languages are rewriting parts in Rust, more often than not, these are available in Rust as well.
Why use C++? What's the benefit over Rust here?
If you already have substantial experience with C++, this could be a good option. But I'd say nowadays that learning to use Rust *well* is much easier than learning to use C++ *well*. And the ecosystem, even if it's a lot less mature, I'd say is already better in Rust for these use-cases.
Indeed, here security (generally safety) is a secondary concern and is not the main reason for choosing Rust, although welcome. It's just that Rust has everything that C++ gives you, but in a more modern and ergonomic package. Although, again, I can see how someone already steeped in C/C++ for years might not feel that, and reasonably so. But I think I can farely safely say that Rust is just "a better C++" from the perspective of someone starting from scratch now.