On the surface this seems like a terrible idea:
FFmpeg is mountains of extremely complex C code whose entire job is processing untrusted inputs.
Choosing to make such code network-enabled if you can't trust your inputs, I would recommend to sandbox if at all possible. Otherwise you are asking for trouble.
Thank you for your comment!
The usecase for something like this is when you control both sides, server & client. There is some basic HMAC auth built into each request.
> I would recommend to sandbox if at all possible.
Since the server is a standard binary that doesn't need any special permissions, you could create the most locked down user in your server that only has access to a limit set of files and the GPUs and it'll work just fine. This is encouraged.