Well, the server just responds with acknowledgement information, the client is doing most of the work and most of the complex parts of RTMP are hand waved and faked by both the client and the server because no one cares about how Flash used to work.
So somewhat unlikely to be able to exploit it but have at it hass.
I'd imagine it'd be easier to exploit the server side, actually.
Yeah, but the server is most likely running the whole thing in a docker container with no permissions to do anything. Big companies security teams tend to require that when opening non-trivial third party code up to the internet.
The client on the other hand I would guess is running it's code as root, or at least something with full GPU access.