You should look into how other companies and tools that use FFMPEG handle this situation.

I wonder if you can keep your application itself closed source, but make an open-source adapter that handles the interaction with FFMPEG.

I'm not super familiar with open source licensing, and IANAL, so make sure to do your own research :)

As an example, I believe Audacity required me to install ffmpeg manually myself, and add it to my path. This is slightly different since Audacity itself is also open source. But could be helpful to reference.