Sounds like something a better model armed with ffmpeg would already be able to do. Run an analysis on loudness along the track, detect when speech begins/ends (with whisper) around loud segments, and compress or cut those parts out