> streaming it over the interwebs isn't the issue

The hope is that you stream over LAN not the interwebs!

> I always get a speedup from moving the files off my external hard-drive

Based on your description, it does seem like your ffmpeg may be IO limited