How large are we talking here?
I'm currently sitting at 35k~ tracks, primarily FLAC with some 320kbps mp3 sprinkled in. I also use Jellyfin for anime and have a LOT of shows on there. Haven't noticed any issues yet, so I'm curious where your tipping point was.
Jellyfin after the 10.X update changed their database. This introduced a major performance issue with SOME large music libraries. There doesn't seem to be a hard set of criteria besides "some large music libraries suffer from bad performance degradation". I finally managed to upgrade after just brute forcing it and trying several times. It finally just worked.