Presumably the optimal size for survival when humans aren't applying pressure via fishing is bigger. Perhaps "ruining" is hyperbolic, but this is making the fish less fit for their environment.
It is making them less fit for their environment? I can see how it is making them less fit for the environment they used to be in before they were subject to large scale human fishing, but that's not their environment now.
If that’s the case it should return if we stopped fishing.
...less fit for their environment excluding human fishing pressure.
Their environment is one where humans prey on them, especially on the large ones. It's a selection pressure like any other and becoming smaller on average makes them more fit for that environment.