But I assume it wouldn't take much more training to make them good again - just a matter of picking the right loss function, since this sort of stuff is probably all encoded in the last layer or two anyway.