I think the genetic variation you get when times are plenty will only hover around the mean, though, until some restraint pushes it a specific direction.

Here is a simplistic example of what I am trying to say:

Imagine you have a population of creatures, and they have some gene (or combination of genes) that controls how much cold they can tolerate. Some can tolerate very cold weather, some can barely tolerate any cold weather, and some are in the middle. They all can live happily, and mate randomly, meaning the individuals who have a high cold tolerance will (on average, since most other individuals by definition will have lower cold tolerance) mate with an individual who has lower cold tolerance. In other words, the next generation will genetically regress to the mean.

This continues on as long as all the individuals can survive at basically the same rate.

Now, imagine there is suddenly a very cold winter, and the individuals who can't tolerate cold die off. Now, there aren't any (or many) individuals with low cold tolerance for the higher tolerance individuals to mate with, meaning it won't regress back to that mean (or more accurately, it will regress back towards the new mean based on a population without the susceptible to cold individuals). Now, that genetic variation you get from generation to generation might reach new extremes that it never would have gotten to during times of plenty.