This is true, you only grow when you have nothing to do. At least, nothing that other people are telling you to do. If there's something you want to learn really bad I highly recommend taking a sabbatical and just spending the whole year learning that topic deeply. You can get to the bleeding edge of most topics in one year of study, especially ones adjacent to what you already know. I did this in my 20s and can't wait for the stars to align to be able to do this again.