I don't think you know what you're talking about. If the difficulty lowers at a lower rate than miners leaving then the difficulty rate will stop dropping.