There is caching to consider. Switching models throws away the cached tokens.