alt Hacker NewsI think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
I think it should, as you dont need to use the encoder layer on the new tokens, you just read the embedding from the cache. that's why cache reads are cheaper
I meant, it shouldn't affect the resulting LLM output. It's a performance optimization that doesn't change the behavior.