alt Hacker Newsfor layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
for layer parallelism (e.g. to get more vram) the bandwidth between layers is essentially nothing (like 16kb per token I think), so I don't think x4 would even be a problem!
Good point. It's much more of an issue when running dense models with tensor parallelism. In that case, I'd look for an MoE model instead.