alt Hacker NewsYou can layer models across multiple GPUs for inference. If you pooled it into one system, that would be 72gb of some pretty damn fast memory. Would be great for dense models in the 30-50b range.
You can layer models across multiple GPUs for inference. If you pooled it into one system, that would be 72gb of some pretty damn fast memory. Would be great for dense models in the 30-50b range.