Can I run this on my RTX 5090?
Yes they have quants for 32GB and 20GB use-cases (including mmproj and kv cache + context)
alt Hacker News
Yes they have quants for 32GB and 20GB use-cases (including mmproj and kv cache + context)