alt Hacker NewsWe don't have lot of GPUs available right now, but it is not crazy hard to get it running on our MI300x. Depending on your quant, you probably want a 4x.
ssh admin.hotaisle.app
Yes, this should be made easier to just get a VM with it pre-installed. Working on that.
Unless using docker, if vllm is not provided and built against ROCm dependencies it’s going to be time consuming.
It took me quite some time to figure the magic combination of versions and commits, and to build each dependency successfully to run on an MI325x.