alt Hacker News> it allows you to run the big models, at decent token speeds
Without CUDA, being an AMD GPU. Big warning depending on the tools you want to use.
> it allows you to run the big models, at decent token speeds
Without CUDA, being an AMD GPU. Big warning depending on the tools you want to use.
There's solutions for that though.
https://docs.scale-lang.com/stable/
https://github.com/vosen/ZLUDA
It's not perfect but it's a start towards unification. In the end though, we're at the same crossroads that graphics drivers were in 2013 with the sunsetting of OpenGL by Apple and the announcement of Vulkan by Khronos Group. CUDA has been around for a while and only recently has it gotten attention from the other chip makers. Thank goodness for open source and the collective minds that participate.