alt Hacker NewsDoes it support flash attention? Use tensor cores? Can I write custom kernels?
UPD. found no evidence that it supports tensor cores, so it's going to be many times slower than implementations that do.
Does it support flash attention? Use tensor cores? Can I write custom kernels?
UPD. found no evidence that it supports tensor cores, so it's going to be many times slower than implementations that do.
Yes, when you use the PTX backend it supports Tensor Cores.It has also implementation for flash attention. You can also write your own kernels, have a look here: https://github.com/beehive-lab/GPULlama3.java/blob/main/src/... https://github.com/beehive-lab/GPULlama3.java/blob/main/src/...