I meant quantized. You can get an EXL3 quantized version thats almost the same performance.
My experience running quantised models is that they look the same at first but are all noticeably worse once the context size gets beyond anything trivial.
alt Hacker News
My experience running quantised models is that they look the same at first but are all noticeably worse once the context size gets beyond anything trivial.