You can get something relatively capable and usable with your configuration and Strata: https://github.com/Niko1221/Strata
Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity.
alt Hacker News
Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity.