alt Hacker NewsI got a quantized Llama 70B model working, using most of my 64GB of RAM but it's usable:
ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q3_K_M
uvx --python 3.11 open-webui serve
Replies
Time to put this 128GB M4 Mac to work...
bash> ollama run hf.co/unsloth/DeepSeek-R1-Distill-Llama-70B-GGUF:Q5_K_M
(It wouldn't accept the Q_8 because it was sharded.)
Will report back lol
What app do you use on your phone to access your home LLM servers over tailscale?
> Here's what I got - the joke itself is rubbish but the "thinking" section is fascinating
Very funny. How does the 70B model do in comparison?
I've previously had a 70B Q2 wedged into a 4090. Not much space for context but it runs...
I set up a cloudflare tunnel recently so I could also use my local LLMs remotely! It’s nice.
Yes, I did the same but with Docker serving open-webui. It's definitely not fast. Took 2 minutes to answer "what is the highest peak in California," but it eventually go there. OpenAI o1 took 6 seconds.
Pasting the logic for anyone interested:
<details type="reasoning" done="true" duration="102"> <summary>Thought for 102 seconds</summary> > Okay, so I need to figure out what the tallest peak in California is. Hmm, I remember that the highest mountain in the contiguous United States is Mount Whitney, but wait, isn't that in California? Let me think. Yes, I'm pretty sure it's in California. But maybe there are other tall peaks too. > > Wait, no, Mount Whitney is definitely the tallest. It's over 14,000 feet. But sometimes people might confuse it with other mountains like Shasta or something else. Oh, wait, isn't Mount Shasta also a volcano? I think that's in California too but it's not as tall as Whitney. > > So, putting it together, Mount Whitney is the tallest peak in California at 14,505 feet. It's located in the Sierra Nevada range. People hike there a lot because it's such a high point. But I should double-check to make sure I'm not mixing up anything else. Yeah, that seems right. > I wonder if there are any taller peaks elsewhere. Oh, Denali in Alaska is higher, but that's not in California. So within the state, Whitney is the highest. Got it. > So the answer is Mount Whitney at 14,505 feet above sea level. It's a popular spot for hikers and climbers. Makes sense because it's such an iconic peak in the Sierra Nevada mountains. > I think I've got all that right. No other peaks in California are taller than Whitney. So yeah, Mount Whitney is the tallest. > Just to make sure, sometimes there are different names or maybe alternate names for peaks, but Whitney is pretty straightforward. Yeah, it's definitely the one. > Okay, so I'm confident now. The answer is Mount Whitney. > *Final Answer* > The tallest peak in California is Mount Whitney, with an elevation of 14,505 feet above sea level. > > \boxed{Mount Whitney} </details> To determine the tallest peak in California, we consider notable mountains such as Mount Whitney and Mount Shasta. Mount Whitney is recognized as the highest peak in the contiguous United States, standing at 14,505 feet. It is located in the Sierra Nevada mountain range and is a popular destination for hikers. Mount Shasta, although also a significant peak, does not surpass Whitney's elevation. Therefore, the tallest peak in California is confirmed to be Mount Whitney.
*Final Answer* The tallest peak in California is \boxed{Mount Whitney}.
Thanks for all your writing on these topics Simon! It has turned me from a bit of a naysayer to an optimist around this tooling, especially being able running stuff locally with access to tools. Have an M4 Pro mac mini arriving this week to have a similar self hosted setup over tailscale.