alt Hacker NewsTraining a tiny LLM for fun using Rust/Candle - I constantly tweak stuff and keep track of results in a spreadsheet and work on generating a bigger corpus with LLMs. It's a project for fun, so I don't care about finding actual human generated text, I'd rather craft data in the format I want using LLMs - Probably not the best practice, but I can sleep properly despite doing that.
My favorite output so far is that I asked it what life was and in a random stroke of genius, it answered plainly: "It is.".
It's able to answer simple questions where the answer is in the question with up to 75% accuracy. Example success: 'The car was red. Q: What was red? ' |> 'the car' - Example failure: 'The stars twinkled at night. Q: What twinkled at night? ' |> 'the night'.
So nothing crazy, but I'm learning and having fun. My current corpus is ~17mb of stories, generated encyclopedia content, json examples, etc. JSON content is new from this weekend and the model is pretty bad at it so far, but I'm curious to see if I can get it somewhere interesting in the next few weeks.