alt Hacker NewsShow HN: I trained a 125M model to autocomplete piano on-device
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15).
The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device.
The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
Comments
I think this is a great project and very HN. Not sure why the comments are so focused on the deliverable- you learned way more and had a much more interesting experience.
One think I didn't see mentioned in the post- maybe I missed it- how large was the data? How many samples did you use to pretrain and post-train
Hearing the start of Für Elise, and then it being taken in an incredibly different direction, is surprisingly disconcerting.
Interesting. I do feel letting a machine generating notes is taking the joy out of improvisation, is it not?
I think one of the great, early, joys of learning a piano is gaining the following intuitions: The seemingly harder path of learning sheet is actually faster. Your mind _should_ learn to think in two dimensions Spatial — where fingers go — and Time — pitch and tempo – ** when learning. The _internalization_ of Space and Time queues guide the fingers in a dance that is vastly satisfying. This skill leads you to the final part of the journey that is improvisation and the one more exciting than what i am on now.
---
**
Space: Your finger placement on keys right, e.g. knowing how to go from landmark/anchor notes(mid-c, G, F etc) and then go to the others above and below it. Crudely this is some what like typing from your landmark f and j qwerty keyboard
Time: The out singing/verbalizing of the notes/beats on a time measure as you play them(per the time measure). e.g. you can say out loud 1-2-3-4 for 4/4 measure, if the measure has quarter notes say out loud. And `1-e-and-a-2-e-and-a-3-e-and-a-4-e-and-a` for a 4/4 with 1/16th note granularity. Do this as you play the notes and you get a sense of tempo.
Reminds me of this project to generate every melody possible algorithmically in order to fight music copyright lawsuits. https://allthemusic.info/
Reminds me of Francois Pachet’s Continuator (all the way back in 2003, using hierarchical markov models)
I would love something like that, except that I play the melody, and it produces proper 3-4 part accompaniment, preferably in good baroque style. Extra bonus if it could also write it into a file in a format suitable for music editing programs.
Ah, MIDI files. The only type of music you could realistically download from the internet back in the day, and you had to wake up at ungodly hours so that your dialup modem would not rack up a massive phone bill.
This is really fun. Scaler 3 starts with a chord progression and lets you break it down into musical performances and parts. Useful for ideation when producing.
Would be fun to get a midi clock going and play some chords on my piano and have my synth start jamming along with the bass and my keyboard doing some performance. Or any combination of the above.
Für Elsevier Journal Access Library pass out candy for safety this halloween your baby off milk shake it off.
Are the weights of the model available?
Should tokens be multiple Midi notes? Why or why not?
How would you expand this to support elements like attack ("velocity of the key-down" in piano speak), grace notes, timing etc. Would each of those be part of this model or another model? How would you model an arbitrary element (pedal, duration, etc...)
The idea is awesome! :) However there's definitely much room for improvement, first of all rythm and composition (so there's some sense of musical form).
Oh, a cool idea! I just tried it, works pretty well. Kudos!
One feature request:
Instead of playing the AI-generated audio solely through the iPhone's speakers, add an option to send the audio as midi notes to a device (probably the same one you received the mini notes from).
Running a 125M model on-device at that speed is impressive. How much did you have to optimize the model to get that performance on an iPhone?
This is so amazing, can you improve the quality of generation at the cost of notes per seconds ? No one can play 108 notes/sec anyways, maybe you can train the model to do CoT for better quality
Amazing idea! Gonna hook this up to my little synthesizer and blast some square wave arpeggiated ML music!
This is honestly astonishing and the first "AI music" I've heard that has the potential to sound beautiful. I always thought that MIDI would be a perfect format for this. Glad to see this person make it happen!
Talking about AI music with some live human MIDI inputs, Magenta Realtime 2 was released a few weeks ago and is pretty fun.
Pretty Impressive!!
Even after a few years deep into AI, I find your application absolutely magic. This is very inspiring, thank you for sharing.
I don’t have MIDI. How about whistling or playing the piano via microphone? Sounds easy. Another 6 month rabbit hole? :)
Would you be willing to share about how much it cost to train a model like this?
Very cool! Can you say a little bit about the size of the DPO training examples and how long training took?
Cool work. I tried using LLMs to parse sheet music and they are really bad.
This is really awesome thanks for sharing
really incredible work! great use case, impeccable learning strategy, congrats!
> Eventually I used Gemini 3.5 Flash for pairwise evaluation
But, but… wouldn't that be… (gasp) DISTILLATION?
Fun project!
Gemma 4 E2B was too heavy for your needs?
[dead]
What is the goal here?
You have trained a model, presumably on other people's work, to improvise on an initial kernel of musical material. In doing so you:
1) remove the opportunity for one to develop their own skills in extending musical ideas 2) rip off other people's work wholesale 3) remove what is the most rewarding part of music (creating and shaping it through refinement and technique).
This sort of “autocomplete” is actually fundamental to how classical composers were trained.
For anyone interested, I’d highly recommend reading Robert Gjerdingen’s article Gebrauchs-Formulas. https://www.researchgate.net/publication/259731561_Gebrauchs...
You can also listen to the transcript of four Russian composers, including Rachmaninoff, playing this pattern recognition and generation game at a dinner party in the late 1800’s: https://youtu.be/PlFPOWuwBHI?is=EKBK7QQkJs4MsTCU
Composers at the time could do this just by looking at sheet music and audiating, without using a piano.