logoalt Hacker News

Show HN: I made an audio-to-MIDI model 60x more efficient

170 points • by hidude562 • last Tuesday at 1:03 AM • 31 comments • view on HN

I released dettato, my open weight audio to midi transcription model. It beats muscriptor 1.3b on every set and is roughly 60x smaller, without training on copyrighted recordings. Feel free to ask questions


Comments

thih9 • today at 4:48 PM

Congrats on the launch!

The top illustration looks like an Angry Birds character with an ear. Fine for a one time blog post but if you plan to reuse it then you may want to change the design.

➕ show 2 replies
SoleilAbsolu • today at 6:27 PM

Curious what's the performance for both note duration, and note pitch? How does it do for tunings/temperaments outside 12TET? Very refreshing to not violate copyright in this day and age amongst all the greedy grabbers!

peaseagee • today at 2:03 PM

Congrats on training on open and public domain recordings. Thanks for proving this kind of work can be done in a more ethical way.

➕ show 1 reply
dimmuborgir • today at 6:27 PM

Is there a way to force it to a single instrument? I uploaded audio files of pure piano music and the output midi files are split into piano, guitar and drum parts.

locusofself • today at 6:08 PM

Very cool, I have been working on a DAW for several months that is getting close to a beta release. I'd love to incorporate this.

As a guitarist who can't play keys well, I'd love to use something like this to help me transcribe string arrangements.

simedw • today at 6:25 PM

Very cool, I wonder if my MIDI generation model would improve if I trained it on the MIDI files your model generates.

epiccoleman • today at 2:52 PM

That is awesome! I'll definitely have to check it out. I was very impressed with muscriptor when I used it in a recent project (a conference talk I gave). It was an AI talk with a musical component, and the final "scene" in the talk involved my own guitar playing. I used Muscriptor to transcribe that playing for categorization as part of a "guitar solo generator" that recut a new solo out of my stock phrases each time. Really fun little demo for a bigger project.

Anyway, it would be awesome to be able to do those sorts of transcriptions better and faster - so I'll definitely give this a whirl. Maybe I can dig up the pipeline scripts from processing my licks for that talk and see how dettato slots in.

rogerdickey • today at 4:37 PM

all the examples seem to be classical music or largely instrumental stuff. how well does it work for things like pop music, rap?

pier25 • today at 4:45 PM

Very cool. This is the sort of stuff I want to see from AI.

oever • today at 1:59 PM

Audio-to-MIDI sounds very ambitious. How do you verify that the midi fits the audio? Does it result in useful beat detection? E.g. I'd very much like to have a way to map audio offset in a recording to labels in sheet music. But even beat detection in fairly standard jazz recordings fails often.

Does the model know when a piece (whole piece, not a section) is 3/4 or 4/4 th and can it give the onsets of all measures? Or all the midi notes just based on time and not organized rhythmically?

➕ show 1 reply
gpugreg • today at 3:18 PM

Nice! Worked surprisingly well.

You might want to implement deduplication for your website so impatient users do not convert the same mp3 multiple times (at multiple times the price) when they click "run audio to midi" and the server takes a bit longer to respond.

➕ show 1 reply
gregzem • last Tuesday at 11:40 PM

Congrats with SOTA model. How good is it in recognition of the instruments which are overlapping by frequencies? Does is provide note offset and velocity in the transcribed MIDI? Thanks!

jsphweid • today at 2:26 PM

I've been out of the loop since Magenta's MT3 model. This feels a lot better and significantly smaller. Very impressive, dude. And thank you!

mycall • today at 2:05 PM

I'm still waiting for someone to convert beatboxing to midi in realtime. That would be a game changer for that art.

➕ show 1 reply
ch_sm • today at 2:52 PM

Just tried it with some home-recorded songs (pop/rock) — this works great!

awinter-py • today at 4:01 PM

at long last we have solved karaoke

chahmed312 • today at 2:14 PM

Is it fully open source?

JodieBenitez • today at 2:03 PM

I need this in Bitwig !

➕ show 2 replies
eduardoagarcia • today at 4:02 PM

super interesting, I'll have to check this out tonight when I get home!

th1nhng0 • today at 4:51 PM

just in time when I need it. Thank you so much

IAmGraydon • today at 2:56 PM

Excellent! I'll try this tonight. The timing couldn't be better as I've been researching audio->MIDI models for some personal projects recently.

hongyong • today at 3:54 PM

nice, this is what i've been looking for

pikann22 • today at 2:23 PM

crazy how a 20m param model actually beats the 1,3b muscriptor. awesome work!

saascostlab_off • today at 3:07 PM

[flagged]

grd_dev • today at 4:18 PM

[dead]

staff000 • last Tuesday at 4:25 PM

[dead]