I've built a similar video editor and have been considering pure client side implementation vs transcoding into a known format beforehand, went with transcoding for wider format support and easier video playback implementation.

I'm interested in how you handle demuxing different container formats any which ones are supported?

I get "Audio decode failed: your browser cannot decode the audio in "41b1aee9-ac65-43f6-b020-e8fed77c3c72_webm.bin". Try re-encoding the file with AAC audio." for a WEBM with no audio.

h264/aac MP4 works, is that demuxed with mp4box.js? I noticed seeking (clicking or scrubbing on timeline) initializes a new VideoDecoder and destroys the previous one for every new frame, leading to abysmal performance as you lose decoder state and a lot of decoding work has to be repeated. Plus the decoder reinitialization time. Is that because the demuxing logic doesn't give precise access to encoded frames? iirc mp4box.js didn't support that last time I checked.