Midi reconstruction from Synthesia video clips (and similar ones)

One day, while sitting on YouTube, searching for interesting training melodies, I came across videos from Synthesia, some of which I really liked, decided to download and learn ... =) But alas, as it turned out, there are videos, but no one was eager to upload midi files = (

Sowing for Google decided to see if there are ready-made solutions that would suit me, but alas, from the fact that I found only audio converters in midi, which upset me a little ... Without thinking twice, I decided that to restore MIDI frame rate in video clips will be enough ..., and I decided to try to implement this business ....

I didn’t want to write everything from scratch, so I decided what I would do with the ready-made components that Debian GNU / Linux provides me, from what python was best suited.

At the beginning of the implementation I decided that I would use ready-made (pulled out of video clips) pictures, but after the first uploads I realized that there was no point ... It turned out that it was very slow and also consumes significant resources on a screw ... Then I decided to try out for myself the new thing OpenCV (I really wanted to feel for a long time), it turned out that OpenCV works very well with the video stream, providing all the functions I need (count pixel, display frames and text).

For example, opening a video file and getting one frame can be described by two lines:

vidcap = cv2.VideoCapture('test.mp4') success,image = vidcap.read() 

And if you want, you can immediately dump the frames on the screw:

 cv2.imwrite("/tmp/frame%d.jpg" % frame, image) 

After some time, I wrote the function of generating the positions of the virtual keyboard keys, and displaying them (in the form of rectangles) over the image of the stream and unloading the picture produced the following:

So having decided that frame by frame, when reading an image from a video stream, I will read active notes from the virtual keys position (only those notes that are active are pixels that match the reference color or not far from it) and send them to midi. It was impossible to simply register the notes, as if the situation was on an ordinary midi keyboard, only a little simpler ... I checked it on the video, looked at how many notes came across (and there were quite a few of them) I thought not bad, it only remained to figure out how to write notes to the file, looking for a bit , I found an excellent python python midiutil package. After some time I was able to record notes in the midi. As it turned out, python-midiutil is a very simple and very easy-to-use sachet. For example, creating a file and adding notes is done in a couple of lines:

 mf.addTrackName(track, time, "Sample Track") mf.addTempo(track, time, 60 ) mf.addNote(track, channel, pitch, keytime, duration, volume) with open(outputmid, 'wb') as outf: mf.writeFile(outf) 

Downloading the resulting midi to the LMMS turned out to be quite successful. The first thing I restored a couple of favorite tunes. Then it became clear that the function of generating key positions was not very convenient from roller to roller, their location changed, I decided that I would do the GUI, I made a simple one, but with the function of key placement

I think that this program can be useful to many, because I put everything on the githaba