SCORE! is a cooperation between the VU University Amsterdam and the national institute Sound and Vision. Its aim is to use simple unsupervised analysis of audio and video, to create meaningful mappings between the two, through latent-space representations.
We focus on using these mappings as a creative tool: for instance to automaticvally generate a soundtrack to a piece of video. However, the methods we use have much broader applicability, for instance in information retrieval, e-humanities or general multimedia manipulation.
Some preliminary examples of video-to-audio mapping can be found here.
The MIDI-LD cloud is a vast archive of MIDI music, annotated and interlinked using Linked Data standards. We will aim to use this as our archive for the audio part of the project.
The NWO KIEM project has officially been concluded. You can read the final report here, and a second report with future recommendations here. We currently have multiple student projects underway, to investigate various aspects of the SCORE! research question. After these have concluded, we will decide on any future directions for the project.
We will present some preliminary results from Score at the Rewire instriumewntal shifts symposium today. The slides of our lightning talk (including links to videos) can be found here.
SCORE! now has a webpage. We are currently investigating the use of Variational Autoencoders for MIDI music, through several MSc student projects. We are also investigating the use of VAEs for the visual part of the project, starting with the Sound and Vision archive. More updates soon.