Concatenative Synthesis Using Score-Aligned Transcription

MIDI files are automatically aligned to audio files to provide a transcription of the audio. The audio file can then be segmented at note onsets and offsets to create a sequence of  sound units. Each unit is labeled with the set of pitches that are sounding. This database can then be used to synthesize a MIDI file. The MIDI file is segmented in a similar way to obtain a sequence of chords. The database is searched for matching chords which are output.

The result preserves the approximate harmony and pitch content (to the extent that good matches are found in the database), but the actual timbres depend on what is in the database. The juxtaposition of sound units can be pretty chaotic but strangely in tune with the original harmonic progression.