Computer Accompaniment

Getting a computer music system to listen to a live performer and synchronize an existing accompaniment.

See also Music Understanding

Back to Bibliography by Subject

Dannenberg, ``An On-Line Algorithm for Real-Time Accompaniment,'' in Proceedings of the 1984 International Computer Music Conference, Computer Music Association, (June 1985), 193-198.

This was the first of my papers on computer accompaniment. It presents algorithms for following a monophonic soloist in a score and synchronizing the accompaniment.

ABSTRACT: Real-time accompaniment solves most of the synchronization problems inherent in taped accompanimnent; however, this new approach requires the computer to have the ability to follow the soloist. Three subproblems arise: detecting and processing input from the live performer, matching this input against a score of expected input, and generating the timing information necessary to control the generation of the accompaniment. It is expected that the live solo performance will contain mistakes or be imperfectly detected by the computer, so it is necessary to allow for performance mistakes when matching the actual solo against the score. An efficient dynamic programming algorithm for finding the best match between solo performance and the score is presented. In producing the accompaniment, it is necessary to generate a time reference that varies in speed according to the soloist. The notion of virtual time is proposed as a solution to this problem. Finally, experience wtih two computer systems that produce real-time accompaniment is summarized.

PDF Version

Dannenberg, Method and apparatus for providing coordinated accompaniment for a performance. US Patent #4745836, 1988.

The first computer accompaniment patent. The patent describes monophonic and polyphonic accompaniment systems.

Buxton, Dannenberg, and Vercoe, ``The Computer as Accompanist,'' in CHI '86 Conference Proceedings (April 1986), ACM/SIGCHI, pp. 41-43.

A short paper (actually presented as a panel discussion) with statements from me, Barry, and Bill.

Bloch and Dannenberg, ``Real-Time Accompaniment of Polyphonic Keyboard Performance,'' in Proceedings of the 1985 International Computer Music Conference, (August 1985), pp. 279-290.

Expands previous work with 2 algorithms for matching polyphonic (keyboard) to a stored score, and also describes various techniques for making accompaniment timing more musical.

PDF Version

Grubb and Dannenberg, ``Automated Accompaniment of Musical Ensembles,'' in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI, (1994), pp. 94-99.

ABSTRACT: This paper describes a computer accompaniment system capable of providing musical accompaniment for an ensemble of performers. The system tracks the performance of each musician in the ensemble to determine the current score location and tempo of the ensemble. ``Missing parts'' in the composition (i.e. the accompaniment) are synthesized and synchronized to the ensemble. The paper presents an overview of the component problems of automated musical accompaniment and discusses solutions and their implementation. The system has been tested with solo performers as well as ensembles having as many as three performers.

PDF Version

Real-Time Scheduling

Dannenberg and Mont-Reynaud, ``Following an Improvisation in Real Time,'' in Proceedings of the International Computer Music Conference, Computer Music Association, (August 1987), pp. 241-248.

Expands previous work by addressing the problem of following solos improvised over fixed chord progressions rather than fixed note sequences. New beat-tracking and matching algorithms are introduced.

Dannenberg and Mukaino, ``New Techniques for Enhanced Quality of Computer Accompaniment,'' in Proceedings of the International Computer Music Conference, Computer Music Association, (September 1988), pp. 243-249.

Introduces the idea of using multiple matchers centered at different locations to make following more robust. Also deals with trills, glissandi, and grace notes, which cannot be matched note-for-note.

ABSTRACT: A computer accompaniment system has been extended with several techniques to solve problems associated with reliably recognizing and following a real-time performance. One technique is the use of an additional matcher to consider alternatives when the performance could be interpreted in two ways. Another is to delay corrective action when data from the matcher is suspect. Methods for handling grace notes, trills, and glissandi are also presented. The implementation of these and other techniques to produce an enhanced computer accompaniment system is discussed.

[Adobe Acrobat (PDF) Version] [Postscript Version]

Grubb and Dannenberg, ``Automated Accompaniment of Musical Ensembles,'' in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI, (1994), pp. 94-99.

Grubb and Dannenberg, ``Automating Ensemble Performance,'' in Proceedings of the 1994 International Computer Music Conference, International Computer Music Association, (September 1994), pp. 63-69.

Grubb and Dannenberg, ``Computer Performance in an Ensemble,'' in 3rd International Conference for Music Perception and Cognition Proceedings, European Society for the Cognitive Sciences of Music, Liege, Belgium, pp. 57-60, 1994.

These three papers discuss the problem of score following when there are multiple players. At any given time, some players may not be playing, some may be lost, and others are providing good information. These papers cover similar material in different levels of detail and for different audiences. Probably the ICMC paper is best, although the AAAI paper assumes a more CS and less musical background.

Dannenberg, Automated musical accompaniment with multiple input sensors. US Patent #5521324, 1994.

This patent is related to the ensemble accompaniment research reported in the papers mentioned above. Here is the abstract: The present invention is directed to an apparatus and method for automating accompaniment to an ensemble's performance. The apparatus is comprised of a plurality of input devices with each input device producing an input signal containing information related to an ensemble's performance. A plurality of tracking devices is provided, with each tracking device being responsive to one of the input signals. Each tracking device produces a position signal indicative of a score position when a match is found between the input signal and the score and a tempo estimate. A first voting device is responsive to each of the position signals for weighting each of the position signals. The weighting may be based on the frequency with which it changes and the proximity of its score position to each of the other score positions represented by each of the other position signals. The same weighting factors are then applied to the tempo estimate associated with that position signal. After the position signals and tempo estimates have been weighted, the voter device calculates a final ensemble score position signal in response to the weighted position signals and a final ensemble tempo based on the weighted tempo estimates. A scheduler is responsive to the final score position and final ensemble tempo for outputting an accompaniment corresponding thereto.

Grubb and Dannenberg, ``A Stochastic Method of Tracking a Vocal Performer,'' in Proceedings of the International Computer Music Conference, San Francisco: International Computer Music Association, (1997), pp 301-308.

Lorin Grubb's Ph.D. thesis introduced a completely new approach to score following in which the position in the score is represented by a probability density function. This and the following paper report on this work, but you should also read Lorin's Ph.D. thesis (available from CMU SCS) for the whole story.

Abstract. Automated accompaniment systems are computer systems that "play along" with a solo musician or a group of musicians when given the score for a piece of music. These systems must be able to "listen" to the live musicians by tracking their progress through the score in real-time. Accurate tracking of vocal performers (as opposed to instrumentalists) is a particularly challenging case. In this paper, we present a tracking method based upon a statistical model of vocal performnances. This technique incorporates both information obtained from real-time signal processing of a performance (such as fundamental pitch) and information describing the performer's movement through the score (namely tempo and elapsed time). We present a description of how this model is incorporated as part of a system currently used to accompany vocal performers. We provide a preliminary evaluation of its ability to estimate score position of a performer.

[Adobe Acrobat (PDF) Version]

Grubb and Dannenberg, “Enhanced Vocal Performance Tracking Using Multiple Information Sources,” in Proceedings of the International Computer Music Conference, San Francisco: International Computer Music Association, (1998), pp. 37-44.

ABSTRACT: We describe an enhanced statistical method for tracking vocal performances in real time. This approach integrates multiple measurements of the sound signal that are relevant to determining score position. The information is combined using a statistical model that incorporates distributions estimated from actual vocal performances. We present results from using different combinations of signal measurements to track both recorded and live performances. Results of this testing indicate some requirements for successfully using multiple information sources to improve automated vocal performance tracking.

[Adobe Acrobat (PDF) Version]

Grubb and Dannenberg, System and method for stochastic score following. US Patent #5913259, 1997.

This patent is related to the vocal accompaniment work described in the paper mentioned above and in Lorin's thesis. Here is the abstract: The present invention is directed to a computer implemented method for stochastic score following. The method includes the step of calculating a probability function over a score based on at least one observation extracted from a performance signal. The method also includes the step of determining a most likely position in the score based on the calculating step.

Dannenberg and Raphael, “Music Score Alignment and Computer Accompaniment,” in Communications of the ACM, Volume 49, Number 8 (August 2006), pp. 38-43.

ABSTRACT: We focus here on a form of machine listening known as music score matching, score following, or score alignment. Here we seek a correspondence between a symbolic music representation and an audio performance of the same music, identifying the onset times of all relevant musical “events” in the audio -- usually notes. There are two different versions of the problem, usually called “off-line” and “on-line.”

[Adobe Acrobat (PDF) Version]

Dannenberg, ``New Interfaces for Popular Music Performance,'' in Seventh International Conference on New Interfaces for Musical Expression: NIME 2007 New York, New York, NY: New York University, June 2007, pp. 130-135.

ABSTRACT: Augmenting performances of live popular music with computer systems poses many new challenges. Here, “popular music” is taken to mean music with a mostly steady tempo, some improvisational elements, and largely predetermined melodies, harmonies, and other parts. The overall problem is studied by developing a framework consisting of constraints and subproblems that any solution should address. These problems include beat acquisition, beat phase, score location, sound synthesis, data preparation, and adaptation. A prototype system is described that offers a set of solutions to the problems posed by the framework, and future work is suggested.

[Adobe Acrobat (PDF) Version]

Dannenberg, ``Computer Coordination With Popular Music: A New Research Agenda,'' in Proceedings of the Eleventh Biennial Arts and Technology Symposium at Connecticut College, (March 2008).

ABSTRACT: Computer accompaniment is now a well-established field of study, but nearly all work in this area assumes a Western classical music tradition in which timing is considered a flexible framework that performers manipulate for expression. This has important implications for the music, accompaniment, and performance style, and therefore the fundamental design of automatic accompaniment systems. Popular music has a very different set of assumptions and requirements for (human or computer) performers. In particular, the tempo is generally very stable, which would appear to simplify the synchronization problem, but synchronization must be quite precise and the music is not strictly notated, making coordination more difficult. The practical requirements for computer coordination with popular music create many problems that have not been addressed by research or practice. Some preliminary investigations in this area are reported, and future opportunities for research are discussed.

[Adobe Acrobat (PDF) Version]

Dannenberg, ``A Vision of Creative Computation in Music Performance,'' in Proceedings of the Second International Conference on Computational Creativity, Mexico City, Mexico, April 2011, pp. 84-89.

Abstract: Human Computer Music Performance (HCMP) is the integration of computer performers into live popular music. At present, HCMP exists only in very limited forms, due to a lack of understanding of how computer performers might operate in the context of live music and a lack of supporting research and technoloyg. The present work examines recent experimental systems as well as existing music performance practice to envision a future performance practice that involves computers as musicians. The resulting findings provide motivation as well as specific research objectivs that will enable new creative practice in music.

[Adobe Acrobat (PDF) Version]