Computer Accompaniment

Getting a computer music system to listen to a live performer and synchronize an existing accompaniment.

See also Music Understanding

Back to Bibliography by Subject

Dannenberg, “An On-Line Algorithm for Real-Time Accompaniment,” in Proceedings of the 1984 International Computer Music Conference, Computer Music Association, (June 1985), 193-198.

This was the first of my papers on computer accompaniment. It presents algorithms for following a monophonic soloist in a score and synchronizing the accompaniment.

ABSTRACT: Real-time accompaniment solves most of the synchronization problems inherent in taped accompanimnent; however, this new approach requires the computer to have the ability to follow the soloist. Three subproblems arise: detecting and processing input from the live performer, matching this input against a score of expected input, and generating the timing information necessary to control the generation of the accompaniment. It is expected that the live solo performance will contain mistakes or be imperfectly detected by the computer, so it is necessary to allow for performance mistakes when matching the actual solo against the score. An efficient dynamic programming algorithm for finding the best match between solo performance and the score is presented. In producing the accompaniment, it is necessary to generate a time reference that varies in speed according to the soloist. The notion of virtual time is proposed as a solution to this problem. Finally, experience wtih two computer systems that produce real-time accompaniment is summarized.

PDF Version

Dannenberg, Method and apparatus for providing coordinated accompaniment for a performance. US Patent #4745836, 1988.

The first computer accompaniment patent. The patent describes monophonic and polyphonic accompaniment systems.

Buxton, Dannenberg, and Vercoe, “The Computer as Accompanist,” in CHI '86 Conference Proceedings (April 1986), ACM/SIGCHI, pp. 41-43.

A short paper (actually presented as a panel discussion) with statements from me, Barry, and Bill.

Bloch and Dannenberg, “Real-Time Accompaniment of Polyphonic Keyboard Performance,” in Proceedings of the 1985 International Computer Music Conference, (August 1985), pp. 279-290.

Expands previous work with 2 algorithms for matching polyphonic (keyboard) to a stored score, and also describes various techniques for making accompaniment timing more musical.

PDF Version

Grubb and Dannenberg, “Automated Accompaniment of Musical Ensembles,” in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI, (1994), pp. 94-99.

ABSTRACT: This paper describes a computer accompaniment system capable of providing musical accompaniment for an ensemble of performers. The system tracks the performance of each musician in the ensemble to determine the current score location and tempo of the ensemble. “Missing parts” in the composition (i.e. the accompaniment) are synthesized and synchronized to the ensemble. The paper presents an overview of the component problems of automated musical accompaniment and discusses solutions and their implementation. The system has been tested with solo performers as well as ensembles having as many as three performers.

PDF Version

Real-Time Scheduling

Dannenberg and Mont-Reynaud, “Following an Improvisation in Real Time,” in Proceedings of the International Computer Music Conference, Computer Music Association, (August 1987), pp. 241-248.

Expands previous work by addressing the problem of following solos improvised over fixed chord progressions rather than fixed note sequences. New beat-tracking and matching algorithms are introduced.

Dannenberg and Mukaino, “New Techniques for Enhanced Quality of Computer Accompaniment,” in Proceedings of the International Computer Music Conference, Computer Music Association, (September 1988), pp. 243-249.

Introduces the idea of using multiple matchers centered at different locations to make following more robust. Also deals with trills, glissandi, and grace notes, which cannot be matched note-for-note.

ABSTRACT: A computer accompaniment system has been extended with several techniques to solve problems associated with reliably recognizing and following a real-time performance. One technique is the use of an additional matcher to consider alternatives when the performance could be interpreted in two ways. Another is to delay corrective action when data from the matcher is suspect. Methods for handling grace notes, trills, and glissandi are also presented. The implementation of these and other techniques to produce an enhanced computer accompaniment system is discussed.

[Adobe Acrobat (PDF) Version] [Postscript Version]

Grubb and Dannenberg, “Automated Accompaniment of Musical Ensembles,” in Proceedings of the Twelfth National Conference on Artificial Intelligence, AAAI, (1994), pp. 94-99.

Grubb and Dannenberg, “Automating Ensemble Performance,” in Proceedings of the 1994 International Computer Music Conference, International Computer Music Association, (September 1994), pp. 63-69.

Grubb and Dannenberg, “Computer Performance in an Ensemble,” in 3rd International Conference for Music Perception and Cognition Proceedings, European Society for the Cognitive Sciences of Music, Liege, Belgium, pp. 57-60, 1994.

These three papers discuss the problem of score following when there are multiple players. At any given time, some players may not be playing, some may be lost, and others are providing good information. These papers cover similar material in different levels of detail and for different audiences. Probably the ICMC paper is best, although the AAAI paper assumes a more CS and less musical background.

Dannenberg, Automated musical accompaniment with multiple input sensors. US Patent #5521324, 1994.

This patent is related to the ensemble accompaniment research reported in the papers mentioned above. Here is the abstract: The present invention is directed to an apparatus and method for automating accompaniment to an ensemble's performance. The apparatus is comprised of a plurality of input devices with each input device producing an input signal containing information related to an ensemble's performance. A plurality of tracking devices is provided, with each tracking device being responsive to one of the input signals. Each tracking device produces a position signal indicative of a score position when a match is found between the input signal and the score and a tempo estimate. A first voting device is responsive to each of the position signals for weighting each of the position signals. The weighting may be based on the frequency with which it changes and the proximity of its score position to each of the other score positions represented by each of the other position signals. The same weighting factors are then applied to the tempo estimate associated with that position signal. After the position signals and tempo estimates have been weighted, the voter device calculates a final ensemble score position signal in response to the weighted position signals and a final ensemble tempo based on the weighted tempo estimates. A scheduler is responsive to the final score position and final ensemble tempo for outputting an accompaniment corresponding thereto.

Grubb and Dannenberg, “A Stochastic Method of Tracking a Vocal Performer,” in Proceedings of the International Computer Music Conference, San Francisco: International Computer Music Association, (1997), pp 301-308.

Lorin Grubb's Ph.D. thesis introduced a completely new approach to score following in which the position in the score is represented by a probability density function. This and the following paper report on this work, but you should also read Lorin's Ph.D. thesis (available from CMU SCS) for the whole story.

Abstract. Automated accompaniment systems are computer systems that "play along" with a solo musician or a group of musicians when given the score for a piece of music. These systems must be able to "listen" to the live musicians by tracking their progress through the score in real-time. Accurate tracking of vocal performers (as opposed to instrumentalists) is a particularly challenging case. In this paper, we present a tracking method based upon a statistical model of vocal performnances. This technique incorporates both information obtained from real-time signal processing of a performance (such as fundamental pitch) and information describing the performer's movement through the score (namely tempo and elapsed time). We present a description of how this model is incorporated as part of a system currently used to accompany vocal performers. We provide a preliminary evaluation of its ability to estimate score position of a performer.

[Adobe Acrobat (PDF) Version]

Grubb and Dannenberg, “Enhanced Vocal Performance Tracking Using Multiple Information Sources,” in Proceedings of the International Computer Music Conference, San Francisco: International Computer Music Association, (1998), pp. 37-44.

ABSTRACT: We describe an enhanced statistical method for tracking vocal performances in real time. This approach integrates multiple measurements of the sound signal that are relevant to determining score position. The information is combined using a statistical model that incorporates distributions estimated from actual vocal performances. We present results from using different combinations of signal measurements to track both recorded and live performances. Results of this testing indicate some requirements for successfully using multiple information sources to improve automated vocal performance tracking.

[Adobe Acrobat (PDF) Version]

Grubb and Dannenberg, System and method for stochastic score following. US Patent #5913259, 1997.

This patent is related to the vocal accompaniment work described in the paper mentioned above and in Lorin's thesis. Here is the abstract: The present invention is directed to a computer implemented method for stochastic score following. The method includes the step of calculating a probability function over a score based on at least one observation extracted from a performance signal. The method also includes the step of determining a most likely position in the score based on the calculating step.

Dannenberg and Raphael, “Music Score Alignment and Computer Accompaniment,” in Communications of the ACM, Volume 49, Number 8 (August 2006), pp. 38-43.

ABSTRACT: We focus here on a form of machine listening known as music score matching, score following, or score alignment. Here we seek a correspondence between a symbolic music representation and an audio performance of the same music, identifying the onset times of all relevant musical “events” in the audio -- usually notes. There are two different versions of the problem, usually called “off-line” and “on-line.”

[Adobe Acrobat (PDF) Version]

Dannenberg, “New Interfaces for Popular Music Performance,” in Seventh International Conference on New Interfaces for Musical Expression: NIME 2007 New York, New York, NY: New York University, June 2007, pp. 130-135.

ABSTRACT: Augmenting performances of live popular music with computer systems poses many new challenges. Here, “popular music” is taken to mean music with a mostly steady tempo, some improvisational elements, and largely predetermined melodies, harmonies, and other parts. The overall problem is studied by developing a framework consisting of constraints and subproblems that any solution should address. These problems include beat acquisition, beat phase, score location, sound synthesis, data preparation, and adaptation. A prototype system is described that offers a set of solutions to the problems posed by the framework, and future work is suggested.

[Adobe Acrobat (PDF) Version]

Dannenberg, “Computer Coordination With Popular Music: A New Research Agenda,” in Proceedings of the Eleventh Biennial Arts and Technology Symposium at Connecticut College, (March 2008).

ABSTRACT: Computer accompaniment is now a well-established field of study, but nearly all work in this area assumes a Western classical music tradition in which timing is considered a flexible framework that performers manipulate for expression. This has important implications for the music, accompaniment, and performance style, and therefore the fundamental design of automatic accompaniment systems. Popular music has a very different set of assumptions and requirements for (human or computer) performers. In particular, the tempo is generally very stable, which would appear to simplify the synchronization problem, but synchronization must be quite precise and the music is not strictly notated, making coordination more difficult. The practical requirements for computer coordination with popular music create many problems that have not been addressed by research or practice. Some preliminary investigations in this area are reported, and future opportunities for research are discussed.

[Adobe Acrobat (PDF) Version]

Dannenberg, “A Vision of Creative Computation in Music Performance,” in Proceedings of the Second International Conference on Computational Creativity, Mexico City, Mexico, April 2011, pp. 84-89.

Abstract: Human Computer Music Performance (HCMP) is the integration of computer performers into live popular music. At present, HCMP exists only in very limited forms, due to a lack of understanding of how computer performers might operate in the context of live music and a lack of supporting research and technoloyg. The present work examines recent experimental systems as well as existing music performance practice to envision a future performance practice that involves computers as musicians. The resulting findings provide motivation as well as specific research objectivs that will enable new creative practice in music.

[Adobe Acrobat (PDF) Version]

Gold and Dannenberg, “A Reference Architecture and Score Representation for Popular Music Human-Computer Music Performance Systems,” in Proceedings of the International Conference on New Interfaces for Musical Expression, Oslo, Norway, May 2011, pp. 36-39.

Abstract: Popular music (characterized by improvised instrumental parts, beat and measure-level organization, and steady tempo) poses challenges for human-computer music performance (HCMP). Pieces of music are typically rearrangeable on-the-fly and involve a high degree of variation from ensemble to ensemble, and even between rehearsal and performance. Computer systems aiming to participate in such ensembles must therefore cope with a dynamic high-level structure in addition to the more traditional problems of beat-tracking, score-following, and machine improvisation. There are many approaches to integrating the components required to implement dynamic human-computer music performance systems. This paper presents a reference architecture designed to allow the typical sub-components (e.g. beat-tracking, tempo prediction, improvisation) to be integrated in a consistent way, allowing them to be combined and/or compared systematically. In addition, the paper presents a dynamic score representation particularly suited to the demands of popular music performance by computer.

[Adobe Acrobat (PDF) Version]

Dannenberg, “A Virtual Orchestra for Human-Computer Music Performance,” in Proceedings of the International Computer Music Conference, San Francisco, California, August 2011, pp. 185-188.

Abstract: A virtual orchestra was implemented to augment a human jazz big band with the addition of violins, violas, and cellos. In an effort to maximize musical quality, the orchestra uses a high-quality studio recording of each individual part, yet adapts to the tempo of the band in real time using a custom multi-channel PSOLA algorithm. The tempo is tracked through a foot-tapping interface, with additional cueing that allows for deviation from the nominal score and recovery from counting errors. Output is through an array of 8 highquality studio monitors arranged to provide the illusion of many point sources configured as an acoustic ensemble.

[Adobe Acrobat (PDF) Version]

Dannenberg, Jin, Gold, Sandu, Palliyaguru, Robertson, Stark, and Kleinberger, “Human-Computer Music Performance: From Synchronized Accompaniment to Musical Partner,” in Proceedings of the Sound and Music Computing Conference, Stockholm, Sweden, July 2013, pp. 277-283.

Abstract: Live music performance with computers has motivated many research projects in science, engineering, and the arts. In spite of decades of work, it is surprising that there is not more technology for, and a better understanding of the computer as music performer. We review the development of techniques for live music performance and outline our efforts to establish a new direction, Human-Computer Music Performance (HCMP), as a framework for a variety of coordinated studies. Our work in this area spans performance analysis, synchronization techniques, and interactive performance systems. Our goal is to enable musicians to incorporate computers into performances easily and effectively through a better understanding of requirements, new techniques, and practical, performance-worthy implementations. We conclude with directions for future work.

[Adobe Acrobat (PDF) Version]

Dannenberg, “Human-Computer Music Performance: A Brief History and Future Prospects,” Journal of the Acoustical Society of America, 135(4) (March 2014), p. 2376.

This is an abstract only. There is no full paper publication. The published text is the following:

Abstract: Computer accompaniment began in the eighties as a technology to synchronize computers to live musicians by sensing, following, and adapting to expressive musical performances. The technology has progressed from systems where performances were modeled as sequences of discrete symbols, i.e., pitches, to modern systems that use continuous probabilistic models. Although score following techniques have been a common focus, computer accompaniment research has addressed many other interesting topics, including the musical adjustment of tempo, the problem of following an ensemble of musicians, and making systems more robust to unexpected mistakes by performers. Looking toward the future, we find that score following is only one of many ways musicians use to synchronize. Score following is appropriate when scores exist and describe the performance accurately, and where timing deviations are to be followed rather than ignored. In many cases, however, especially in popular music forms, tempo is rather steady, and performers improvise many of their parts. Traditional computer accompaniment techniques do not solve these important music performance scenarios. The term Human-Computer Music Performance (HCMP) has been introduced to cover a broader spectrum of problems and technologies where humans and computers perform music together, adding interesting new problems and directions for future research.

Dannenberg, Gold, Liang, and Xia, “Methods and Prospects for Human-Computer Music Performance of Popular Music,” Computer Music Journal, 38(2) (Summer 2014), pp. 36-50.

Abstract: Computers are often used in popular music performance, but most often in very restricted ways, such as keyboard synthesizers where musicians are in complete control or pre-recorded or sequenced music where musicians follow the computer’s drums or click-track. An interesting and yet little-explored possibility is the computer as highly autonomous popular music performer capable of joining a mixed ensemble of computers and humans. Considering the skills and functional requirements of musicians leads to a number of predictions about future Human-Computer Music Performance (HCMP) systems for popular music. We describe a general architecture for such systems and describe some early implementations and our experience with them.

[Adobe Acrobat (PDF) Version]

Dannenberg, Gold, Liang, and Xia, “Active Scores: Representation and Synchronization in Human-Computer Music Performance of Popular Music,” Computer Music Journal, 38(2) (Summer 2014), pp. 51-62.

Abstract: Computers have the potential to significantly extend the practice of popular music based on steady tempo and mostly determined form. However, there are significant challenges to overcome due to constraints including accurate timing based on beats and adherence to a form or structure in spite of possible changes that might be made even during the performance. We describe an approach to synchronization across media that takes into account latency due to communication delays and audio buffering. We also address the problem of mapping from a conventional score with repeats and other structures to an actual performance which can involve both “flattening” the score and rearranging it, as is common in popular music. Finally, we illustrate the possibilities of the score as a bi-directional user interface in a real-time music performance system, allowing the user to direct the computer through a digitally displayed score, and allowing the computer to indicate position back to human performers.

[Adobe Acrobat (PDF) Version]

Xia, Wang, Dannenberg, and Gordon, “Spectral Learning for Expressive Interactive Ensemble Performance,” in Proceedings of the 16th International Society for Music Information Retrieval Conference, October 2015, pp. 816-822.

Abstract: We apply machine learning to a database of recorded ensemble performances to build an artificial performer that can perform music expressively in concert with human musicians. We consider the piano duet scenario and focus on the interaction of expressive timing and dynamics. We model different performers musical expression as coevolving time series and learn their interactive relationship from multiple rehearsals. In particular, we use a spectral method, which is able to learn the correspondence not only between different performers but also between the performance past and future by reduced-rank partial regressions. We describe our model that captures the intrinsic interactive relationship between different performers, present the spectral learning procedure, and show that the spectral learning algorithm is able to generate a more human-like interaction.

[Adobe Acrobat (PDF) Version]