Machine accompaniment

Machine accompaniment, also called computer accompaniment or automatic accompaniment, is the use of a computer program as a musical partner to live musicians, usually playing a supporting part that adapts to them as they perform. Christopher Raphael distinguishes several kinds of system: some process the musician's own audio, some improvise alongside the player, as in jazz or call-and-response playing, and some perform a precomposed part while following a soloist, as an orchestra does in a concerto.^[1:1] Systems of the last kind rely on score following, the real-time tracking of a performer's position in a known score, and computer accompaniment has been the most successful application of that technique.^[2:1]

Accompaniment systems have been used in contemporary art music, where they synchronize live electronics with performers, and in music education.^[2:2][3:1] Since the late 2010s, researchers have also built machine learning systems that generate accompaniment to a melody in real time without a score.^[4:1]

History

In early pieces combining live performers with electronic sound, by composers such as Bruno Maderna, Karlheinz Stockhausen, and Mario Davidovsky, the electronic part was fixed on tape, and performers had to keep in time with it by following a click track or listening closely.^[3:1] Roger Dannenberg argued that real-time accompaniment would solve most of these synchronization problems, provided that the computer could follow the soloist.^[5:1]

The first score followers were presented in 1984, independently, by Dannenberg and by Barry Vercoe; both worked from MIDI-style note events rather than audio. Vercoe's Synthetic Performer explicitly tracked and responded to the performer's tempo.^[3:1] Dannenberg divided the problem into three parts: detecting what the soloist plays, matching that input against a score of expected events, and generating an accompaniment that follows the soloist. Because performances contain mistakes and detection is imperfect, the matching has to tolerate errors. Drawing on the time-warping techniques used in speech recognition, he proposed an efficient dynamic programming algorithm that finds the best match between the performance and the score, defined as their longest common subsequence.^[5:1]

From the mid-1980s, composers working in real-time electronics, beginning with experiments by Philippe Manoury and Pierre Boulez, used score followers to synchronize live instruments with a pre-written score and trigger electronic events during performance. The composers Jean-Claude Risset and Marco Stroppa criticized this approach.^[3:1]

Later systems moved from matching note sequences to analyzing audio directly, and adopted probabilistic methods. Hidden Markov models allowed score followers to be trained on rehearsals so that they followed more accurately and responded with less delay. Another approach represented the score position as a continuous probability distribution rather than a set of discrete states.^[2:3]

How score-based systems work

Dannenberg and Raphael describe most computer accompaniment systems as having four parts: audio feature extraction, score following, accompaniment control, and sound synthesis. The synthesis stage either drives a synthesizer or resynthesizes prerecorded audio.^[2:4]

Score following itself is the real-time form of score alignment. Offline alignment can use an entire recording, but a score follower must process audio as it arrives, with no look-ahead.^[2:1] A simple score follower represents the score as a sequence of states and uses the Viterbi algorithm to find the most likely path through them given noisy pitch estimates. Because estimating several simultaneous pitches is unreliable, systems for polyphonic music instead compare the spectrum expected from the score with the spectrum of the audio, often reduced to chroma vectors that summarize energy in each of the 12 pitch classes.^[2:3]

The accompaniment control stage decides when to play. Because every musical event is detected with some delay, a system that waits to hear a note before responding will always be late. Accompaniment systems therefore predict the soloist's future timing from the score, the score follower's output, and past performances. Early systems used hand-written rules for this prediction; later systems used Bayesian networks.^[2:4] Raphael reports detection delays of about 30–90 milliseconds, enough to ruin the ensemble if the accompaniment lags consistently by that much. He also notes that a score follower must not report a note before it has happened: if it does, the accompaniment can arrive at a shared point in the music ahead of the soloist, forcing the soloist to struggle to regain control of the performance.^[1:2]

A related approach applies to music without a regular pulse, as in some works of the postwar avant-garde, where the score may specify only points at which soloist and accompaniment coincide. Here a system can simply wait for the relevant solo events and then respond, the approach taken by IRCAM's score follower.^[1:3]

Notable systems

Music Plus One

Raphael's Music Plus One, also called the Informatics Philharmonic, takes its name from the Music Minus One play-along records that inspired it. It accompanies soloists in concertos and other classical works.^[1:2] Raphael developed it with students and faculty at Indiana University's Jacobs School of Music, where about 200 student pianists study the concerto repertoire but the school's orchestras perform only two piano concertos a year with student soloists.^[1:3]

The system has three modules. "Listen" is a hidden Markov model score follower; "Predict" models musical timing with a Gaussian model similar to a Kalman filter; and "Play" produces the orchestra's sound by time-stretching an orchestra-only recording with a phase vocoder, continually adjusting its playback rate to reach each scheduled note on time.^[1:2] The system has to perform reasonably well with no rehearsal, and it improves with rehearsal: after each session it re-estimates its timing model to match the soloist's interpretation, training on the most recent rehearsals because that interpretation keeps evolving.^[1:4]

Antescofo

Arshia Cont's Antescofo, developed at IRCAM, is designed for mixed works that combine live instruments with electronics. It holds the instrumental and electronic scores together and executes the electronic score in synchronization with the live performance. It couples an audio agent, which tracks the score position, with a tempo agent that predicts the performer's tempo, and it provides a score language in which composers can specify timing and interaction.^[3:2]

Educational software

Computer accompaniment has been used in music teaching. The Piano Tutor used score following to turn pages automatically, then aligned the student's performance with the score to identify skipped notes, extra notes, and timing errors and give feedback. The commercial program SmartMusic offered accompaniments for more than 30,000 pieces for wind players, singers, and beginning string players. According to Dannenberg and Raphael, only small-scale studies of its effects had been carried out, but they suggested that practicing with accompaniment improved students' skill, confidence, motivation, and practice time.^[2:2]

Improvised and generative accompaniment

Not all accompaniment systems follow a fixed score. Some generate an accompaniment offline after receiving a complete melody: MySong produces chords for a sung melody, and SingSong produces full backing tracks. Interactive systems for improvisation take other approaches. George Lewis's Voyager uses rules to decide how to listen and respond, OMax recombines material the musician has just played, and ImproteK and its successor DJazz coordinate improvisation through a shared chord progression such as a jazz standard. Neural network systems include BachDuet, which adds a bass line in counterpoint to a melody, and SongDriver, which generates chords to a melody in real time.^[4:1]

Generating accompaniment in real time is harder than generating it offline, because each chord must be chosen before the melody note it accompanies is known. Yusong Wu and colleagues found that models trained only to imitate existing songs adapt poorly when a live player makes mistakes or does something unexpected. Their system, ReaLchords, is fine-tuned with reinforcement learning, using reward models that judge how well chords fit the melody and a "teacher" model that can see the future melody.^[4:2] Even so, an offline model that could see the whole melody still matched the harmony of human-written accompaniments more closely than any online model did.^[4:3]

The same constraint applies to rhythm: accompaniment that must respond to a live player without a score depends on online beat tracking, which estimates beats using only the audio heard so far.^[6:1]

Limitations and debate

Raphael's system learns its musical interpretation by imitating the soloist, which suits the concerto, where the soloist leads. It is less suited to situations in which the accompanist must make independent musical choices. One example is accompanying young students, where a human pianist would also model steady rhythm and lead when necessary; another is opera, where the orchestra is often on an equal footing with the singers. Raphael reports that the system performed worst when it served as rehearsal pianist for an opera.^[1:4] He also notes that computers can play extremely fast and rhythmically complex parts but are weak at producing aesthetically satisfying interpretations, which makes music written for human–computer partnerships, such as pieces by Jan Beran for Music Plus One, an attractive use of the technology.^[1:3]

Dannenberg and Raphael identify a deeper computer understanding of musical aesthetics as the main obstacle to wider acceptance of accompaniment systems. Responding to concerns that such systems could displace human musicians, they argue that this is unlikely to be the main effect, and that the systems will instead make music-making accessible to more people and increase demand for performances by human musicians.^[2:5]