Score following

Score following is the task of tracking, in real time, where a live performer is in a known musical score, so that a computer can stay synchronized with the performance.^[1:1] It is the real-time form of score alignment: whereas offline alignment can use a complete recording, a score follower must process the performance as it arrives, with no look-ahead.^[2:1]

The best-known application of score following is machine accompaniment, in which a computer plays a part that follows a live soloist; Roger Dannenberg and Christopher Raphael described it in 2006 as the technique's biggest success so far. Other applications include coordinating opera supertitles and stage lighting with a performance, score-based pitch correction, and turning pages automatically.^[2:1] In contemporary music, score followers are used to synchronize live electronics with performers.^[3:1]

History

The first score followers were presented in 1984 by Dannenberg and by Barry Vercoe, and worked from MIDI-style note events.^[3:1] Dannenberg's algorithm compared the stream of notes detected from the performer with the sequence of notes expected from the score. To tolerate the performer's mistakes and detection errors, it used dynamic programming to find the best match between the two, defined as their longest common subsequence.^[4:1]

At IRCAM in Paris, research on score following was begun by Vercoe and Lawrence Beauregard in 1983 and continued by Miller Puckette and Philippe Manoury. Nicola Orio, Serge Lemouton, and Diemo Schwarz counted at least 15 pieces composed at IRCAM between 1987 and 1997 that relied on score following, including Manoury's Sonus ex Machina and En echo and Pierre Boulez's Anthèmes II and ...explosante-fixe....^[1:1]

The earliest systems matched sequences of notes; later systems analyzed the audio signal directly and adopted probabilistic methods. Hidden Markov models allowed score followers to be trained on rehearsals so that they followed more accurately and with less delay, and another approach represented the score position as a continuous probability distribution rather than a set of discrete states.^[2:2] In the 2010s, researchers began applying deep learning to follow performances directly in images of printed music.^[5:1][6:1]

How score followers work

A score follower first extracts features from the performer's sound, such as pitch, spectrum, and amplitude, and then matches them against an internal model of the score being followed. A separate list of actions specifies what the computer should do when particular points in the score are reached, such as starting a sound or transformation. Following Vercoe, Orio and colleagues describe three tasks for a score follower: to listen, to perform, and to learn from previous performances.^[1:2]

Matching methods

A basic way to align a performance with a score is to represent the score as a sequence of states and use the Viterbi algorithm to find the most likely path through them given noisy estimates of pitch.^[2:2] Because estimating several simultaneous pitches is unreliable, systems for polyphonic music instead compare the spectrum expected from the score with the spectrum of the audio, often reduced to chroma vectors that summarize the energy in each of the 12 pitch classes.^[2:2] Hidden Markov models can handle the many kinds of unpredictability in a live performance, can model the whole spectrum of the signal rather than relying on error-prone pitch detection, and can be trained on example performances.^[1:2]

Another approach aligns the live audio with audio synthesized from the score. Andreas Arzt, Gerhard Widmer, and Simon Dixon built a system of this kind for turning pages automatically, extending Dixon's on-line version of dynamic time warping. It needed no training on a particular performer and was designed to cope with expressive changes of tempo, with performers taking or skipping repeats, and with mistakes and restarts; in live tests with a pianist it worked reliably even when the pianist made errors.^[7:1]

Timing and latency

A score follower must balance accuracy against delay. Raphael reports detection delays of about 30–90 milliseconds in his accompaniment system, and warns that a follower must not report a note before it has actually begun: if it does, an accompaniment can arrive at a shared point in the music ahead of the soloist.^[8:1] Some systems model the performer's tempo explicitly. Arshia Cont's Antescofo, for example, couples an audio agent with a tempo agent, outputs the performer's tempo in real time, and executes an electronic score in synchronization with the live performance.^[3:2]

Challenges

Orio and colleagues identified two main sources of difficulty: performers can play something different from the score, and real-time analysis of audio, especially pitch detection, is prone to error. As of 2003, highly polyphonic instruments and ensembles were still hard to follow, many systems tracked only pitch, and score followers were not reliable enough to be used in concert without a human operator supervising them.^[1:3] Arzt and colleagues add that a practical follower should adapt to how a musician plays without prior training, and should recognize when a performer ignores repeats or jumps to another section.^[7:1]

Most score followers also need the score in a symbolic, machine-readable form such as MusicXML or MIDI. That representation must be created either by hand or by optical music recognition, which Matthias Dorfer, Arzt, and Widmer described in 2016 as still highly unreliable, especially for complex music such as orchestral scores.^[5:1]

Following sheet music images

To avoid the need for a symbolic score, Dorfer and colleagues trained a convolutional neural network that takes an image of a line of sheet music and a short excerpt of audio and predicts where in the image the excerpt is being played. Their proof of concept was limited to monophonic piano music and one staff at a time.^[5:1] Florian Henkel and colleagues later framed score following in sheet music images as a reinforcement learning problem, training agents that listen to audio, read the score image, and move along it. The agents were trained only on audio synthesized from scores, but first experiments on 16 recordings of human pianists suggested that they were beginning to generalize to real performances.^[6:1]

Evaluation

Score followers can be evaluated subjectively, by judging whether important events are recognized with a delay acceptable for the musical effect the composer intended, or objectively, by comparing the follower's output with a reference alignment. Objective measures include the number of missed events, the number of false detections, the delay in detecting events, and the time a follower needs to recover after a performer plays wrong notes. How many errors and how much delay are tolerable depends on the application and the musical style.^[1:4]

Applications

Besides accompaniment and live electronics, score following has been used in music education. The Piano Tutor used it to turn pages on screen as a student played and then aligned the performance with the score to find skipped and extra notes for feedback.^[2:3]

Wikidata: Q7435317