Score following: State of the art and new developments — Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03)

Claims

1. definition-history
   Score following is the synchronisation of a computer with a performer playing a known musical score, often addressed as real-time automatic accompaniment. At IRCAM, research on it began with Barry Vercoe and Lawrence Beauregard in 1983 and was continued by Miller Puckette and Philippe Manoury; at least 15 IRCAM pieces from 1987 to 1997 relied on it, including Manoury's Sonus ex Machina and En echo and Boulez's Anthèmes II and Explosante-fixe.

   "Score following is the synchronisation of a computer with a performer playing a known musical score. [...] Score following is often addressed as "real-time automatic accompaniment". [...] Many pieces have been composed relying on score following techniques. For instance, at Ircam we can count at least 15 pieces between 1987 and 1997, such as Sonus ex Machina and En echo by Philippe Manoury, and Anthèmes II and Explosante-fixe by Pierre Boulez. [...] At Ircam, the research on score following was initiated by Barry Vercoe and Lawrence Beauregard as soon as 1983. It was continued by Miller Puckette and Philippe Manoury"

   Locator: section: Abstract, 1 · Quote language: en
2. difficulties
   The two main difficulties in score following are musicians' errors and departures from the score, and errors in real-time signal analysis such as pitch detection. As of 2003, highly polyphonic instruments and ensembles remained problematic, systems often tracked only pitch, and followers were not robust enough to run without a human operator supervising the performance.

   "The two most important difficulties are related to possible sources of mismatch between the human and the synthetic performer: On the one hand, musicians can make errors, i.e. playing something differing from the score [...]. On the other hand, all real-time analysis of musical signals, and in particular pitch detection algorithms, are prone to error. [...] Although it is possible to follow instruments with low polyphony, such as the violin, highly polyphonic instruments or even a group of instruments are still problematic. Often, only the pitch parameter is taken into account [...]. Finally, the follower is not always robust enough; in some particular musical configurations, the score follower fails, which means that it always needs a constant supervision by a human operator during the performance of the piece."

   Locator: section: 1 · Quote language: en
3. evaluation
   Score followers can be evaluated subjectively, by checking whether events are recognised with a latency that respects the composer's intentions, or objectively against reference alignments, counting missed and falsely detected events and measuring latency; the time needed to recover after a performer's wrong notes is another measure. Tolerable error depends on the application and musical style.

   "A subjective or qualitative evaluation of a score follower means that the important performance events are recognised with a latency that respects the intention of the composer [...]. error labels can be due to the miss of a correct label at a given moment, or to the false alarm of a label incorrectly given. [...] Again, the tolerable number of mistakes and latencies of the follower largely depend on the kind of application and the type of musical style involved. [...] human errors will occur, suggesting as another measure the time needed by the score follower to recover from an error situation, that is, to resynchronise itself after a number of wrong notes are played."

   Locator: section: 4 · Quote language: en
4. structure-hmm
   A score follower extracts features such as pitch, spectrum, and amplitude from the performer's sound and matches them against a model of the target score; an actions score specifies what the accompaniment does at labelled positions. Following Vercoe, a follower has three tasks: listen, perform, and learn. Statistical followers based on hidden Markov models can handle the unpredictability of performance, model the whole spectrum, and be trained on example performances.

   "In a pre-processing step, the system extracts some features (e.g. pitch, spectrum, amplitude) from the sound produced by the musician. [...] The model is matched with the incoming data in the follower, while the actions score represents the actions that the accompaniment has to perform at some specified positions [...]. According to Vercoe, the score follower has to fulfill three tasks: Listen-Perform-Learn. [...] HMMs can deal with the several levels of unpredictability typical of performed music and they can model complex features, without requiring pre-processing techniques that are prone to errors like any pitch detectors or midi sensors. For instance, in our approach, the whole frequency spectrum of the signal is modeled. [...] One of the advantages of a score following system based on a statistical model is that it can learn using well-known training methods."

   Locator: section: 2–2.3 · Quote language: en