Music score alignment and computer accompaniment — Communications of the ACM

Claims

1. applications-education
   Much computer accompaniment research has been motivated by contemporary art music. It is also used in music education: the Piano Tutor used score following to turn pages and identify errors for feedback, and the commercial SmartMusic software included accompaniments for over 30,000 pieces; small-scale studies indicated that students who practice with accompaniment improve in skill, confidence, motivation, and practice time.

   "Much research in computer accompaniment has been motivated by contemporary art music. [...] Computer accompaniment is also used in music education systems, including the Piano Tutor and SmartMusic. [...] As the student performs the music, real-time score following is used to track the performance and automatically turn pages by updating the display. After the performance, score alignment is used to identify all skipped notes and extra notes. [...] SmartMusic [...] includes accompaniments for over 30,000 pieces of music for wind instruments, vocalists, and beginning string players. [...] Only small-scale studies have been undertaken, but they indicate that students who practice with accompaniment improve in performance skill and confidence, motivation, and practice time."

   Locator: section: Current Applications of Score Following · Quote language: en
2. components
   Most computer accompaniment systems have four components: audio feature extraction, score following, accompaniment control, and music synthesis (from a synthesizer or by resynthesizing prerecorded audio). Because all events are detected with latency, a system that waits to hear a note before responding will always be late; accompaniment control must instead extrapolate future timing from the score, the score follower's output, and past performances. Early systems used hand-coded rules for this; later ones used belief networks.

   "Most computer accompaniment systems are organized roughly along the lines of Figure 3, which shows four fairly independent components: audio feature extraction, score following, accompaniment control, and music synthesis. [...] The fourth, music synthesis, generates the actual accompaniment sound, either by means of a note-based sound synthesizer or by resynthesizing prerecorded audio on-the-fly. [...] An accompaniment system, (or person), that tries to achieve synchrony by waiting to hear a note, and then responding, will always be late, since all musical events are detected with latency. A better approach coordinates parts by extrapolating the future evolution based on the detected note onsets, as well as other information. [...] Early accompaniment systems used hand-coded rules to guide this extrapolation. [...] More recently, belief networks address the challenges of automatic learning and extrapolation from incomplete information."

   Locator: section: On-Line Matching and Computer Accompaniment · Quote language: en
3. future-aesthetics
   Dannenberg and Raphael identify capturing a deeper understanding of musical aesthetics as the paramount challenge for broader acceptance of accompaniment systems. Addressing concern that such machines will displace musicians, they doubt this will be the predominant effect, expecting instead to make music more accessible and to increase demand for all-human music.

   "broader acceptance of this technology requires that we overcome significant challenges. Paramount among these is capturing a deeper understanding of musical aesthetics by the computer. [...] While some express concern that musicians will be displaced by these machines, we doubt that this will be the predominant effect. On the contrary, we believe that accompaniment systems will make music more accessible to more people, fostering a wider appreciation and love for music-making. We expect that the effect will be an increased demand for "all-human" music, both from listeners and participants."

   Locator: section: The Future of Score Matching · Quote language: en
4. matching-methods
   Score matching can model the score as a graph of states and use the Viterbi algorithm (dynamic programming) to find the optimal alignment with noisy pitch estimates. Because multi-pitch estimation is unreliable, polyphonic alignment systems compare spectra predicted from the score with observed spectra, often reduced to 12-element chroma vectors. The earliest accompaniment systems matched note sequences; hidden Markov models later allowed training on rehearsals; another approach represents score position as a continuous probability density.

   "Readers are likely familiar with the Viterbi Algorithm, which uses dynamic programming to solve this problem without actually enumerating every path. [...] Unfortunately, multi-pitch estimation is unreliable and still an active research area, so today's score-alignment systems use representations based on the distributions of frequencies in the signal, also known as the spectrum. [...] A popular technique for comparing spectra from scores to spectra in music audio is to reduce the detailed spectrum to a 12-element chroma vector [...]. The earliest computer accompaniment systems matched note sequences rather than spectral data, reducing data rates and computation. [...] Hidden Markov models have been explored for score following, allowing systems to be trained using rehearsals to follow more accurately and to respond with less latency. [...] Another approach avoids the problem of discrete states by representing score position as a continuous probability density function."

   Locator: section: Offline Score Matching; Variations and Alternate Approaches · Quote language: en
5. online-offline
   Score alignment finds the correspondence between a symbolic score and an audio performance. Offline alignment uses the complete performance; its real-time counterpart, score following, processes audio as it arrives with no look-ahead. Computer accompaniment, which generates a flexible accompaniment that follows a live soloist, is the biggest success of on-line score following; other applications include coordinating opera supertitles and lighting, score-based pitch correction, and automatic page turning.

   "Here we seek a correspondence between a symbolic music representation and an audio performance of the same music [...]. Off-line matching uses the complete performance to estimate the correspondence between audio data and symbolic score. [...] Off-line score matching's real-time cousin, sometimes called score-following, processes the audio data on-line as the signal is acquired, thus, no "look ahead" is possible. [...] The application dearest to the hearts of both authors is the accompaniment system, which generates a flexible musical accompaniment that follows a live soloist. [...] Other applications include the automatic coordination of audio-visual equipment with musical performance, such as opera supertitles and lighting effects, as well as real-time score-based audio enhancement e.g. pitch correction, and automatic page turners. [...] The biggest success of on-line score following so far has been computer accompaniment."

   Locator: section: Introduction; On-Line Matching and Computer Accompaniment · Quote language: en