Beat tracking

Beat tracking is the task of automatically deriving, from a music audio signal, a sequence of time points that correspond to where a human listener would tap their foot along with the music.^[1:1] It is a core task in music information retrieval and is closely related to tempo estimation and to downbeat tracking, which finds the first beat of each bar.^[2:1]

Beat positions are used in applications such as beat-synchronized computer graphics and lighting,^[3:1] and they support higher-level tasks such as segmenting a piece into sections, music analysis, and automated DJ mixing.^[2:1] They are also needed to measure microtiming, the small deviations of notes from an even metrical grid.^[4:1] Early systems combined hand-designed signal processing with explicit rules or optimization; since the mid-2010s, the best-performing systems have used neural networks.^{[1:2][2:2][5:1]}

The problem

Daniel Ellis describes beat tracking as satisfying two constraints at once. The chosen beat times should generally fall at moments where the audio indicates a beat, for example at the onset of a note, and they should be spaced at a locally constant interval, since that regular spacing is what defines the musical pulse.^[1:1]

The task is ambiguous in several ways. Masataka Goto identified three main problems for audio beat tracking: detecting cues in the signal, interpreting those cues to infer the beat structure, and dealing with ambiguous interpretations, since no single sound necessarily marks the beat position.^[3:2] Beats also sit within a metrical hierarchy that includes faster subdivisions and slower levels such as half-notes and bars.^[3:1][2:1] Human listeners do not always agree on which level is "the beat": in the tapping data used for the 2006 MIREX beat-tracking evaluation, listeners' taps for each excerpt fell into two groups at faster and slower metrical levels, related by a factor of two or three. As a result, Ellis notes, no beat tracker can score close to 100% agreement with such ground truth.^[1:3]

Approaches

Symbolic and early audio systems

Many early beat-tracking systems took as input MIDI-like symbolic representations of note events rather than audio recordings.^[3:1] In 1998, Eric Scheirer described an audio method that splits the signal into a small number of frequency bands and passes them through banks of parallel comb filters. The method works causally, so it can predict when upcoming beats will occur, and in a short validation it performed similarly to human listeners.^[6:1]

Goto's 2001 system tracked beats in real time from popular-music recordings, with or without drums, and recognized three levels of the metrical hierarchy: quarter notes, half notes, and bars. It combined knowledge of note onsets, chord changes, and drum patterns, and used multiple agents to evaluate competing hypotheses about the beat structure in parallel. It assumed music in 4/4 time with a roughly constant tempo.^[3:1][3:2]

Onset strength and periodicity

Many systems first compute an onset strength function from the audio, which peaks where new notes begin, and then look for periodic structure in it. Ellis's 2007 system estimates a single global tempo and then uses dynamic programming to find the sequence of beat times that best balances high onset strength against consistency with that tempo. It achieved just under 60% beat accuracy on the MIREX 2006 development data and could follow tempo changes of about ±10% around the estimated tempo.^[1:2]

Music with soft note onsets or continuously changing tempo is especially hard for these methods. Peter Grosche and Meinard Müller's predominant local pulse representation addresses this by fitting a sinusoid to the local periodicity of the onset function at each point in time and summing these sinusoids, producing a pulse curve that stays robust under tempo changes and weak onsets.^[7:1]

Neural networks

Sebastian Böck, Florian Krebs, and Gerhard Widmer used a recurrent neural network operating directly on spectrograms to model several metrical levels at once, distinguishing beats from downbeats. A dynamic Bayesian network then fits these predictions to bars of variable length. The system achieved state-of-the-art results across a wide range of genres.^[2:2] Matthew Davies and Böck later showed that temporal convolutional networks could match the state of the art on many beat-tracking datasets while being well suited to parallel training on large datasets and requiring relatively few parameters.^[5:1][5:1]

Evaluation

Beat trackers are evaluated by comparing their output with beat times annotated by people. Commonly used measures include an F-measure, which counts an estimated beat as correct if it falls within a tolerance window around an annotated beat (70 milliseconds by default in the widely used mir_eval library); continuity-based scores, which measure how much of the beat sequence is continuously correct; and others such as Cemgil's score, Goto's score, McKinney's P-score, and information gain.^[8:1] Downbeat tracking is usually reported with an F-measure at a ±70 ms tolerance.^[2:3]

Comparing published systems is difficult because studies often use different metrics and datasets. Evaluation collections include sets built specifically from hard-to-track pieces, such as the SMC dataset, and collections of non-Western music, including Carnatic, Cretan, and Turkish music.^[2:3]

Limitations

Some systems build in assumptions that fit some repertoires better than others; Goto's system, for example, assumed 4/4 time and a roughly constant tempo.^[3:2] Rhythmic conventions outside the music a system was developed on can also mislead it: Fuentes and colleagues report that beat-tracking models not trained on Uruguayan candombe usually displace the beat by one sixteenth note, because of an accent in candombe's rhythmic pattern.^[4:2]