Beat tracking

Beat tracking is the task of automatically deriving, from a music audio signal, a sequence of time points that correspond to where a human listener would tap their foot along with the music.^[1:1] It is a core task in music information retrieval and is closely related to tempo estimation and to downbeat tracking, which finds the first beat of each bar.^[2:1]

Beat positions are used in applications such as beat-synchronized computer graphics and lighting,^[3:1] and they support higher-level tasks such as segmenting a piece into sections, music analysis, and automated DJ mixing.^[2:1] They are also needed to measure microtiming, the small deviations of notes from an even metrical grid.^[4:1] Early systems combined hand-designed signal processing with explicit rules or optimization.^[5:1][1:2] Since the mid-2010s, the best-performing systems have used neural networks.^[2:2][6:1] Most systems analyze a complete recording after the fact, but some track beats online, in real time, using only the audio heard so far.^[7:1]

The problem

Daniel Ellis describes beat tracking as satisfying two constraints at once. The chosen beat times should generally fall at moments where the audio indicates a beat, for example at the onset of a note, and they should be spaced at a locally constant interval, since that regular spacing is what defines the musical pulse.^[1:1]

The task is ambiguous in several ways. Masataka Goto identified three main problems for audio beat tracking: detecting cues in the signal, interpreting those cues to infer the beat structure, and dealing with ambiguous interpretations, since no single sound necessarily marks the beat position.^[3:2] Beats also sit within a metrical hierarchy that includes faster subdivisions and slower levels such as half-notes and bars.^[3:1][2:1] Human listeners do not always agree on which level is "the beat": in the tapping data used for the 2006 MIREX beat-tracking evaluation, listeners' taps for each excerpt fell into two groups at faster and slower metrical levels, related by a factor of two or three. As a result, Ellis notes, no beat tracker can score close to 100% agreement with such ground truth.^[1:3]

Approaches

Symbolic and early audio systems

Many early beat-tracking systems took as input MIDI-like symbolic representations of note events rather than audio recordings.^[3:1] In 1998, Eric Scheirer described an audio method that splits the signal into a small number of frequency bands and passes them through banks of parallel comb filters. The method works causally, so it can predict when upcoming beats will occur, and in a short validation it performed similarly to human listeners.^[5:1]

Goto's 2001 system tracked beats in real time from popular-music recordings, with or without drums, and recognized three levels of the metrical hierarchy: quarter notes, half notes, and bars. It combined knowledge of note onsets, chord changes, and drum patterns, and used multiple agents to evaluate competing hypotheses about the beat structure in parallel. It assumed music in 4/4 time with a roughly constant tempo.^[3:1][3:2]

Onset strength and periodicity

Many systems first compute an onset strength function from the audio, which peaks where new notes begin, and then look for periodic structure in it. Ellis's 2007 system estimates a single global tempo and then uses dynamic programming to find the sequence of beat times that best balances high onset strength against consistency with that tempo. It achieved just under 60% beat accuracy on the MIREX 2006 development data and could follow tempo changes of about ±10% around the estimated tempo.^[1:2]

Music with soft note onsets or continuously changing tempo is especially hard for these methods. Peter Grosche and Meinard Müller's predominant local pulse representation addresses this by fitting a sinusoid to the local periodicity of the onset function at each point in time and summing these sinusoids, producing a pulse curve that stays robust under tempo changes and weak onsets.^[8:1]

Neural networks

Sebastian Böck, Florian Krebs, and Gerhard Widmer used a recurrent neural network operating directly on spectrograms to model several metrical levels at once, distinguishing beats from downbeats. A dynamic Bayesian network then fits these predictions to bars of variable length. The system achieved state-of-the-art results across a wide range of genres.^[2:2] Matthew Davies and Böck later showed that temporal convolutional networks could match the state of the art on many beat-tracking datasets while being well suited to parallel training on large datasets and requiring relatively few parameters.^[9:1]

Most high-accuracy neural systems from this period passed the network's output through the dynamic Bayesian network introduced by Böck and colleagues, which restricts tempo to a fixed range and the number of beats per bar to a short list of values. Francesco Foscarin, Jan Schlüter, and Gerhard Widmer argued that this step is bound to fail on pieces with time-signature changes, tempos outside the supported range, or unusual numbers of beats per bar.^[6:1] Their 2024 system, Beat This!, drops the dynamic Bayesian network and aims for generality by training on many datasets, including solo instrument recordings, pieces with time-signature changes, and classical music with large tempo variations; its network alternates convolutions with transformers. It surpassed earlier systems in F-measure but scored worse on continuity measures, and its authors noted that it can still fail, especially on difficult and underrepresented genres.^[6:2]

Online tracking

Most beat trackers work offline: they analyze a complete recording and can use audio that comes after a given moment to decide where a beat falls. Online or real-time beat tracking must instead be causal, estimating beats from past and present audio only, as required by applications such as live accompaniment systems.^[7:1] Scheirer's comb-filter method and Goto's 2001 system were both designed to work causally.^[5:1][3:1]

Many later online systems repeatedly applied an offline model to a sliding window of recent audio, which Mojtaba Heydari, Frank Cwitkowitz, and Zhiyao Duan note can produce discontinuous beat predictions and a delay before the first window is filled. Their BeatNet system (2021) instead uses a causal neural network followed by particle filters, and infers the meter as it goes rather than requiring the time signature in advance.^[7:1] On the GTZAN dataset it outperformed other online trackers but trailed Böck and colleagues' offline system by 3.65 percentage points on beat tracking and 4.9 on downbeat tracking.^[7:2]

Heydari and Duan identify three challenges for real-time rhythm analysis: staying accurate without access to future audio, keeping latency low, especially on low-powered devices, and generalizing to music without strong percussion. Their BeatNet+ system (2024) addresses the third by training an auxiliary branch of the network on the same music with the drums removed, so that the main network learns a representation less dependent on percussion, and adds methods for adapting the model to isolated singing voices and non-percussive music.^[10:1]

Evaluation

Beat trackers are evaluated by comparing their output with beat times annotated by people. Commonly used measures include an F-measure, which counts an estimated beat as correct if it falls within a tolerance window around an annotated beat (70 milliseconds by default in the widely used mir_eval library); continuity-based scores, which measure how much of the beat sequence is continuously correct; and others such as Cemgil's score, Goto's score, McKinney's P-score, and information gain.^[11:1] Downbeat tracking is usually reported with an F-measure at a ±70 ms tolerance.^[2:3]

Comparing published systems is difficult because studies often use different metrics and datasets. Evaluation collections include sets built specifically from hard-to-track pieces, such as the SMC dataset, and collections of non-Western music, including Carnatic, Cretan, and Turkish music.^[2:3]

Datasets

Because the audio in large annotated collections is hard to share, researchers have used different datasets in an ad hoc way, which complicates comparison. Beat-tracking datasets differ in whether they contain short excerpts or full tracks, how many people annotated them and how the annotations were corrected, whether they cover many styles or one, whether they include non-Western music, and whether pieces were chosen for being hard to track. When MIREX introduced a downbeat-estimation task in 2014, it used six datasets: The Beatles; Hardcore, Jungle, Drum and Bass (HJDB); Turkish; Ballroom; Carnatic; and Cretan.^[12:1]

The Harmonix Set (2019) provides beat, downbeat, and section annotations for 912 tracks of Western popular music, most with a stable tempo and 4/4 time. It distributes annotations and identifiers for locating the recordings rather than the audio itself.^[12:2] GuitarSet (2018) provides acoustic guitar recordings annotated with beats and downbeats along with chords, string and fret positions, and playing style.^[13:1] Beat This! was trained on 15 collections, including Ballroom, Hainsworth, SMC, HJDB, The Beatles, Harmonix, GuitarSet, and a candombe dataset, and tested on GTZAN.^[6:3]

GTZAN was assembled for genre-recognition research, and Sturm documented repetitions, mislabelings, and distortions in it.^[14:1] Foscarin and colleagues found errors in its beat annotations as well, along with pieces where even experts would struggle to agree on a single annotation and pieces where it is unclear there is a beat to track at all.^[6:4]

Limitations

Some systems build in assumptions that fit some repertoires better than others; Goto's system, for example, assumed 4/4 time and a roughly constant tempo.^[3:2] Rhythmic conventions outside the music a system was developed on can also mislead it: Fuentes and colleagues report that beat-tracking models not trained on Uruguayan candombe usually displace the beat by one sixteenth note, because of an accent in candombe's rhythmic pattern.^[4:2]

Norwegian Hardanger fiddle music poses a harder case. In some of its styles, bars are made of beats of unequal length whose proportions vary from bar to bar, and beat onsets are often not clearly accented. Olivier Lartillot and colleagues tested the madmom beat tracker, from Böck and colleagues, against beat annotations made by the performers and by expert musicians. For every tune, the predicted beats did not correspond to the annotated ones, and the two coincided only at isolated points.^[15:1]

Even recent systems share this weakness: the developers of Beat This! note that it can still fail on difficult and underrepresented genres, and they argue that new annotated data for such genres is crucial for further progress.^[6:2][6:4]