Chord recognition

Commons ↗

Chord recognition, also called automatic chord estimation (ACE) or chord detection, is the task of automatically identifying the chords in a piece of music and when they occur. A chord recognition system splits a recording into segments and assigns a chord label, such as C major or A minor, to each one.^[1:1] The most detailed versions of the task also identify each chord's bass note and its exact start time and duration.^[2:1] It is a standard task in music information retrieval (MIR), and most audio-based systems start from chroma features, which summarize how much energy a recording has in each of the 12 pitch classes.^[3:1]

Takuya Fujishima's 1999 system, which compared "pitch class profiles" computed from audio with templates for different chord types, started an active line of research.^[4:1][5:1] Over the following two decades the field moved from systems built on hand-coded musical knowledge to systems trained on data, and more recently it has been dominated by deep learning.^[5:2][5:3] Consumer services such as Chordify use chord recognition to show people the chords of songs they want to play.^[6:1]

Major and minor triads are recognized more reliably than more complex chords, and results have improved enough that ambiguity in the chord labels themselves has become a research topic.^[5:4] Expert annotators who label the same songs often disagree, which limits how meaningfully systems can be evaluated against any single reference.^[7:1][8:1]

The task

Output and chord vocabularies

A chord recognizer's output is a sequence of time segments, each with a chord label drawn from a fixed vocabulary of possible labels. The most common simplification is the 24 major and minor triads, one of each on every root note. Bello and Pickens used this vocabulary in 2005, and Meinard Müller and Christof Weiß describe the choice as convenient but problematic from a musical point of view.^[9:1][1:2] Later systems use larger vocabularies: one described in 2015 has 157 classes (13 chord qualities on each of 12 roots, plus "no chord"), and one from 2017 has 170.^[3:2][10:1]

Larger vocabularies run into a long tail of rare chords. In the Billboard dataset of American popular music, annotators used 230 different chord qualities, or 2,749 distinct chords counting every root.^[11:1] The most common 10% of those qualities cover 93.86% of the data, while the rarest half together cover only 0.35%.^[11:2] Across popular-music datasets, the five most common chord types account for over 80% of the data, an imbalance that affects both how systems are trained and how they are evaluated.^[5:5]

Chord labels

In 2005, Christopher Harte and colleagues noted that there was no standard method for writing chord annotations, and that the lack of annotated collections made chord recognition systems hard to evaluate and compare.^[12:1] In their proposed text syntax, a label gives the root, a colon, the chord's quality, and optionally a bass note after a slash. The quality can be written as an explicit list of scale degrees or as a shorthand, so that C:min7 is equivalent to C:(b3,5,b7). A single N means "no chord", for silence or for untuned, possibly percussive sounds.^{[12:2][12:3][12:4]} Evaluation software such as mir_eval parses labels of this form: G:maj(6)/5, for example, is broken down into a root (G), a set of active pitch classes (a major triad with an added sixth), and a bass note (the fifth).^[13:1]

Why it is difficult

Chords in recordings are not neat sets of pitches. Each note also produces harmonics, so in real recordings the energy is spread across the chroma bands in an unstructured way rather than in the clean pattern of a written chord.^[1:3] A single note can put substantial energy into the bands of its fifth and its major third, which often leads systems to confuse major and minor chords.^[8:2] Matthias Mauch and Simon Dixon point out that the bass note's overtones in particular fall at the frequencies where other notes have their fundamentals.^[14:1] Inversions, chords with a note other than the root in the bass, are especially hard: in one 2010 baseline system they were recognized about 40 percentage points less accurately than the same chords in root position.^[14:2]

Choosing a vocabulary involves a trade-off. Many errors come from forcing music into an oversimplified set of chords, but adding more chord classes also creates more chances for confusion between them.^[8:3]

Approaches

Until the spread of deep learning, most systems shared a common architecture. They computed chroma features from short frames of audio, optionally processed them further, compared each frame with a set of chord models, and finally smoothed the resulting chord likelihoods into a sequence of labels.^[3:1]

Chroma and templates

The traditional approach before Fujishima's work was to transcribe the individual notes first and then infer the chord from them. Fujishima argued that this approach suffered from transcription errors caused by noise and overlapping harmonics.^[4:2] His 1999 system skipped note identification. It computed a Pitch Class Profile, a 12-dimensional vector of the intensity of each semitone pitch class, and compared it with templates for 27 chord types at every possible root.^[4:1][4:3] Because a chord usually lasts for several frames, the system also averaged profiles over time to reduce noise.^[4:4] It correctly distinguished all 27 chords played on an electronic keyboard, and reached 94% accuracy on a 50-second excerpt of Smetana's Moldau, although templates tuned for keyboard input did not work on recorded music and vice versa.^[4:5][4:6] Fujishima noted that the profile discards information about the bass note, and listed bass-note tracking and the use of musical context among his plans for future work.^[4:7][4:8] Subsequent systems quickly reduced the 27 chord types to two or four types of triad.^[5:6]

In the simplest form of the template method, each template is an idealized chroma vector: the C major template has ones at C, E and G and zeros elsewhere, and templates for the other roots are obtained by shifting it.^[1:4] Matching each frame on its own, however, produces many rapid, random-looking chord changes.^[15:1]

Probabilistic sequence models

In 2003, Alexander Sheh and Daniel Ellis used hidden Markov models (HMMs), in which the chords are hidden states that change over time and produce the observed features. Their models could be trained with the expectation–maximization algorithm from just the sequence of chords in each training song, without the time of each chord change.^[16:1] On 20 early Beatles songs, the system aligned known chord sequences with the audio at about 75% frame-level accuracy, but the authors judged its recognition of unknown songs not yet accurate enough to give usable transcriptions.^[16:2][16:3] Pitch class profiles performed up to four times better as features than mel-frequency cepstral coefficients.^[16:4]

One benefit of HMMs is smoothing: if the model gives a high probability to staying on the same chord from one frame to the next, the recognizer tends to stay on the current chord rather than switching at every fluctuation.^[15:2] Juan Pablo Bello and Jeremy Pickens (2005) built musical knowledge into this framework. They averaged chroma within each beat rather than over fixed frames, and set the initial transition probabilities so that chords close together on the circle of fifths were more likely to follow each other.^[9:2][9:3] Keeping their musically defined chord models fixed during training, rather than letting training adjust them, raised accuracy on 28 Beatles recordings from 42.93% to 75.04%.^[9:4]

Labelling training data by hand is laborious. Kyogu Lee and Malcolm Slaney (2008) avoided it by running automatic harmony analysis on symbolic music files and synthesizing audio from the same files, which gave them audio features perfectly aligned with chord labels. They trained a separate HMM for each of the 24 major and minor keys, so that choosing the best-fitting model also identified the key.^[17:1][17:2]

Mauch and Dixon (2010) modelled more of the musical context at once. Their six-layer dynamic Bayesian network jointly estimates metrical position, key, chord and bass note, with parameters set from expert musical knowledge rather than learned from data.^[18:1][18:2] It classified 71% of chords correctly on the 176 songs of the 2008 MIREX evaluation; each added context variable improved the results, and modelling the bass helped most.^[18:3][18:4] It still recognized chords in first inversion as such only 15% of the time.^[18:5] In a separate 2010 paper, they improved the input features by first making an approximate transcription of the notes, using non-negative least squares. This reached 80% accuracy on the 2009 MIREX data, compared with 74% for the best system submitted that year.^[14:3][14:4]

Deep learning

Jonggwon Park and colleagues identify a convolutional neural network by Eric Humphrey and Juan Pablo Bello, trained to classify major and minor chords, as the first deep-learning chord recognition system.^[19:1] A later convolutional network described by Humphrey and Bello in 2015 takes patches of about one second of a constant-Q spectrum as input, with its output likelihoods rescaled after training to counter the severe imbalance between common and rare chords.^[3:3][3:4]

Filip Korzeniowski and Gerhard Widmer (2016) argued that chroma vectors could in principle carry enough information for chord recognition, but that standard extractors produce noisy features. Using chroma vectors derived from the ground-truth annotations, a simple classifier recognized 97% of the chords in the Beatles songs when reduced to major and minor.^[20:1][20:2] They trained a neural network to compute chroma from audio instead; on 383 songs it scored 78.8% weighted chord symbol recall for major and minor chords, against 69.2% for standard constant-Q chroma.^[20:3]

Large vocabularies make the shortage of training examples for rare chords more acute.^[10:2] Brian McFee and Bello (2017) trained a convolutional-recurrent network over 170 chord classes to predict not only each chord label but also its root and quality, so that the model could exploit the structural relationships between chords. The model remained biased toward major and minor chords, performed poorly on the rarest class (minor-major seventh chords, 0.03% of the data), and did not predict inversions.^{[10:3][10:4][10:5]} Junyan Jiang and colleagues (2019) break each chord label into components such as the triad, bass note and seventh, each with a much smaller vocabulary, recognize the components with a single multitask network, and reassemble them into a full label.^[11:3]

Networks that classify each frame independently tend to produce too many chord changes, so systems often add a decoder, such as an HMM or a conditional random field, to smooth the predicted sequence.^[11:4][19:2] Park and colleagues (2019) used a bi-directional transformer instead, which needs only one training stage and no separate decoder, and obtained results comparable to systems that use one. Analysis of its attention patterns suggested that it had learned to divide the music into chord segments.^[19:3][19:4]

Recognition from symbolic music

Chord recognition can also start from symbolic music, such as MIDI files or MusicXML scores, in which each note's pitch, onset, offset and metrical position is given directly.^[21:1] One difficulty is telling apart the notes of the underlying chord from figuration notes such as passing and neighbouring tones. Kristen Masada and Razvan Bunescu argue that identifying such notes requires features of whole candidate segments rather than of short spans, and they use metrical position as evidence: chord changes tend to fall on strong beats, and passing and neighbouring tones on weak ones.^[21:2][21:3] They evaluated their system on Bach chorales, Mozart and Beethoven theme-and-variation sets, textbook excerpts, and pop and rock songs. It performed worse on the variations, with their many figurations and varied rhythms, than on the mostly note-against-note chorales.^[21:4][21:5]

Evaluation

Campaigns and metrics

The Music Information Retrieval Evaluation eXchange (MIREX) ran an Audio Chord Detection task in 2008, testing pretrained systems on 176 Beatles songs.^[22:1] MIREX typically measures chord symbol recall, the proportion of a song's duration for which the estimated chord matches the reference. When averaging over a collection it weights each song by its length, giving the weighted chord symbol recall (WCSR).^[2:2][13:2]

What counts as a match depends on how finely chords are compared. Following Johan Pauwels and Geoffroy Peeters, MIREX in 2013 computed recall under five vocabulary mappings, ranging from roots only to seventh chords with inversions.^[2:3] The documentation of the mir_eval library states that "there is no single 'right' way to compare two sequences of chord labels", and the library implements each of the conventional comparison rules.^[23:1] Under its "mirex" rule, for example, an estimate counts as correct if it shares at least three pitch classes with the reference, while under its "tetrads" rule A:7 matches A:9 but not A:maj7.^[23:2][23:3] MIREX also scores how well chord boundaries are placed, measuring over- and under-segmentation.^[2:4] The developers of mir_eval describe evaluation methodology for the task as an ongoing topic of discussion.^[13:3]

Datasets

Harte's hand-labelled transcriptions of all 180 songs on the Beatles' 12 studio albums were made publicly available and became widely used for evaluation.^[24:1][24:2] They later became part of the Isophonics collection, along with annotations of songs by Queen, Carole King and Zweieck.^[3:5] In 2011, 174 of the 210 songs (83%) in the MIREX dataset were by the Beatles, which John Ashley Burgoyne and colleagues argued made it impossible to draw general conclusions about harmony in other music.^[25:1] Their McGill Billboard dataset instead sampled more than a thousand songs at random from the US Billboard Hot 100 chart between 1958 and 1991. Two annotators transcribed each song, and a third compared and combined their versions.^[25:2][25:3] More than 700 of its annotations have been made public.^[3:6] MIREX 2013 evaluated systems on both collections and on a fresh set of 200 unreleased songs.^[2:5] Pauwels and colleagues note that most available annotated data is English-language pop music, which makes its representativeness questionable.^[5:7]

Subjectivity and ground truth

Evaluation assumes a correct reference annotation, but for chords this is often unclear: Müller and Weiß note that, in general, no well-defined ground truth exists and even music experts may disagree.^[8:1] Studies of annotator agreement report average agreement on chord roots between 76% and 94%, with much lower agreement on some individual songs.^[5:8] Yizhao Ni and colleagues (2013) found that five annotators of 20 Beatles and Queen songs disagreed on roughly 10% of harmonic annotations.^[7:2] For the Beatles' "I Saw Her Standing There", which has several independent reference annotations, no two agreed on more than 65% at the level of four-note chords.^[3:7]

In 2019, Hendrik Vincent Koops and colleagues had four expert annotators label the same 50 songs. On average the annotators overlapped on only 73% of labels using the major/minor vocabulary and 54% using the most detailed labels, and agreed on only 76% of root notes.^[7:1][7:3] They describe this as a harmonic "subjectivity ceiling", an upper bound for evaluation, and report that leading systems score about 10% above it, which they suggest means the systems are overfitting to a single annotator's subjective view.^[7:4][7:5]

Humphrey and Bello (2015) drew four broader conclusions about the field's methods: that not all music content is suited to chord estimation; that conventional comparison methods struggle to capture the relationships between chords; that research has mixed the partly conflicting goals of chord transcription and chord recognition; and that the subjectivity of chord perception may make objective ground truth and evaluation untenable.^[3:8]

Applications

Chordify, launched in January 2013, is a web service that transcribes the chords of uploaded audio files or YouTube and SoundCloud links so that people can play along on an instrument.^[26:1][6:1] Its HarmTrace software computes chord candidates at each beat. Where the audio clearly matches a chord it uses that chord, and otherwise a model of Western tonal harmony selects the chord that best fits the harmonic rules.^[6:2] According to the company, by 2015 it had more than 250,000 registered users and had analysed 3 million songs, and it had added a feature that lets users correct mistakes in the recognized chords.^[26:1][26:2]

Chord changes can also serve as evidence for other MIR tasks. Masataka Goto's 2001 real-time beat tracking system, for example, used knowledge of chord changes, along with note onsets and drum patterns, to infer the beat structure of popular music.^[27:1]