Music information retrieval

Music information retrieval (MIR) is the interdisciplinary field concerned with extracting, organizing, searching, and recommending information from music. It draws on signal processing, machine learning, information retrieval, musicology, music psychology, and library science. MIR deals with music in several forms: audio recordings, symbolic encodings such as scores and MIDI files, and the metadata and listening data that surround them.^[1][2]

Typical MIR tasks range from identifying a specific recording from a short excerpt, to finding songs by humming a melody, to classifying music by genre or mood, transcribing audio into notation, separating the instruments in a mix, and recommending music to listeners. Early content-based systems relied on hand-designed audio features; since the 2010s, deep learning techniques have become widely adopted in MIR research.^[3][4]

History

An early use of the phrase appears in the title of Michael Kassler's 1966 article "Toward Musical Information Retrieval" in Perspectives of New Music.^[5]

In the mid-1990s, research on retrieving music from audio queries became more prominent. A 1995 system by Asif Ghias and colleagues let users find a melody in a database by humming it: the system transcribed the hummed query into a coarse pitch contour and matched it against stored melodies using approximate string matching.^[6]

The field consolidated around 2000 with the first ISMIR conference, held in Plymouth, Massachusetts, on October 23–25, 2000.^[7] Joe Futrelle and J. Stephen Downie reviewed the interdisciplinary research issues raised at the first three ISMIR meetings, from 2000 to 2002.^[8] The International Society for Music Information Retrieval now oversees the annual ISMIR conference and publishes the journal Transactions of the International Society for Music Information Retrieval (TISMIR), established to complement the conference proceedings.^[9]

Beginning in 2005, the Music Information Retrieval Evaluation eXchange (MIREX) provided a community-based framework for formally evaluating MIR systems on shared tasks in an annual cycle. Reviewing its first three years, Downie observed that both MIREX and MIR as a whole had a strong bias toward audio-based approaches, and that spectral-based approaches, after driving steady advances, appeared to be reaching their limits of effectiveness, a limitation known as the "glass ceiling" problem.^[10:1]

Representations and features

Michael Casey and colleagues characterize MIR systems along a scale of specificity. High-specificity systems match instances of particular audio content, such as identifying an exact recording. Mid-specificity systems match higher-level musical features such as melody, as in cover-song identification. Low-specificity systems match global, statistical properties of the query, as in genre retrieval.^[3:1]

Content-based systems typically begin with low-level features computed over short frames of the audio signal, then aggregate them over longer time spans or group them into musically meaningful segments.^[3] Common representations include time–frequency transforms such as the short-time Fourier transform and constant-Q transform; spectral envelope descriptors such as mel-frequency cepstral coefficients, which summarize timbre; chroma features, which summarize energy across the 12 pitch classes of Western music; and onset and novelty functions used for rhythm and structure analysis.^[11][12]

Symbolic representations encode notes, durations, and other score information explicitly. They support tasks such as melodic similarity, harmonic analysis, and pattern discovery without the uncertainty introduced by audio analysis, but they require that a transcription already exist.^[1][3]

Tasks

Identification and matching

Audio fingerprinting identifies a specific recording from a short, possibly noisy excerpt. Avery Wang's algorithm, developed for the Shazam service, reduces a spectrogram to a sparse "constellation" of spectral peaks and indexes pairs of peaks as hashes. This lets the system identify recordings in a large database within milliseconds, even with background noise or several overlapping tracks.^[13]

Mid-specificity matching tasks include query by humming,^[6] cover-song and version identification,^[3] and audio-to-score alignment, which synchronizes a recording with its notated score.^[12] Cross-modal approaches link audio directly to images of sheet music without an intermediate symbolic transcription; Matthias Dorfer and colleagues trained convolutional networks to embed short excerpts of audio and sheet-music images in a shared space, supporting both score identification from audio queries and retrieval of performances from a score.^[14:1]

Classification and tagging

Music genre classification has been one of the most-studied MIR tasks.^[15] George Tzanetakis and Perry Cook's 2002 paper proposed feature sets describing timbral texture, rhythmic content, and pitch content for classifying audio by genre.^[16] The dataset associated with it, GTZAN, became the most-used public dataset for evaluating genre recognition. Bob Sturm later catalogued faults in GTZAN—repetitions, mislabelings, and distortions—and showed that they affect different systems in different ways, which challenges the interpretability of results derived from it.^[17:1]

Related tasks include mood and emotion recognition, instrument recognition, and auto-tagging, in which a system predicts free-form descriptive labels for a track.^[2][4]

Transcription and analysis

Automatic music transcription converts an audio recording into some form of music notation. It comprises several subtasks, including multi-pitch estimation, onset and offset detection, instrument recognition, beat and rhythm tracking, interpretation of expressive timing and dynamics, and score typesetting.^[18:1] Other analysis tasks include beat and tempo tracking, chord recognition, key detection, and music structure analysis, which segments a piece into sections such as verses and choruses.^[11][12]

Source separation

Music source separation attempts to recover individual instruments or voices from a mixed recording. Separated tracks, often called stems, let listeners and engineers rebalance a mix—for example, making vocals louder or suppressing an unwanted sound—upmix stereo recordings to surround formats, or move an instrument to a different spatial position.^[19:1]

Recommendation and discovery

Music recommendation combines content-based analysis with collaborative filtering and information about listeners. The growth of streaming services, which make very large catalogs available to listeners, has made recommendation a major application area. Markus Schedl and colleagues argue that a central open challenge is building and evaluating recommenders that go beyond user–item interactions and content descriptors to model listeners' needs, preferences, and intentions.^[20:1] Large datasets such as the Million Song Dataset, which provides audio-derived features and metadata for 1 million tracks but no audio, were created in part to support research at commercial scale.^[21:1]

Methods

Early content-based MIR largely followed a pipeline of hand-designed features followed by a statistical classifier or similarity measure.^[3] In 2013, Eric Humphrey, Juan Bello, and Yann LeCun argued for feature learning and deep architectures as new directions for the field, in place of hand-crafted representations.^[22] Convolutional neural networks and recurrent neural networks subsequently became standard tools for tagging, transcription, and separation.^[4][18][19]

More recent work trains large models with self-supervised learning on unlabeled music audio, then adapts them to many downstream tasks. MERT, for example, is pretrained by predicting masked portions of audio using pseudo-labels from two teacher models: an acoustic teacher based on a residual-vector-quantized variational autoencoder, and a musical teacher based on the constant-Q transform. Its authors evaluated it on 14 music-understanding tasks.^[23:1]

Evaluation

Evaluation has been a recurring concern in MIR. Downie traced many of the field's challenges to the intellectual property issues surrounding music. Among other effects, these issues left researchers unable to test their approaches against MIREX's collections outside the annual evaluation cycle.^[10:2]

Reviewing roughly 500 published works on genre recognition, Sturm argued that the task had progressed little in a decade because most evaluations lacked validity: they did not show that a system was using criteria relevant to genre. He found similar problems in research on music emotion recognition and auto-tagging.^[15:1]

Ethical and social dimensions

Andre Holzapfel, Bob Sturm, and Mark Coeckelbergh argue that MIR technology is not value-neutral, because design choices give it unintended and ethically relevant implications. They highlight biases introduced by the algorithms, datasets, and evaluation measures used in MIR, and the cultural relativity of copyright: tools such as cover-song detection and automated rights management build on Anglo-American legal concepts that may treat other musical traditions unfairly. They also point to problems of dataset availability and quality in MIR research, and to the remoteness of technology development from end users.^[24:1]

Wikidata: Q1163335