Automatic harmonisation

Automatic harmonisation is the use of computers to supply harmony for a given musical line, usually a melody. In popular-music settings this typically means choosing a sequence of chords to accompany the melody;^[1:1] in the classical tradition it often means composing the remaining voices of a piece in four parts, as in the harmonisation of chorale melodies.^[2:1][3:1] The task is hard to define precisely because a melody can be harmonised in many acceptable ways, and which harmonisation sounds best is subjective and depends on genre and context.^[1:1]

Early systems encoded the rules of harmony found in music theory treatises and searched for solutions that satisfied them.^[2:2] Later systems learned from examples instead, using hidden Markov models and then neural networks.^[3:1][4:1]

The problem

François Pachet and Pierre Roy define automatic harmonisation as producing musical arrangements from given melodies in a way that satisfies the rules of harmony, and note that most research has concerned tonal music. Treatises on harmony state their rules mostly as prohibitions. In four-part writing for soprano, alto, tenor, and bass, for example, each voice must stay within its range, voices should not cross, certain melodic intervals such as the tritone are forbidden, and two voices should not move in parallel fifths or octaves from one chord to the next. Pachet and Roy stress that obeying these rules does not guarantee a musically interesting result; in their analogy, the rules are merely syntactic.^[2:1]

Composition teaching gives several versions of the problem: harmonising a given melody, realising an unfigured bass, realising a figured bass whose figures indicate the chords (as keyboard players once did in real time), or finding only a bass line for a melody. When one voice is fixed, the task can be treated as a constraint satisfaction problem, in which each note to be chosen is a variable restricted to its voice's range and the rules of harmony are constraints.^[2:3]

Chorale harmonisation, a traditional exercise in the training of classical musicians, asks for three further parts to accompany a given melody.^[3:1] The 389 chorale harmonisations by Johann Sebastian Bach, in which he set well-known Lutheran hymn tunes for four voices, have become a standard body of examples for computer systems.^[4:1]

Rule-based and constraint-based systems

According to Pachet and Roy, Bill Schottstaedt described a complete four-part harmonisation system in 1984, based on Johann Joseph Fux's counterpoint treatise. Kemal Ebcioğlu's system of 1987 was probably the first to produce high-quality four-part music entirely automatically. It contained about 350 rules representing Bach's style, written in a constraint logic programming language designed for the task, but it was difficult to understand and assess because it was implemented in a proprietary language and contained many interrelated modules.^[2:2] Ebcioğlu himself acknowledged that its harmonisations did not sound like Bach, apart from occasional Bach-like patterns and cadences.^[4:1]

Writing in 2001, Pachet and Roy considered the technical problem of four-voice harmonisation solved by constraint satisfaction techniques, but regarded the problem of producing musically interesting results as unsolved.^[2:4]

Learning from examples

As an alternative to hand-written rules, Hermann Hild and colleagues used several neural networks in 1992, each trained for a different part of the task; Pachet and Roy reported that such learning approaches had produced excellent results.^[4:1][2:4] Moray Allan and Christopher Williams trained hidden Markov models on Bach's chorales, producing a system that learns from examples and can compose new harmonisations.^[3:1] Hidden Markov models were also used in MySong, which chose chords for a melody sung into a microphone; in a study, its chords received ratings similar to chords chosen by musicians.^[5:1]

Deep learning systems followed. DeepBach, trained on Bach's chorales, generates music by repeatedly resampling notes rather than composing from beginning to end, and lets users fix particular notes, rhythms, or cadences before generating the rest; it was released with a plugin for the MuseScore notation editor.^[4:1] Coconet, a convolutional neural network trained to complete partial scores, likewise avoids composing in a fixed order; its authors point out that human composers work nonlinearly and that rules of counterpoint can contradict one another.^[6:1]

Yin-Cheng Yeh and colleagues compared approaches to chord-based melody harmonisation on 9,226 melody–chord pairs, testing template matching, a hidden Markov model, a genetic algorithm, and two deep learning models. In a listening study with 202 participants, the deep learning models generally performed better. A model that took account of the harmonic function of chords performed best, and listeners did not rate it significantly lower than the original human-written progressions for interest or overall quality, though they did for coherence. Among the other automatic methods, the genetic algorithm produced the most interesting progressions.^[1:1]

Real-time harmonisation

Many electronic keyboards offer basic automatic harmonisation, such as one-touch chords. Pachet and Roy note that these typically pick a chord for each note without considering the preceding melody, since real-time operation rules out the search methods used by offline systems.^[2:4] More recent research on machine accompaniment has addressed real-time harmonisation with machine learning. ReaLchords generates chords to accompany a melody as it is played, choosing each chord before hearing the melody note it will accompany.^[7:1] In its evaluation, a model that could see the whole melody in advance still fitted its chords to the melody more closely than any of the real-time models.^[7:2]

Evaluation

Because there is no single correct harmonisation, systems are often evaluated by listeners. The DeepBach authors ran a "Bach or computer" test in which 1,272 people judged whether excerpts were by Bach or generated by a model.^[4:1] MySong's developers compared its chords with chords chosen by musicians,^[5:1] and Yeh and colleagues combined listener ratings with objective measures.^[1:1]