FROM AGPEDIA — AGENCY THROUGH KNOWLEDGE

Computational auditory scene analysis

Computational auditory scene analysis (CASA) is the field of study that attempts to replicate auditory scene analysis, the human ability to form separate perceptual representations of the sources in a sound mixture, in machines.[1:1] It grew from Albert Bregman's account of auditory scene analysis, published in 1990, and is motivated in part by practical uses of sound separation, including speech recognition in noise, hearing prostheses and automatic music transcription.[2:1][2:2]

Some CASA systems closely model the known stages of auditory processing, while others take a more functional approach.[1:2] A common output is a time–frequency mask, which keeps the parts of a sound's spectrogram that belong to the target and removes the rest; DeLiang Wang proposed the ideal binary mask as a main computational goal of the field.[3:1][4:1] In speech and music separation, supervised learning and deep learning have since largely replaced hand-designed grouping rules.[5:1][6:1] The broader study of how machines can make sense of everyday sound, often called machine listening, is described as related to CASA.[7:1]

History

Early systems

Mitchel Weintraub's 1985 Stanford doctoral thesis defined sound separation as "the processing of an incoming acoustic signal which assists in the recognition of each of the sounds that are present in the listener's environment."[8:1] His computational model, built to separate two simultaneous talkers, took as its input Richard F. Lyon's computational model of the cochlea, and began by tracking the pitch period of each voice.[8:2][8:3][8:4] Guy Brown and Wang credit Weintraub as the first to use a time–frequency mask in a CASA system.[1:3]

Lyon's 1983 binaural model combined corresponding frequency channels from the two ears by cross-correlation and used quickly varying gains to separate sound fragments into streams for different sources; he presented it as "a first step" towards human performance in localization and separation.[9:1][9:2][9:3] The same paper called a picture of the cochlear model's output a cochleagram, resembling a spectrogram with a distorted frequency scale and finer time resolution.[9:4]

Wang and Brown name later multistage models by Cooke, David Mellinger, Brown and Cooke, and Daniel Ellis.[10:1] Mellinger's 1991 thesis described a model that groups parts of the spectrum over time into separate sound events, with filters tuned to find and separate the notes in musical sound.[11:1][11:2] Brown's 1992 Sheffield thesis, titled Computational auditory scene analysis: a representational approach, separated speech from a variety of interfering noises by simulating the auditory periphery and then grouping auditory objects with similar periodicity, onset time or offset time.[12:1][12:2][12:3] These systems set out to duplicate the organizing function of the auditory system, to separate speech from interference or the instruments in ensemble music.[13:1]

In his 1996 thesis, Ellis criticized such "data-driven" systems: they interpret a sound the same way regardless of context, and cannot infer a sound whose evidence is hidden by other sounds. He proposed a "prediction-driven" alternative that reconciles the observed features with the predictions of an internal model of the sound sources.[13:2][13:3] Writing in 1999, Brown and Wang described CASA systems as "inspired by auditory function but do not model it closely", relying on symbolic search or high-level inference, and judged that CASA remained "an unsolved problem for real-time applications such as automatic speech recognition."[14:1][14:2]

Building blocks

Peripheral filtering

Many CASA systems begin by modeling the frequency analysis of the inner ear. The gammatone filter, an approximation to the impulse responses recorded from auditory nerve fibers, is often used for this stage.[1:4] Roy Patterson and colleagues described a gammatone filterbank in 1987 to perform the initial frequency analysis in models of hearing and speech perception, noting that it predicts human masking data accurately and runs almost as fast as the simpler filterbanks then used in speech recognition.[15:1][15:2] It agrees well with both physiological and psychoacoustic estimates of frequency selectivity.[10:2] A typical system of the late 1990s used 128 gammatone filters with center frequencies spaced on the equivalent rectangular bandwidth (ERB) scale between 80 Hz and 5 kHz.[14:3]

Pitch and the correlogram

Grouping by pitch, a cue taken from human auditory scene analysis, relies on representations of periodicity. The correlogram is a two-dimensional image in which each row is the running short-time autocorrelation of one cochlear channel.[16:1] Brown and Wang trace it to Licklider's theory of pitch: Weintraub gave the first computational implementation, calling it an "auto-coincidence" representation, and Malcolm Slaney and Lyon later introduced the term correlogram.[1:5][16:2]

Grouping by oscillatory correlation

In 1995 Wang and David Terman proposed a network of oscillators as a foundation for oscillatory correlation, a theory of how features are bound into objects, and as a framework for segmenting scenes:[17:1] each object is represented by a group of oscillators that oscillate in synchrony, and different objects by groups that are desynchronized from each other.[17:2] Wang and Brown applied this to speech in 1999, representing each stream as a population of synchronized oscillators.[10:3]

Speech segregation systems

Guoning Hu and Wang's 2004 system separated voiced speech by treating resolved and unresolved harmonics differently, grouping the unresolved ones by their common amplitude modulation.[18:1][18:2] Measured against the ideal binary mask, it retained on average about 96% of the target speech energy while letting through about 3% noise.[18:3] Because it grouped by pitch alone, it handled only voiced speech; the authors saw unvoiced speech as "the biggest challenge" for separation from a single microphone.[18:4] Nicoleta Roman, Wang and Brown instead used the interaural time and intensity differences between two ears' signals, learned by a supervised method; the system produced masks very close to ideal ones, and in listening tests its intelligibility gain grew as the signal-to-noise ratio fell, reaching 61% at −10 dB.[19:1][19:2][19:3]

Time–frequency masking and the ideal binary mask

Time–frequency masking originated in CASA and separates sources in the time–frequency domain.[3:1] The ideal binary mask (IBM) is a binary matrix marking each time–frequency unit 1 where the target is stronger than the interference and 0 otherwise.[4:2] It requires knowing the target and interference before they are mixed, so it serves as a reference: Wang and colleagues proposed it as an explicit criterion for evaluating CASA systems, and Roman and colleagues argued that it sets a ceiling on the performance of all binary masks.[3:2][19:4][20:1] The same mixture yields different masks depending on which sound is chosen as the target.[4:3]

Listening studies found that the IBM can restore intelligibility in severe noise:

In a 2008 review, Wang reported that IBM processing gives larger benefits for hearing-impaired listeners than for normal-hearing ones, and makes the two groups' intelligibility comparable.[3:3][3:4] Kjems and colleagues suggested this could make the IBM a candidate for hearing aids, "provided that the IBM can be approximated sufficiently well."[23:3] Wang noted that no masking study had yet targeted hearing aids directly.[3:5]

Supervised learning and deep learning

With the ideal binary mask as the goal, separating speech becomes binary classification, an elementary form of supervised learning.[24:1] In 2013, Eric Healy and colleagues used deep neural networks to estimate masks from noisy speech, training on speech not used in testing; intelligibility improved for both normal-hearing and hearing-impaired listeners, most for the hearing-impaired.[20:1][20:2][20:3] The authors describe it as the first single-microphone algorithm shown to improve intelligibility for hearing-impaired listeners in background noise, while stressing that it was "far from ready" for use in devices.[20:4][20:5]

Yuxuan Wang, Arun Narayanan and DeLiang Wang compared training targets in 2014. The IBM had been the traditional target because of its simplicity and large intelligibility gains, but ratio masks, including the ideal ratio mask, which is closely related to the frequency-domain Wiener filter, performed better on objective measures of intelligibility and quality.[25:1][25:2][25:3] They found that masks outperformed direct estimation of the speech spectrum, partly because masks operate directly on the observed mixture.[25:4] Wang and Jitong Chen note that the IBM was the first training target in supervised speech separation, but not the only one.[24:2]

Separating talkers

Separating several talkers of unknown identity raised a further difficulty. Dong Yu and colleagues write that before deep learning, CASA was the most popular approach to multi-talker separation, using often semi-manually designed grouping rules to estimate a mask, and that CASA and other methods "led to very limited success" on such mixtures.[5:1][5:2] A network with one output per talker faces the permutation problem: nothing tells it which talker belongs to which output.[5:3]

Music

Music source separation is the task of isolating the stems (such as voice, bass, drums and other accompaniment) that were recorded separately and mixed into a song.[28:1] Alexandre Défossez and colleagues trace the computational work back to Bregman's framework, which "spawned its computational counterpart".[28:2] Research has recently concentrated on separating four sources in a supervised way, using the MUSDB18 dataset of 150 full-length tracks (about 10 hours) split into bass, drums, vocals and other.[29:1][6:2]

At the 2018 Signal Separation Evaluation Campaign (SiSEC), most submitted music systems used deep learning; the best ones were not significantly different from oracle methods for separating the accompaniment, leading the organizers to suggest that automatic karaoke could "be considered solved to a large extent", while vocal separation still had room for improvement.[6:1][6:3] The ideal binary mask served there as one of the oracle methods.[6:4] At that campaign the best system predicting waveforms directly reached an average SDR (signal-to-distortion ratio, see Evaluation) of 3.2, against 5.3 for the best spectrogram-masking system; Défossez and colleagues' waveform model Demucs reached 6.3 on average.[28:3][28:4] A later Hybrid Demucs lets the model decide which domain suits each source and combine both; it ranked first at the 2021 Music Demixing Challenge among systems trained only on MUSDB, with 7.32 dB SDR.[29:2][29:3] Each representation leaves its own artifacts: Défossez notes that spectrogram methods can make drum and bass attacks sound hollow, while waveform separation of vocals can add "crunchy static noise".[29:4]

Machine listening

CASA's methods have also been applied to everyday sound. Dan Stowell and colleagues describe such "machine listening" tasks as coming under the umbrella of CASA, a name that refers back to Bregman's work, though they note that CASA is often taken to imply an approach that parallels the stages of processing in human hearing or mimics its observed phenomena.[30:1][30:2] In 2012–2013 they organized the first challenge on Detection and Classification of Acoustic Scenes and Events (DCASE), which has since become an annual benchmarking effort.[30:3][31:1]

Sound event detection aims to recognize what is happening in an audio recording and when.[31:2] Natural environments are polyphonic, with several sounds active at once and no fixed rules for how they co-occur.[31:3] One strategy is to separate the sources first and analyze each separately, an approach Stowell and colleagues compare to auditory streaming in Bregman's model.[30:4] Acoustic scene classification assigns a recording a label identifying the environment in which it was made.[7:2]

Evaluation

Separation systems are commonly evaluated with the BSS Eval metrics, which measure the ratio of the target source to distortion, artifacts and interference (SDR, SAR and SIR).[6:5] Jonathan Le Roux and colleagues argue that the SDR as implemented in the BSS Eval toolkit has often been misused, especially for single-channel separation, noting that it completely forgives channel errors that a 512-tap time-invariant filter can account for; they propose a simpler scale-invariant SDR (SI-SDR).[32:1][32:2][32:3] Conv-TasNet was trained to maximize a scale-invariant signal-to-noise ratio directly.[27:4]

  1. ^ ↗ definition ^ ↗ closely-or-functional ^ ↗ weintraub-first-mask ^ ↗ gammatone-front-end ^ ↗ correlogram-origin Brown, Guy J.; Wang, DeLiang (2005). Separation of speech by computational auditory scene analysis. Speech Enhancement. Springer. https://doi.org/10.1007/3-540-27489-8_16 https://pnlwang.github.io/papers/Brown-Wang05.pdf.
  2. ^ ↗ bregman-framework ^ ↗ casa-named Brown, Guy J. Computational Auditory Scene Analysis: Principles, Algorithms and Applications (companion website: About the book). University of Sheffield. https://staffwww.dcs.shef.ac.uk/people/G.Brown/casabook/index.html.
  3. ^a ^b ↗ originated-in-casa ^ ↗ ibm-criterion ^ ↗ larger-for-hi ^ ↗ hi-nh-comparable ^ ↗ no-hearing-aid-studies Wang, DeLiang (2008-12). Time-frequency masking for speech separation and its potential for hearing aid design. Trends in Amplification. https://doi.org/10.1177/1084713808326455 https://pmc.ncbi.nlm.nih.gov/articles/PMC4111459/.
  4. ^ ↗ proposed-goal ^ ↗ definition ^ ↗ target-dependent Wang, DeLiang (2005). On Ideal Binary Mask As the Computational Goal of Auditory Scene Analysis. Speech Separation by Humans and Machines. Kluwer Academic Publishers, Norwell, MA. https://doi.org/10.1007/0-387-22794-6_12 https://pnlwang.github.io/papers/Wang05.pdf.
  5. ^a ^b ↗ casa-before-deep-learning ^ ↗ limited-success ^ ↗ permutation-problem ^ ↗ method Yu, Dong; Kolbæk, Morten; Tan, Zheng-Hua; Jensen, Jesper (2017-03). Permutation invariant training of deep models for speaker-independent multi-talker speech separation. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2017.7952154 https://arxiv.org/abs/1607.00325.
  6. ^a ^b ↗ deep-learning-shift ^ ↗ musdb18 ^ ↗ karaoke ^ ↗ ibm-oracle ^ ↗ bss-eval Stöter, Fabian-Robert; Liutkus, Antoine; Ito, Nobutaka (2018). The 2018 Signal Separation Evaluation Campaign. Latent Variable Analysis and Signal Separation (LVA/ICA 2018), Lecture Notes in Computer Science. Springer. https://doi.org/10.1007/978-3-319-93764-9_28 https://arxiv.org/abs/1804.06267.
  7. ^ ↗ machine-listening ^ ↗ definition Barchiesi, Daniele; Giannoulis, Dimitrios; Stowell, Dan; Plumbley, Mark D. (2015-05). Acoustic Scene Classification: Classifying environments from the sounds they produce. IEEE Signal Processing Magazine. https://doi.org/10.1109/MSP.2014.2326181 https://arxiv.org/abs/1411.3715.
  8. ^ ↗ defines-separation ^ ↗ two-talkers ^ ↗ lyon-front-end ^ ↗ four-steps Weintraub, Mitchel (1985-08). A theory and computational model of auditory monaural sound separation. Stanford University, Department of Electrical Engineering. https://www.ee.columbia.edu/~dpwe/papers/Weintraub85-phd.pdf.
  9. ^ ↗ cross-correlation ^ ↗ streams ^ ↗ first-step ^ ↗ cochleagram Lyon, Richard F. (1983). A computational model of binaural localization and separation. ICASSP ’83: IEEE International Conference on Acoustics, Speech, and Signal Processing. https://doi.org/10.1109/ICASSP.1983.1171927 https://www.dicklyon.com/tech/Scans/ICASSP83_Binaural-Lyon.pdf.
  10. ^ ↗ multistage-models ^ ↗ gammatone-agreement ^ ↗ stream-as-oscillators Wang, DeLiang L.; Brown, Guy J. (1999-05). Separation of speech from interfering sounds based on oscillatory correlation. IEEE Transactions on Neural Networks. https://doi.org/10.1109/72.761727 https://pnlwang.github.io/papers/Wang-Brown.tnn99.pdf.
  11. ^ ↗ groups-into-events ^ ↗ musical-sounds Mellinger, David K. (1991-12). Event formation and separation in musical sound. Stanford University, Department of Computer Science (CCRMA Report STAN-M-77). https://ccrma.stanford.edu/files/papers/stanm77.pdf.
  12. ^ ↗ separates-speech ^ ↗ periphery ^ ↗ grouping Brown, Guy Jason (1992-09). Computational auditory scene analysis: a representational approach. University of Sheffield, Department of Computer Science. https://etheses.whiterose.ac.uk/id/eprint/2982/1/DX202847.pdf.
  13. ^ ↗ early-systems ^ ↗ data-driven-weakness ^ ↗ prediction-driven Ellis, Daniel P. W. (1996-06). Prediction-driven computational auditory scene analysis. Massachusetts Institute of Technology, Cambridge, MA. https://www.ee.columbia.edu/~dpwe/pubs/pdcasa.pdf.
  14. ^ ↗ not-close-models ^ ↗ unsolved ^ ↗ gammatone-128 Brown, Guy J.; Wang, DeLiang L. (2000). An oscillatory correlation framework for computational auditory scene analysis. Advances in Neural Information Processing Systems 12. MIT Press. https://pnlwang.github.io/papers/Brown-Wang.nips99.pdf.
  15. ^ ↗ purpose ^ ↗ advantages Patterson, Roy; Nimmo-Smith, Ian; Holdsworth, John; Rice, Peter (1987-12). An efficient auditory filterbank based on the gammatone function. MRC Applied Psychology Unit, Cambridge. https://www.pdn.cam.ac.uk/system/files/documents/SVOSAnnexB1988.pdf.
  16. ^ ↗ correlogram-definition ^ ↗ licklider Slaney, Malcolm; Lyon, Richard F. (1990). A perceptual pitch detector. International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1990). https://doi.org/10.1109/ICASSP.1990.115684 https://www.dicklyon.com/tech/Hearing/PerceptualPitch-SlaneyLyon.pdf.
  17. ^ ↗ foundation ^ ↗ oscillatory-correlation Wang, DeLiang; Terman, David (1995-01). Locally excitatory globally inhibitory oscillator networks. IEEE Transactions on Neural Networks. https://doi.org/10.1109/72.363423 https://pnlwang.github.io/papers/Wang-Terman.tnn95.pdf.
  18. ^ ↗ resolved-unresolved ^ ↗ am ^ ↗ results ^ ↗ voiced-only Hu, Guoning; Wang, DeLiang (2004-09). Monaural speech segregation based on pitch tracking and amplitude modulation. IEEE Transactions on Neural Networks. https://doi.org/10.1109/TNN.2004.832812 https://pnlwang.github.io/papers/Hu-Wang.tnn04.pdf.
  19. ^ ↗ binaural-cues ^ ↗ masks-close ^ ↗ intelligibility ^ ↗ ibm-ceiling Roman, Nicoleta; Wang, DeLiang; Brown, Guy J. (2003-10). Speech segregation based on sound localization. The Journal of the Acoustical Society of America. https://doi.org/10.1121/1.1610463 https://pnlwang.github.io/papers/RWB.jasa03.pdf.
  20. ^a ^b ↗ estimated-masks ^ ↗ dnn ^ ↗ gains ^ ↗ first-monaural ^ ↗ not-ready Healy, Eric W.; Yoho, Sarah E.; Wang, Yuxuan; Wang, DeLiang (2013-10). An algorithm to improve speech recognition in noise for hearing-impaired listeners. The Journal of the Acoustical Society of America. https://doi.org/10.1121/1.4820893 https://pnlwang.github.io/papers/HYWW.jasa13.pdf.
  21. ^ ↗ near-perfect ^ ↗ energetic-masking ^ ↗ minus-6-db Brungart, Douglas S.; Chang, Peter S.; Simpson, Brian D.; Wang, DeLiang (2006-12). Isolating the energetic component of speech-on-speech masking with ideal time-frequency segregation. The Journal of the Acoustical Society of America. https://doi.org/10.1121/1.2363929 https://pnlwang.github.io/papers/BCSW.jasa06.pdf.
  22. ^ ↗ minus-10-db ^ ↗ plateau ^ ↗ false-alarms ^ ↗ pattern-matters Li, Ning; Loizou, Philipos C. (2008-03). Factors influencing intelligibility of ideal binary-masked speech: implications for noise reduction. The Journal of the Acoustical Society of America. https://doi.org/10.1121/1.2832617 https://pubmed.ncbi.nlm.nih.gov/18345855/.
  23. ^ ↗ sparse-and-dense ^ ↗ target-mask ^ ↗ hearing-aid-candidate Kjems, Ulrik; Boldt, Jesper B.; Pedersen, Michael S.; Lunner, Thomas; et al. (2009-09). Role of mask pattern in intelligibility of ideal binary-masked noisy speech. The Journal of the Acoustical Society of America. https://doi.org/10.1121/1.3179673 https://pnlwang.github.io/papers/KBPLW.jasa09.pdf.
  24. ^ ↗ ibm-classification ^ ↗ ibm-not-only-target Wang, DeLiang; Chen, Jitong (2018-10). Supervised Speech Separation Based on Deep Learning: An Overview. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/TASLP.2018.2842159 https://arxiv.org/abs/1708.07524.
  25. ^ ↗ ibm-traditional ^ ↗ ratio-masks-better ^ ↗ irm ^ ↗ masks-on-mixture Wang, Yuxuan; Narayanan, Arun; Wang, DeLiang (2014-12). On training targets for supervised speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/TASLP.2014.2352935 https://pmc.ncbi.nlm.nih.gov/articles/PMC4293540/.
  26. ^ ↗ casa-lineage ^ ↗ embeddings ^ ↗ permutation-invariant ^ ↗ results Hershey, John R.; Chen, Zhuo; Le Roux, Jonathan; Watanabe, Shinji (2015-08-18). Deep clustering: Discriminative embeddings for segmentation and separation. arXiv. https://arxiv.org/abs/1508.04306.
  27. ^ ↗ tf-drawbacks ^ ↗ time-domain ^ ↗ beats-ideal-masks ^ ↗ si-snr Luo, Yi; Mesgarani, Nima (2019-08). Conv-TasNet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. https://doi.org/10.1109/TASLP.2019.2915167 https://arxiv.org/abs/1809.07454.
  28. ^ ↗ stems ^ ↗ bregman-lineage ^ ↗ spectrogram-masking-ahead ^ ↗ result Défossez, Alexandre; Usunier, Nicolas; Bottou, Léon; Bach, Francis (2019). Music Source Separation in the Waveform Domain. arXiv. https://arxiv.org/abs/1911.13254.
  29. ^ ↗ four-sources ^ ↗ two-domains ^ ↗ mdx ^ ↗ artifacts Défossez, Alexandre (2021). Hybrid Spectrogram and Waveform Source Separation. Proceedings of the MDX Workshop. https://arxiv.org/abs/2111.03600.
  30. ^ ↗ machine-listening-under-casa ^ ↗ human-centric ^ ↗ dcase-origin ^ ↗ separation-strategy Stowell, Dan; Giannoulis, Dimitrios; Benetos, Emmanouil; Lagrange, Mathieu; et al. (2015-10). Detection and Classification of Acoustic Scenes and Events. IEEE Transactions on Multimedia. https://doi.org/10.1109/TMM.2015.2428998 https://openaccess.city.ac.uk/id/eprint/13650/.
  31. ^ ↗ dcase-annual ^ ↗ definition ^ ↗ polyphony Mesaros, Annamaria; Heittola, Toni; Virtanen, Tuomas; Plumbley, Mark D. (2021-09). Sound Event Detection: A tutorial. IEEE Signal Processing Magazine. https://doi.org/10.1109/MSP.2021.3090678 https://arxiv.org/abs/2107.05463.
  32. ^ ↗ sdr-misused ^ ↗ filter-flaw ^ ↗ si-sdr Le Roux, Jonathan; Wisdom, Scott; Erdogan, Hakan; Hershey, John R. (2019-05). SDR – Half-baked or Well Done? ICASSP 2019: IEEE International Conference on Acoustics, Speech and Signal Processing. https://doi.org/10.1109/ICASSP.2019.8683855 https://arxiv.org/abs/1811.02508.

Wikidata: Q5157303

Available in