Learning deep models from synthetic data for extracting dolphin whistle contours

Pu Li; Xiaobai Liu; Kaitlin Palmer; Erica Fleishman; Douglas Michael Gillespie; Eva-Marie Nosal; Yu Shiu; Holger Klinck; Danielle Cholewiak; Tyler Helble; Marie Roch

doi:10.1109/IJCNN48605.2020.9206992

Learning deep models from synthetic data for extracting dolphin whistle contours

Pu Li, Xiaobai Liu^*, Kaitlin Palmer, Erica Fleishman, Douglas Michael Gillespie, Eva-Marie Nosal, Yu Shiu, Holger Klinck, Danielle Cholewiak, Tyler Helble, Marie Roch

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We present a learning-based method for extracting whistles of toothed whales (Odontoceti) in hydrophone recordings. Our method represents audio signals as time-frequency spectrograms and decomposes each spectrogram into a set of time-frequency patches. A deep neural network learns archetypical patterns (e.g., crossings, frequency modulated sweeps) from the spectrogram patches and predicts time-frequency peaks that are associated with whistles. We also developed a comprehensive method to synthesize training samples from background environments and train the network with minimal human annotation effort. We applied the proposed learn-from-synthesis method to a subset of the public
Detection, Classification, Localization, and Density Estimation (DCLDE) 2011 workshop data to extract whistle confidence maps, which we then processed with an existing contour extractor to produce whistle annotations. The F1-score of our best synthesis method was 0.158 greater than our baseline whistle extraction algorithm (~25% improvement) when applied to common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.

Original language	English
Title of host publication	2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings
Publisher	IEEE Computer Society
Number of pages	10
ISBN (Electronic)	9781728169262
DOIs	https://doi.org/10.1109/IJCNN48605.2020.9206992
Publication status	Published - Jul 2020
Event	IEEE World Congress on Computational Intelligence (IEEE WCCI) - 2020 International Joint Conference on Neural Networks (IJCNN 2020) - Glasgow, United Kingdom Duration: 19 Jul 2020 → 24 Jul 2020

Publication series

Name	Proceedings of the International Joint Conference on Neural Networks

Conference

Conference	IEEE World Congress on Computational Intelligence (IEEE WCCI) - 2020 International Joint Conference on Neural Networks (IJCNN 2020)
Abbreviated title	IJCNN
Country/Territory	United Kingdom
City	Glasgow
Period	19/07/20 → 24/07/20

Keywords

Whistle contour extraction
Deep neural network
Data synthesis
Acoustic
Odontocetes

Access to Document

10.1109/IJCNN48605.2020.9206992

PID6435539
Copyright © 2020 IEEE. This work has been made available online in accordance with publisher policies or with permission. Permission for further reuse of this content should be sought from the publisher or the rights holder. This is the author created accepted manuscript following peer review and may differ slightly from the final published version. The final published version of this work is available at https://ieeexplore.ieee.org
Accepted author manuscript, 3.08 MBLicence: Unspecified

Cite this

Li, P., Liu, X., Palmer, K., Fleishman, E., Gillespie, D. M., Nosal, E.-M., Shiu, Y., Klinck, H., Cholewiak, D., Helble, T., & Roch, M. (2020). Learning deep models from synthetic data for extracting dolphin whistle contours. In 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings Article 9206992 (Proceedings of the International Joint Conference on Neural Networks). IEEE Computer Society. https://doi.org/10.1109/IJCNN48605.2020.9206992

@inproceedings{bbb67a66ee56485890d4201454b839fb,

title = "Learning deep models from synthetic data for extracting dolphin whistle contours",

abstract = "We present a learning-based method for extracting whistles of toothed whales (Odontoceti) in hydrophone recordings. Our method represents audio signals as time-frequency spectrograms and decomposes each spectrogram into a set of time-frequency patches. A deep neural network learns archetypical patterns (e.g., crossings, frequency modulated sweeps) from the spectrogram patches and predicts time-frequency peaks that are associated with whistles. We also developed a comprehensive method to synthesize training samples from background environments and train the network with minimal human annotation effort. We applied the proposed learn-from-synthesis method to a subset of the publicDetection, Classification, Localization, and Density Estimation (DCLDE) 2011 workshop data to extract whistle confidence maps, which we then processed with an existing contour extractor to produce whistle annotations. The F1-score of our best synthesis method was 0.158 greater than our baseline whistle extraction algorithm (~25% improvement) when applied to common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.",

keywords = "Whistle contour extraction, Deep neural network, Data synthesis, Acoustic, Odontocetes",

author = "Pu Li and Xiaobai Liu and Kaitlin Palmer and Erica Fleishman and Gillespie, {Douglas Michael} and Eva-Marie Nosal and Yu Shiu and Holger Klinck and Danielle Cholewiak and Tyler Helble and Marie Roch",

year = "2020",

month = jul,

doi = "10.1109/IJCNN48605.2020.9206992",

language = "English",

series = "Proceedings of the International Joint Conference on Neural Networks",

publisher = "IEEE Computer Society",

booktitle = "2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings",

address = "United States",

note = "IEEE World Congress on Computational Intelligence (IEEE WCCI) - 2020 International Joint Conference on Neural Networks (IJCNN 2020), IJCNN ; Conference date: 19-07-2020 Through 24-07-2020",

}

Li, P, Liu, X, Palmer, K, Fleishman, E, Gillespie, DM, Nosal, E-M, Shiu, Y, Klinck, H, Cholewiak, D, Helble, T & Roch, M 2020, Learning deep models from synthetic data for extracting dolphin whistle contours. in 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings., 9206992, Proceedings of the International Joint Conference on Neural Networks, IEEE Computer Society, IEEE World Congress on Computational Intelligence (IEEE WCCI) - 2020 International Joint Conference on Neural Networks (IJCNN 2020), Glasgow, United Kingdom, 19/07/20. https://doi.org/10.1109/IJCNN48605.2020.9206992

Learning deep models from synthetic data for extracting dolphin whistle contours. / Li, Pu; Liu, Xiaobai; Palmer, Kaitlin et al.
2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings. IEEE Computer Society, 2020. 9206992 (Proceedings of the International Joint Conference on Neural Networks).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Learning deep models from synthetic data for extracting dolphin whistle contours

AU - Li, Pu

AU - Liu, Xiaobai

AU - Palmer, Kaitlin

AU - Fleishman, Erica

AU - Gillespie, Douglas Michael

AU - Nosal, Eva-Marie

AU - Shiu, Yu

AU - Klinck, Holger

AU - Cholewiak, Danielle

AU - Helble, Tyler

AU - Roch, Marie

PY - 2020/7

Y1 - 2020/7

N2 - We present a learning-based method for extracting whistles of toothed whales (Odontoceti) in hydrophone recordings. Our method represents audio signals as time-frequency spectrograms and decomposes each spectrogram into a set of time-frequency patches. A deep neural network learns archetypical patterns (e.g., crossings, frequency modulated sweeps) from the spectrogram patches and predicts time-frequency peaks that are associated with whistles. We also developed a comprehensive method to synthesize training samples from background environments and train the network with minimal human annotation effort. We applied the proposed learn-from-synthesis method to a subset of the publicDetection, Classification, Localization, and Density Estimation (DCLDE) 2011 workshop data to extract whistle confidence maps, which we then processed with an existing contour extractor to produce whistle annotations. The F1-score of our best synthesis method was 0.158 greater than our baseline whistle extraction algorithm (~25% improvement) when applied to common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.

AB - We present a learning-based method for extracting whistles of toothed whales (Odontoceti) in hydrophone recordings. Our method represents audio signals as time-frequency spectrograms and decomposes each spectrogram into a set of time-frequency patches. A deep neural network learns archetypical patterns (e.g., crossings, frequency modulated sweeps) from the spectrogram patches and predicts time-frequency peaks that are associated with whistles. We also developed a comprehensive method to synthesize training samples from background environments and train the network with minimal human annotation effort. We applied the proposed learn-from-synthesis method to a subset of the publicDetection, Classification, Localization, and Density Estimation (DCLDE) 2011 workshop data to extract whistle confidence maps, which we then processed with an existing contour extractor to produce whistle annotations. The F1-score of our best synthesis method was 0.158 greater than our baseline whistle extraction algorithm (~25% improvement) when applied to common dolphin (Delphinus spp.) and bottlenose dolphin (Tursiops truncatus) whistles.

KW - Whistle contour extraction

KW - Deep neural network

KW - Data synthesis

KW - Acoustic

KW - Odontocetes

U2 - 10.1109/IJCNN48605.2020.9206992

DO - 10.1109/IJCNN48605.2020.9206992

M3 - Conference contribution

AN - SCOPUS:85093866240

T3 - Proceedings of the International Joint Conference on Neural Networks

BT - 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings

PB - IEEE Computer Society

T2 - IEEE World Congress on Computational Intelligence (IEEE WCCI) - 2020 International Joint Conference on Neural Networks (IJCNN 2020)

Y2 - 19 July 2020 through 24 July 2020

ER -

Li P, Liu X, Palmer K, Fleishman E, Gillespie DM, Nosal EM et al. Learning deep models from synthetic data for extracting dolphin whistle contours. In 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings. IEEE Computer Society. 2020. 9206992. (Proceedings of the International Joint Conference on Neural Networks). doi: 10.1109/IJCNN48605.2020.9206992

Learning deep models from synthetic data for extracting dolphin whistle contours

Abstract

Publication series

Conference

Keywords

Access to Document

Fingerprint

Cite this