Learning stage-wise GANs for whistle extraction in time-frequency spectrograms

Pu Li, Marie A. Roch, Holger Klinck, Erica Fleishman, Douglas Gillespie, Eva-Marie Nosal, Yu Shiu, Xiaobai Liu

Research output: Contribution to journalArticlepeer-review


Whistle contour extraction aims to derive animal whistles from time-frequency spectrograms as polylines. For toothed whales, whistle extraction results can serve as the basis for analyzing animal abundance, species identity, and social activities. During the last few decades, as long-term recording systems have become affordable, automated whistle extraction algorithms were proposed to process large volumes of recording data. Recently, a deep learning-based method demonstrated superior performance in extracting whistles under varying noise conditions. However, training such networks requires a large amount of labor-intensive annotation, which is not available for many species. To overcome this limitation, we present a framework of stage-wise generative adversarial networks (GANs), which compile new whistle data suitable for deep model training via three stages: generation of background noise in the spectrogram, generation of whistle contours, and generation of whistle signals. By separating the generation of different components in the samples, our framework composes visually promising whistle data and labels even when few expert annotated data are available. Regardless of the amount of human-annotated data, the proposed data augmentation framework leads to a consistent improvement in performance of the whistle extraction model, with a maximum increase of 1.69 in the whistle extraction mean F1-score. Our stage-wise GAN also surpasses one single GAN in improving whistle extraction models with augmented data. The data and code will be available at https://github.com/Paul-LiPu/CompositeGAN_WhistleAugment.
Original languageEnglish
Pages (from-to)9302-9314
Number of pages13
JournalIEEE Transactions on Multimedia
Early online date1 Mar 2023
Publication statusPublished - 31 Mar 2023


  • Electrical and electronic engineering
  • Computer Science applications
  • Media technology
  • Signal processing


Dive into the research topics of 'Learning stage-wise GANs for whistle extraction in time-frequency spectrograms'. Together they form a unique fingerprint.

Cite this