Investigating multisensory integration in emotion recognition through bio-inspired computational models

Esma Mansouri Benssassi, Juan Ye*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)
3 Downloads (Pure)


Emotion understanding represents a core aspect of human communication. Our social behaviours are closely linked to expressing our emotions and understanding others emotional and mental states through social signals. The majority of the existing work proceeds by extracting meaningful features from each modality and applying fusion techniques either at a feature level or decision level. However, these techniques are incapable of translating the constant talk and feedback between different modalities. Such constant talk is particularly important in continuous emotion recognition, where one modality can predict, enhance and complement the other. This paper proposes three multisensory integration models, based on different pathways of multisensory integration in the brain; that is, integration by convergence, early cross-modal enhancement, and integration through neural synchrony. The proposed models are designed and implemented using third-generation neural networks, Spiking Neural Networks (SNN). The models are evaluated using widely adopted, third-party datasets and compared to state-of-the-art multimodal fusion techniques, such as early, late and deep learning fusion. Evaluation results show that the three proposed models have achieved comparable results to the state-of-the-art supervised learning techniques. More importantly, this paper demonstrates plausible ways to translate constant talk between modalities during the training phase, which also brings advantages in generalisation and robustness to noise.
Original languageEnglish
Number of pages13
JournalIEEE Transactions on Affective Computing
VolumeEarly Access
Early online date19 Aug 2021
Publication statusE-pub ahead of print - 19 Aug 2021


  • Spiking neural network
  • Multisensory integration
  • Emotion recognition
  • Neural synchrony
  • Graph neural network


Dive into the research topics of 'Investigating multisensory integration in emotion recognition through bio-inspired computational models'. Together they form a unique fingerprint.

Cite this