Multimodal latent emotion recognition from micro-expression and physiological signal

This paper aims at the benefits of incorporating multimodal data for improving latent emotion recognition accuracy, focusing on micro-expression (ME) and physiological signals (PS). In particular, we propose a novel multimodal learning framework that combines ME and PS information. This framework includes a novel 1D separable and mixable depthwise inception CNN, tailored to extract informative features from diverse physiological signals effectively. Additionally, we develop a standardised normal distribution weighted feature fusion methodology, which facilitates the reconstruction of feature maps from different frames within micro-expression videos. To achieve comprehensive multimodal learning, we introduce guided attention modules that assist recognising latent emotions from micro-expressions (including colour and depth information) and physiological signals. Our empirical results show that the proposed approach outperforms the benchmark method, with the weighted fusion method and guided attention modules both contributing to enhanced performance.