PoseR: a deep learning toolbox for decoding animal behavior

Pierce Mullen, Beatrice Bowlby, Holly C Armstrong, Maarten Frans Zwart

Research output: Working paperPreprint

Abstract

The actions of animals provide a window into how their minds work. Recent advances in deep learning are providing powerful approaches to recognize patterns of animal movement from video recordings, including markerless pose estimation models. However, tools to efficiently parse coordinates of animal position and pose into meaningful semantic behavioral labels are lacking. Here, we present PoseRecognition (PoseR), a behavioral decoder leveraging state- of-the-art action recognition models using spatio-temporal graph convolutional networks. We show that it can be used to decode animal behavior quickly and accurately from pose estimations, using zebrafish larvae and mice as model organisms. PoseR can be accessed using a Napari plugin, which facilitates efficient behavioral extraction, annotation, model training and deployment. We have simplified the workflow of behavioral analysis after pose estimation, transforming coordinates of animal position and pose into meaningful semantic behavioral labels, using methods designed for fast and accurate behavioral extraction, annotation, model training and deployment. Furthermore, we contribute a novel method for unsupervised clustering of behaviors and provide open-source access to our zebrafish datasets and models. The design of our tool ensures scalability and versatility for use across multiple species and contexts, improving the efficiency of behavioral analysis across fields.
Original languageEnglish
PublisherbioRxiv
Number of pages28
DOIs
Publication statusPublished - 8 Apr 2023

Fingerprint

Dive into the research topics of 'PoseR: a deep learning toolbox for decoding animal behavior'. Together they form a unique fingerprint.

Cite this