Automated behavioral annotation in rodent trials using AI frameworks

Abstract

Behavioral assays in rodents are fundamental tools for studying cognition, anxiety, and neurological disorders, yet manual annotation of animal behaviors remains labor-intensive, subjective, and difficult to scale. This dissertation presents an automated framework for behavioral annotation in rodent trials that integrates deep learning, pose estimation, and rule-based spatial reasoning to produce accurate, reproducible, and analysis-ready behavioral data from video recordings. The framework is developed primarily for the Novel Object Recognition Test (NORT) and is further extended to the Elevated Plus Maze (EPM) and Marble Burying Test (MBT), demonstrating its adaptability across multiple behavioral paradigms. The proposed workflow combines fine-tuned YOLOv11 pose estimation for rat keypoint detection, heuristic-based behavioral inference using spatial relationships between animals and experimental objects, and YOLOv11 image classification models for frame-level behavior recognition. Keypoint models were trained to detect the rat nose and tail-base, enabling estimation of body orientation, posture, and object proximity. Multiple behavior-classification strategies, including heuristic-based, deep learning-based, and hybrid approaches, were evaluated for automated identification of behaviors such as object interaction, standing, freezing, resting, and marble interaction. The resulting framework also generates analysis-ready outputs, including behavioral summaries, heatmaps, ethograms, and quantitative metrics for downstream neuroscience analyses. A central feature of the framework is the separation of generic learned behavioral recognition from task-specific spatial interpretation, enabling visual information to be shared across related experimental configurations while retaining explicit object- and region-specific behavioral outputs. Experimental evaluation demonstrated that a hybrid approach combining a unified YOLOv11 behavior classification model with keypoint-based heuristic refinement achieved the best overall performance across two-object and five-object NORT configurations while generalizing effectively to varying experimental conditions. The framework was subsequently applied to over 500 NORT videos to enable large-scale analysis of object exploration, novel object and place recognition, rearing, and resting/freezing behaviors across experimental groups and testing sessions. Additional adaptations for different strains of rats, EPM analysis, and MBT further demonstrate the flexibility of the proposed methodology for diverse rodent behavioral assays. This work provides a unified, scalable framework for automated rodent behavior annotation that substantially reduces manual scoring effort while maintaining high analytical utility. By integrating modern computer vision techniques with interpretable spatial reasoning, the proposed approach advances automated behavioral neuroscience and establishes a flexible foundation for future studies involving additional behavioral paradigms, animal strains, and experimental environments.