A Neuro-Symbolic Framework for Interpreting Human Behavior in Social Media Using the Behavioral Patterns Ontology.
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Understanding and interpreting human behavior is inherently challenging because behaviors are often complex, context-dependent, and not always directly observable. This challenge becomes even greater in social media environments, where behavioral information is expressed across multiple modalities, including spoken language, written text, visual content, and contextual cues. Social media videos therefore provide a rich source of behavioral data, but integrating these heterogeneous signals into consistent behavioral interpretations remains difficult. Human behaviors are also often not stated explicitly and must instead be inferred from combinations of multimodal evidence. Although multimodal large language models (MLLMs) provide a promising means of integrating such diverse information, behavioral interpretation requires more than recognizing visual and textual cues; these cues must also be mapped consistently to domain-relevant behavioral concepts. When MLLMs are applied without structured knowledge, they may generate unsupported or overly broad behavioral interpretations. To address this challenge, this work develops a neuro-symbolic framework that grounds behavioral interpretation in domain-specific structured knowledge. The framework is based on the Behavioral Patterns Ontology (BPO), which was developed to formally represent behavioral concepts, definitions, relationships, and hierarchical structure. The BPO was applied to TikTok videos in collaboration with domain experts from the Psychology Department to create human-validated behavioral annotations that serve as the ground-truth dataset for further automated behavioral interpretation. To evaluate whether incorporating structured behavioral knowledge improves this process, the research compares two few-shot prompting conditions: a Label-Only condition, in which models are provided with the annotation vocabulary without explicit BPO semantics, and a BPO-Guided condition, in which models are additionally provided with ontology-based definitions and semantic relationships. The performance of these two conditions is compared across four multimodal large language models—Google Gemini 3.1 Pro Preview, OpenAI GPT-5.6 Sol, Meta Llama 4 Scout 17B-16E Instruct and Qwen3-VL-8B-Instruct—to examine the effect of structured domain knowledge on the interpretation of human behavior in social media videos. Our results show that integrating structured BPO knowledge improved multi-label behavioral annotation performance for certain MLLMs; however, improvements varied across models and input modalities, indicating that the benefits of ontology-guided prompting were model- and input-dependent.