<link rel="stylesheet" href="styles.cf390c6bfbe7555f.css">
dc.contributor.authorBaloch, Maria
dc.date.accessioned2026-11-17T16:30:18Z
dc.date.graduationmonthDecember
dc.date.issued2026
dc.description.abstractUnderstanding and interpreting human behavior is inherently challenging because behaviors are often complex, context-dependent, and not always directly observable. This challenge becomes even greater in social media environments, where behavioral information is expressed across multiple modalities, including spoken language, written text, visual content, and contextual cues. Social media videos therefore provide a rich source of behavioral data, but integrating these heterogeneous signals into consistent behavioral interpretations remains difficult. Human behaviors are also often not stated explicitly and must instead be inferred from combinations of multimodal evidence. Although multimodal large language models (MLLMs) provide a promising means of integrating such diverse information, behavioral interpretation requires more than recognizing visual and textual cues; these cues must also be mapped consistently to domain-relevant behavioral concepts. When MLLMs are applied without structured knowledge, they may generate unsupported or overly broad behavioral interpretations. To address this challenge, this work develops a neuro-symbolic framework that grounds behavioral interpretation in domain-specific structured knowledge. The framework is based on the Behavioral Patterns Ontology (BPO), which was developed to formally represent behavioral concepts, definitions, relationships, and hierarchical structure. The BPO was applied to TikTok videos in collaboration with domain experts from the Psychology Department to create human-validated behavioral annotations that serve as the ground-truth dataset for further automated behavioral interpretation. To evaluate whether incorporating structured behavioral knowledge improves this process, the research compares two few-shot prompting conditions: a Label-Only condition, in which models are provided with the annotation vocabulary without explicit BPO semantics, and a BPO-Guided condition, in which models are additionally provided with ontology-based definitions and semantic relationships. The performance of these two conditions is compared across four multimodal large language models—Google Gemini 3.1 Pro Preview, OpenAI GPT-5.6 Sol, Meta Llama 4 Scout 17B-16E Instruct and Qwen3-VL-8B-Instruct—to examine the effect of structured domain knowledge on the interpretation of human behavior in social media videos. Our results show that integrating structured BPO knowledge improved multi-label behavioral annotation performance for certain MLLMs; however, improvements varied across models and input modalities, indicating that the benefits of ontology-guided prompting were model- and input-dependent.
dc.description.advisorHande McGinty
dc.description.degreeMaster of Science
dc.description.departmentDepartment of Computer Science
dc.description.levelMasters
dc.description.sponsorshipThis work was funded by the Division of Workplace Programs, Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration (SAMHSA), and the Department of Defense (DoD), through the National Laboratory Certification Program contract.
dc.identifier.urihttps://hdl.handle.net/2097/47539
dc.language.isoen_US
dc.subjectKnowledge Graphs
dc.subjectBehavioral Patterns Ontology
dc.subjectMultimodel Large Language Model
dc.subjectSocial Media
dc.subjectKNowledge Acquisition and Representation Methodology
dc.titleA Neuro-Symbolic Framework for Interpreting Human Behavior in Social Media Using the Behavioral Patterns Ontology.
dc.typeThesis
local.embargo.terms2027-11-16

Files

Original bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
MariaBaloch2026.pdf
Size:
3.75 MB
Format:
Adobe Portable Document Format

License bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
license.txt
Size:
1.65 KB
Format:
Item-specific license agreed upon to submission
Description: