Residual-driven ensemble boosting for recommender systems: statistical methods for temporal dynamics and text modeling

Abstract

There is limited research on recommender systems that jointly model temporal dynamics and textual context within a unified framework. Existing approaches typically either neglect temporal effects (e.g., CDL, Mult-VAE) or underutilize textual information (e.g., BERT4Rec), while hybrid methods often lack a principled integration of collaborative filtering and deep language models. To address this gap, we propose a two-stage framework that combines temporal collaborative filtering with deep text representations. In the first stage, we employ a multinomial collaborative filtering model that treats ratings as categorical variables, extending the temporal framework of Koren and Bell by incorporating item-specific time bins, user bias drift, and implicit feedback. In the second stage, residuals from the first model are modeled using a text-based component that leverages embeddings derived from user reviews. These two stages are combined additively at the log-odds level to form an ensemble base learner. This base learner is further embedded within a LogitBoost architecture, which iteratively refines predictions through sequential residual modeling. The proposed framework addresses key challenges in recommender systems, including sparse userâ item interactions, non-random missing data, and popularity bias, by decomposing the feature space into structured and unstructured components. A preliminary ensemble implementation is introduced in Chapter 3, where BERT embeddings of review text are incorporated into a multinomial classifier within the base learner. More advanced implementations are developed in Chapters 4 and 5, resulting in the CF-BERT-Boost and CF-Tweedie-BiLSTM-Boost models, respectively. These models differ in their choice of embeddings and classifiers in the second stage. Specifically, CF-BERT-Boost employs pretrained BERT embeddings within a multinomial classification framework, while CF-Tweedie-BiLSTM-Boost utilizes pretrained global word embeddings derived from a Shared-Parameter Alternating Tweedie representation, combined with a BiLSTM model. Empirical evaluation on Amazon review datasets demonstrates consistent improvements over benchmark models. The preliminary ensemble achieves mean squared error (MSE) values ranging from 0.593 to 0.668 across five collaborative filtering variants, representing improvements of 6.3% to 15.6%. The CF-BERT-Boost model outperforms benchmarks across all seven variants, with MSE reductions of 6.3% to 15.7% on the 2014 dataset and 5.8% to 30.2% on the 2023 dataset. The CF-Tweedie-BiLSTM-Boost model further improves performance, reducing MSE by 12.8% to 22.1% on the 2014 dataset and by 24.2% to 46.1% on the 2023 dataset. The LogitBoost optimization procedure demonstrates stable convergence across a range of learning rates and iteration settings. Overall, this work establishes a rigorous statistical framework for dynamic recommender systems by effectively integrating ensemble learning with natural language processing. It provides a unified approach for handling high-dimensional, heterogeneous data and addresses critical limitations in existing hybrid recommendation methods.