Engineering enhancements for movie recommender systems

Abstract

The evolution of the World Wide Web has resulted in extremely large amounts of information. As a consequence, users are faced with the problem of information overload: they have difficulty in identifying and selecting items of interest to them, such as books, movies, blogs, bookmarks, etc. Recommender systems can be used to address the information over-load problem by suggesting potentially interesting or useful items to users. Many existing recommender systems rely on the collaborative filtering technology. Among other domains, collaborative filtering systems have been widely used in e-commerce and they have proven to be very successful. However, in recent years the number of users and items available in e-commerce has grown tremendously, challenging recommender systems with scalability issues. To address such issues, we use canopy/clustering techniques and Hadoop MapReduce distributed framework to implement user-based and item-based recommender systems. We evaluate our implementations in the context of movie recommendation. Generally, standard rating prediction schemes work by identifying similar users/items. We propose a novel rating prediction scheme, which makes use of dissimilar users/items, in addition to the similar ones, and experimentally show that the new prediction scheme produces better results than the standard prediction scheme. Finally, we engineer two new approaches for clustering-based collaborative filtering that can make use of movie synopsis and user information. Specifically, in the first approach, we perform user-based clustering using movie synopsis, together with user demographic data. In the second approach, we perform item-based clustering using movie synopsis, together with user quotes about movies. Experimental results show that the movie synopsis and user demographic data can be effectively used to improve the rating predictions made by a recommender system. However, user quotes are too vague and do not produce better predictions.