Spark平台下KNN-ALS模型推荐算法
|
摘要
考虑Spark大数据平台内存计算框架在迭代计算的优势,提出Spark平台下KNN-ALS模型的推荐算法.针对矩阵分解算法只考虑隐含信息而忽视相似度信息的缺陷,将相似度信息加入评分预测中,并采用适合并行化的交替最小二乘法进行模型最优.在MovieLens数据集上的实验表明:该算法能够提高协同过滤推荐算法在大数据集下的处理效率,且加速比也达到并行处理的线性要求,相比其他方法有较好的精度.
Taking into account the memory computing advantage of Spark framework in iterative computation,the KNN-ALS model of recommendation algorithm based on Spark is proposed in this paper.The matrix factorization algorithm only considers the implicit information but ignores the similarity information,the model adds the similarity information into the rating prediction and then use the method of alternating least squares to optimize the model.From the experiments on the MovieLens dataset,the algorithm can improve the processing efficiency of the collaborative filtering algorithm in large data set,and also got a regular parameter of speedup in parallel processing.Furthermore,the proposed model have better accuracy than other methods.
Taking into account the memory computing advantage of Spark framework in iterative computation,the KNN-ALS model of recommendation algorithm based on Spark is proposed in this paper.The matrix factorization algorithm only considers the implicit information but ignores the similarity information,the model adds the similarity information into the rating prediction and then use the method of alternating least squares to optimize the model.From the experiments on the MovieLens dataset,the algorithm can improve the processing efficiency of the collaborative filtering algorithm in large data set,and also got a regular parameter of speedup in parallel processing.Furthermore,the proposed model have better accuracy than other methods.
引文
[1]AMATRIAIN X.Past,present,and future of recommender systems:An industry perspective[C]∥International Conference on Intelligent User Interfaces.Boston:ACM,2016:211-214.DOI:10.1145/2959100.2959144.
[2]JANNACH D,FELFERNIDG A,FRIEDRICH G.推荐系统[M].蒋凡,译.北京:人民邮电出版社,2013:1-8.
[3]孟祥武,刘树栋,张玉洁,等.社会化推荐系统研究[J].软件学报,2015,26(6):1356-1372.DOI:10.13328/j.cnki.jos.004831.
[4]KOREN Y,BELL R.Advances in collaborative filtering[M].Berlin:Springer,2011:145-186.DOI:10.1007/978-1-4899-7637-6_3.
[5]SU Xiaoyuan,KHOSHGOFTAAR T M.A survey of collaborative filtering techniques[J].Advances in Artificial Intelligence,2009(12):1-19.DOI:10.1155/2009/421425.
[6]SARWAR B,KARYPIS G,KONSTAN J,et al.Item-based collaborative filtering recommendation algorithms[C]∥International Conference on World Wide Web.New York:ACM,2001:285-295.DOI:10.1145/371920.372071.
[7]BILLSUS D,PAZZANI M J.Learning collaborative information filters[C]∥Proceedings of the Fifteenth International Conference on Machine Learning.San Francisco:Morgan Kaufmann Publishers Inc,1998:46-54.
[8]SALAKHUTDINOV R,MNIH A.Probabilistic matrix factorization[C]∥Proceedings of the 20th International Conference on Neural Information Processing Systems.Vancouver:Curran Associates Inc,2007:1257-1264.
[9]SALAKHUTDINOV R,MNIH A.Bayesian probabilistic matrix factorization using Markov chain Monte Carlo[C]∥International Conference on Machine Learning.New York:ACM,2008:880-887.DOI:10.1145/1390156.1390267.
[10]WANG Chong,BLEI D M.Collaborative topic modeling for recommending scientific articles[C]∥Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York:ACM,2011:448-456.DOI:10.1145/2020408.2020480.
[11]WANG Hao,WANG Naiyan,YEUNG D Y.Collaborative deep learning for recommender systems[C]∥Proceeding KDD′15Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York:ACM,2015:1235-1244.DOI:10.1145/2783258.2783273.
[12]ZHANG Zike,ZHOU Tao,ZHANG Yicheng.Tag-aware recommender systems:A state-of-the-art survey[J].Journal of Computer Science and Technology,2011,26(5):767-777.DOI:10.1007/S11390-011-0176-1.
[13]KOREN Y,BELL R,VOLINSKY C.Matrix factorization techniques for recommender systems[J].Computer,2009,42(8):30-37.DOI:10.1109/MC.2009.263.
[14]SYMEONIDIS P.Matrix and tensor decomposition in recommender systems[C]∥ACM Conference on Recommender Systems.Boston:ACM,2016:429-430.DOI:10.1145/2959100.2959195.
[15]张川.基于矩阵分解的协同过滤推荐算法研究[D].长春:吉林大学,2013:29-50.
[16]PARASCHAKIS D.Recommender systems from an industrial and ethical perspective[C]∥ACM Conference on Recommender Systems.New York:ACM,2016:463-466.DOI:10.1145/2959100.2959101.
[17]WHITE T.Hadoop权威指南[M].3版.华东师范大学数据科学与工程学院,译.北京:清华大学出版社,2015.
[18]杜江,张铮,张杰鑫,等.MapReduce并行编程模型研究综述[J].计算机科学,2015,42(增刊1):2635-2642.
[19]KARAU H,KONWINSKI A,WENDELL P,et al.Spark快速大数据分析[M].王道远,译.北京:人民邮电出版社,2015:187-209.
[20]李卫平,杨杰.基于随机梯度矩阵分解的社会网络推荐算法[J].计算机应用研究,2014,31(6):1654-1656.DOI:10.3969/j.issn.1001-3695.2014.06.011.
[21]ZHOU Yunhong,WILKINSON D,SCHREIBER R,et al.Large-scale parallel collaborative filtering for the netflix prize[C]∥Proceedings of the 4th International Conference on Algorithmic Aspects in Information and Management.Berlin:Springer-Verlag,2008:337-348.DOI:10.1007/978-3-540-68880-8_32.
[2]JANNACH D,FELFERNIDG A,FRIEDRICH G.推荐系统[M].蒋凡,译.北京:人民邮电出版社,2013:1-8.
[3]孟祥武,刘树栋,张玉洁,等.社会化推荐系统研究[J].软件学报,2015,26(6):1356-1372.DOI:10.13328/j.cnki.jos.004831.
[4]KOREN Y,BELL R.Advances in collaborative filtering[M].Berlin:Springer,2011:145-186.DOI:10.1007/978-1-4899-7637-6_3.
[5]SU Xiaoyuan,KHOSHGOFTAAR T M.A survey of collaborative filtering techniques[J].Advances in Artificial Intelligence,2009(12):1-19.DOI:10.1155/2009/421425.
[6]SARWAR B,KARYPIS G,KONSTAN J,et al.Item-based collaborative filtering recommendation algorithms[C]∥International Conference on World Wide Web.New York:ACM,2001:285-295.DOI:10.1145/371920.372071.
[7]BILLSUS D,PAZZANI M J.Learning collaborative information filters[C]∥Proceedings of the Fifteenth International Conference on Machine Learning.San Francisco:Morgan Kaufmann Publishers Inc,1998:46-54.
[8]SALAKHUTDINOV R,MNIH A.Probabilistic matrix factorization[C]∥Proceedings of the 20th International Conference on Neural Information Processing Systems.Vancouver:Curran Associates Inc,2007:1257-1264.
[9]SALAKHUTDINOV R,MNIH A.Bayesian probabilistic matrix factorization using Markov chain Monte Carlo[C]∥International Conference on Machine Learning.New York:ACM,2008:880-887.DOI:10.1145/1390156.1390267.
[10]WANG Chong,BLEI D M.Collaborative topic modeling for recommending scientific articles[C]∥Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York:ACM,2011:448-456.DOI:10.1145/2020408.2020480.
[11]WANG Hao,WANG Naiyan,YEUNG D Y.Collaborative deep learning for recommender systems[C]∥Proceeding KDD′15Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York:ACM,2015:1235-1244.DOI:10.1145/2783258.2783273.
[12]ZHANG Zike,ZHOU Tao,ZHANG Yicheng.Tag-aware recommender systems:A state-of-the-art survey[J].Journal of Computer Science and Technology,2011,26(5):767-777.DOI:10.1007/S11390-011-0176-1.
[13]KOREN Y,BELL R,VOLINSKY C.Matrix factorization techniques for recommender systems[J].Computer,2009,42(8):30-37.DOI:10.1109/MC.2009.263.
[14]SYMEONIDIS P.Matrix and tensor decomposition in recommender systems[C]∥ACM Conference on Recommender Systems.Boston:ACM,2016:429-430.DOI:10.1145/2959100.2959195.
[15]张川.基于矩阵分解的协同过滤推荐算法研究[D].长春:吉林大学,2013:29-50.
[16]PARASCHAKIS D.Recommender systems from an industrial and ethical perspective[C]∥ACM Conference on Recommender Systems.New York:ACM,2016:463-466.DOI:10.1145/2959100.2959101.
[17]WHITE T.Hadoop权威指南[M].3版.华东师范大学数据科学与工程学院,译.北京:清华大学出版社,2015.
[18]杜江,张铮,张杰鑫,等.MapReduce并行编程模型研究综述[J].计算机科学,2015,42(增刊1):2635-2642.
[19]KARAU H,KONWINSKI A,WENDELL P,et al.Spark快速大数据分析[M].王道远,译.北京:人民邮电出版社,2015:187-209.
[20]李卫平,杨杰.基于随机梯度矩阵分解的社会网络推荐算法[J].计算机应用研究,2014,31(6):1654-1656.DOI:10.3969/j.issn.1001-3695.2014.06.011.
[21]ZHOU Yunhong,WILKINSON D,SCHREIBER R,et al.Large-scale parallel collaborative filtering for the netflix prize[C]∥Proceedings of the 4th International Conference on Algorithmic Aspects in Information and Management.Berlin:Springer-Verlag,2008:337-348.DOI:10.1007/978-3-540-68880-8_32.