字符级卷积神经网络短文本分类算法
|
摘要
由于短文本具有长度短、特征稀疏以及上下文依赖性强等特点,传统方法对其直接进行分类精度有限。针对该问题,提出了一种基于字符级嵌入的卷积神经网络(CNN)和长短时记忆网络(LSTM)相结合的神经网络模型进行短文本的分类。该模型同时包括了高速公路网络(Highway networks)框架,用于缓解深度神经网络训练时的困难,提高分类的准确性。通过对几种数据集的测试,结果表明提出的模型在短文本分类任务中优于传统模型和其他基于CNN的分类模型。
Since short text is characterized of the short length, sparse features and strong context dependency, the traditional models have a limited precision. Motivated by this, this article offers an empirical exploration on a character-level model which implements a combination of Convolutional Neural Network(CNN)and Long Short-Term Memory neural networks(LSTM)for short text classification. Including the highway networks framework so that it can address the difficult of training and improve the accuracy of classification. The evaluations on several datasets show that the proposed model outperforms the traditional and CNN-based models on short text classification mission.
Since short text is characterized of the short length, sparse features and strong context dependency, the traditional models have a limited precision. Motivated by this, this article offers an empirical exploration on a character-level model which implements a combination of Convolutional Neural Network(CNN)and Long Short-Term Memory neural networks(LSTM)for short text classification. Including the highway networks framework so that it can address the difficult of training and improve the accuracy of classification. The evaluations on several datasets show that the proposed model outperforms the traditional and CNN-based models on short text classification mission.
引文
[1] Liwicki M,Graves A,Bunke H,et al.A novel approachto on-line handwriting recognition based on bidirectionallong short-term memory networks[C]//Proc 9th Int Confon Document Analysis and Recognition,2007,1:367-371.
[2] Sutskever I,Vinyals O,Le Q V.Sequence to sequencelearning with neural networks[C]//Advances in NeuralInformation Processing Systems,2014:3104-3112.
[3] Tai K S,Socher R,Manning C D.Improved semanticrepresentations from tree-structured long short-term memorynetworks[J].arXiv:1500.00075,2015.
[4] Zhu X,Sobihani P,Guo H.Long short-term memoryover recursive structures[C]//Proceedings of the 32ndInternational Conference on Machine Learning(LCML-15),2015:1604-1612.
[5] Kalchbrenner N,Grefenstette E,Blunsom P.A convolu-tional neural network for modelling sentences[C]//Pro-ceedings of the Association for Computational Linguistics(ACL),2014:655-665.
[6] Kim Y.Convolutional neural networks for sentence classi-fication[C]//Proceedings of the Conference on EmpiricalMethods in Natural Language Processing(EMNLP),2014:1746-1751.
[7] Zhang Y,Wallace B.A sensitivity analysis of(and practi-tioners’ guide to)convolutional neural networks forsentence classification[J].Computer Science,2015.
[8] Yin W,Schütze H.Multichannel variable-size convolutionfor sentence classification[C]//Proceedings of CoNLL,2015:204-214.
[9]陈钊,徐睿峰,桂林,等.结合卷积神经网络和词语情感序列特征的中文情感分析[J].中文信息学报,2015,29(6):172-178.
[10] Zhang X,Zhao J,Lecun Y.Character-level convolutionalnetworks for text classification[C]//International Confer-ence on Neural Information Processing Systems,2015:649-657.
[11] Kim Y,Jernite Y,Sontag D,et al.Character-aware neurallanguage models[C]//Proceedings of the Association forthe Advancement of Artificial Intelligence(AAAI),2016:2741-2749.
[12] Santos C N,Gatti M A.Deep convolutional neural net-works for sentiment analysis of short texts[C]//Proc ofInternational Conference on Computational Linguistics,2014:69-78.
[13] Johnson R,Zhang T.Effective use of word order for textcategorization with convolutional neural networks[C]//Proc of North American Chapter of the Associationfor Computational Linguistics,2014:103-112.
[14]刘龙飞,杨亮,张绍武,等.基于卷积神经网络的微博情感倾向性分析[J].中文信息学报,2015,29(6):159-165.
[15] Jarrett K,Kavukcuoglu K,Ranzato M,et al.What isthe best multi-stage architecture for object recognition[C]//2009 IEEE 12th International Conference on ComputerVision,2009.
[16] Krizhevsky A,Sutskever I,Hinton G E.ImageNet classi-fication with deep convolutional neural networks[C]//International Conference on Neural Information Pro-cessing,2012.
[17] Srivastava R K,Greff K,Schmidhuber J.Training verydeep networks[J].Computer Science,2015:1-5.
[18] Mandelbaum A,Shalev A.Word embeddings and their usein sentence classification tasks[J].Co RR,abs/1610.08229,2016.
[19] Liu D R,Chuang S P,Lee H Y.Attention-based memoryselection recurrent network for language modeling[J].arXiv preprint arXiv:1611.08656,2016.
[20] Dauphin Y N,Fan A,Auli M,et al.Language modelingwith gated convolutional networks[J].arXiv:1612.08083,2016:1-5.
[2] Sutskever I,Vinyals O,Le Q V.Sequence to sequencelearning with neural networks[C]//Advances in NeuralInformation Processing Systems,2014:3104-3112.
[3] Tai K S,Socher R,Manning C D.Improved semanticrepresentations from tree-structured long short-term memorynetworks[J].arXiv:1500.00075,2015.
[4] Zhu X,Sobihani P,Guo H.Long short-term memoryover recursive structures[C]//Proceedings of the 32ndInternational Conference on Machine Learning(LCML-15),2015:1604-1612.
[5] Kalchbrenner N,Grefenstette E,Blunsom P.A convolu-tional neural network for modelling sentences[C]//Pro-ceedings of the Association for Computational Linguistics(ACL),2014:655-665.
[6] Kim Y.Convolutional neural networks for sentence classi-fication[C]//Proceedings of the Conference on EmpiricalMethods in Natural Language Processing(EMNLP),2014:1746-1751.
[7] Zhang Y,Wallace B.A sensitivity analysis of(and practi-tioners’ guide to)convolutional neural networks forsentence classification[J].Computer Science,2015.
[8] Yin W,Schütze H.Multichannel variable-size convolutionfor sentence classification[C]//Proceedings of CoNLL,2015:204-214.
[9]陈钊,徐睿峰,桂林,等.结合卷积神经网络和词语情感序列特征的中文情感分析[J].中文信息学报,2015,29(6):172-178.
[10] Zhang X,Zhao J,Lecun Y.Character-level convolutionalnetworks for text classification[C]//International Confer-ence on Neural Information Processing Systems,2015:649-657.
[11] Kim Y,Jernite Y,Sontag D,et al.Character-aware neurallanguage models[C]//Proceedings of the Association forthe Advancement of Artificial Intelligence(AAAI),2016:2741-2749.
[12] Santos C N,Gatti M A.Deep convolutional neural net-works for sentiment analysis of short texts[C]//Proc ofInternational Conference on Computational Linguistics,2014:69-78.
[13] Johnson R,Zhang T.Effective use of word order for textcategorization with convolutional neural networks[C]//Proc of North American Chapter of the Associationfor Computational Linguistics,2014:103-112.
[14]刘龙飞,杨亮,张绍武,等.基于卷积神经网络的微博情感倾向性分析[J].中文信息学报,2015,29(6):159-165.
[15] Jarrett K,Kavukcuoglu K,Ranzato M,et al.What isthe best multi-stage architecture for object recognition[C]//2009 IEEE 12th International Conference on ComputerVision,2009.
[16] Krizhevsky A,Sutskever I,Hinton G E.ImageNet classi-fication with deep convolutional neural networks[C]//International Conference on Neural Information Pro-cessing,2012.
[17] Srivastava R K,Greff K,Schmidhuber J.Training verydeep networks[J].Computer Science,2015:1-5.
[18] Mandelbaum A,Shalev A.Word embeddings and their usein sentence classification tasks[J].Co RR,abs/1610.08229,2016.
[19] Liu D R,Chuang S P,Lee H Y.Attention-based memoryselection recurrent network for language modeling[J].arXiv preprint arXiv:1611.08656,2016.
[20] Dauphin Y N,Fan A,Auli M,et al.Language modelingwith gated convolutional networks[J].arXiv:1612.08083,2016:1-5.