基于抗白噪声理论的支持向量机
|
摘要
作为上世纪九十年代兴起的一种新的机器学习技术,支持向量机(Support Vector Machine,SV M)在许多领域都取得了成功的应用。但它的应用其实大多局限于常见的标准化或者说“干净”的数据分布情况,对于在实际应用中不得不面对的一些数据分布不合常规或者说不“理想”的机器学习问题,比如:受噪声干扰的数据集分类,不确定性输入信息学习算法、不平衡数据集分类、半监督型数据学习等,传统型支持向量机的学习性能则表现得不尽人意,有时甚至根本达不到人们所期望的学习效果,这在很大程度上影响了支持向量机向更大范围的推广和应用。针对这些问题,本文就受噪声干扰的支持向量机学习算法进行了研究,给出了较理想的解决方案。在简单回顾标准支持向量机及其数学基础之后,本文重点研究了抗白噪声理论的支持向量机学习算法。针对某些训练样本存在输入信息带有噪声的问题,通过引入白噪声,高斯白噪声及核函数的概念,结合支持向量机的特性,提出了解决抗高斯白噪声的支持向量机分类算法。该类算法在传统支持向量机的基础上,通过改造原有的经验风险,来减弱或抑制噪声。也可利用核函数的特性,通过调节核函数中的参数,使其达到抗噪的目的。
Content: As one kind of new machine learning technology rose inthe early 1990s,support vector machine(SV M)has been extensivelystudied and has shown remarkable success in many applications. How-ever,the success of SV M is only limited in standard or ideal datadistribution status,when faced with non-ideal or exception data dis-tribution eases,the classical SV M performance dissatisfactory,andcan not meet the expected learning demands. Which in?uenced theSV Ms further extension and application in a great extent. In thelight of these problems of SV M,in this paper,we study the SV Malgorithms to cope with the non-ideal data distribution learning prob-lems,and give the suitable solutions.
After reviewing the standard support vector machine and its math-ematical foundation. We introduce some conception about white noise,white Gaussian noise. Then,we study the classification algorithms ofSV M on the situation when training group is intervened with noise.This Problem has universal significance in practical applications,dueto the limit of subjective and objective condition,we can hardly en-sure that all the input information of training instances are clear orprecise,on the contrary,there are often mixed up with some un-certain or noisy information inside the training samples. By trans-forming the experience risk measurement of the traditional SupportVector Machine algorithm. We proposed gray information supportvector machine classification algorithm respectively,which uses theexperience risk measurement number to represent the uncertain infor-mation,transforms the noise information input to vector form,andextends traditional operation to new function. We also can adjustingthe parameters of kernel function. Using this that we also can con-trol or decrease the random noise. We give a new SV M model,in the model,if we know the probability of noise,we can construct thecontrol noise of function. At last,there is a concrete model that con-trol White Gaussian Noise is given.
Content: As one kind of new machine learning technology rose inthe early 1990s,support vector machine(SV M)has been extensivelystudied and has shown remarkable success in many applications. How-ever,the success of SV M is only limited in standard or ideal datadistribution status,when faced with non-ideal or exception data dis-tribution eases,the classical SV M performance dissatisfactory,andcan not meet the expected learning demands. Which in?uenced theSV Ms further extension and application in a great extent. In thelight of these problems of SV M,in this paper,we study the SV Malgorithms to cope with the non-ideal data distribution learning prob-lems,and give the suitable solutions.
After reviewing the standard support vector machine and its math-ematical foundation. We introduce some conception about white noise,white Gaussian noise. Then,we study the classification algorithms ofSV M on the situation when training group is intervened with noise.This Problem has universal significance in practical applications,dueto the limit of subjective and objective condition,we can hardly en-sure that all the input information of training instances are clear orprecise,on the contrary,there are often mixed up with some un-certain or noisy information inside the training samples. By trans-forming the experience risk measurement of the traditional SupportVector Machine algorithm. We proposed gray information supportvector machine classification algorithm respectively,which uses theexperience risk measurement number to represent the uncertain infor-mation,transforms the noise information input to vector form,andextends traditional operation to new function. We also can adjustingthe parameters of kernel function. Using this that we also can con-trol or decrease the random noise. We give a new SV M model,in the model,if we know the probability of noise,we can construct thecontrol noise of function. At last,there is a concrete model that con-trol White Gaussian Noise is given.
引文
[1] Cortes C ,Vapnik.V.Support Vectot Networks[J].Machine Learning,1995,20:273-295
[2] Vapnik. V.The Nature of statistical learning theory. Springer, NY, 1995.张学工译,统计学习理论的本质,清华大学出版社,2000
[3] Vapnik.V.Statistical learning theory[M].New York:Wiley,1998
[4] Vapnik.V.An overview of statistical learning theory[J].IEEE Trans,Neural Net-works,1999,10(5)
[5] BartlettP.L,Shawe-Taylor J.Generalization performance on support vec-tor machines and other pattern classifiers[C].InB.scholkopf,C.Burges,andA.SmolaEds.Advances in Kernel Methods-Support Vector Learning,Cambridge,MA:MITP ress,1999
[6] Joachims T.Text categorization with support vector machines.Learning with manyrelevant features[C] In Proceedings of the European Conferenceon Machine Learn-ing Berlin:Springe 1998:137-142
[7] Gish H ,Schimdt M.Text-indepenten speaker identification[J].BEE Transon SignalProcessing Magazine 1994.42(1),1832
[8] Osuna E,Freund R,Girosi F.Improved training algorithm for support vector ma-chines[C].In 7th IEEE workshop on Neural Networks for signal Processing NNSP’97IEEE 1997,276-285
[9] Heisele B, etal. Hierarchical classification and feature reduction for fast face detec-tion with support vectro machines[J].Pattern Recognition 2003,36,2007-2017
[10] Walavalkar L, et al. Support Vector learning for gender classification using au-dio and visual cues[J]. International Journal of Pattern Recognition and ArtificialIntelligence.2003,17(3),417-439
[11] Mike F.James R.Computer intrusion detection with classification and anomaly de-tection using SVMs[J]. International Journal of Pattern Recognition and ArtificialInteligence.2003,17(3),441-458
[12] Mukherjee S.Classifying Microarray Data Using Support Vector Machines[C].InA practical approach to microarray data analysis Kluwer Acadamic PublishersBoston.M A.2003-09,166-185
[13] Brown M,Lewis G.H,Gunn R.S.Linear spectral mixture models and support vectormachines for remote sensing[J].IEEE Transon Geoscience and Remote Sensing,1998
[14] Bryant M,Garber F.SVM classifier applied to the MSTA Rpublic data set[C].InA lgorithms for Synthetic Aperture Radar Imagery VI Proceedings of the SPIE3721,1999,355-360
[15] Vapnik V, Lemer A. Patera recognition using generalized portrait method. Au-tomation and Remote Control,24,1963
[16] Vapnik V, Chervoknenkis A.Y. The necessary and suficient conditions for the uni-form convergence of averages to their expected values.Teoriya Veroyatnostei I EePrimeniniya,1981,26(3):543-564
[17] Vapnik V.Estimation of dependence based oempirical data.New York,Springer-Verlag,1982
[18] Boser B,Guyon I,Vapnik V.A training algorithm for optimal margin classifiers,FifthAnnual Workshop on Computational Learning Theory.Pittsburgh:ACM Press,1992
[19] ConesC C,Vapnik V.The soft margin classifier,Technical memorandum I 1359-931209-18TM,Bell Labs,1993
[20] Vapnik V, Golowich S, Smola A. Support vector method for function approxi-mation, regression estima tion, and signal processing.In :M.Mozer,M .Jordan,T.Petsche(eds).Neural Information Porcessing Systems ,MIT Press,1997
[21] ScholkopfB,BargesC,VapnikV.Incorporating invariances in support vec-tor learning machines,In:vonde MalsburgC, von SeelenW,VorbruggenJC etal(eds).Artificial Neural Networks ICANN’96, Spingers Lecture Notes in Com-puter Science,Berlin,1996,11(12):47-52
[22] Scholkopf B .Comparing support vector machines with Gaussian kernelsto radial basis function classifier IEEE Transactions on signal process-ing,November1997,45(11)
[23] Scholkopf B,Smola A ,MullerK R.Kernel principal component analysis,In :Proc.ofICANN’97,1997: 583- 589
[24] Scholkopf B,Smola A.A tutorial on support vector regression[R].NeuroCOLT2Technical Report Series NC2-TR-1998-030,October,1998
[25] Scholkopf B,Smola A, Vapnik V. Priorores in Neural Information Proceeding Sys-tems 10,M knowledge in support vector;kernels Jordan, M.Kearns,and S.Solla,EdsCambridge,MA,MIT Press,1998:640-646;
[26] Scholkopf B,Williamson R,Shawe-Taylor J.Single-class support vector ma-chines.In J.Buhmann,W Maass,H. Riter and N.Tishby,editors,UnsupervisedLearning,Dagstubl-Semianr-Report235,1999:19 -20;
[27] ScholkopfB,SmolaA,Muller K R.Kernel principal component analysis.In ScholkopfB,Barges C,and Smola A, editors,Advances in Kernel Methods-Support VectorLearning.MIT Press,Cambridge,MA,1999:327-352;
[28] Scholkopf B,Smola A,Williamson R.C,et al.New support vectorAgorithms[J].Neural Computation 2000,12(5): 1207-1245;
[29] Scholkopf B,Plat J.C,Shawe-Taylor J,et al.Estimating the support of a high-dimensional distribution Neural Computation,2001,13(7):1443-1471;
[30] Smola A. Generalization bounds for convex combinations of kernel functions. AlexJ. Smola, GMD NeuroCOLT2 Technical Report series,NC2-TR-1998-020,July,1998;
[31] SmolaA .Learning with kernels,Thesis,1998
[32] OsunaE ,Freund R,Girosi F.Support vector machines:training and applications.AIMemo 1602,MITALL Lab ,1997
[33] Friess T.-T, Christianimi CN, Campbell C. The kernel adatron algorithm: a fastand simple learning procedure for support vector machines.In Proceeding of 15thIntl.C on Machine Learning.Morgan Kaufman Publishers,1998
[34] Mangasarian O.L, Musicant D.R. Successive overrelaxation for support vector ma-chines. IEEE Trans. Neural Networks,1999,10(5):1032-1037
[35] Suykens J,Vandewalle J.Least square support vector machine classifiers.Neural Pro-cessing Leters,1999, 9(3):293-300
[36] Keerthi S.S,et al. A Fast Iterative Nearest Point Algorithm for Support VectorMachine Classifier Design.TR-ISL-99-03Dept.of CS and Auto.Indian Institute ofScience Bangalore,India,1999;
[37] ScholkopfB,SmolaA,WilliamsonR.C,et al.New support vector Agorithnw[J].NeuralComputation 2000,12(5):1217-1245
[38] ScholkopfB ,PlatJ C,Shawe-Taylor J,et al.Estimating the support of ahigh-dimensional distribution Neural Computation,2001,13(7):1443-1471
[39] Chew Hong-Gunn, Bogner Robert E,Lim Cheng-Chew.Dual nu-support vector ma-chine with error rate and training size biasing[J]Proceedings of 26th IEEE ICASSP2001,Salt Lake City,USA,2001:1269 -1272;
[40] Chew Hong-Gunn, Crisp D.J, Bogner R. E.et al.Target detection in radar im-agery using support vector machines with training size biasing[A].In:Proceedingsof the sixth international conference on control,Automation,Robotics and Vi-sion[C],Singapore,2000
[41] Lin Chun-Pu,Wang Sheng-De.Fuzzy support vector machines[J].IEEE Transaction-son Neural Networks, 2002, 13(2):464-471
[42] Suykens J,Branbanter J.D,LukasL ,et al.Weighted least squares support vector ma-chines:robustness and spare approximation[J]. eurocomputing,2002,48(1):85 -105
[43] Roobaert D.Direct SVM:a fast and simple support vector ma-chine perception[A].Proceedings of IEEE Signal Processing Society;Workshop[C].Sydney,Australia:IEEE,2000,1:356-365
[44] LeeY-J,Maugasarian O L.R SVM :Reduced support vector machines.In Proceedingsof the First SIAM International Conference on Data Mining,2001
[45] Mangasarian O. L,Musicant D. R. Lagrangian support vector machines. Journal ofMachine Learning Research,2001,1:161-177
[46] Knerr S, et al.Single-layer learning revisited: A stepwise procedure for build-ing and training;ncurd network .In Fogehuan Soalie et al,Neuor comput-ing:Algorithms,Architectures and Applications,NATOASI. Spring,1990
[47] WestonJ,WatkinsC.Multi-class suppor vector machines.In M.Verleysen,editor,Proceedings ESANN, Brussels,1999
[48] Blanz V,Vapnik V,Barges C.Multicalss discrimination with an extended supportvector machine.Talk given at Bell Labs, 1995
[49] Platt J, et al. Large Margin DAGs for Multiclass Classification, in Advances inNeural Information Processing Systems12,MIT Press,2000,547-553
[50] Joachims T. Estimating the generalization performance of an SVM eficientily. InP.Langley, editor, Proceedings of the 17th International Conference on MachineLearning, pages 431-438, San Franscisco, California, 2000. Morgan Kaufmann.
[51] Vapnik V, Chapelle O. Bounds on error expectation for support vector machinesNeural Computation, 12(g), 2000
[52] Ayat N.E, Cheriet M, Remaki L, et al. KMOD-a new support vector ma-chine kernel with moderate decreasing for patern recognition,application to dig-itimage recognition[A].Proceedings of 6th Int Conf on Document Analysis andRecognition[C].Seatle,USA:IEEE,2001,1215-1211
[53] Amaxi S,Wu S. Aninformation-geometrical method for improving the perfor-mance of support vector machine classifiers[A].Porceedings of Artificial neuralnetworks[J].1999:85-90
[54] ChapelleO ,Vapnik V,Bacsquest O,et al.Choosing multiple parameters for supportvector machines[J]. Machine Learning,2002,46(1):131-159
[55] Vapnik,V.N.,LevinE,Le Cun Y.Measuring the VC dimension of a leamingmachine.Neural Computation[J].(6):851 876,1994
[56] Burges C J C.A tutorial on support vector machines for pattem recognition.DataMining and Knowledge Diseovery[R].2(2),1998.
[57]边肇棋,张学工。模式识别(第二版)[M].北京:清华大学出版社,2000。
[58]邓乃扬,田英杰著。数据挖掘中的新方法-支持向量机[M].北京:科学技术出版社,2004,63-71.
[59] KeerthiS S,Lin C J.Asymptotic behaviors of support vector machines with Gaussiankernel[J]. Neural Computation,2003,15(7):1667O1689
[2] Vapnik. V.The Nature of statistical learning theory. Springer, NY, 1995.张学工译,统计学习理论的本质,清华大学出版社,2000
[3] Vapnik.V.Statistical learning theory[M].New York:Wiley,1998
[4] Vapnik.V.An overview of statistical learning theory[J].IEEE Trans,Neural Net-works,1999,10(5)
[5] BartlettP.L,Shawe-Taylor J.Generalization performance on support vec-tor machines and other pattern classifiers[C].InB.scholkopf,C.Burges,andA.SmolaEds.Advances in Kernel Methods-Support Vector Learning,Cambridge,MA:MITP ress,1999
[6] Joachims T.Text categorization with support vector machines.Learning with manyrelevant features[C] In Proceedings of the European Conferenceon Machine Learn-ing Berlin:Springe 1998:137-142
[7] Gish H ,Schimdt M.Text-indepenten speaker identification[J].BEE Transon SignalProcessing Magazine 1994.42(1),1832
[8] Osuna E,Freund R,Girosi F.Improved training algorithm for support vector ma-chines[C].In 7th IEEE workshop on Neural Networks for signal Processing NNSP’97IEEE 1997,276-285
[9] Heisele B, etal. Hierarchical classification and feature reduction for fast face detec-tion with support vectro machines[J].Pattern Recognition 2003,36,2007-2017
[10] Walavalkar L, et al. Support Vector learning for gender classification using au-dio and visual cues[J]. International Journal of Pattern Recognition and ArtificialIntelligence.2003,17(3),417-439
[11] Mike F.James R.Computer intrusion detection with classification and anomaly de-tection using SVMs[J]. International Journal of Pattern Recognition and ArtificialInteligence.2003,17(3),441-458
[12] Mukherjee S.Classifying Microarray Data Using Support Vector Machines[C].InA practical approach to microarray data analysis Kluwer Acadamic PublishersBoston.M A.2003-09,166-185
[13] Brown M,Lewis G.H,Gunn R.S.Linear spectral mixture models and support vectormachines for remote sensing[J].IEEE Transon Geoscience and Remote Sensing,1998
[14] Bryant M,Garber F.SVM classifier applied to the MSTA Rpublic data set[C].InA lgorithms for Synthetic Aperture Radar Imagery VI Proceedings of the SPIE3721,1999,355-360
[15] Vapnik V, Lemer A. Patera recognition using generalized portrait method. Au-tomation and Remote Control,24,1963
[16] Vapnik V, Chervoknenkis A.Y. The necessary and suficient conditions for the uni-form convergence of averages to their expected values.Teoriya Veroyatnostei I EePrimeniniya,1981,26(3):543-564
[17] Vapnik V.Estimation of dependence based oempirical data.New York,Springer-Verlag,1982
[18] Boser B,Guyon I,Vapnik V.A training algorithm for optimal margin classifiers,FifthAnnual Workshop on Computational Learning Theory.Pittsburgh:ACM Press,1992
[19] ConesC C,Vapnik V.The soft margin classifier,Technical memorandum I 1359-931209-18TM,Bell Labs,1993
[20] Vapnik V, Golowich S, Smola A. Support vector method for function approxi-mation, regression estima tion, and signal processing.In :M.Mozer,M .Jordan,T.Petsche(eds).Neural Information Porcessing Systems ,MIT Press,1997
[21] ScholkopfB,BargesC,VapnikV.Incorporating invariances in support vec-tor learning machines,In:vonde MalsburgC, von SeelenW,VorbruggenJC etal(eds).Artificial Neural Networks ICANN’96, Spingers Lecture Notes in Com-puter Science,Berlin,1996,11(12):47-52
[22] Scholkopf B .Comparing support vector machines with Gaussian kernelsto radial basis function classifier IEEE Transactions on signal process-ing,November1997,45(11)
[23] Scholkopf B,Smola A ,MullerK R.Kernel principal component analysis,In :Proc.ofICANN’97,1997: 583- 589
[24] Scholkopf B,Smola A.A tutorial on support vector regression[R].NeuroCOLT2Technical Report Series NC2-TR-1998-030,October,1998
[25] Scholkopf B,Smola A, Vapnik V. Priorores in Neural Information Proceeding Sys-tems 10,M knowledge in support vector;kernels Jordan, M.Kearns,and S.Solla,EdsCambridge,MA,MIT Press,1998:640-646;
[26] Scholkopf B,Williamson R,Shawe-Taylor J.Single-class support vector ma-chines.In J.Buhmann,W Maass,H. Riter and N.Tishby,editors,UnsupervisedLearning,Dagstubl-Semianr-Report235,1999:19 -20;
[27] ScholkopfB,SmolaA,Muller K R.Kernel principal component analysis.In ScholkopfB,Barges C,and Smola A, editors,Advances in Kernel Methods-Support VectorLearning.MIT Press,Cambridge,MA,1999:327-352;
[28] Scholkopf B,Smola A,Williamson R.C,et al.New support vectorAgorithms[J].Neural Computation 2000,12(5): 1207-1245;
[29] Scholkopf B,Plat J.C,Shawe-Taylor J,et al.Estimating the support of a high-dimensional distribution Neural Computation,2001,13(7):1443-1471;
[30] Smola A. Generalization bounds for convex combinations of kernel functions. AlexJ. Smola, GMD NeuroCOLT2 Technical Report series,NC2-TR-1998-020,July,1998;
[31] SmolaA .Learning with kernels,Thesis,1998
[32] OsunaE ,Freund R,Girosi F.Support vector machines:training and applications.AIMemo 1602,MITALL Lab ,1997
[33] Friess T.-T, Christianimi CN, Campbell C. The kernel adatron algorithm: a fastand simple learning procedure for support vector machines.In Proceeding of 15thIntl.C on Machine Learning.Morgan Kaufman Publishers,1998
[34] Mangasarian O.L, Musicant D.R. Successive overrelaxation for support vector ma-chines. IEEE Trans. Neural Networks,1999,10(5):1032-1037
[35] Suykens J,Vandewalle J.Least square support vector machine classifiers.Neural Pro-cessing Leters,1999, 9(3):293-300
[36] Keerthi S.S,et al. A Fast Iterative Nearest Point Algorithm for Support VectorMachine Classifier Design.TR-ISL-99-03Dept.of CS and Auto.Indian Institute ofScience Bangalore,India,1999;
[37] ScholkopfB,SmolaA,WilliamsonR.C,et al.New support vector Agorithnw[J].NeuralComputation 2000,12(5):1217-1245
[38] ScholkopfB ,PlatJ C,Shawe-Taylor J,et al.Estimating the support of ahigh-dimensional distribution Neural Computation,2001,13(7):1443-1471
[39] Chew Hong-Gunn, Bogner Robert E,Lim Cheng-Chew.Dual nu-support vector ma-chine with error rate and training size biasing[J]Proceedings of 26th IEEE ICASSP2001,Salt Lake City,USA,2001:1269 -1272;
[40] Chew Hong-Gunn, Crisp D.J, Bogner R. E.et al.Target detection in radar im-agery using support vector machines with training size biasing[A].In:Proceedingsof the sixth international conference on control,Automation,Robotics and Vi-sion[C],Singapore,2000
[41] Lin Chun-Pu,Wang Sheng-De.Fuzzy support vector machines[J].IEEE Transaction-son Neural Networks, 2002, 13(2):464-471
[42] Suykens J,Branbanter J.D,LukasL ,et al.Weighted least squares support vector ma-chines:robustness and spare approximation[J]. eurocomputing,2002,48(1):85 -105
[43] Roobaert D.Direct SVM:a fast and simple support vector ma-chine perception[A].Proceedings of IEEE Signal Processing Society;Workshop[C].Sydney,Australia:IEEE,2000,1:356-365
[44] LeeY-J,Maugasarian O L.R SVM :Reduced support vector machines.In Proceedingsof the First SIAM International Conference on Data Mining,2001
[45] Mangasarian O. L,Musicant D. R. Lagrangian support vector machines. Journal ofMachine Learning Research,2001,1:161-177
[46] Knerr S, et al.Single-layer learning revisited: A stepwise procedure for build-ing and training;ncurd network .In Fogehuan Soalie et al,Neuor comput-ing:Algorithms,Architectures and Applications,NATOASI. Spring,1990
[47] WestonJ,WatkinsC.Multi-class suppor vector machines.In M.Verleysen,editor,Proceedings ESANN, Brussels,1999
[48] Blanz V,Vapnik V,Barges C.Multicalss discrimination with an extended supportvector machine.Talk given at Bell Labs, 1995
[49] Platt J, et al. Large Margin DAGs for Multiclass Classification, in Advances inNeural Information Processing Systems12,MIT Press,2000,547-553
[50] Joachims T. Estimating the generalization performance of an SVM eficientily. InP.Langley, editor, Proceedings of the 17th International Conference on MachineLearning, pages 431-438, San Franscisco, California, 2000. Morgan Kaufmann.
[51] Vapnik V, Chapelle O. Bounds on error expectation for support vector machinesNeural Computation, 12(g), 2000
[52] Ayat N.E, Cheriet M, Remaki L, et al. KMOD-a new support vector ma-chine kernel with moderate decreasing for patern recognition,application to dig-itimage recognition[A].Proceedings of 6th Int Conf on Document Analysis andRecognition[C].Seatle,USA:IEEE,2001,1215-1211
[53] Amaxi S,Wu S. Aninformation-geometrical method for improving the perfor-mance of support vector machine classifiers[A].Porceedings of Artificial neuralnetworks[J].1999:85-90
[54] ChapelleO ,Vapnik V,Bacsquest O,et al.Choosing multiple parameters for supportvector machines[J]. Machine Learning,2002,46(1):131-159
[55] Vapnik,V.N.,LevinE,Le Cun Y.Measuring the VC dimension of a leamingmachine.Neural Computation[J].(6):851 876,1994
[56] Burges C J C.A tutorial on support vector machines for pattem recognition.DataMining and Knowledge Diseovery[R].2(2),1998.
[57]边肇棋,张学工。模式识别(第二版)[M].北京:清华大学出版社,2000。
[58]邓乃扬,田英杰著。数据挖掘中的新方法-支持向量机[M].北京:科学技术出版社,2004,63-71.
[59] KeerthiS S,Lin C J.Asymptotic behaviors of support vector machines with Gaussiankernel[J]. Neural Computation,2003,15(7):1667O1689