基于大数据的时间序列异常点检测研究
|
摘要
针对传统时间序列异常点检测方法在处理大量数据时检测精度与效率低下的缺陷,文中提出一种基于大数据技术的全新时间序列异常点检测方法。首先介绍了传统时间序列异常点检测方法并分析了其缺陷。其次介绍了基于大数据方法的理论推导,包括特征提取、奇异点检测及异常点判别,具体为采用大数据方法将海量序列分解为周期分量、趋势分量、随机误差分量及突发分量四个不同分量,对不同分量进行特征提取并根据特征提取结果进行奇异点检测,并在此基础上利用序列特点判别奇异点是否为异常点。最后通过实验分析对比验证大数据方法的可行性与效率。实验结果表明,基于大数据方法的时间序列异常点检测相比于传统的方法具有更高的检测精度与更快的检测速率。
According to the detection accuracy and efficiency limitation of traditional time series outlier detection methods when dealing with a large amount of data,a newtime series outlier detection method is put forward,which is based on the big data technology. Firstly,the traditional time series outlier detection methods are introduced,analysis of their defects. Secondly,it introduces the theoretical derivation of big data method in this paper,which can be divided into feature extraction,abnormal detection and outlier distinguish. The massive series is decomposed into four different components,including periodic component,trend component,random error component and burst component. Then the feature is extracted to four components and abnormal detection is made according to the result of extraction. On this basis it determines whether abnormal point is outlier by series characteristic. Finally,the feasibility and efficiency of big data approach is verified by experiment analysis and comparison. The results showthat the big data method has higher precision and rate compared with traditional methods.
According to the detection accuracy and efficiency limitation of traditional time series outlier detection methods when dealing with a large amount of data,a newtime series outlier detection method is put forward,which is based on the big data technology. Firstly,the traditional time series outlier detection methods are introduced,analysis of their defects. Secondly,it introduces the theoretical derivation of big data method in this paper,which can be divided into feature extraction,abnormal detection and outlier distinguish. The massive series is decomposed into four different components,including periodic component,trend component,random error component and burst component. Then the feature is extracted to four components and abnormal detection is made according to the result of extraction. On this basis it determines whether abnormal point is outlier by series characteristic. Finally,the feasibility and efficiency of big data approach is verified by experiment analysis and comparison. The results showthat the big data method has higher precision and rate compared with traditional methods.
引文
[1]曹忠虔.时间序列异常检测的研究[D].成都:电子科技大学,2012.
[2]郭春.基于数据挖掘的网络入侵检测关键技术研究[D].北京:北京邮电大学,2014.
[3]Box G E P.时间序列分析——预测与控制[M].上海:机械工业出版社,2011.
[4]杨金伟.基于距离和信息熵的不确定异常点检测研究[D].昆明:云南大学,2011.
[5]刘良旭,乐嘉锦,乔少杰,等.基于轨迹点局部异常度的异常点检测算法[J].计算机学报,2011,34(10):1966-1975.
[6]刘丹丹,陈启军,森一之.线性回归模型的多离群点检测方法及节能应用[J].信息与控制,2013,42(6):765-771.
[7]胡世杰,钱宇宁,严如强.基于概率密度空间划分的符号化时间序列分析及其在异常诊断中的应用[J].振动工程学报,2014,27(5):780-784.
[8]苏卫星,朱云龙,胡琨元,等.基于模型的过程工业时间序列异常值检测方法[J].仪器仪表学报,2012,33(9):2080-2087.
[9]杨越,胡汉平,熊伟,等.一种基于超统计理论的非平稳时间序列异常点检测方法研究[J].计算机科学,2011,38(6):93-95.
[10]王佳玮.决策支持中基于时间序列数据的异常点检测[D].合肥:中国科学技术大学,2014.
[11]陈敏.基于BP神经网络的混沌时间序列预测模型研究[D].长沙:中南大学,2007.
[12]崔万照,朱长纯,保文星,等.基于模糊模型支持向量机的混沌时间序列预测[J].物理学报,2005,54(7):3009-3018.
[13]庄雪鹏.基于小波的时间序列中异常点的检测[D].南京:南京大学,2013.
[14]张建平,李斌,刘学军,等.基于Hadoop的异常传感数据时间序列检测[J].传感技术学报,2014,27(12):1659-1665.
[15]王骏,钟富礼,王士同,等.基于移相加权球面单簇聚类的周期时间序列异常检测[J].自动化学报,2011,37(8):984-992.
[16]张玉飞,董永贵.一种时间序列异常检测用参数化熵滤波器[J].机械工程学报,2011,47(22):13-18.
[17]张蕾.非线性时间序列的高阶统计特征提取和趋势分析[D].沈阳:沈阳航空航天大学,2012.
[18]龚祝平.混沌时间序列的平均周期计算方法[J].系统工程,2010,28(12):111-113.
[19]韩娜.聚类算法在时间序列中的研究与应用[D].广州:广东工业大学,2011.
[20]闫秋艳,夏士雄.一种无限长时间序列的分段线性拟合算法[J].电子学报,2010,38(2):443-448.
[21]Rasheed F,Alhajj R.A framework for periodic outlier pattern detection in time-series sequences[J].IEEE Transactions on Cybernetics,2014,44(5):569-582.
[22]Buzzi-Ferraris G,Manenti F.Outlier detection in large data sets[J].Computers and Chemical Engineering,2010,35:388
[23]Li S T,Cheng Y C.A stochastic HMM-based forecasting model for fuzzy time series[J].IEEE Transactions on Cybernetics,2010,40(5):1255-1266.
[2]郭春.基于数据挖掘的网络入侵检测关键技术研究[D].北京:北京邮电大学,2014.
[3]Box G E P.时间序列分析——预测与控制[M].上海:机械工业出版社,2011.
[4]杨金伟.基于距离和信息熵的不确定异常点检测研究[D].昆明:云南大学,2011.
[5]刘良旭,乐嘉锦,乔少杰,等.基于轨迹点局部异常度的异常点检测算法[J].计算机学报,2011,34(10):1966-1975.
[6]刘丹丹,陈启军,森一之.线性回归模型的多离群点检测方法及节能应用[J].信息与控制,2013,42(6):765-771.
[7]胡世杰,钱宇宁,严如强.基于概率密度空间划分的符号化时间序列分析及其在异常诊断中的应用[J].振动工程学报,2014,27(5):780-784.
[8]苏卫星,朱云龙,胡琨元,等.基于模型的过程工业时间序列异常值检测方法[J].仪器仪表学报,2012,33(9):2080-2087.
[9]杨越,胡汉平,熊伟,等.一种基于超统计理论的非平稳时间序列异常点检测方法研究[J].计算机科学,2011,38(6):93-95.
[10]王佳玮.决策支持中基于时间序列数据的异常点检测[D].合肥:中国科学技术大学,2014.
[11]陈敏.基于BP神经网络的混沌时间序列预测模型研究[D].长沙:中南大学,2007.
[12]崔万照,朱长纯,保文星,等.基于模糊模型支持向量机的混沌时间序列预测[J].物理学报,2005,54(7):3009-3018.
[13]庄雪鹏.基于小波的时间序列中异常点的检测[D].南京:南京大学,2013.
[14]张建平,李斌,刘学军,等.基于Hadoop的异常传感数据时间序列检测[J].传感技术学报,2014,27(12):1659-1665.
[15]王骏,钟富礼,王士同,等.基于移相加权球面单簇聚类的周期时间序列异常检测[J].自动化学报,2011,37(8):984-992.
[16]张玉飞,董永贵.一种时间序列异常检测用参数化熵滤波器[J].机械工程学报,2011,47(22):13-18.
[17]张蕾.非线性时间序列的高阶统计特征提取和趋势分析[D].沈阳:沈阳航空航天大学,2012.
[18]龚祝平.混沌时间序列的平均周期计算方法[J].系统工程,2010,28(12):111-113.
[19]韩娜.聚类算法在时间序列中的研究与应用[D].广州:广东工业大学,2011.
[20]闫秋艳,夏士雄.一种无限长时间序列的分段线性拟合算法[J].电子学报,2010,38(2):443-448.
[21]Rasheed F,Alhajj R.A framework for periodic outlier pattern detection in time-series sequences[J].IEEE Transactions on Cybernetics,2014,44(5):569-582.
[22]Buzzi-Ferraris G,Manenti F.Outlier detection in large data sets[J].Computers and Chemical Engineering,2010,35:388
[23]Li S T,Cheng Y C.A stochastic HMM-based forecasting model for fuzzy time series[J].IEEE Transactions on Cybernetics,2010,40(5):1255-1266.