达梦数据库中大规模数据可扩展并行算法
|
摘要
达梦(DM)数据库中的数据规模大且维度复杂,为了在有限的条件下尽可能满足用户对DM数据库功能的需求,提出一种新的DM数据库中大规模数据可扩展并行算法。不可扩展并行算法包括朴素并行、典型并行与逻辑并行三种处理规则,新算法将这三种处理规则结合起来实现数据自主运算,令每个运算节点均拥有三种处理模式,采用有向图将大规模数据划分为局部数据,并分配到处理器上,通过设置数据处理优先等级,完成流水线形式的数据处理过程,赋予并行算法强大的可扩展性。实验结果表明,新算法具有较强的可扩展性,负债均衡能力强。
The Dameng( DM) data in the large scale and complex dimensions,to as much as possible in limited conditions to meet the needs of users of the DM database function,a scalable parallel algorithm of large-scale new data was proposed in the DM database. Not a scalable parallel algorithm including simple parallel,parallel and parallel three kinds of typical logic processing rules,a new algorithm of the three kinds of rules to combine data independent operations,so that each computation node has three processing modes,using directed graph divide the large-scale data into local data,and assigned to the processor,through set the priority of data processing,to complete the pipeline in the form of data processing,with strong scalability of parallel algorithms. The experimental results show that the new algorithm has strong scalability and excellent debt balance ability.
The Dameng( DM) data in the large scale and complex dimensions,to as much as possible in limited conditions to meet the needs of users of the DM database function,a scalable parallel algorithm of large-scale new data was proposed in the DM database. Not a scalable parallel algorithm including simple parallel,parallel and parallel three kinds of typical logic processing rules,a new algorithm of the three kinds of rules to combine data independent operations,so that each computation node has three processing modes,using directed graph divide the large-scale data into local data,and assigned to the processor,through set the priority of data processing,to complete the pipeline in the form of data processing,with strong scalability of parallel algorithms. The experimental results show that the new algorithm has strong scalability and excellent debt balance ability.
引文
1刘智翔,方勇,宋安平,等.基于MRT-LBM方法的大规模可扩展并行计算研究[J].计算机研究与发展,2016,53(5):1156-1165Liu Zhixiang,Fang Yong,Song Anping,et al.Large scale scalable parallel computing based on LBM with multiple-relaxation-time model[J].Computer Research and Development,2016,53(5):1156-1165
2赵飞,苏忠.一致性哈希算法在数据库集群上的拓展应用[J].成都信息工程学院学报,2015,30(1):52-58Zhao Fei,Su Zhong.The expansion research of consistent hash algorithm on parallel analysis database cluster[J].Journal of Chengdu University of Information Technology,2015,30(1):52-58
3张滨,乐嘉锦.基于列存储的MapReduce并行连接算法[J].计算机工程,2014,40(8):70-75Zhang Bin,Le Jiajin.MapReduce parallel join algorithm based on column-store[J].Computer Engineering,2014,40(8):70-75
4李志辉,蒋新宇,吴俊林,等.求解Boltzmann模型方程高性能并行算法在航天跨流域空气动力学应用研究[J].计算机学报,2016,39(9):1801-1811Li Zhihui,Jiang Xinyu,Wu Junlin,et al.Study on high performance parallel algorithm for spacecraft reentry aerodynamics in the whole of flow regimes using Boltzmann model equation[J].Chinese Journal of Computers,2016,39(9):1801-1811
5范宣华,陈璞,吴瑞安,等.基于Jacobi-Davidson算法的大规模模态分析并行计算研究[J].振动与冲击,2014,33(1):203-208Fan Xuanhua,Chen Pu,Wu Ruian,et al.Parallel computing of largescale modal ananlysis based on Jacobi-Davidson algorithm[J].Journal of Vibration and Shock,2014,33(1):203-208
6范协裕,任应超.开源关系数据库集群的并行空间连接算法实现[J].计算机系统应用,2016,25(10):233-239Fan Xieyu,Ren Yingchao.Research and realization on parallel spatial join query algorithm based on open source RDBMS cluster[J].Computer Systems&Applications,2016,25(10):233-239
7时钢.基于Mipmap的大规模地形绘制算法与仿真[J].计算机仿真,2015,32(2):270-274Shi Gang.A rendering algorithm and simulation for large scale terrain based on mipmap[J].Computer Simulation,2015,32(2):270-274
8徐仕超,黎建辉.基于数据库集群的海量RDF数据联合查询系统的研究与实现[J].科研信息化技术与应用,2016,7(1):24-35Xu Shichao,Li Jianhui.Research and implementation of massive RDFdata federated query system based on database cluster[J].E-science Technology&Application,2016,7(1):24-35
9何玉新.增广链修复下大数据并行搜索聚类算法[J].科技通报,2016,32(3):109-113He Yuxin.Parallel search clustering algorithm of large based on data augmented chain repair[J].Bulletin of Science and Technology,2016,32(3):109-113
10赵长海,王狮虎,罗国安,等.高度可扩展的3D叠前Kirchhoff时间偏移并行算法[J].计算机研究与发展,2015,52(4):869-878Zhao Changhai,Wang Shihu,Luo Guoan,et al.A highly scalable parallel algorithm for 3D prestack Kirchhoff time migration[J].Journal of Computer Research and Development,2015,52(4):869-878
11亢晓丽,亢晓琛.大规模DEM数据并行可视域分析算法研究[J].计算机测量与控制,2014,22(6):1970-1972Kang Xiaoli,Kang Xiaochen.Parallel viewshed analysis on large scale DEM data[J].Computer Measurement&Control,2014,22(6):1970-1972
12王瑞峰,张小花,张迎春.移动数据库中数据复制同步处理策略的研究[J].计算机工程与应用,2016,52(1):61-65Wang Ruifeng,Zhang Xiaohua,Zhang Yingchun.Research on synchronous processing strategy of data repli-cation in mobile database[J].Computer Engineering and Applications,2016,52(1):61-65
13贺正红,周娅,文缔尧,等.面向HBase的大规模数据加载研究[J].计算机系统应用,2016,25(6):231-237He Zhenghong,Zhou Ya,Wen Diyao,et al.Research on large scale data loading based on HBase[J].Computer Systems&Applications,2016,25(6):231-237
14石岚,杨聪.基于数据库的咸阳职业技术学院科研论文统计分析[J].电脑知识与技术,2016,12(1):16-17Shi Lan,Yang Cong.Statistical analysis of papers undersigned Xianyang vocational technical college based CNKI[J].Computer Knowledge and Technology,2016,12(1):16-17
15田野,佟皓萌.千兆以太网中CRC-32的并行实现[J].电子设计工程,2016,24(15):112-114Tian Ye,Tong Haomeng.Parallel implementation of CRC-32 in gigabit ethernet[J].Electronic Design Engineering,2016,24(15):112-114
16王青,谭良,杨显华.基于Spark的Apriori并行算法优化实现[J].郑州大学学报(理学版),2016,48(4):60-64Wang Qing,Tan Liang,Yang Xianhua.Optimization of apriori parallel algorithm based on spark[J].Journal of Zhengzhou University:(Science Edition),2016,48(4):60-64
17 Zhang J B,Li T R,Pan Y,et al.Parallel and incremental algorithm for knowledge update based on rough sets in cloud platform[J].American Journal of Pathology,2015,10(6):827-834
18张明锋,杨国伟,郑冠男,等.基于任意多面体网格的NavierStokes方程并行求解器[J].科学技术与工程,2016,16(18):101-105Zhang Mingfeng,Yang Guowei,Zheng Guannan,et al.A parallel solver for Navier-Stokes equation based on arbitrary polyhedral grids[J].Science Technology and Engineering,2016,16(18):101-105
19刘纪平,吴立新,董春,等.一种大规模空间数据流式并行处理方法研究[J].测绘科学,2016,41(1):89-93Liu Jiping,Wu Lixin,Dong Chun,et al.Study on a streaming parallel method for massive spatial data processing[J].Science of Surveying and Mapping,2016,41(1):89-93
20杨林青,李湛,牟雁超,等.面向大规模数据集的并行化Top-k Skyline查询算法[J].计算机科学与探索,2015,9(8):897-905Yang Linqing,Li Zhan,Mou Yanchao,et al.Algorithm of parallel Top-k Skyline queries for large data set[J].Journal of Frontiers of Computer Science&Technology,2015,9(8):897-905
2赵飞,苏忠.一致性哈希算法在数据库集群上的拓展应用[J].成都信息工程学院学报,2015,30(1):52-58Zhao Fei,Su Zhong.The expansion research of consistent hash algorithm on parallel analysis database cluster[J].Journal of Chengdu University of Information Technology,2015,30(1):52-58
3张滨,乐嘉锦.基于列存储的MapReduce并行连接算法[J].计算机工程,2014,40(8):70-75Zhang Bin,Le Jiajin.MapReduce parallel join algorithm based on column-store[J].Computer Engineering,2014,40(8):70-75
4李志辉,蒋新宇,吴俊林,等.求解Boltzmann模型方程高性能并行算法在航天跨流域空气动力学应用研究[J].计算机学报,2016,39(9):1801-1811Li Zhihui,Jiang Xinyu,Wu Junlin,et al.Study on high performance parallel algorithm for spacecraft reentry aerodynamics in the whole of flow regimes using Boltzmann model equation[J].Chinese Journal of Computers,2016,39(9):1801-1811
5范宣华,陈璞,吴瑞安,等.基于Jacobi-Davidson算法的大规模模态分析并行计算研究[J].振动与冲击,2014,33(1):203-208Fan Xuanhua,Chen Pu,Wu Ruian,et al.Parallel computing of largescale modal ananlysis based on Jacobi-Davidson algorithm[J].Journal of Vibration and Shock,2014,33(1):203-208
6范协裕,任应超.开源关系数据库集群的并行空间连接算法实现[J].计算机系统应用,2016,25(10):233-239Fan Xieyu,Ren Yingchao.Research and realization on parallel spatial join query algorithm based on open source RDBMS cluster[J].Computer Systems&Applications,2016,25(10):233-239
7时钢.基于Mipmap的大规模地形绘制算法与仿真[J].计算机仿真,2015,32(2):270-274Shi Gang.A rendering algorithm and simulation for large scale terrain based on mipmap[J].Computer Simulation,2015,32(2):270-274
8徐仕超,黎建辉.基于数据库集群的海量RDF数据联合查询系统的研究与实现[J].科研信息化技术与应用,2016,7(1):24-35Xu Shichao,Li Jianhui.Research and implementation of massive RDFdata federated query system based on database cluster[J].E-science Technology&Application,2016,7(1):24-35
9何玉新.增广链修复下大数据并行搜索聚类算法[J].科技通报,2016,32(3):109-113He Yuxin.Parallel search clustering algorithm of large based on data augmented chain repair[J].Bulletin of Science and Technology,2016,32(3):109-113
10赵长海,王狮虎,罗国安,等.高度可扩展的3D叠前Kirchhoff时间偏移并行算法[J].计算机研究与发展,2015,52(4):869-878Zhao Changhai,Wang Shihu,Luo Guoan,et al.A highly scalable parallel algorithm for 3D prestack Kirchhoff time migration[J].Journal of Computer Research and Development,2015,52(4):869-878
11亢晓丽,亢晓琛.大规模DEM数据并行可视域分析算法研究[J].计算机测量与控制,2014,22(6):1970-1972Kang Xiaoli,Kang Xiaochen.Parallel viewshed analysis on large scale DEM data[J].Computer Measurement&Control,2014,22(6):1970-1972
12王瑞峰,张小花,张迎春.移动数据库中数据复制同步处理策略的研究[J].计算机工程与应用,2016,52(1):61-65Wang Ruifeng,Zhang Xiaohua,Zhang Yingchun.Research on synchronous processing strategy of data repli-cation in mobile database[J].Computer Engineering and Applications,2016,52(1):61-65
13贺正红,周娅,文缔尧,等.面向HBase的大规模数据加载研究[J].计算机系统应用,2016,25(6):231-237He Zhenghong,Zhou Ya,Wen Diyao,et al.Research on large scale data loading based on HBase[J].Computer Systems&Applications,2016,25(6):231-237
14石岚,杨聪.基于数据库的咸阳职业技术学院科研论文统计分析[J].电脑知识与技术,2016,12(1):16-17Shi Lan,Yang Cong.Statistical analysis of papers undersigned Xianyang vocational technical college based CNKI[J].Computer Knowledge and Technology,2016,12(1):16-17
15田野,佟皓萌.千兆以太网中CRC-32的并行实现[J].电子设计工程,2016,24(15):112-114Tian Ye,Tong Haomeng.Parallel implementation of CRC-32 in gigabit ethernet[J].Electronic Design Engineering,2016,24(15):112-114
16王青,谭良,杨显华.基于Spark的Apriori并行算法优化实现[J].郑州大学学报(理学版),2016,48(4):60-64Wang Qing,Tan Liang,Yang Xianhua.Optimization of apriori parallel algorithm based on spark[J].Journal of Zhengzhou University:(Science Edition),2016,48(4):60-64
17 Zhang J B,Li T R,Pan Y,et al.Parallel and incremental algorithm for knowledge update based on rough sets in cloud platform[J].American Journal of Pathology,2015,10(6):827-834
18张明锋,杨国伟,郑冠男,等.基于任意多面体网格的NavierStokes方程并行求解器[J].科学技术与工程,2016,16(18):101-105Zhang Mingfeng,Yang Guowei,Zheng Guannan,et al.A parallel solver for Navier-Stokes equation based on arbitrary polyhedral grids[J].Science Technology and Engineering,2016,16(18):101-105
19刘纪平,吴立新,董春,等.一种大规模空间数据流式并行处理方法研究[J].测绘科学,2016,41(1):89-93Liu Jiping,Wu Lixin,Dong Chun,et al.Study on a streaming parallel method for massive spatial data processing[J].Science of Surveying and Mapping,2016,41(1):89-93
20杨林青,李湛,牟雁超,等.面向大规模数据集的并行化Top-k Skyline查询算法[J].计算机科学与探索,2015,9(8):897-905Yang Linqing,Li Zhan,Mou Yanchao,et al.Algorithm of parallel Top-k Skyline queries for large data set[J].Journal of Frontiers of Computer Science&Technology,2015,9(8):897-905
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |