并行系统零拷贝通信优化技术的研究
摘要
集群系统在近年来已经逐渐成为并行计算实现载体的主流。随着计算结点性能的不断提高,集群系统的通信性能已经成为影响并行计算性能的一个重要因素。为了使得高性能计算达到实用化,并且满足计算节点间通信需求,就必须提高通信系统的性能。但是新型的高速网络常常价格昂贵,而传统的低效网络又因为不能满足高性能通信需要而得不到有效利用。因此,对并行集群通信底层的通信效率进行提高,将已有的通信资源充分利用起来是目前的研究热点之一。本文围绕着如何提高并行集群的通信效率展开研究工作,主要内容包括:
1.阐述了并行计算技术的应用优势,并行计算的重要载体集群系统的发展,说明了通信系统的发展对集群系统计算效率的提高和适用范围的扩大具有重要意义。着重研究了应用于并行计算的高性能通信优化技术。在分析消息传递过程的内核代码基础上,分别从应用层通信、系统层通信、UDP/IP协议通信三方面,对并行集群系统通信时间开销进行研究。根据轻量级通信协议与零拷贝通信协议的特点,有针对性的提出了通信性能优化的方法,提高通信底层的通信效率。
2.在现有的集群试验环境基础上,使现有的零拷贝技术更具有实用价值和应用性能。通过对静态缓冲区分配、动态缓冲区分配策略进行分析比较,并且针对复杂的应用层协议,提出一种不依赖操作系统或特殊硬件支持的,可自控的缓冲区管理的集群零拷贝通信优策略。经过并行测试、Ping-pong测试、socket通信传输速率测试、cpu利用率测试等试验。结果表明,经过优化后的通信系统和原有系统相比在数据传输速率和cpu利用率上均有所提高。
Recently, Cluster system has become the main trend in parallel computing. With the development of the Computing node performance, Cluster system of communication performance has become an important factor of parallel computing performance. In order to promote the practice of High Performance Computing and content the demand of Computing node communication, it must promote the communication system performance. The new High Speed Network is always very costly, but the traditional inefficient network can not be got the effective utilization, because it can not satisfy the demand of the High performance Communications. So we must promote the communication efficiency for the parallel system communication, and using the communication resources adequately has been a research hotspot.
In this paper, we studied how to promotes the communication efficiency, The main content is:
1. Expound the application advantage of Parallel computing technology and the development of the important carrier of cluster system in parallel computing. Explain the important significance for the development of the cluster computing efficiency and applicable scope, because of the development of the communication system. Study High-performance communication optimization technology which is used on the parallel computing. Analysing the kernel code in the process of message transmission, we analysed the process of messages from the network communication, the system layer communication, the overhead of UDP/IP communication. Based on the feature of the lightweight communications protocol and Zero copy communication protocol, we proposed the method for optimizing the performance of the system communication. Improve the efficiency of communication.
2. Based on the existing test environment, make zero copy technology have more practical value and application properties. By comparing static buffer allocation with dynamic buffer allocation, and based on the complicated application layer protocol, we proposed a cluster strategy of zero copy communication, which is independent of a operating system or a special hardware, and has an intelligent buffer management. Through the parallel test, Ping-pong test, socket test and the CPU utilization test, the data transfer rate and the CPU utilization of the communication system which is optimized is better than that without optimizing.
1.阐述了并行计算技术的应用优势,并行计算的重要载体集群系统的发展,说明了通信系统的发展对集群系统计算效率的提高和适用范围的扩大具有重要意义。着重研究了应用于并行计算的高性能通信优化技术。在分析消息传递过程的内核代码基础上,分别从应用层通信、系统层通信、UDP/IP协议通信三方面,对并行集群系统通信时间开销进行研究。根据轻量级通信协议与零拷贝通信协议的特点,有针对性的提出了通信性能优化的方法,提高通信底层的通信效率。
2.在现有的集群试验环境基础上,使现有的零拷贝技术更具有实用价值和应用性能。通过对静态缓冲区分配、动态缓冲区分配策略进行分析比较,并且针对复杂的应用层协议,提出一种不依赖操作系统或特殊硬件支持的,可自控的缓冲区管理的集群零拷贝通信优策略。经过并行测试、Ping-pong测试、socket通信传输速率测试、cpu利用率测试等试验。结果表明,经过优化后的通信系统和原有系统相比在数据传输速率和cpu利用率上均有所提高。
Recently, Cluster system has become the main trend in parallel computing. With the development of the Computing node performance, Cluster system of communication performance has become an important factor of parallel computing performance. In order to promote the practice of High Performance Computing and content the demand of Computing node communication, it must promote the communication system performance. The new High Speed Network is always very costly, but the traditional inefficient network can not be got the effective utilization, because it can not satisfy the demand of the High performance Communications. So we must promote the communication efficiency for the parallel system communication, and using the communication resources adequately has been a research hotspot.
In this paper, we studied how to promotes the communication efficiency, The main content is:
1. Expound the application advantage of Parallel computing technology and the development of the important carrier of cluster system in parallel computing. Explain the important significance for the development of the cluster computing efficiency and applicable scope, because of the development of the communication system. Study High-performance communication optimization technology which is used on the parallel computing. Analysing the kernel code in the process of message transmission, we analysed the process of messages from the network communication, the system layer communication, the overhead of UDP/IP communication. Based on the feature of the lightweight communications protocol and Zero copy communication protocol, we proposed the method for optimizing the performance of the system communication. Improve the efficiency of communication.
2. Based on the existing test environment, make zero copy technology have more practical value and application properties. By comparing static buffer allocation with dynamic buffer allocation, and based on the complicated application layer protocol, we proposed a cluster strategy of zero copy communication, which is independent of a operating system or a special hardware, and has an intelligent buffer management. Through the parallel test, Ping-pong test, socket test and the CPU utilization test, the data transfer rate and the CPU utilization of the communication system which is optimized is better than that without optimizing.
引文
[1]杨晓奇.郑启龙.陈国良等.国产万亿次高性能计算机KD_50_的通信优化.小型微型计算机,2009,8(30):1474.
[2]Regnier G.Makineni S.et al.TCP Onloading for Data Center Servers[J].IEEE ComputerMagazine,2004,37(11):48-58.
[3]高飞.并行系统高效多通道通信技术的研究:(硕士学位论文).大连:辽宁师范大学,2008.
[4]Chen Zhihui, Ma Jie, Chen Guoliang, et al.Study on IP supporting over user level protocol BCL 23 .Journal of Software,2003,14(09):163021634.
[5]heng Weimin, Tang Zhizhong. Computer architecture.Beijing: Tsinghua Publisher, 1998.
[6]王佰玲.方滨兴.云晓春.零拷贝报文捕获平台的研究.计算机学报,2005,28(1):46-52.
[7]李大斌.并行数据库通讯组件零拷贝技术的研究与实现:(硕士学位论文).哈尔滨:哈尔滨工业大学,2005.
[8]可向民.龚正虎.夏建东.零拷贝技术及其实现的研究.计算机工程与科学,2000,5(22):17-20,21.
[9]夏鸿.程克非.基于零拷贝技术的千兆网络性能优化研究.微计算机信息,2008,24(36):155-157.
[10]余胜生.初莹莹.周敬利等.基于RDMA协议的零拷贝技术研究.计算机工程与应用,2004,3(40):126-128.
[11]Eicken T, Culler D E, et al. Active messages: A mechanism for integrated communication and computation.In: Proceedings of the 19th International Symposium on Computer Architecture,1992.
[12]Guo Qingping,Yakup Paker. Concurrent communication and granularity assessment for a transputer-based multiprocessor system. Journal of Computer Systems Science&Engineering,1990,5(1):18-20.
[13]刘方爱.乔香珍.刘志勇.并行计算模型的层次分析及性能评价.计算机科学:2000,27(8):1-4.
[14]聂晓娜.模型生物大分子电荷分布并行处理的研究:(硕士学位论文).大连:辽宁师范大学,2009.
[15]刘青昆.聂晓娜.马丽.宫利东.Cholesky分解并行算法的性能评测.辽宁师范大学学报(自然科学版),2009,1(32):58-60.
[16]宋佳.谢湘楠.冉蜀阳. TCP/IP协议的零拷贝和简化实现.中国测试技术,2007,1(33):114-117.
[17]毛永捷.用户级通信在软件分布式共享存储系统中的应用:(博士学位论文).北京:中国科学院研究生院(计算技术研究所).2000.
[18]刘天华.陈袅.朱宏峰等.Linux可加载内核模块机制的研究与应用.微计算机信息,2007,23(7-2):48-49.
[19]倪继利.linux内核分析及编程,北京:电子工业出版社,2005.
[20]张兆心.方滨兴.胡铭曾.支持IDS的高速网络信息获取体系结构.北京邮电大学学报,2006,29(2):118-122.
[21]于晓.王家礼.嵌入式系统网络实时性能的优化.电子测量与仪器学报,2009,3(23):62-63.
[22]俞晓明.郭莉.TCP/IP协议处理中的缓冲区优化及实现.计算机工程,2006,32(8):62-63.
[23]宋丽华.张晓彤.王沁等.一种高效嵌人式协议栈缓冲区管理机制.小型微型计算机系统,2008,29(1):1-5.
[24]王培东.吴显伟.一种自适应的嵌入式协议栈缓冲区管理机制.计算机应用研究,2009,26(6):2254-2256.
[25]姚崎.刘吉强.韩臻等.面向多核处理器的linux网络报文缓冲区重用机制研究[J].通信学报,2009,30(9):102-108.
[26]Druschel P, Peterson L L.Fbufs: A High-bandwidth Cross-domain Transfer Facility [C].Fourteenth ACM Symposium on Operating Systems Principles,1993-12:189-202.
[27]Clark D D, Tennenhouse D L. Architectural Considerations for a New Generation of Protocols[C]. Proceedings of the SIGGOMM’90 Symposium,1990-09:200-208.
[2]Regnier G.Makineni S.et al.TCP Onloading for Data Center Servers[J].IEEE ComputerMagazine,2004,37(11):48-58.
[3]高飞.并行系统高效多通道通信技术的研究:(硕士学位论文).大连:辽宁师范大学,2008.
[4]Chen Zhihui, Ma Jie, Chen Guoliang, et al.Study on IP supporting over user level protocol BCL 23 .Journal of Software,2003,14(09):163021634.
[5]heng Weimin, Tang Zhizhong. Computer architecture.Beijing: Tsinghua Publisher, 1998.
[6]王佰玲.方滨兴.云晓春.零拷贝报文捕获平台的研究.计算机学报,2005,28(1):46-52.
[7]李大斌.并行数据库通讯组件零拷贝技术的研究与实现:(硕士学位论文).哈尔滨:哈尔滨工业大学,2005.
[8]可向民.龚正虎.夏建东.零拷贝技术及其实现的研究.计算机工程与科学,2000,5(22):17-20,21.
[9]夏鸿.程克非.基于零拷贝技术的千兆网络性能优化研究.微计算机信息,2008,24(36):155-157.
[10]余胜生.初莹莹.周敬利等.基于RDMA协议的零拷贝技术研究.计算机工程与应用,2004,3(40):126-128.
[11]Eicken T, Culler D E, et al. Active messages: A mechanism for integrated communication and computation.In: Proceedings of the 19th International Symposium on Computer Architecture,1992.
[12]Guo Qingping,Yakup Paker. Concurrent communication and granularity assessment for a transputer-based multiprocessor system. Journal of Computer Systems Science&Engineering,1990,5(1):18-20.
[13]刘方爱.乔香珍.刘志勇.并行计算模型的层次分析及性能评价.计算机科学:2000,27(8):1-4.
[14]聂晓娜.模型生物大分子电荷分布并行处理的研究:(硕士学位论文).大连:辽宁师范大学,2009.
[15]刘青昆.聂晓娜.马丽.宫利东.Cholesky分解并行算法的性能评测.辽宁师范大学学报(自然科学版),2009,1(32):58-60.
[16]宋佳.谢湘楠.冉蜀阳. TCP/IP协议的零拷贝和简化实现.中国测试技术,2007,1(33):114-117.
[17]毛永捷.用户级通信在软件分布式共享存储系统中的应用:(博士学位论文).北京:中国科学院研究生院(计算技术研究所).2000.
[18]刘天华.陈袅.朱宏峰等.Linux可加载内核模块机制的研究与应用.微计算机信息,2007,23(7-2):48-49.
[19]倪继利.linux内核分析及编程,北京:电子工业出版社,2005.
[20]张兆心.方滨兴.胡铭曾.支持IDS的高速网络信息获取体系结构.北京邮电大学学报,2006,29(2):118-122.
[21]于晓.王家礼.嵌入式系统网络实时性能的优化.电子测量与仪器学报,2009,3(23):62-63.
[22]俞晓明.郭莉.TCP/IP协议处理中的缓冲区优化及实现.计算机工程,2006,32(8):62-63.
[23]宋丽华.张晓彤.王沁等.一种高效嵌人式协议栈缓冲区管理机制.小型微型计算机系统,2008,29(1):1-5.
[24]王培东.吴显伟.一种自适应的嵌入式协议栈缓冲区管理机制.计算机应用研究,2009,26(6):2254-2256.
[25]姚崎.刘吉强.韩臻等.面向多核处理器的linux网络报文缓冲区重用机制研究[J].通信学报,2009,30(9):102-108.
[26]Druschel P, Peterson L L.Fbufs: A High-bandwidth Cross-domain Transfer Facility [C].Fourteenth ACM Symposium on Operating Systems Principles,1993-12:189-202.
[27]Clark D D, Tennenhouse D L. Architectural Considerations for a New Generation of Protocols[C]. Proceedings of the SIGGOMM’90 Symposium,1990-09:200-208.