分布式存储系统中数据访问子系统研究及实现
|
摘要
随着计算机和互联网技术的不断发展,网络信息呈现爆炸性的增长,如何解决互联网上信息的存储和利用是一个很有挑战性的问题,分布式存储方式由此应运而生,是当前学术界研究的热点问题,所以研究分布式存储系统有着重大的理论意义和现实意义。
分布式存储系统有两种类型,一种是P2P模式,另一种是非P2P模式。本文研究的是基于P2P模式的分布式存储系统。基于P2P模式的分布式存储系统由大量对等的存储节点构成整个系统,存储节点的性能决定了系统的整体性能,如何提高存储节点的性能是一个重要的研究方向。因此,本文集中研究分布式存储系统中存储节点的网络性能,具体内容包括吞吐率和CPU占用率。
本文的工作主要包括以下二个部分:
一、提出了ONPK方案,该方案可以提高节点的吞吐率和降低CPU的占用率。
如何提高存储节点的网络数据吞吐率,同时又能降低CPU占用率,本文提出了以下解决方法,简称为ONPK方案(The Optimizing of the Network Performance based on the Linux Kernel),该方法通过以下三个措施来实现目标:
减少数据传输时的复制次数;
减少系统调用次数和程序上下文切换次数;
预先构造数据头,降低协议栈的处理开销。
实验表明,ONPK方案对于千兆以太网的网络性能有着明显的改善,在保持740Mbps的发送速度的基础上,CPU的占用率平均可以降低11%。
二、研究和实现数据访问子系统。
在ONPK方案的基础上,研究和实现了一个数据访问子系统,该数据访问子系统采用P2P模式,支持断点续传和多点下载。
本论文的研究是国家863计划“分布式宽带业务制作协同环境的开发”(2008AA01A318)课题中分布式存储系统的一部分。
As the development of computers and the Internet, the information on Internet is showing a explosive growth. How to solve the storage and use of information is a very challenging problem.The distributed storage system appears, and it’s the hot-spot issues in academic. So the study of the distributed storage system is a very important issue.
There are two styles of the distributed storage system.One is based on the P2P, while another is not. The distributed storage system in this paper is based on P2P. It contains a large number of P2P storage nodes .Those P2P storage nodes determine the performance of the overall system.How to improve the performance of these nodes is an important issue. Therefore, this paper focuses on the network performance of the istributed storage system. It includes the throughput and CPU utilization.This paper mainly includes the following two parts:
Firstly, we propose a method called ONPK, which can not only improve the throughput but also reduce the CPU utilization.How to improve the throughput, while reduce the CPU utilization is a very difficult problem .In this paper, we propose the following solution, called ONPK (The Optimizing of the Network Performance based on the Linux Kernel).the method can solve the problem through the following three measures:
1. To reduce the times of replication when sending data;
2. To reduce the number of system calls and context switches;
3. Pre-constructed header, to reduce the processing overhead of the protocol stack.The experiments show that the mothed can improve the Gigabit Ethernet performance significantly.For instance, it can maintain a speed of 740Mbps while reduce CPU utilization by 11% on average.
Secondly, research and implement of Data Access Subsystem.Based on the ONPK, we research and implement a data access subsystem. It is based on the P2P technology and support for continuous download and downloads from many nodes.
In this paper, the study was a part of project: The distributed production of broadband services in the development of collaborative environment.
分布式存储系统有两种类型,一种是P2P模式,另一种是非P2P模式。本文研究的是基于P2P模式的分布式存储系统。基于P2P模式的分布式存储系统由大量对等的存储节点构成整个系统,存储节点的性能决定了系统的整体性能,如何提高存储节点的性能是一个重要的研究方向。因此,本文集中研究分布式存储系统中存储节点的网络性能,具体内容包括吞吐率和CPU占用率。
本文的工作主要包括以下二个部分:
一、提出了ONPK方案,该方案可以提高节点的吞吐率和降低CPU的占用率。
如何提高存储节点的网络数据吞吐率,同时又能降低CPU占用率,本文提出了以下解决方法,简称为ONPK方案(The Optimizing of the Network Performance based on the Linux Kernel),该方法通过以下三个措施来实现目标:
减少数据传输时的复制次数;
减少系统调用次数和程序上下文切换次数;
预先构造数据头,降低协议栈的处理开销。
实验表明,ONPK方案对于千兆以太网的网络性能有着明显的改善,在保持740Mbps的发送速度的基础上,CPU的占用率平均可以降低11%。
二、研究和实现数据访问子系统。
在ONPK方案的基础上,研究和实现了一个数据访问子系统,该数据访问子系统采用P2P模式,支持断点续传和多点下载。
本论文的研究是国家863计划“分布式宽带业务制作协同环境的开发”(2008AA01A318)课题中分布式存储系统的一部分。
As the development of computers and the Internet, the information on Internet is showing a explosive growth. How to solve the storage and use of information is a very challenging problem.The distributed storage system appears, and it’s the hot-spot issues in academic. So the study of the distributed storage system is a very important issue.
There are two styles of the distributed storage system.One is based on the P2P, while another is not. The distributed storage system in this paper is based on P2P. It contains a large number of P2P storage nodes .Those P2P storage nodes determine the performance of the overall system.How to improve the performance of these nodes is an important issue. Therefore, this paper focuses on the network performance of the istributed storage system. It includes the throughput and CPU utilization.This paper mainly includes the following two parts:
Firstly, we propose a method called ONPK, which can not only improve the throughput but also reduce the CPU utilization.How to improve the throughput, while reduce the CPU utilization is a very difficult problem .In this paper, we propose the following solution, called ONPK (The Optimizing of the Network Performance based on the Linux Kernel).the method can solve the problem through the following three measures:
1. To reduce the times of replication when sending data;
2. To reduce the number of system calls and context switches;
3. Pre-constructed header, to reduce the processing overhead of the protocol stack.The experiments show that the mothed can improve the Gigabit Ethernet performance significantly.For instance, it can maintain a speed of 740Mbps while reduce CPU utilization by 11% on average.
Secondly, research and implement of Data Access Subsystem.Based on the ONPK, we research and implement a data access subsystem. It is based on the P2P technology and support for continuous download and downloads from many nodes.
In this paper, the study was a part of project: The distributed production of broadband services in the development of collaborative environment.
引文
独孤求真。RTL8139网卡驱动程序分析。www.Osplay.org,Addylee2004@163.com
李斌,辛海红,胡铭兽,2006,一种高效的用户级通信协议的研究与应用,计算机工程2006 年1月。
刘天华,陈枭,朱宏峰,刘骏,2007,Linux可加载内核模块机制的研究与应用,<<微型计算机信息>>(嵌入式与SOC),2007年第23卷第7-2期。
罗杰文,Peer-To-Peer综述,http://www.intsci.ac.cn/users/luojw/P2P/ch01.html。王圣,苏金树,TCP加速技术研究综述,软件学报2007,Vol.15,No.11。
王小峰,时向泉,苏金树,2008,一种TCP/IP卸载的数据零拷贝传输方法,计算机工程与科学Vol.30,No. 2,2008。
张晓军,崔建,马皓等,2007,Socket缓冲区回收利用的网络性能研究.,《中国教育网络》, 2007年第02期。
张宇雷,黄皓,2007,基于网络处理器的零拷贝技术,计算机应用研究,2007年第1期。
周敬利,王志华,姜明华,徐漾,余胜生,基于TCP/IP卸载引擎的千兆网卡。计算机工程第30卷第4期。
周敬利,杨芳,汪雪磊,2004,千兆以太网中通信系统优化策略的研究,计算机工程与应用2004.4。
Alan Baldus . TOE: TCP/IP Offload Engine relieves CPU burden. www.sbei.com
Alliance whitepaper . Introduction to TCP/IP Offload Engine (TOE) 10 Gigabit Ethernet. http://www.10gea.org/SP0502IntroToTOE_F.pdf
Brice Goglin, Olivier Gl¨uck, and Pascale Vicat-Blanc Primet, 2005, An Efficient network API for in-Kernel Applications in Clusters. IEEE International Conference on Cluster Computing (Cluster 2005) Boston, Massachusetts, USA, September 27 - 30, 2005
Edwards A, Muir S, 1995, Experiences implementing a high performance TCP in user-space. In:Proc . of SIGCOMM’95 .New York ;ACM press ,1995.196~205
Gokhan Memik,William H. Mangione-Smith, 2002, A Flexible Accelerator for Layer 7 Networking Applicatins. Design Automation Conference, 2002. Proceedings. 39th
Giljae Lee, Yoonjoo Kwon ,Kwangjong Cho etc, 2008, Performance Evaluation of Gigabit Ethernet Interfaces Advanced Communication Technology, 2008. ICACT 2008. 10th International Conference on Volume 1, 17-20 Feb. 2008 Page(s):114– 117
Hankook Jang, Sang-Hwa Chung, Dae-Hyum Yoo, 2008, Implementation of an Efficient RDMAMechanism Tightly Coupled with a TCP/IP Offload Engine. Industrial Embedded Systems, 2008. SIES 2008. International Symposium on page(s): 82-88
H. V. Shah, C. Pu, and R. S. Madukkarumukumana, 1999, High Performance Sockets and RPC over Virtual Interface (VI) Architecture. In CANPC workshop (held in conjunction with HPCA Conference), pages 91-107, 1999.
Jonathan Lemon, Zhe Wang, Zheng Yang etc, 2003, A New Kernel Interface for High-Performance Internet Streaming Servers.http://2003.iwcw.org/papers/lemon.pdf
Kasenna Inc. Kasenna Streaming Accelerator. www.kasenna.com
Kyoung Park, et. Al, 2006, A Network I/0 Offloading Architecture for Terminal-Initiated Traffics. The Journal of Korean Institute of Next Generation Computing, Vol. 2, No. 3, pp.27-36, Sep. 2006
Liu Tianhua, Zhu Hongfeng, Chang Guira,Zhou, Chuansheng, 2006, Research and Implementation of Zero-Copy Technology in Linux. IMSCCS '06: Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences - Volume 2 (IMSCCS'06) - Volume 02
M-VIA, A High Performance Modular VIA for Linux. http://www.nersc.gov/ research/FTG/via.
Mapp G, Pope S, The design and implementation of a high-speed user-space transport protocol. In :Proc. Of the 1st Int’l Workshop on High Performance Protocol Architectures.http://citeseer.ist.psu.edu/metzler94design.html
Nathan L, Binkert Ali G, Saidi Steven K. Reinhardt , Integrated Network Interfaces for High-Bandwidth TCP/IP. Proceedings of the 12th international conference on Architectural support for programming languages and operating systems
N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik,C. L. Seitz,J. N. Seizovic, and .W. K. Su. Myrinet: A Gigabit-per-Second Local Area Network. Micro,IEEE Volume: 15, Issue: 1 On page(s): 29-36
P. Shivam, P. Wyckoff, and D. Panda. EMP: Zero-copy OSbypass NIC-driven Gigabit Ethernet Message Passing. In SC’01.
P. Shivam, P.Wyckoff, and D. Panda, 2002 , Can User Level Protocols Take Advantage of Multi-CPU NICs? In IPDPS’02.
Pavan Balaji, Piyush Shivam ,Pete Wyckoff etc, 2002, High Performance User Level Socket over Gigabit Ethernet. Cluster Computing, 2002. Proceedings. 2002 IEEE International Conference on 23-26 Sept. 2002 Page(s):179– 186
Soo-Cheol Oh, et. Al, 2006, An Effective Linux Kernel Module supporting TCP/IP ffload engine on Grid. Proceedings of the 5th International Conference on Grid and cooperativeComputing, pp.228-235, Oct. 2006
Thekkath CA, Nguyen TD, Moyt B etc, 1993, Implementing network protocols at user level. I EEE/ACM Trans on Networking ,1993,1(5):554~565
W. Richard Stevens. UNIX Network Programming.
Yuhyeon Bak, Kapdong Kin, Youngju Lee etc, 2006, Design and Implementation of High Performance SMART for HD Video On Demand. 2006 Interational Conference on Hybrid Information Technology
李斌,辛海红,胡铭兽,2006,一种高效的用户级通信协议的研究与应用,计算机工程2006 年1月。
刘天华,陈枭,朱宏峰,刘骏,2007,Linux可加载内核模块机制的研究与应用,<<微型计算机信息>>(嵌入式与SOC),2007年第23卷第7-2期。
罗杰文,Peer-To-Peer综述,http://www.intsci.ac.cn/users/luojw/P2P/ch01.html。王圣,苏金树,TCP加速技术研究综述,软件学报2007,Vol.15,No.11。
王小峰,时向泉,苏金树,2008,一种TCP/IP卸载的数据零拷贝传输方法,计算机工程与科学Vol.30,No. 2,2008。
张晓军,崔建,马皓等,2007,Socket缓冲区回收利用的网络性能研究.,《中国教育网络》, 2007年第02期。
张宇雷,黄皓,2007,基于网络处理器的零拷贝技术,计算机应用研究,2007年第1期。
周敬利,王志华,姜明华,徐漾,余胜生,基于TCP/IP卸载引擎的千兆网卡。计算机工程第30卷第4期。
周敬利,杨芳,汪雪磊,2004,千兆以太网中通信系统优化策略的研究,计算机工程与应用2004.4。
Alan Baldus . TOE: TCP/IP Offload Engine relieves CPU burden. www.sbei.com
Alliance whitepaper . Introduction to TCP/IP Offload Engine (TOE) 10 Gigabit Ethernet. http://www.10gea.org/SP0502IntroToTOE_F.pdf
Brice Goglin, Olivier Gl¨uck, and Pascale Vicat-Blanc Primet, 2005, An Efficient network API for in-Kernel Applications in Clusters. IEEE International Conference on Cluster Computing (Cluster 2005) Boston, Massachusetts, USA, September 27 - 30, 2005
Edwards A, Muir S, 1995, Experiences implementing a high performance TCP in user-space. In:Proc . of SIGCOMM’95 .New York ;ACM press ,1995.196~205
Gokhan Memik,William H. Mangione-Smith, 2002, A Flexible Accelerator for Layer 7 Networking Applicatins. Design Automation Conference, 2002. Proceedings. 39th
Giljae Lee, Yoonjoo Kwon ,Kwangjong Cho etc, 2008, Performance Evaluation of Gigabit Ethernet Interfaces Advanced Communication Technology, 2008. ICACT 2008. 10th International Conference on Volume 1, 17-20 Feb. 2008 Page(s):114– 117
Hankook Jang, Sang-Hwa Chung, Dae-Hyum Yoo, 2008, Implementation of an Efficient RDMAMechanism Tightly Coupled with a TCP/IP Offload Engine. Industrial Embedded Systems, 2008. SIES 2008. International Symposium on page(s): 82-88
H. V. Shah, C. Pu, and R. S. Madukkarumukumana, 1999, High Performance Sockets and RPC over Virtual Interface (VI) Architecture. In CANPC workshop (held in conjunction with HPCA Conference), pages 91-107, 1999.
Jonathan Lemon, Zhe Wang, Zheng Yang etc, 2003, A New Kernel Interface for High-Performance Internet Streaming Servers.http://2003.iwcw.org/papers/lemon.pdf
Kasenna Inc. Kasenna Streaming Accelerator. www.kasenna.com
Kyoung Park, et. Al, 2006, A Network I/0 Offloading Architecture for Terminal-Initiated Traffics. The Journal of Korean Institute of Next Generation Computing, Vol. 2, No. 3, pp.27-36, Sep. 2006
Liu Tianhua, Zhu Hongfeng, Chang Guira,Zhou, Chuansheng, 2006, Research and Implementation of Zero-Copy Technology in Linux. IMSCCS '06: Proceedings of the First International Multi-Symposiums on Computer and Computational Sciences - Volume 2 (IMSCCS'06) - Volume 02
M-VIA, A High Performance Modular VIA for Linux. http://www.nersc.gov/ research/FTG/via.
Mapp G, Pope S, The design and implementation of a high-speed user-space transport protocol. In :Proc. Of the 1st Int’l Workshop on High Performance Protocol Architectures.http://citeseer.ist.psu.edu/metzler94design.html
Nathan L, Binkert Ali G, Saidi Steven K. Reinhardt , Integrated Network Interfaces for High-Bandwidth TCP/IP. Proceedings of the 12th international conference on Architectural support for programming languages and operating systems
N. J. Boden, D. Cohen, R. E. Felderman, A. E. Kulawik,C. L. Seitz,J. N. Seizovic, and .W. K. Su. Myrinet: A Gigabit-per-Second Local Area Network. Micro,IEEE Volume: 15, Issue: 1 On page(s): 29-36
P. Shivam, P. Wyckoff, and D. Panda. EMP: Zero-copy OSbypass NIC-driven Gigabit Ethernet Message Passing. In SC’01.
P. Shivam, P.Wyckoff, and D. Panda, 2002 , Can User Level Protocols Take Advantage of Multi-CPU NICs? In IPDPS’02.
Pavan Balaji, Piyush Shivam ,Pete Wyckoff etc, 2002, High Performance User Level Socket over Gigabit Ethernet. Cluster Computing, 2002. Proceedings. 2002 IEEE International Conference on 23-26 Sept. 2002 Page(s):179– 186
Soo-Cheol Oh, et. Al, 2006, An Effective Linux Kernel Module supporting TCP/IP ffload engine on Grid. Proceedings of the 5th International Conference on Grid and cooperativeComputing, pp.228-235, Oct. 2006
Thekkath CA, Nguyen TD, Moyt B etc, 1993, Implementing network protocols at user level. I EEE/ACM Trans on Networking ,1993,1(5):554~565
W. Richard Stevens. UNIX Network Programming.
Yuhyeon Bak, Kapdong Kin, Youngju Lee etc, 2006, Design and Implementation of High Performance SMART for HD Video On Demand. 2006 Interational Conference on Hybrid Information Technology