基于混合P2P结构的网格存储模型的研究
摘要
网格是利用高速国际互联网把地理上广泛分布的各种资源,包括计算资源、存储资源、带宽资源、软件资源、数据资源、信息资源、知识资源等连成一个逻辑整体,就像一台超级计算机一样为用户提供一体化信息和应用服务。
随着计算机技术和Internet的发展,网络上出现大量闲散的自治结点,用何种方法能够有效而可靠的利用这些闲散结点的剩余资源,成为今年来研究的热点之一。
本文首先讨论和总结了分布式存储的发展历史和现状,并在对P2P技术的研究现状进行了系统、全面的分析和总结的基础上,设计了一种基于混合型P2P技术的网格存储模型。该存储模型将大量分散的结点组织成一个逻辑网络。在系统内部,结点被分为服务器结点,超级结点和一般结点,并结合热点文件和超级结点的特点,对原有的方根复制策略进行了改进,提出了一种混合型的副本备份策略,即在一般结点上存储的数据按照方根复制策略进行备份;在超级结点上,利用RAID技术备份热点文件的副本,经过仿真试验证明,在维持相同可用性的前提下;超级结点上采用RAID技术备份文件副本的策略,能够比单纯的方根复制策略有效的节省有限的网络空间资源。
Grid is a kind of the Technology that using high-speed Internet on a broad geographical-distributed of various resources,including computing resources,storage resources,bandwidth resources, software resources,data resources,information resources,knowledge resources,as a logical Whole thing,just as a supercomputer as to provide users with the integration services of information and application.
With the computer technology and Internet development,there are large number of idle nodes appeared on the net.In which effective and reliable way to be able to use the remaining resources of the idle nodes is a hot research topic in recently years.
First,this paper discusses and summarizes the development and actuality of distributed storage systems.Based on systematically analyzing and summarizing the relevant works on peer-to-peer technology,it designs a Hybrid P2P based distributed storage system. Organizes large numbers of nodes distributed in Internet into a united overlay network.The system divide into three kinds of node,there are server node,super node and general node.Combined with the haracteristics of hot documents and super-node,we modify the original square-root replication strategy,and then present a new mixed-type replication of backup strategy.That is,using the square-root replication in the general node as the backup strategy;and using the RAID technology based on the hot-spot document in the super-node.Simulation test proved that the premise of maintaining the same availability, compared with the original strategy,the mixed-type replication of backup strategy can be more effective in saving the limited resources in cyberspace.
随着计算机技术和Internet的发展,网络上出现大量闲散的自治结点,用何种方法能够有效而可靠的利用这些闲散结点的剩余资源,成为今年来研究的热点之一。
本文首先讨论和总结了分布式存储的发展历史和现状,并在对P2P技术的研究现状进行了系统、全面的分析和总结的基础上,设计了一种基于混合型P2P技术的网格存储模型。该存储模型将大量分散的结点组织成一个逻辑网络。在系统内部,结点被分为服务器结点,超级结点和一般结点,并结合热点文件和超级结点的特点,对原有的方根复制策略进行了改进,提出了一种混合型的副本备份策略,即在一般结点上存储的数据按照方根复制策略进行备份;在超级结点上,利用RAID技术备份热点文件的副本,经过仿真试验证明,在维持相同可用性的前提下;超级结点上采用RAID技术备份文件副本的策略,能够比单纯的方根复制策略有效的节省有限的网络空间资源。
Grid is a kind of the Technology that using high-speed Internet on a broad geographical-distributed of various resources,including computing resources,storage resources,bandwidth resources, software resources,data resources,information resources,knowledge resources,as a logical Whole thing,just as a supercomputer as to provide users with the integration services of information and application.
With the computer technology and Internet development,there are large number of idle nodes appeared on the net.In which effective and reliable way to be able to use the remaining resources of the idle nodes is a hot research topic in recently years.
First,this paper discusses and summarizes the development and actuality of distributed storage systems.Based on systematically analyzing and summarizing the relevant works on peer-to-peer technology,it designs a Hybrid P2P based distributed storage system. Organizes large numbers of nodes distributed in Internet into a united overlay network.The system divide into three kinds of node,there are server node,super node and general node.Combined with the haracteristics of hot documents and super-node,we modify the original square-root replication strategy,and then present a new mixed-type replication of backup strategy.That is,using the square-root replication in the general node as the backup strategy;and using the RAID technology based on the hot-spot document in the super-node.Simulation test proved that the premise of maintaining the same availability, compared with the original strategy,the mixed-type replication of backup strategy can be more effective in saving the limited resources in cyberspace.
引文
[1].徐恪,叶明江,胡懋智.P2P技术现状及未来发展[J].中兴通讯技术,2007,12(6).6-10
[2].冯雪芬,郑秋霞.P2P技术的应用与发展[J].应用技术,2007,6:56-59
[3].郑亚梅.网格技术的应用及展望[J].辽宁教育行政学院学报,2005(2):129-130
[4].Douceur J R,Bolosky W J.A large Scale Study of file system Contents.SIGMETRICS'99,1999,27(1):59-70
[5].朱永庆.P2P在Internet中的发展策略[J].广东通信技术,2008,5,52-55
[6].逄锦勇,马宁.P2P网络技术的发展应用和未来[J].科技信息,2005,博士·专家论坛,26-28
[7].胡红宇.P2P技术发展综述[J].软件导刊,2008,8,3-4
[8].宋丽娜.关于P2P技术的发展与思考[J].山东电大学报,2007,(1),33-34
[9].田荣华,卢显良,侯孟书,王晓斌.P2P分布式存储系统[J].计算机科学,2007,(34),47-48
[10].Napster http://www.napster.com[EB/OL].1999
[11].Stefan Saroiu,P.krishna Gummadi,Steven D.Gribble.A Measurement Study of Peer-to-Peer File Sharing System.In:proc Multimedia Computing and Networking[J].MMCN'02,2002,156-170
[12].Stefan Saroiu,P.krishna Gummadi,Steven D.Gribble.Measuring and Analyzing the Characteristics of Napster and Gnutella Hosts[J].Multimedia Systems,2003,9(2):170-184
[13].陈贵海,李振华著.对等网络:结构、应用于设计[M].清华大学出版社.2007
[14].Gnutella.www.Gnutella.com[EB/OL].2001
[15].Metei Ripeanu.Peer-to-Peer Architecture Case Study:Gnutella Network[J].In:IEEE 99 P2P'01,2001,99-100
[16].Metei Ripeanu,Adriana Iamnitchi,Ian Foster.Mapping the Gnutella Network Properties of Large scale peer-to-peer systems and implications for system design[J],IEEE Internet Computing,2002,6(2),50-57
[17].Gbutella.Gnutella to the Bandwidth Barrier and Beyond.http://lambda.cs.yale.edu/cs425/doc/gnutella.html[DB/OL].2001
[18].Eytan Adar,Bernardi A.Huberman.Free Riding on Gnutella[J].2000,First Monday,5(10),October,134-139
[19].罗杰文.Peer to Peer(P2P)综述,http://www.instsci.ac.cn/users/loujw/Papers/p2p.htm,2005
[20].OceanStore.http://oceanstore.org[EB/OL].2006
[21].C Gred Plaxton,Rajmohan Rajaraman,Andre W.Richa.Accessing Nearby Copies of Replicated Objects in a Distributed Environment[J],1997,In:SPAA'97,314-326
[22].Sean Rhea,Brighten Godfrey,Brad Karp,John Kubiatowicz,Sylvia Ratnasamy,Scott Shenker,Ion Stociam,Harlan Yu.OpenDHT:A Public DHT Service and Its Uses[J].In:SIGCOMM'05,2005,73-84
[23].Ben Y.Zhao,Ling Huang,Jeremy Stribking,Scan C.Rhea,Anthony D,Joseph,John D.Kubiatowicz.Tapestry:An Infrastructure for Fault-tolerant Wide-area Location and Routing[J].IEEE Journal on Selected Areas in Communications,2004,22(1),41-53
[24].Chord.Http://pdos.csail.mit.edu/chord[DB/OL].2006
[25].Ion Stoica,Robert Morris,Daviad R.Karger,M.Frans Kaashoek,Hari Balakrishnam.Chord:A Scalable Peer-to-Peer Lookup Service for Internet Applications[J].In:SIGCOMM'01,2001,149-160
[26].Ion Stocia,Robert Morrisz,David Liben-Nowell,David R.Kargerz,M.Frans Kaashoekz,Frank Dabekz,Hari Balakrishnanz.Chord:a Scalable Peer-to-Peer lookup Protocal for Internet Applications[J],IEEE/ACM Transactions on Networking,2003,11(1):17-32
[27].司徒俊峰.Internet的小世界网络研究[J].情报技术,2004,(12),86-88
[28].李玲娟,姬同亮,王汝传.小世界现象在P2P网络中的应用研究[J].南京邮电大学学报(自然科学版),2006,vol.26,No.4,86-89
[29].安德森 著,乔江涛 译.长尾理论[M].中信出版社,2006
[30].BitTorrent.www.bittorrent.com[DB/OL].2006
[31].J.A.Pouwelse,P.Garbacki,D.H.J.Epema,H.J.Sips.A Measurement Study of the BitTorrent Peer-to-Peer File-Sharing System[J],In:Parallel and Distributed Systems Group,Delft University of Technology,The NetherLands,2004
[32].Bram Cohen.Incentives Build Robustness in BitTorrent[J].In:WorkShop on Economics of P2P Systems,Berkely,CA,USA,2003,VOL.6
[33].MAZE.http://maze.tianwang.com/maze_guide_6.htm[DB/OL].2009-3-20
[34].刘翰宇,肖明忠,代亚非,李晓明.活跃型用户对P2P文件共享系统可用性的影响[J].软件学报,2006,Vol.17,No.10,2087-2095
[35].QQ.www.qq.com[EB/OL].2009
[36].eDonkey,www.verycd.com[EB/OL].2009
[37].Kurl Tutschku.A Measurement-based Traffic Profile of the eDonkey Filesharing Service[J].In:Proc 5~(th) Passive and Active Measurement Workshop(PAM'04),2004,12-21
[38].田萍芳,鲁宏伟,秦磊华.网络存储技术[J].计算机与数字工程,2004.VOL(32),38-41
[39].Dave Anderson.Network Attached storage is inevitable[R].Proc.Of 30th Hawai International Conf.On System Science,Jan,1977
[40].赵文辉,徐俊,周家林,李晨著.网络存储技术[M].清华大学出版社,2005
[41]付长冬,舒继武,沈美明,郑纬民.网络存储体系结构的发展和研究[J].小型微型计算机,2004,Vol 125 No.4,486-489
[42].亚轩.网络存储新技术iSCSI[J],现代通信,2001(10):12-13
[43].Russel Sandberg.The Sun Network Filesystem:Design,Implementation and Experience[J].Proceedings of the 1987 Summer Usenix Conference,Summer 1987,University of California Press,Pages 300-314
[44].朱文琦.基于网格技术的统计调查前景展望[J].工作视点,2006,(10),67-68
[45].Ian Foster,Carl Kesselman.金海,等译.网格计算[M].北京电子工业出版社,2004
[46].Globus Tookits.http://www.globus.org/[EB/OL],2009
[47].Raman,R.Matcharnking Frameworks for Distributed Resource Management[D].Ph.D.thesis,University of Wisconsin,2000
[48].Bent,J.,Venkataramani,V.,LeRoy,N.,A.,Stanley,J.,Arpaci-Dusseau,A.C.,Arpaci-Dusseau,R.H.,and Livny.M.Flexibility,manageability,and performance in a grid storage appliance[J],in 11~(th) IEEE International Symposium on High Performance Distributed Computing,IEEE Computer Society Press,Los Alamitos,CA,2002
[49].Dongarra,J.,Foster,I.,Fox,G.,Gropp,W.,Kennedy,K.,Torczon,L.,and White,A.Sourcebook of Parallel Computing[M].Morgan Kaufmann,San Francisco,2002
[50].刘轶群.基于Maze的混合型超级结点架构设计和实现[D].硕士学位论文,北京大学,2007
[51].戴晓露.基于P2P的广域存储系统中副本管理相关问题的研究[D].硕士学位论文,浙江大学,2007
[52].高田,刘方爱.一种含超级结点的P2P模型在数据复制中的应用[J].山东科学,2007,(20),57-60
[53].D.A.Patterson,G.Gibson,and R.H.Katz.A case for redundant arrays of inexpensive disks(raid)[J].In Proc.SIGMOD,1988
[2].冯雪芬,郑秋霞.P2P技术的应用与发展[J].应用技术,2007,6:56-59
[3].郑亚梅.网格技术的应用及展望[J].辽宁教育行政学院学报,2005(2):129-130
[4].Douceur J R,Bolosky W J.A large Scale Study of file system Contents.SIGMETRICS'99,1999,27(1):59-70
[5].朱永庆.P2P在Internet中的发展策略[J].广东通信技术,2008,5,52-55
[6].逄锦勇,马宁.P2P网络技术的发展应用和未来[J].科技信息,2005,博士·专家论坛,26-28
[7].胡红宇.P2P技术发展综述[J].软件导刊,2008,8,3-4
[8].宋丽娜.关于P2P技术的发展与思考[J].山东电大学报,2007,(1),33-34
[9].田荣华,卢显良,侯孟书,王晓斌.P2P分布式存储系统[J].计算机科学,2007,(34),47-48
[10].Napster http://www.napster.com[EB/OL].1999
[11].Stefan Saroiu,P.krishna Gummadi,Steven D.Gribble.A Measurement Study of Peer-to-Peer File Sharing System.In:proc Multimedia Computing and Networking[J].MMCN'02,2002,156-170
[12].Stefan Saroiu,P.krishna Gummadi,Steven D.Gribble.Measuring and Analyzing the Characteristics of Napster and Gnutella Hosts[J].Multimedia Systems,2003,9(2):170-184
[13].陈贵海,李振华著.对等网络:结构、应用于设计[M].清华大学出版社.2007
[14].Gnutella.www.Gnutella.com[EB/OL].2001
[15].Metei Ripeanu.Peer-to-Peer Architecture Case Study:Gnutella Network[J].In:IEEE 99 P2P'01,2001,99-100
[16].Metei Ripeanu,Adriana Iamnitchi,Ian Foster.Mapping the Gnutella Network Properties of Large scale peer-to-peer systems and implications for system design[J],IEEE Internet Computing,2002,6(2),50-57
[17].Gbutella.Gnutella to the Bandwidth Barrier and Beyond.http://lambda.cs.yale.edu/cs425/doc/gnutella.html[DB/OL].2001
[18].Eytan Adar,Bernardi A.Huberman.Free Riding on Gnutella[J].2000,First Monday,5(10),October,134-139
[19].罗杰文.Peer to Peer(P2P)综述,http://www.instsci.ac.cn/users/loujw/Papers/p2p.htm,2005
[20].OceanStore.http://oceanstore.org[EB/OL].2006
[21].C Gred Plaxton,Rajmohan Rajaraman,Andre W.Richa.Accessing Nearby Copies of Replicated Objects in a Distributed Environment[J],1997,In:SPAA'97,314-326
[22].Sean Rhea,Brighten Godfrey,Brad Karp,John Kubiatowicz,Sylvia Ratnasamy,Scott Shenker,Ion Stociam,Harlan Yu.OpenDHT:A Public DHT Service and Its Uses[J].In:SIGCOMM'05,2005,73-84
[23].Ben Y.Zhao,Ling Huang,Jeremy Stribking,Scan C.Rhea,Anthony D,Joseph,John D.Kubiatowicz.Tapestry:An Infrastructure for Fault-tolerant Wide-area Location and Routing[J].IEEE Journal on Selected Areas in Communications,2004,22(1),41-53
[24].Chord.Http://pdos.csail.mit.edu/chord[DB/OL].2006
[25].Ion Stoica,Robert Morris,Daviad R.Karger,M.Frans Kaashoek,Hari Balakrishnam.Chord:A Scalable Peer-to-Peer Lookup Service for Internet Applications[J].In:SIGCOMM'01,2001,149-160
[26].Ion Stocia,Robert Morrisz,David Liben-Nowell,David R.Kargerz,M.Frans Kaashoekz,Frank Dabekz,Hari Balakrishnanz.Chord:a Scalable Peer-to-Peer lookup Protocal for Internet Applications[J],IEEE/ACM Transactions on Networking,2003,11(1):17-32
[27].司徒俊峰.Internet的小世界网络研究[J].情报技术,2004,(12),86-88
[28].李玲娟,姬同亮,王汝传.小世界现象在P2P网络中的应用研究[J].南京邮电大学学报(自然科学版),2006,vol.26,No.4,86-89
[29].安德森 著,乔江涛 译.长尾理论[M].中信出版社,2006
[30].BitTorrent.www.bittorrent.com[DB/OL].2006
[31].J.A.Pouwelse,P.Garbacki,D.H.J.Epema,H.J.Sips.A Measurement Study of the BitTorrent Peer-to-Peer File-Sharing System[J],In:Parallel and Distributed Systems Group,Delft University of Technology,The NetherLands,2004
[32].Bram Cohen.Incentives Build Robustness in BitTorrent[J].In:WorkShop on Economics of P2P Systems,Berkely,CA,USA,2003,VOL.6
[33].MAZE.http://maze.tianwang.com/maze_guide_6.htm[DB/OL].2009-3-20
[34].刘翰宇,肖明忠,代亚非,李晓明.活跃型用户对P2P文件共享系统可用性的影响[J].软件学报,2006,Vol.17,No.10,2087-2095
[35].QQ.www.qq.com[EB/OL].2009
[36].eDonkey,www.verycd.com[EB/OL].2009
[37].Kurl Tutschku.A Measurement-based Traffic Profile of the eDonkey Filesharing Service[J].In:Proc 5~(th) Passive and Active Measurement Workshop(PAM'04),2004,12-21
[38].田萍芳,鲁宏伟,秦磊华.网络存储技术[J].计算机与数字工程,2004.VOL(32),38-41
[39].Dave Anderson.Network Attached storage is inevitable[R].Proc.Of 30th Hawai International Conf.On System Science,Jan,1977
[40].赵文辉,徐俊,周家林,李晨著.网络存储技术[M].清华大学出版社,2005
[41]付长冬,舒继武,沈美明,郑纬民.网络存储体系结构的发展和研究[J].小型微型计算机,2004,Vol 125 No.4,486-489
[42].亚轩.网络存储新技术iSCSI[J],现代通信,2001(10):12-13
[43].Russel Sandberg.The Sun Network Filesystem:Design,Implementation and Experience[J].Proceedings of the 1987 Summer Usenix Conference,Summer 1987,University of California Press,Pages 300-314
[44].朱文琦.基于网格技术的统计调查前景展望[J].工作视点,2006,(10),67-68
[45].Ian Foster,Carl Kesselman.金海,等译.网格计算[M].北京电子工业出版社,2004
[46].Globus Tookits.http://www.globus.org/[EB/OL],2009
[47].Raman,R.Matcharnking Frameworks for Distributed Resource Management[D].Ph.D.thesis,University of Wisconsin,2000
[48].Bent,J.,Venkataramani,V.,LeRoy,N.,A.,Stanley,J.,Arpaci-Dusseau,A.C.,Arpaci-Dusseau,R.H.,and Livny.M.Flexibility,manageability,and performance in a grid storage appliance[J],in 11~(th) IEEE International Symposium on High Performance Distributed Computing,IEEE Computer Society Press,Los Alamitos,CA,2002
[49].Dongarra,J.,Foster,I.,Fox,G.,Gropp,W.,Kennedy,K.,Torczon,L.,and White,A.Sourcebook of Parallel Computing[M].Morgan Kaufmann,San Francisco,2002
[50].刘轶群.基于Maze的混合型超级结点架构设计和实现[D].硕士学位论文,北京大学,2007
[51].戴晓露.基于P2P的广域存储系统中副本管理相关问题的研究[D].硕士学位论文,浙江大学,2007
[52].高田,刘方爱.一种含超级结点的P2P模型在数据复制中的应用[J].山东科学,2007,(20),57-60
[53].D.A.Patterson,G.Gibson,and R.H.Katz.A case for redundant arrays of inexpensive disks(raid)[J].In Proc.SIGMOD,1988