基于Eucalyptus的对象存储系统的设计与实现
|
摘要
基于对象的存储是一种新的网络存储模式,它融合了SAN的高性能和NAS的跨平台两个优点,发展前景广阔。它将90%左右的数据管理下移动智能存储设备,缓解了元数据服务器和文件系统的压力,在系统的可扩展性、安全性以及智能化存储方面取得了突破。设计基于对象的文件系统,结合对象控制器,构成对象存储系统,应用到Eucalyptus中,提供网络存储服务。
分析了Eucalyptus的软件框架,在此基础上针对组件访问接口、网络服务等级协议和虚拟组网三部分进行理论分析,提出改进意见。详细分析Eucalyptus的存储组件Walrus,提供用对象存储系统替换Walrus的理论依据。设计基于对象的可扩展的文件系统:EXOFS。它和传统的文件系统不同,对文件的管理不再是以数据块的形式,而是以对象的形式,对象包括对象属性和数据,对象属性用来描述对象和数据的信息,存放在数据区的首部,方便对象属性的扩展。采用的大、小对象的设计方式,合理的利用磁盘的空间;对象ID采用增量法对ID号进行递增,加上通用标识符构成一个完整的ID,方便了对象的识别。测试结果表明:对象存储系统较之Walrus的硬盘IO速度提升了20%,对象存储系统较之Walrus的磁盘空间利用率提升了10%。
Object-based storage is a new network storage model. It combines high performance of SAN and cross-platform of NAS. It moves 90% of the data management down intelligent storage devices, easing pressure of metadata server and file system. It is a breakthrough in scalability, security and intelligent storage of the system. Design object-based file system, with the object controller, condtitutes the object storage system and provides network storage services.
Analysis the software framework of Eucalyptus, and suggest improvement for access component interface, network service level agreement and virtual network. Detailed analysis the storage component of Eucalyptus Walrus, provide the theoretical basis for replacement Walrus. Design object-based scalable file system: EXOFS. It is different from the traditional file systems. File management is no longer in the form of data blocks, but in the form of object. Object includes object attribute and data. Object attribute describe the information of object and data, store in the first of data area.Object includes big object and small object, can rational use disk space. Use incremental methods to increase object ID, coupled with a general-purpose identifier constitutes a complete object ID, can identify object ID easily. The results show that: disk IO of object storage system is fastern than Walrus. Disk space utilization of object storage system is improved by 10 percentage points.
分析了Eucalyptus的软件框架,在此基础上针对组件访问接口、网络服务等级协议和虚拟组网三部分进行理论分析,提出改进意见。详细分析Eucalyptus的存储组件Walrus,提供用对象存储系统替换Walrus的理论依据。设计基于对象的可扩展的文件系统:EXOFS。它和传统的文件系统不同,对文件的管理不再是以数据块的形式,而是以对象的形式,对象包括对象属性和数据,对象属性用来描述对象和数据的信息,存放在数据区的首部,方便对象属性的扩展。采用的大、小对象的设计方式,合理的利用磁盘的空间;对象ID采用增量法对ID号进行递增,加上通用标识符构成一个完整的ID,方便了对象的识别。测试结果表明:对象存储系统较之Walrus的硬盘IO速度提升了20%,对象存储系统较之Walrus的磁盘空间利用率提升了10%。
Object-based storage is a new network storage model. It combines high performance of SAN and cross-platform of NAS. It moves 90% of the data management down intelligent storage devices, easing pressure of metadata server and file system. It is a breakthrough in scalability, security and intelligent storage of the system. Design object-based file system, with the object controller, condtitutes the object storage system and provides network storage services.
Analysis the software framework of Eucalyptus, and suggest improvement for access component interface, network service level agreement and virtual network. Detailed analysis the storage component of Eucalyptus Walrus, provide the theoretical basis for replacement Walrus. Design object-based scalable file system: EXOFS. It is different from the traditional file systems. File management is no longer in the form of data blocks, but in the form of object. Object includes object attribute and data. Object attribute describe the information of object and data, store in the first of data area.Object includes big object and small object, can rational use disk space. Use incremental methods to increase object ID, coupled with a general-purpose identifier constitutes a complete object ID, can identify object ID easily. The results show that: disk IO of object storage system is fastern than Walrus. Disk space utilization of object storage system is improved by 10 percentage points.
引文
[1] Seigle. M. Storage area network in video applications. SMPTE Journal, April, 2001. 110(4): 236~239
[2] Chang-Soo Kim, Gyoung-Bae Kim and Bum-Joo Shin. Volume management in SAN environment. CPADS 2001. 500~505
[3] Gibson G, Meter R. Network attached storage architecture. Communications of the ACM, 2000, 43(11): 37 ~ 45
[4] Robert Gray, Bill North, Vernon Turner. Storage Network Management and Virtualization. IDC. August 2002. 1~5
[5] Luo Xinguo, Jiangling Zhang. Study on a Network Storage System. In Proceedings of International Symposium on Multidisciplines, China, 1992. 151~155
[6] Weiss A. Computing in the Clouds. Network. 2007, 11(4): 16 ~ 25
[7] Buyya R, Yeo CS. Market-Oriented Cloud Computing: Vision, Hype, and Reality for Delivering IT Services as Computing Utilities. Proceedings of the 2008 IEEE International Conference on High Performance Computing and Communications. 2008, 5(10): 5 ~ 13
[8] Brian Hayes. Cloud Computing. Communications of the ACM. 2009, 51(7): 9 ~ 11
[9] Carl Hewitt. ORGs for Scalable, Robust, Privacy-Friendly Client Cloud Computing. IEEE Internet Computing. 2008, 12(5): 96 ~ 99
[10]王昊鹏,刘旺盛.虚拟化技术在云计算中的应用初探.电脑知识与技术. 2008, 3(5): 1554 ~ 1565
[11] Frank E. Future View: The New Tech Ecosystems of Cloud, Cloud Services, and Cloud Computing. Forrester Research. 2008
[12] Rich Uhlig, Gil Neiger, et al. Intel Virtualization Technology. IEEE Computer Society. 2005. 48 ~ 56
[13] Azagury A, Dreizin V, Factor M, et al. Towards an object store. In: Titsworth F, ed. Proceedings 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies. New York: IEEE, 2003. 165 ~ 176
[14]董耀祖,周正伟.基于x86架构的系统虚拟机技术与应用.上海交通大学. 2006, 32(13): 51 ~ 73
[15] Intel Coporation. Intel vanderpool techonology specification. Intel TechnologySpecification, 2005
[16] Ruth P, Jiang X, Xu D. Virtual Distributed Environments in a Share Infrastructure. IEEE Computer Society. 2005. 63 ~ 69
[17] Goldberg R. Survey of Virtual Machine Research. IEEE Computer. 1974, 7(6): 34 ~ 35
[18] Nagle D, Ganger G, Butler J, et al. Network Support for Network-Attached Storage. In: Anon, ed. Proceedings of 1999 Hot Interconnects Symposium. New York: IEEE, 1999. 245 ~ 253
[19] Private Clouds on the Horizon. InformationWeek Analytics. April 13, 2009
[20] Randy George. Cloud Storage’s Top Uses. Information Week. 2008
[21] Phillips B. Have storage area networks come of age. IEEE Computer, 1998, 31(7): 10 ~ 12
[22] Mary Weier. Amazon Launches Private Cloud Services. Information Week. August 26, 2009
[23] Nicholas Hoover J and Richard Martin. Demystifying the Cloud. Information Week Research&Reports, 2008. 30 ~ 37
[24] Fielding R T and Taylor R N. Principled Design of the Modern Web Architecture. ACM Transactions on Internet Technology (TOIT). 2002, 2(2): 115 ~ 150
[25] Schatz M.C. Cloud Burst: highly sensitive read mapping with MapReduce. Bioinformatics, 2009. 1363 ~ 1369
[26] Mesnier M, Ganger G R, Riedel E. Object-based storage: pushing more functionality into storage. Potentials, 2005, 24(2): 31 ~ 34
[27] Goldchleger A, Kon F, Goldman A, et al. Checkpointing BSP parallel applications on the InteGrade Grid middleware. Concurrency and Comp.: Practice and Experience. 2006, 18(6): 567 ~ 579
[28] Elnozahy M, Alvisi L, Wang Y M, et al. Survey of Rollback-recovery Protocols in Message-passing Systems. ACM Computing Surveys. 2002, 34(3): 375 ~ 408
[29] Frey J, Tannenbaum T, Livny M, et al. Condor-g: A computation management agent for multi-institutional grids. Cluster Computing. 2002, 5(3): 8 ~ 12
[30] Rabin M O. Efficient dispersal of information for security, load balancing, and fault tolerance. Journal of the ACM. 1989, 36(2): 335 ~ 348
[31] Stoica I, Morris R, Karger D, et al. Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Transactions on Networking. 2003, 11(1):17 ~ 32
[32] Nethercote N and Seward J. Valgrind: A program supervision framework. Electronic Notes in Theoretical Computer Science. 2003, 89(2): 15 ~ 18
[33] Gibson G A, Nagle D F, Amiri K, et al. A cost-effective, High-bandwidth storage architecture. ACM Sigplan, 1998, 33(11): 92 ~ 103
[34] Steinke S, Wehmeyer L, Lee B, etal. Assigning program and data objects to scratchpad for energy reduction. In Proceedings of the 2002 Design, Automation and Test in Europe Conference and Exhibition, 2002. 409 ~ 417
[35] Alvarez G A. Minerva: An Automated Resource Provisioning Tool for Large-Scale Storage Systems. ACM Transactions on Computer Systems. 2001, 19(4): 483 ~ 518
[36] Zhu Q and Zhou Y. Power-Aware Storage Cache Management. IEEE Transactions on Computers. 2005, 54(5): 18 ~ 20
[37] Attiya H, Bar-Noy A, and Dolev D. Sharing memory robustly in message-passing systems. Journal of the ACM. 1995, 42(1): 124 ~ 142
[38] Herlihy M. Wait-free synchronization. ACM Transactions on Programming Languages and Systems. 1991, 13(1): 124 ~ 149
[39] Hospodor A D, Miller E L. Interconnection Architectures for Petabyte-Scale High- Performance Storage Systems. Communications Magazine, 2003, 41(8): 72 ~ 75
[40] Rivest R L, Shamir A, and Adleman L M. A method for obtaining digital signatures and public-key cryptosystems. Communications of the ACM, 1978, 21(2): 120 ~ 126
[41] Mesnier M, Ganger G R, and Riedel E. Object-based storage. IEEE Communications Magazine. 2003, 41(5): 84 ~ 90
[42] Weil S A. Leveraging intra-object locality with EBOFS. UCSC CMPS-290S Project Report, May 2004
[43] Mu Fei, Shu Jiwu, Li Bigang, et al. Multi-dimensional Storage QoS Guarantees for an Object-Based Storage System.International Conference on Computational Science. 2006, 3(8): 687 ~ 694
[44] Prashant J, Shenoy, Pawan Goyal, et al. Issues in Multimedia Server Design. ACM Computing Surveys (CSUR). 1995, 27(4): 636 ~ 639
[45] Jones K A and Wulf W A. Towards the design of secure systems. Software Practice and Experience. 1975, 5(4): 321 ~ 336
[2] Chang-Soo Kim, Gyoung-Bae Kim and Bum-Joo Shin. Volume management in SAN environment. CPADS 2001. 500~505
[3] Gibson G, Meter R. Network attached storage architecture. Communications of the ACM, 2000, 43(11): 37 ~ 45
[4] Robert Gray, Bill North, Vernon Turner. Storage Network Management and Virtualization. IDC. August 2002. 1~5
[5] Luo Xinguo, Jiangling Zhang. Study on a Network Storage System. In Proceedings of International Symposium on Multidisciplines, China, 1992. 151~155
[6] Weiss A. Computing in the Clouds. Network. 2007, 11(4): 16 ~ 25
[7] Buyya R, Yeo CS. Market-Oriented Cloud Computing: Vision, Hype, and Reality for Delivering IT Services as Computing Utilities. Proceedings of the 2008 IEEE International Conference on High Performance Computing and Communications. 2008, 5(10): 5 ~ 13
[8] Brian Hayes. Cloud Computing. Communications of the ACM. 2009, 51(7): 9 ~ 11
[9] Carl Hewitt. ORGs for Scalable, Robust, Privacy-Friendly Client Cloud Computing. IEEE Internet Computing. 2008, 12(5): 96 ~ 99
[10]王昊鹏,刘旺盛.虚拟化技术在云计算中的应用初探.电脑知识与技术. 2008, 3(5): 1554 ~ 1565
[11] Frank E. Future View: The New Tech Ecosystems of Cloud, Cloud Services, and Cloud Computing. Forrester Research. 2008
[12] Rich Uhlig, Gil Neiger, et al. Intel Virtualization Technology. IEEE Computer Society. 2005. 48 ~ 56
[13] Azagury A, Dreizin V, Factor M, et al. Towards an object store. In: Titsworth F, ed. Proceedings 20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies. New York: IEEE, 2003. 165 ~ 176
[14]董耀祖,周正伟.基于x86架构的系统虚拟机技术与应用.上海交通大学. 2006, 32(13): 51 ~ 73
[15] Intel Coporation. Intel vanderpool techonology specification. Intel TechnologySpecification, 2005
[16] Ruth P, Jiang X, Xu D. Virtual Distributed Environments in a Share Infrastructure. IEEE Computer Society. 2005. 63 ~ 69
[17] Goldberg R. Survey of Virtual Machine Research. IEEE Computer. 1974, 7(6): 34 ~ 35
[18] Nagle D, Ganger G, Butler J, et al. Network Support for Network-Attached Storage. In: Anon, ed. Proceedings of 1999 Hot Interconnects Symposium. New York: IEEE, 1999. 245 ~ 253
[19] Private Clouds on the Horizon. InformationWeek Analytics. April 13, 2009
[20] Randy George. Cloud Storage’s Top Uses. Information Week. 2008
[21] Phillips B. Have storage area networks come of age. IEEE Computer, 1998, 31(7): 10 ~ 12
[22] Mary Weier. Amazon Launches Private Cloud Services. Information Week. August 26, 2009
[23] Nicholas Hoover J and Richard Martin. Demystifying the Cloud. Information Week Research&Reports, 2008. 30 ~ 37
[24] Fielding R T and Taylor R N. Principled Design of the Modern Web Architecture. ACM Transactions on Internet Technology (TOIT). 2002, 2(2): 115 ~ 150
[25] Schatz M.C. Cloud Burst: highly sensitive read mapping with MapReduce. Bioinformatics, 2009. 1363 ~ 1369
[26] Mesnier M, Ganger G R, Riedel E. Object-based storage: pushing more functionality into storage. Potentials, 2005, 24(2): 31 ~ 34
[27] Goldchleger A, Kon F, Goldman A, et al. Checkpointing BSP parallel applications on the InteGrade Grid middleware. Concurrency and Comp.: Practice and Experience. 2006, 18(6): 567 ~ 579
[28] Elnozahy M, Alvisi L, Wang Y M, et al. Survey of Rollback-recovery Protocols in Message-passing Systems. ACM Computing Surveys. 2002, 34(3): 375 ~ 408
[29] Frey J, Tannenbaum T, Livny M, et al. Condor-g: A computation management agent for multi-institutional grids. Cluster Computing. 2002, 5(3): 8 ~ 12
[30] Rabin M O. Efficient dispersal of information for security, load balancing, and fault tolerance. Journal of the ACM. 1989, 36(2): 335 ~ 348
[31] Stoica I, Morris R, Karger D, et al. Chord: A scalable peer-to-peer lookup protocol for internet applications. IEEE/ACM Transactions on Networking. 2003, 11(1):17 ~ 32
[32] Nethercote N and Seward J. Valgrind: A program supervision framework. Electronic Notes in Theoretical Computer Science. 2003, 89(2): 15 ~ 18
[33] Gibson G A, Nagle D F, Amiri K, et al. A cost-effective, High-bandwidth storage architecture. ACM Sigplan, 1998, 33(11): 92 ~ 103
[34] Steinke S, Wehmeyer L, Lee B, etal. Assigning program and data objects to scratchpad for energy reduction. In Proceedings of the 2002 Design, Automation and Test in Europe Conference and Exhibition, 2002. 409 ~ 417
[35] Alvarez G A. Minerva: An Automated Resource Provisioning Tool for Large-Scale Storage Systems. ACM Transactions on Computer Systems. 2001, 19(4): 483 ~ 518
[36] Zhu Q and Zhou Y. Power-Aware Storage Cache Management. IEEE Transactions on Computers. 2005, 54(5): 18 ~ 20
[37] Attiya H, Bar-Noy A, and Dolev D. Sharing memory robustly in message-passing systems. Journal of the ACM. 1995, 42(1): 124 ~ 142
[38] Herlihy M. Wait-free synchronization. ACM Transactions on Programming Languages and Systems. 1991, 13(1): 124 ~ 149
[39] Hospodor A D, Miller E L. Interconnection Architectures for Petabyte-Scale High- Performance Storage Systems. Communications Magazine, 2003, 41(8): 72 ~ 75
[40] Rivest R L, Shamir A, and Adleman L M. A method for obtaining digital signatures and public-key cryptosystems. Communications of the ACM, 1978, 21(2): 120 ~ 126
[41] Mesnier M, Ganger G R, and Riedel E. Object-based storage. IEEE Communications Magazine. 2003, 41(5): 84 ~ 90
[42] Weil S A. Leveraging intra-object locality with EBOFS. UCSC CMPS-290S Project Report, May 2004
[43] Mu Fei, Shu Jiwu, Li Bigang, et al. Multi-dimensional Storage QoS Guarantees for an Object-Based Storage System.International Conference on Computational Science. 2006, 3(8): 687 ~ 694
[44] Prashant J, Shenoy, Pawan Goyal, et al. Issues in Multimedia Server Design. ACM Computing Surveys (CSUR). 1995, 27(4): 636 ~ 639
[45] Jones K A and Wulf W A. Towards the design of secure systems. Software Practice and Experience. 1975, 5(4): 321 ~ 336
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |