基于开源软件一个实用iSCSI Server的实现和性能
|
摘要
面对日益增强的数据存储需求,传统的存储架构凸现出众多问题,网络存储成为一种重要的因应之道。iSCSI是一种新型的网络存储技术,它结合了存储领域和网络领域标准成熟的技术:SCSI和IP/Ethernet,将SCSI协议通过TCP/IP网络进行传输。相比传统的Fibre Channel SAN(Storage Area Network)技术,iSCSI使能了在普遍部署的IP网络上实现低成本的SAN系统,从而成为网络存储迅速普及的关键技术。
iSCSI server是iSCSI应用的核心部件。本文基于开源参考实现,针对其相对实际应用的不足,扩充实现了一个实用的iSCSI server。主要的工作在于:支持多target/LU服务、支持多种常见形式的存储空间、提供灵活的软件结构等。通过多target的支持,该服务器可以为多个主机用户提供独立或共享的存储服务,从而满足集中存储的使用特点。通过支持多种常见形式的存储空间,SCSI/IDE磁盘、磁盘分区、逻辑卷、正规文件等常见形式的存储空间都可以配置为SCSI target设备中的一个逻辑单元,并进而通过iSCSI提供给主机使用,从而满足了基于设备的虚拟存储技术的应用。灵活的软件结构不仅表现在SCSI/iSCSI的分层结构,更表现在一个灵活的I/O接口框架,它可以方便多种I/O接口的扩展和选择使用。
通过性能评测发现该实现:1)选择的write back写策略带来明显的性能优势;2)同主要initiator互操作时表现稳定;3)在多个客户节点访问时表现出更好的扩展性。
并且,我们在蓝鲸SonD动态服务部署系统进行了该iSCSI server的实际应用,经过长期运行,达到了预期效果。
With the increasing data storage needs, traditional storage architecture manifests many problems. Storage networking becomes an important way to handle the challenge. iSCSI is a novel storage networking technology which transports SCSI protocol over TCP/IP network, combining the SCSI and IP/Ethernet technologies, which are standard and mature in storage and network fields respectively. In contrast with traditional Fibre Channel SAN (Storage Area Network) technology, iSCSI enables low-cost SAN to be deployed on ubiquitous IP network, thus becoming a key factor for rapid popularization of networked storage.
iSCSI server is a core component for iSCSI application. There were several open source implementations, but they lack features desired in practice. This thesis presents design and implementation of a practical iSCSI server, which borrows from one of the open source reference implementations and has the following main features:1) supporting many target/LU service; 2) supporting common kinds of storage space; 3) a flexible software architecture. By supporting many targets, this server can provide independent or share storage service for many host users, which satisfies with requirements of centralized storage. By supporting common kinds of storage space, SCSI/IDE disk, disk partition, logical volume, and regular file can be configured as a logical unit of SCSI target device and then be exported to host user by iSCSI, thus allying iSCSI with virtual storage technologies based on device. Flexible software architecture is not only exhibited in SCSI/iSCSI two-layer structure, but also in an I/O interface framework which helps extendsion and selecting of other I/O interfaces.
We do performance test and find out:1) Choosing write back policy obviously brings better write performance; 2) The iSCSI server runs well when interoperating with the main iSCSI initiators; 3) When many client nodes access, the iSCSI server has better scalability.
Moreover, We also introduce its application deployment in Blueware SonD system. The application system has been running for a long term, and accomplished our aims.
iSCSI server是iSCSI应用的核心部件。本文基于开源参考实现,针对其相对实际应用的不足,扩充实现了一个实用的iSCSI server。主要的工作在于:支持多target/LU服务、支持多种常见形式的存储空间、提供灵活的软件结构等。通过多target的支持,该服务器可以为多个主机用户提供独立或共享的存储服务,从而满足集中存储的使用特点。通过支持多种常见形式的存储空间,SCSI/IDE磁盘、磁盘分区、逻辑卷、正规文件等常见形式的存储空间都可以配置为SCSI target设备中的一个逻辑单元,并进而通过iSCSI提供给主机使用,从而满足了基于设备的虚拟存储技术的应用。灵活的软件结构不仅表现在SCSI/iSCSI的分层结构,更表现在一个灵活的I/O接口框架,它可以方便多种I/O接口的扩展和选择使用。
通过性能评测发现该实现:1)选择的write back写策略带来明显的性能优势;2)同主要initiator互操作时表现稳定;3)在多个客户节点访问时表现出更好的扩展性。
并且,我们在蓝鲸SonD动态服务部署系统进行了该iSCSI server的实际应用,经过长期运行,达到了预期效果。
With the increasing data storage needs, traditional storage architecture manifests many problems. Storage networking becomes an important way to handle the challenge. iSCSI is a novel storage networking technology which transports SCSI protocol over TCP/IP network, combining the SCSI and IP/Ethernet technologies, which are standard and mature in storage and network fields respectively. In contrast with traditional Fibre Channel SAN (Storage Area Network) technology, iSCSI enables low-cost SAN to be deployed on ubiquitous IP network, thus becoming a key factor for rapid popularization of networked storage.
iSCSI server is a core component for iSCSI application. There were several open source implementations, but they lack features desired in practice. This thesis presents design and implementation of a practical iSCSI server, which borrows from one of the open source reference implementations and has the following main features:1) supporting many target/LU service; 2) supporting common kinds of storage space; 3) a flexible software architecture. By supporting many targets, this server can provide independent or share storage service for many host users, which satisfies with requirements of centralized storage. By supporting common kinds of storage space, SCSI/IDE disk, disk partition, logical volume, and regular file can be configured as a logical unit of SCSI target device and then be exported to host user by iSCSI, thus allying iSCSI with virtual storage technologies based on device. Flexible software architecture is not only exhibited in SCSI/iSCSI two-layer structure, but also in an I/O interface framework which helps extendsion and selecting of other I/O interfaces.
We do performance test and find out:1) Choosing write back policy obviously brings better write performance; 2) The iSCSI server runs well when interoperating with the main iSCSI initiators; 3) When many client nodes access, the iSCSI server has better scalability.
Moreover, We also introduce its application deployment in Blueware SonD system. The application system has been running for a long term, and accomplished our aims.
引文
[1] Garth A. Gibson, Rodney Van Meter. Network Attached Storage Architecture. COMMUNICATIONS OF THE ACM November 2000, Vol.43, No.11
[2] Greg Schulz, Storage Networking Primer and Update , CMG2002, http://www.cmg.org/measureit/shared/paper326.pdf
[3] Eliezer Levy and Abraham Silberschatz. Distributed File Systems: Concepts and Examples. ACM Computing Surveys, Vol. 22, No. 4, December 1990: 321-374
[4] Sandberg, R., Goldberg, D., Klexman, S., Walsh, D., and Lyone, B.. Design and implementation of the Sun network file system. Proceedings of USENIX 1985 Summer Conference, June 1985: 119-130
[5] SINA NAS work group, CIFS Technical Reference 1.00, http://www.snia.org/tech_activities/CIFS/CIFS-TR-1p00_FINAL.pdf
[6] Gibson, G.A., Nagle, D.F., Amiri, K., Chang, etc, A Case for Network-Attached Secure Disks CMU SCS Technical Report CMU-CS-96-142, September 1996.
[7] Gregory G. Finn, An Integration of Network Communication with Workstation Architecture,ACM Computer Communication Review Oct. 1991
[8] Rodney Van Meter, A Brief Survey of Current Work on Network-Attached Peripherals, ACM Operating Systems Review, January '96.
[9] Gregory G. Finn,Paul Mockapetris, Netstation Architecture: Multi-Gigabit Workstation Network Fabric , InterOp 1994
[10] Rodney Van Meter, Gregory G. Finn , Steve Hotz, VISA: Netstation's Virtual Internet SCSI Adapter , ASPLOS VIII, October 1998.
[11] Nagle, D.F., Ganger, G.R., Butler, J., etc, Network Support for Network-Attached Storage, Hot Interconnects 1999, August 18 - 20, 1999, Stanford University, Stanford, California.
[12] Steve Hotz, Rodney Van Meter, Gregory Finn, Internet Protocols for Network-Attached Peripherals, Sixth NASA Goddard Conference on Mass Storage Systems and Technologies in conjunction with 15th IEEE Symposium on Mass Storage Systems, March 1998.
[13] Julian Satran, Ealan Henis and Pnina Vortman, Trends in Storage Infrastructure , http://www.ssgrr.it/en/ssgrr2002w/papers/156.pdf
[14] J.Satran, K.Meth, C.Sapuntzakis. RFC3720 :iSCSI. http://www.ietf.org/rfc/rfc3720.txt
[15] K. Voruganti , P. Sarkar, An Analysis of Three Gigabit Storage Networking Protocols, pp 259-265, IPCCC 2001.
[16] T10/1157D, SCSI Architecture Model - 2 (SAM-2),http://www.t10.org/ftp/t10/drafts/sam2/sam2r24.pdf
[17] T10/1416-D, SCSI Primary Commands-3,http://www.t10.org/ftp/t10/drafts/spc3/spc3r23.pdf
[18] T10/0996-M,SCSI Block Commands-2, http://www.t10.org/ftp/t10/drafts/sbc2/sbc2r16.pdf
[19] Peter Wang, Robert E.Gilligan, Henry Green, IP SAN-From iSCSI to IP-addressable Ethernet disks, IEEE/NASA MSST2003.
[20] Ashish A. Palekar, Robert D.Russell, Design and impelmentaton of a Linux SCSI target for storage area networks”, 5th Annual Linux Showcase & conference. Atlanta, GA: USENIX,Nov. 2001
[21] UNH-iSCSI, http://sourceforge.net/projects/unh-iscsi/
[22] Cisco Linux initiator, http://sourceforge.net/projects/linux-iscsi/
[23] Ardis iSCSI Target,http://www.ardistech.com/iscsi
[24] iSCSI Enterprise Target, http://sourceforge.net/projects/iscsitarget/
[25] F. Tomonori , O. Masanori, Analysis of iSCSI Target Software, International Workshop on Storage Network Architecture and Parallel I/Os (SNAPI), 2004
[26] 刘振军,许鲁,尹洋,蓝鲸 SonD 动态服务部署系统,计算机学报,2005 年第 5 期
[27] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, A Performance Analysis of the iSCSI Protocol. IEEE/NASA MSST2003
[28] P. Sarkar, K. Voruganti, IP Storage: The Challenge Ahead 19th IEEE Symposium on Mass Storage Systems, pp 35-42, 2002.
[29] Prasenjit Sarkar, Sandeep Uttamchandani, Kaladhar Voruganti, Storage over IP: When Does Hardware Support help? FAST’03.
[30] Peter Radkov, Li Yin, Pawan Goyal, etc, A Performance Comparison of NFS and iSCSI for IP-Networked Storage, Fast04.
[31] Bonnie++, http:// http://www.coker.com.au/bonnie++/
[32] Iozone, http://www.iozone.org/
[33] LMbench, http://sourceforge.net/projects/lmbench
[34] IOMeter,http://www.iometer.org
[2] Greg Schulz, Storage Networking Primer and Update , CMG2002, http://www.cmg.org/measureit/shared/paper326.pdf
[3] Eliezer Levy and Abraham Silberschatz. Distributed File Systems: Concepts and Examples. ACM Computing Surveys, Vol. 22, No. 4, December 1990: 321-374
[4] Sandberg, R., Goldberg, D., Klexman, S., Walsh, D., and Lyone, B.. Design and implementation of the Sun network file system. Proceedings of USENIX 1985 Summer Conference, June 1985: 119-130
[5] SINA NAS work group, CIFS Technical Reference 1.00, http://www.snia.org/tech_activities/CIFS/CIFS-TR-1p00_FINAL.pdf
[6] Gibson, G.A., Nagle, D.F., Amiri, K., Chang, etc, A Case for Network-Attached Secure Disks CMU SCS Technical Report CMU-CS-96-142, September 1996.
[7] Gregory G. Finn, An Integration of Network Communication with Workstation Architecture,ACM Computer Communication Review Oct. 1991
[8] Rodney Van Meter, A Brief Survey of Current Work on Network-Attached Peripherals, ACM Operating Systems Review, January '96.
[9] Gregory G. Finn,Paul Mockapetris, Netstation Architecture: Multi-Gigabit Workstation Network Fabric , InterOp 1994
[10] Rodney Van Meter, Gregory G. Finn , Steve Hotz, VISA: Netstation's Virtual Internet SCSI Adapter , ASPLOS VIII, October 1998.
[11] Nagle, D.F., Ganger, G.R., Butler, J., etc, Network Support for Network-Attached Storage, Hot Interconnects 1999, August 18 - 20, 1999, Stanford University, Stanford, California.
[12] Steve Hotz, Rodney Van Meter, Gregory Finn, Internet Protocols for Network-Attached Peripherals, Sixth NASA Goddard Conference on Mass Storage Systems and Technologies in conjunction with 15th IEEE Symposium on Mass Storage Systems, March 1998.
[13] Julian Satran, Ealan Henis and Pnina Vortman, Trends in Storage Infrastructure , http://www.ssgrr.it/en/ssgrr2002w/papers/156.pdf
[14] J.Satran, K.Meth, C.Sapuntzakis. RFC3720 :iSCSI. http://www.ietf.org/rfc/rfc3720.txt
[15] K. Voruganti , P. Sarkar, An Analysis of Three Gigabit Storage Networking Protocols, pp 259-265, IPCCC 2001.
[16] T10/1157D, SCSI Architecture Model - 2 (SAM-2),http://www.t10.org/ftp/t10/drafts/sam2/sam2r24.pdf
[17] T10/1416-D, SCSI Primary Commands-3,http://www.t10.org/ftp/t10/drafts/spc3/spc3r23.pdf
[18] T10/0996-M,SCSI Block Commands-2, http://www.t10.org/ftp/t10/drafts/sbc2/sbc2r16.pdf
[19] Peter Wang, Robert E.Gilligan, Henry Green, IP SAN-From iSCSI to IP-addressable Ethernet disks, IEEE/NASA MSST2003.
[20] Ashish A. Palekar, Robert D.Russell, Design and impelmentaton of a Linux SCSI target for storage area networks”, 5th Annual Linux Showcase & conference. Atlanta, GA: USENIX,Nov. 2001
[21] UNH-iSCSI, http://sourceforge.net/projects/unh-iscsi/
[22] Cisco Linux initiator, http://sourceforge.net/projects/linux-iscsi/
[23] Ardis iSCSI Target,http://www.ardistech.com/iscsi
[24] iSCSI Enterprise Target, http://sourceforge.net/projects/iscsitarget/
[25] F. Tomonori , O. Masanori, Analysis of iSCSI Target Software, International Workshop on Storage Network Architecture and Parallel I/Os (SNAPI), 2004
[26] 刘振军,许鲁,尹洋,蓝鲸 SonD 动态服务部署系统,计算机学报,2005 年第 5 期
[27] Stephen Aiken, Dirk Grunwald, Andrew R. Pleszkun, A Performance Analysis of the iSCSI Protocol. IEEE/NASA MSST2003
[28] P. Sarkar, K. Voruganti, IP Storage: The Challenge Ahead 19th IEEE Symposium on Mass Storage Systems, pp 35-42, 2002.
[29] Prasenjit Sarkar, Sandeep Uttamchandani, Kaladhar Voruganti, Storage over IP: When Does Hardware Support help? FAST’03.
[30] Peter Radkov, Li Yin, Pawan Goyal, etc, A Performance Comparison of NFS and iSCSI for IP-Networked Storage, Fast04.
[31] Bonnie++, http:// http://www.coker.com.au/bonnie++/
[32] Iozone, http://www.iozone.org/
[33] LMbench, http://sourceforge.net/projects/lmbench
[34] IOMeter,http://www.iometer.org