基于STAF的分布式自动化可用性评测平台的设计与实现
|
摘要
随着云计算、数据中心、虚拟化逐渐深入并影响着信息化进程,高端容错计算机扮演着越来越重要的角色,高端容错计算机系统对可用性有很高的要求,因此对评测高端容错计算机可用性虽然任务艰巨但是具有重要意义。故障注入方法又是评测系统有效性的重要的实验方法,所以需要对故障注入技术进行深入的了解和研究。
本文对国内外一些比较经典的故障注入技术进行了研究对比分析,发现虽然已经这些工具都已经很成熟并得到广泛地应用,但是这些工具测试手段较为单一,不能对目标系统进行全面的测试,而且传统工具的目标系统普遍是IA-32位体系结构的。针对上述问题,本文提出基于组件技术集成各类故障注入工具,实现了对目标系统不同抽象层次上的故障注入的多样化,同时支持故障注入工具“即插即用”的功能;基于平台集成的故障注入工具设计开发相关性测试套件、冗余度测试套件、MTTR测试套件、可用度测试套件,以得到相关指标来有效地评测系统的可用性。
手动生成测试用例延长了测试时间,同时平台的测试环境需要覆盖各种各样的平台,增加了环境搭建的复杂性。考虑上述手动测试存在的问题,本文基于IBM提供的开源的自动化测试框架STAF,探索建立分布式自动化平台,通过动态部署测试环境、测试执行自动化,减少人机交互的次数,提高自动化测试程度,降低人力物力的消耗。最后,本文给出MTTR测试套件和基于MCA故障注入工具的应用实例,并通过对比故障注入手动测试和自动化测试,验证了开发平台的有效性。
As cloud computing, data center and virtualization arebecoming more advanced and affecting informationization, high-end fault-tolerant computers are playing a more and more important role. High-end fault-tolerant computer systems require a high degree of usability. Therefore although evaluating usability isdifficult, it is extremely useful. The faultinjection method is an important methodfor evaluating system effectiveness, so it needs to be studied and understood more deeply.
This paper researches and analyzes domestic and foreign classic fault injection technology. It finds that although these tools are already very advanced and widely used, there are relatively few means of testing. These meanscan’t thoroughly test the target systems, which are commonly IA - 32-bit architectures. In view of the above problems, this paper develops a platform which integrates all kinds of faults injection tools on the target system, which diversifies the means of testing on different levels. At the same time it supports the "plug and play" capability of fault injected tools.Based on the fault injected tools integrated into the platform, designing and developing the correlation test suite, the redundancy test suite, the MTTR test suite, the availability test suite gets the related indicators to effectively evaluate the system availability.
Generating test cases manually extends the time of test. The testing environment of the fault injection platform needs to cover a wide range of platforms, which also increases the complexity of building environment. Considering the manual testing problems, this paper establishes a distributed and automation platform based on Software Testing Automation Framework (STAF). Through the testing environment dynamic deployment, test task automation control and the reduction of the number of the human-computer interaction, improve the degree of testing automation and reduce the consumption of manpower and material resources. Finally, this paper gives the application of the MTTR test suite and the fault injected tool based on MCA.Tthe effectiveness of developed platform has been verified by using the implemented fault-injection platform and test automation system to do the manual and automation tests.
本文对国内外一些比较经典的故障注入技术进行了研究对比分析,发现虽然已经这些工具都已经很成熟并得到广泛地应用,但是这些工具测试手段较为单一,不能对目标系统进行全面的测试,而且传统工具的目标系统普遍是IA-32位体系结构的。针对上述问题,本文提出基于组件技术集成各类故障注入工具,实现了对目标系统不同抽象层次上的故障注入的多样化,同时支持故障注入工具“即插即用”的功能;基于平台集成的故障注入工具设计开发相关性测试套件、冗余度测试套件、MTTR测试套件、可用度测试套件,以得到相关指标来有效地评测系统的可用性。
手动生成测试用例延长了测试时间,同时平台的测试环境需要覆盖各种各样的平台,增加了环境搭建的复杂性。考虑上述手动测试存在的问题,本文基于IBM提供的开源的自动化测试框架STAF,探索建立分布式自动化平台,通过动态部署测试环境、测试执行自动化,减少人机交互的次数,提高自动化测试程度,降低人力物力的消耗。最后,本文给出MTTR测试套件和基于MCA故障注入工具的应用实例,并通过对比故障注入手动测试和自动化测试,验证了开发平台的有效性。
As cloud computing, data center and virtualization arebecoming more advanced and affecting informationization, high-end fault-tolerant computers are playing a more and more important role. High-end fault-tolerant computer systems require a high degree of usability. Therefore although evaluating usability isdifficult, it is extremely useful. The faultinjection method is an important methodfor evaluating system effectiveness, so it needs to be studied and understood more deeply.
This paper researches and analyzes domestic and foreign classic fault injection technology. It finds that although these tools are already very advanced and widely used, there are relatively few means of testing. These meanscan’t thoroughly test the target systems, which are commonly IA - 32-bit architectures. In view of the above problems, this paper develops a platform which integrates all kinds of faults injection tools on the target system, which diversifies the means of testing on different levels. At the same time it supports the "plug and play" capability of fault injected tools.Based on the fault injected tools integrated into the platform, designing and developing the correlation test suite, the redundancy test suite, the MTTR test suite, the availability test suite gets the related indicators to effectively evaluate the system availability.
Generating test cases manually extends the time of test. The testing environment of the fault injection platform needs to cover a wide range of platforms, which also increases the complexity of building environment. Considering the manual testing problems, this paper establishes a distributed and automation platform based on Software Testing Automation Framework (STAF). Through the testing environment dynamic deployment, test task automation control and the reduction of the number of the human-computer interaction, improve the degree of testing automation and reduce the consumption of manpower and material resources. Finally, this paper gives the application of the MTTR test suite and the fault injected tool based on MCA.Tthe effectiveness of developed platform has been verified by using the implemented fault-injection platform and test automation system to do the manual and automation tests.
引文
1 Sun H, Han JJ,Levendel H.Availability Requirement for AFault-Management Server In High-Availability Communication Systems[J]. IEEE Transactions on Reliability.2003, 52(2): 238–244
2 Arlat J,Fabre JC, Rodríguez M, et al. Dependability of COTS Microkernel-based Systems[J]. IEEE Trans. on Computers.2002, 51( 2): 138-163
3 Brown A, Patterson DA. Towards AvailabilityBenchmarks: A Cases Study of Software RAID Systems[C]. In: Proceedings of the 2000 USENIX Annual Technical Conference, San Diego,CA, 2000:263–276
4 Arlat J,Costes A, Crouzet Y, et al. Fault Injection and Dependability Evaluation of Fault-Tolerant Systems[J]. IEEE Trans. on Computers.1993, 42(8): 913-923
5孙峻朝,李运策,杨孝宗.故障注入研究的一种理论框架[J].小型微型计算机, 1999,20(11):816-819.
6 Tai AT, Hecht H, Trivedi KS, et al. Toward Accessibility Enhancement of Dependability Modeling Techniques and Tools[C]. In: Proceedings 0f 27th IEEE InternationalSymposium on Fault Tolerant Computing (FTCS-27),Seattle, WA, 1997: 37-41
7翟季冬.系统可用性基准测试技术综述[J].计算机系统性能测试,2007,16(11):158-163.
8 K. Kanoun, L. Spainhower. Dependability Benchmarking for Computer Systems. John Wiley & Sons, Inc., 2008: 63-79
9王建莹,孙峻朝,李运策,杨孝宗. FIT-1:一个基于硬件的故障注入器的设计与实现.计算机工程与设计. 1998,19(4):14-19
10王建莹,孙峻朝,杨孝宗.容错计算机系统可靠性评估工具:HFI-2故障注入器.电子学报. 1999,27(11):24-26
11秦磊.面向安腾2处理器的故障注入工具设计与实现[D].哈尔滨:哈尔滨工业大学.2010.
12 Vinter J, Aidemark J. An Overview of GOOFI-A Generic Object-Oriented Fault Injection Framework[C]. Technical Report No. 05-07. Department of Computer Science and Engineering CHALMERS UNIVERSITY OF TECHNOLOGY, 2005:5-8
13 Dawson S, Jahanian F, Tung TL. Testing of Fault-Tolerant and Real-Time Distributed Systems via Protocol Fault Injection[C]. In: Proceedings of the 26th IEEE Int. Symp. on Fault Tolerant Computing (FTCS-26), Sendai, Japan, 1996:404-414
14 Karlsson J, Arlat J, Leber G. Application of Three Physical Fault Injection Techniques to the Experimental Assessment of the MARS Architecture[C]. In: Proceedings of the Fifth Ann. IEEE Working Conf. Dependable Computing forCritical Applications. IEEE CS Press, Los, California, 1995:150-161
15 H.Madeira, M.Z.Rela, F.Moreira, et al. RIFLE: A General Purpose Pin-level Fault Injector. Proc. EDCC-1, Berlin Germany, 1994:199-216
16 H.Madeira, J.G.Silva. Experimental Evaluation of the Fail-Silent Behavior in Computers Without Error Masking. In: Proceedings of the 24th IEEE Int. Symp. on Fault Tolerant Computing(FTCS-24), Austin, Texas, 1994:350-359
17 S. Han. Experiment Evaluation of Failure-Detection Schemes in Real-Time Communication Networks. In: Proceedings of the 27th fault tolerant computing, WA, Seattle, 1997: 122-131
18 J. H. Barton. Fault Injection Experiments Using FIAT. IEEE Transactions on Computers. 1990, 39(4): 575-582
19 T.K. Tsai, R.K.Iyer. Measuring Fault Tolerance with theFTAPE Fault Injection Tool. In Proc. of Performance Tools, Heidelberg, Germany, 1995: 26-40
20 G. A. Kanawati. FERRARI: A Flexible Software-Based Fault and Error Injection System. IEEE Transactions on computers. 1995, 44(2): 248-260
21 B. Miller, D. Koski, C. Lee, et al. Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services. ComputerScience Technical Report 1268. University of WisconsinMadison, 1998: 2-4
22 G. A. Kanawati. FERRARI: A Flexible software—Based fault and error injection system.IEEE Transctions on computers. 1995, 44(2): 248-260
23 J. Carreira, H. Madeira, J. Silva. Xception: A Technique for the Experimental Evaluation of Dependabilityin Modern Computers. IEEE Transactions on Software Engineering. 1998, 24(2): 125-136
24谭兰芳.面向单粒子效应的软件故障注入技术研究.国防科学技术大学工学硕士学位论文. 2008: 12-13
25刘伟娜.面向安腾架构的高端容错机故障注入平台的设计与实现[D].哈尔滨:哈尔滨工业大学.2010.
26边维.基于软件实现的故障注入的系统健壮性测试研究[D].哈尔滨:哈尔滨工业大学.2009.
27 Dumitrescu. C. Raicu. I, Ripeanu. M. et al. An Automated Distributed Performance Testing Framework. Grid Computing[J], 2004. Proceeding Fitth IEEE/ACM International Workshop on 8 Nov. 2004: 289-296
28 George Coulouris, Jean Dollimore, Tim Kindberg. Distributed Systems Concepts and Design, Fourth Edition.Pearson Education. 2007.
29薹董嫒.分布式测试管理系统的设计与实现[D].北京:北京邮电大学. 2010
30 Lydia Ash著.李昂,黄江海等译. Web测试指南[M].机械工业出版社, 2004
31蔡志贤.软件自动化测试研究与实践.南京理工大学硕士学位论文. 2005: 4-13
32于秀山,胡兢玉.软件自动化测试:引入、管理与实施.北京:电子工业出版社, 003: 59-78
33朱菊,王志坚,杨雪.基于数据驱动的软件自动化测试框架[J].计算机技术与发展, 2006, 16(5): 68-70.
34朱芳,李曦,赵振西.一种多平台自动化测试工具的设计和实现[J].计算机工程, 2004, 30(24): 186-188.
35 PEI Songwen,WU Baifeng,ZHU Kun,et al.Novel Software Automated Testing System Based onJ2EE[J].Tsinghua Science and Technology,2007,12(s1):51-56.
36严少清,陈革,万年红.软件测试自动化管理系统的设计与实现[J].计算机工程, 2002, 28(9): 152-157.
37李夏安,陈志泊.基于STAF的软件自动化测试系统的研究和实现[J].计算机应用, 2009,29(3): 699—704
38张磊,王晓军.基于STAF框架下的自动化测试[J].计算机技术与发展, 2010, 20(3):116-120
39李先旺,宋豫川,刘飞.基于消息控制的服务流程和功能模型改进研究.制造业自动化. 2008, 30(11): 60-63
40 Bundell G A., Glee, J. MorrisS. Hope, Parr and ILGDromey.Component software: A WhitePaper: PartlI. Techniacl Aspects. SoftwareEngineering Australia
41 Brown,A.W; Wallnau, K.C. The current state of CBSE.IEEE Software. 1998. V01. 15(5). 37-46.
42蔡福喜,黄大贵.基于模块化设计思想的测控系统框架设计.测控技术. 2009, 28(1): 55-61
43李先旺,宋豫川,刘飞.基于消息控制的服务流程和功能模型改进研究.制造业自动化. 2008, 30(11): 60-63
2 Arlat J,Fabre JC, Rodríguez M, et al. Dependability of COTS Microkernel-based Systems[J]. IEEE Trans. on Computers.2002, 51( 2): 138-163
3 Brown A, Patterson DA. Towards AvailabilityBenchmarks: A Cases Study of Software RAID Systems[C]. In: Proceedings of the 2000 USENIX Annual Technical Conference, San Diego,CA, 2000:263–276
4 Arlat J,Costes A, Crouzet Y, et al. Fault Injection and Dependability Evaluation of Fault-Tolerant Systems[J]. IEEE Trans. on Computers.1993, 42(8): 913-923
5孙峻朝,李运策,杨孝宗.故障注入研究的一种理论框架[J].小型微型计算机, 1999,20(11):816-819.
6 Tai AT, Hecht H, Trivedi KS, et al. Toward Accessibility Enhancement of Dependability Modeling Techniques and Tools[C]. In: Proceedings 0f 27th IEEE InternationalSymposium on Fault Tolerant Computing (FTCS-27),Seattle, WA, 1997: 37-41
7翟季冬.系统可用性基准测试技术综述[J].计算机系统性能测试,2007,16(11):158-163.
8 K. Kanoun, L. Spainhower. Dependability Benchmarking for Computer Systems. John Wiley & Sons, Inc., 2008: 63-79
9王建莹,孙峻朝,李运策,杨孝宗. FIT-1:一个基于硬件的故障注入器的设计与实现.计算机工程与设计. 1998,19(4):14-19
10王建莹,孙峻朝,杨孝宗.容错计算机系统可靠性评估工具:HFI-2故障注入器.电子学报. 1999,27(11):24-26
11秦磊.面向安腾2处理器的故障注入工具设计与实现[D].哈尔滨:哈尔滨工业大学.2010.
12 Vinter J, Aidemark J. An Overview of GOOFI-A Generic Object-Oriented Fault Injection Framework[C]. Technical Report No. 05-07. Department of Computer Science and Engineering CHALMERS UNIVERSITY OF TECHNOLOGY, 2005:5-8
13 Dawson S, Jahanian F, Tung TL. Testing of Fault-Tolerant and Real-Time Distributed Systems via Protocol Fault Injection[C]. In: Proceedings of the 26th IEEE Int. Symp. on Fault Tolerant Computing (FTCS-26), Sendai, Japan, 1996:404-414
14 Karlsson J, Arlat J, Leber G. Application of Three Physical Fault Injection Techniques to the Experimental Assessment of the MARS Architecture[C]. In: Proceedings of the Fifth Ann. IEEE Working Conf. Dependable Computing forCritical Applications. IEEE CS Press, Los, California, 1995:150-161
15 H.Madeira, M.Z.Rela, F.Moreira, et al. RIFLE: A General Purpose Pin-level Fault Injector. Proc. EDCC-1, Berlin Germany, 1994:199-216
16 H.Madeira, J.G.Silva. Experimental Evaluation of the Fail-Silent Behavior in Computers Without Error Masking. In: Proceedings of the 24th IEEE Int. Symp. on Fault Tolerant Computing(FTCS-24), Austin, Texas, 1994:350-359
17 S. Han. Experiment Evaluation of Failure-Detection Schemes in Real-Time Communication Networks. In: Proceedings of the 27th fault tolerant computing, WA, Seattle, 1997: 122-131
18 J. H. Barton. Fault Injection Experiments Using FIAT. IEEE Transactions on Computers. 1990, 39(4): 575-582
19 T.K. Tsai, R.K.Iyer. Measuring Fault Tolerance with theFTAPE Fault Injection Tool. In Proc. of Performance Tools, Heidelberg, Germany, 1995: 26-40
20 G. A. Kanawati. FERRARI: A Flexible Software-Based Fault and Error Injection System. IEEE Transactions on computers. 1995, 44(2): 248-260
21 B. Miller, D. Koski, C. Lee, et al. Fuzz Revisited: A Re-examination of the Reliability of UNIX Utilities and Services. ComputerScience Technical Report 1268. University of WisconsinMadison, 1998: 2-4
22 G. A. Kanawati. FERRARI: A Flexible software—Based fault and error injection system.IEEE Transctions on computers. 1995, 44(2): 248-260
23 J. Carreira, H. Madeira, J. Silva. Xception: A Technique for the Experimental Evaluation of Dependabilityin Modern Computers. IEEE Transactions on Software Engineering. 1998, 24(2): 125-136
24谭兰芳.面向单粒子效应的软件故障注入技术研究.国防科学技术大学工学硕士学位论文. 2008: 12-13
25刘伟娜.面向安腾架构的高端容错机故障注入平台的设计与实现[D].哈尔滨:哈尔滨工业大学.2010.
26边维.基于软件实现的故障注入的系统健壮性测试研究[D].哈尔滨:哈尔滨工业大学.2009.
27 Dumitrescu. C. Raicu. I, Ripeanu. M. et al. An Automated Distributed Performance Testing Framework. Grid Computing[J], 2004. Proceeding Fitth IEEE/ACM International Workshop on 8 Nov. 2004: 289-296
28 George Coulouris, Jean Dollimore, Tim Kindberg. Distributed Systems Concepts and Design, Fourth Edition.Pearson Education. 2007.
29薹董嫒.分布式测试管理系统的设计与实现[D].北京:北京邮电大学. 2010
30 Lydia Ash著.李昂,黄江海等译. Web测试指南[M].机械工业出版社, 2004
31蔡志贤.软件自动化测试研究与实践.南京理工大学硕士学位论文. 2005: 4-13
32于秀山,胡兢玉.软件自动化测试:引入、管理与实施.北京:电子工业出版社, 003: 59-78
33朱菊,王志坚,杨雪.基于数据驱动的软件自动化测试框架[J].计算机技术与发展, 2006, 16(5): 68-70.
34朱芳,李曦,赵振西.一种多平台自动化测试工具的设计和实现[J].计算机工程, 2004, 30(24): 186-188.
35 PEI Songwen,WU Baifeng,ZHU Kun,et al.Novel Software Automated Testing System Based onJ2EE[J].Tsinghua Science and Technology,2007,12(s1):51-56.
36严少清,陈革,万年红.软件测试自动化管理系统的设计与实现[J].计算机工程, 2002, 28(9): 152-157.
37李夏安,陈志泊.基于STAF的软件自动化测试系统的研究和实现[J].计算机应用, 2009,29(3): 699—704
38张磊,王晓军.基于STAF框架下的自动化测试[J].计算机技术与发展, 2010, 20(3):116-120
39李先旺,宋豫川,刘飞.基于消息控制的服务流程和功能模型改进研究.制造业自动化. 2008, 30(11): 60-63
40 Bundell G A., Glee, J. MorrisS. Hope, Parr and ILGDromey.Component software: A WhitePaper: PartlI. Techniacl Aspects. SoftwareEngineering Australia
41 Brown,A.W; Wallnau, K.C. The current state of CBSE.IEEE Software. 1998. V01. 15(5). 37-46.
42蔡福喜,黄大贵.基于模块化设计思想的测控系统框架设计.测控技术. 2009, 28(1): 55-61
43李先旺,宋豫川,刘飞.基于消息控制的服务流程和功能模型改进研究.制造业自动化. 2008, 30(11): 60-63