恶意软件的反分析行为检测方法研究
|
摘要
恶意软件是威胁信息安全的重要因素。为了抽取和分析恶意软件运行时的行为,安全产品厂商和分析人员开发出自动化的分析工具。然而,恶意程序会检测出分析工具的存在,逃避分析,即检测出虚拟或模拟的分析环境后,恶意程序会表现出和在真实环境下不一样的行为,比如减少攻击行为,或者立即结束运行等。
为了使恶意程序的反分析能力失效,现有的方法分为两类：第一类方法是构建透明的分析工具,但是由于性能开销太大,不适应当今大规模恶意软件样本分析的需求；第二类方法通过比较不同分析环境中恶意程序行为的差异,检测出恶意程序的反分析行为,但是有准确度低,及需要人工干预等不足之处。
本文采用第二类方法,在已有工作的基础上,提出一种改进的恶意程序反分析行为检测方法,有效地提高了检测的准确度。本文的主要研究工作如下：
1)总结了恶意程序反分析使用的不同类型的技术,以及现有反分析行为检测方法的优势和不足。
2)研究了二进制代码分析技术在恶意软件分析领域的应用,重点描述了本文方法用到的动态二进制切片技术。
3)改进了现有的检测方法。本文的方法能消除外界环境中无关因素的影响,检测出恶意程序真正的反分析行为。采用灵活的比较算法对恶意程序在不同分析环境中执行的行为进行比较,如果行为有差异,利用高效的算法对恶意程序执行的指令序列比较,并自动分析出行为的差异是否由反分析导致。
4)基于上述方法,实现了一个恶意程序反分析行为检测的原型系统。实验结果表明,本文的检测方法能检测出不同类型的反分析技术,如通过检测硬件特征、应用程序和时间开销等方法发现虚拟环境进而逃避分析。并且对于不具有反分析能力的恶意程序,本文的方法具有较好的鲁棒性。
Malware is the root cause of many information security threats. Security companies and researchers develop automated tools to extract and analysis the runtime behaviors of malware samples. Unfortunately, malware is aware of these tools, and looks for evidence of emulated or virtualized analysis environments. If such evidence is found, malware samples reduce their malicious behaviors or simply crash, showing a different "personality" than when executed on real system.
To solve the problem of anti-analysis malware, two kinds of approaches are proposed. One approach is to try to build transparent analysis platforms that are more difficult to detect by malware. But due to the performance overhead, these platforms are not suitable for the analysis of current high-volume malware feeds. Another approach runs malware samples in multiple analysis environments, and detect deviations in behaviors that may indicate anti-analysis. This approach also has some drawbacks, such as low accuracy, manual intervention and so on.
The method used in this thesis falls into the second class. In addition, we made some modifications on existing approaches to improve accuracy of the detection. The main work in this thesis is as follows:
1) We summarized varies of evasion techniques used by malicious programs. We discussed advantages and disadvantages of recent approaches to detect anti-analysis malware.
2) We launched a deep research on applications of binary program analysis in the field of malware analysis, in particularly on dynamic binary slicing technology used in this thesis.
3) We made some modifications on recent approaches to detecting anti-analysis behavior in malware. The approach proposed in this thesis can identify real anti-analysis behavior in malware, through eliminating unrelated differences between multiple analysis environments. A flexible algorithm is employed to compare the traces of system calls executed by malware across different analysis platforms. If a deviation exists, instruction traces are further compared using an efficient algorithm to determine whether the root cause of behavior deviation is anti-analysis or not.
4) Based on the improved detection method, we designed and implemented a prototype system to detect anti-analysis behavior in malware. Experimental results have demonstrated that the approach can detect varies of evasion techniques, include detecting hardware characteristics, applications, time overhead and so on. When detecting malware without anti-analysis capability, our approach has better robustness.
为了使恶意程序的反分析能力失效,现有的方法分为两类：第一类方法是构建透明的分析工具,但是由于性能开销太大,不适应当今大规模恶意软件样本分析的需求；第二类方法通过比较不同分析环境中恶意程序行为的差异,检测出恶意程序的反分析行为,但是有准确度低,及需要人工干预等不足之处。
本文采用第二类方法,在已有工作的基础上,提出一种改进的恶意程序反分析行为检测方法,有效地提高了检测的准确度。本文的主要研究工作如下：
1)总结了恶意程序反分析使用的不同类型的技术,以及现有反分析行为检测方法的优势和不足。
2)研究了二进制代码分析技术在恶意软件分析领域的应用,重点描述了本文方法用到的动态二进制切片技术。
3)改进了现有的检测方法。本文的方法能消除外界环境中无关因素的影响,检测出恶意程序真正的反分析行为。采用灵活的比较算法对恶意程序在不同分析环境中执行的行为进行比较,如果行为有差异,利用高效的算法对恶意程序执行的指令序列比较,并自动分析出行为的差异是否由反分析导致。
4)基于上述方法,实现了一个恶意程序反分析行为检测的原型系统。实验结果表明,本文的检测方法能检测出不同类型的反分析技术,如通过检测硬件特征、应用程序和时间开销等方法发现虚拟环境进而逃避分析。并且对于不具有反分析能力的恶意程序,本文的方法具有较好的鲁棒性。
Malware is the root cause of many information security threats. Security companies and researchers develop automated tools to extract and analysis the runtime behaviors of malware samples. Unfortunately, malware is aware of these tools, and looks for evidence of emulated or virtualized analysis environments. If such evidence is found, malware samples reduce their malicious behaviors or simply crash, showing a different "personality" than when executed on real system.
To solve the problem of anti-analysis malware, two kinds of approaches are proposed. One approach is to try to build transparent analysis platforms that are more difficult to detect by malware. But due to the performance overhead, these platforms are not suitable for the analysis of current high-volume malware feeds. Another approach runs malware samples in multiple analysis environments, and detect deviations in behaviors that may indicate anti-analysis. This approach also has some drawbacks, such as low accuracy, manual intervention and so on.
The method used in this thesis falls into the second class. In addition, we made some modifications on existing approaches to improve accuracy of the detection. The main work in this thesis is as follows:
1) We summarized varies of evasion techniques used by malicious programs. We discussed advantages and disadvantages of recent approaches to detect anti-analysis malware.
2) We launched a deep research on applications of binary program analysis in the field of malware analysis, in particularly on dynamic binary slicing technology used in this thesis.
3) We made some modifications on recent approaches to detecting anti-analysis behavior in malware. The approach proposed in this thesis can identify real anti-analysis behavior in malware, through eliminating unrelated differences between multiple analysis environments. A flexible algorithm is employed to compare the traces of system calls executed by malware across different analysis platforms. If a deviation exists, instruction traces are further compared using an efficient algorithm to determine whether the root cause of behavior deviation is anti-analysis or not.
4) Based on the improved detection method, we designed and implemented a prototype system to detect anti-analysis behavior in malware. Experimental results have demonstrated that the approach can detect varies of evasion techniques, include detecting hardware characteristics, applications, time overhead and so on. When detecting malware without anti-analysis capability, our approach has better robustness.
引文
[1]G. McGraw, J.G. Morrisett. Attacking Malicious Code:A Report to the Infosec Research Council. In Proceedings of IEEE Software.2000:33-41.
[2]McAfee Labs. McAfee Threats Report:Third Quarter 2010. from http://www.mcafee.com/us/resources/reports/rep-quarterly-threat-q3-2010.pdf.2010.
[3]McAfee Labs. McAfee Threats Report:Fourth Quarter 2020.from http://www.mcafee.com/us/resources/reports/rp-quarterly-threat-q4-2010.pdf.2011.
[4]Anubis:Analyzing Unknown Binaries.from http://anubis.seclab.tuwien.ac.at.2009.
[5]CWSandbox. from http://www.cwsandbox.org/.2009.
[6]Joebox:A Secure Sandbox Application for Win-dows. from http://www.joebox.org/.2009.
[7]Norman Sandbox. from http://www.norman.com/technology/norman_sandbox/.2009.
[8]ThreatExpert. from http://www.threatexpert.com/.2009.
[9]U. Bayer, C. Kruegel, E. Kirda. TT Analyze:A Tool for Analyzing Malware. In Proceedings of 15th European Institute for Computer Antivirus Research (EICAR),2006.
[10]P. Ferrie. Attacks on Virtual Machine Emulators. Technical report, Symantec Research White Paper.2006.
[11]P. Ferrie. Attacks on More Virtual Machines. Technical report, Symantec Research White Paper.2007.
[12]D. Quist, V.Smith. Computing, O. Detecting the Presence of Virtual Machines Using the Local Data Table.from http://www.offensivecomputing.net/files/active/0/vm.pdf.
[13]T. Raffetseder, C. Krugel, E. Kirda. Detecting System Emulators. In Proceedings of the 13th Information Security Conference(ISC).2007:1-18..
[14]X. Chen, J. Andersen, Z. Mao, M. Bailey, J. Nazario. Towards an Understanding of Anti-virtualization and Anti-debugging Behavior in Modern Malware. In Proceedings of the International Conference on Dependable Systems and Networks (DSN).2008:177-186.
[15]A. Vasudevan, R. Yerraballi. Cobra:Fine-grained Malware Analysis using Stealth Localized Executions. In Proceedings of the 27nd IEEE Symposium on Security and Privacy (S&P). 2006:264-279.
[16]A. Vasudevan,R. Yerraballi. Stealth Break-points. In Proceedings of the 21st Annual Computer Security Applications Conference (ACSAC).2005:381-392.
[17]A. Dinaburg, P. Royal, M. Sharif, W. Lee. Ether:Malware Analysis via Hardware Virtualization Extensions. In Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS).2008:51-62.
[18]A.M. Nguyen, N. Schear, H. Jung, A. Godiyal, S.T. King, H.D. Nguyen. MAVMM: Lightweight and Purpose Built VMM for Malware Analysis. In:Proceedings of the 25st Annual Computer Security Applications Conference(ACSAC).2009:441-450.
[19]T. Garfinkel, K Adams, A. Warfield, J. Franklin. Compatibility is Not Transparency:VMM Detection Myths and Realities. In Proceedings of the 11th Workshop on Hot Topics in Operating Systems (HotOS-Ⅺ),2007:1-6.
[20]G.Pek, B. Bencsath, L. Buttyan. nEther:In-guest Detection of Out-of-the-guest Malware Analyzers. In Proceeding of the 4th European Workshop on System Security (EUROSEC).2011.
[21]D. Balzarotti, M. Cova, C. Karlberger, E. Kirda, C. Kruegel, G. Vigna. Efficient Detection of Split Personalities in Malware. In Proceedings of the 17th Annual Network and Distributed System Security Symposium(NDSS),2010.
[22]N.M. Johnson, J. Caballero, K.Z. Chen, S. McCamant, P. Poosankam, D. Reynaud, D. Song. Differential Slicing:Identifying Causal Execution Differences for Security Applications. In Proceedings of the 32nd IEEE Symposium on Security and Privacy(S&P).2011:347-362.
[23]M.G. Kang, H. Yin, S. Hanna, S. McCamant, D. Song. Emulating Emulation-Resistant Malware.In Proceedings of the 1st ACM Workshop on Virtual Machine Security (VMsec).2009:11-22.
[24]M. Lindorfer, C. Kolbitsch, P.M. Comparetti. Detecting Environment-Sensitive Malware.In Proceeding of the 14th International Symposium on Recent Advances in Intrusion Detection, MenloPark, California(RAID).2011:338-357.
[25]T.Liston, E.Skoudis. On the Cutting Edge:Thwarting Virtual Machine Detection.from http://handlers.sans.org/tliston/ThwartingVMDetection_Liston_Skoudis.pdf.2006.
[26]A. Moser, C. Kruegel,E. Kirda. Exploring Multiple Execution Paths for Malware Analysis. In Proceedings of the 28nd IEEE Symposium on Security and Privacy(S&P).2007:231-245.
[27]Andreas Moser, Christopher Kruegel, Engin Kirda. Limits of Static Analysis for Malware Detection. In Proceedings of the 23rd Annual Computer Security Applications Conference(ACSAC),2007:421-430.
[28]C. Linn, S. Debray. Obfuscation of Executable Code to Improve Resistance to Static Disassembly. In Proceedings of 10th ACM Conference on Computer and Communications Security(CCS).2003:290-299.
[29]N. Nethercote, J. Seward. Valgrind:A Framework for Heavyweight Dynamic Binary Instrumentation. In Proceedings of the 2007 Conference on Programming Language Design and Implementation (PLDI).2007:89-100.
[30]D. Brumley, C. Hartwig, Z. Liang, J. Newsome,P. Poosankam, D. Song, H. Yin. Automatically Identifying Trigger-based Behavior in Malware. In Proceedings of Botnet Detection.2008:65-88.
[31]J. Rutkowska. Red Pill... or How to Detect VMM Using (almost) One CPU Instruction. from http://invisiblethings.org/papers/redpill.html.2004.
[32]R. Paleari, L. Martignoni, G. F. Roglia, D. Bruschi. A Fistful of Red-pills:How to Automatically Generate Procedures to Detect CPU Emulators. In Proceedings of the 3rd USENIX Workshop on Offensive Technologies (WOOT).2009:2-2.
[33]T. Klein. ScoopyNG-The VMware detection tool. from http://www.trapkit.de/research/vmm/scoopyng/index.html.
[34]K. Kato. VMware Backdoor I/O Port from http://chichat.at.infoseek.co.jp/vmware/backdoor .html.
[35]余冲,王振兴,郭浩然,刘慧生.基于监控器时间开销的虚拟机发现方法.计算机工程,2009,35(22)：47-49.
[36]Intel Corporation. Intel 64 and IA-32 Architectures Software Developer's Manual[R/OL]. from http://www.intel.com/products/processor/manuals/.
[37]S. S. Muchnick. Advanced Compiler Design and Implementation. Morgan Kaufmann,1997.
[38]M. Weiser. Program Slicing. In Proceedings of IEEE Transactions on Software Engin eering(TSE).1984:352-357.
[39]B. Korel, J.W. Laski. Dynamic Program Slicing. In Proceedings of Information processing Letters(IPL).1988:155-163.
[40]B. Korel,J.W. Laski. Dynamic Slicing of Computer Programs. In Proceedings of Journal of Systems and Software(JSS).1990:187-195.
[41]H. Agrawal, J.R. Horgan, Dynamic Program Slicing. In Proceedings of the ACM SIGPLAN'90 Conference on Programming Language Design and Implementation (PLDI).1990:246-256.
[42]T. Ball, S. Horwitz. Slicing Programs with Arbitrary Control-flow. In Proceedings of the 1st International Workshop on Automated Debugging(AADEBUG).1993:206-222.
[43]H. Agrawal, R.A. DeMillo, E.H. Spafford. Dynamic Slicing in the Presence of Unconstrained Pointers. In Proceedings of the 4th Symposium on Testing, Analysis, and Verif ic ation(TAV).1991:60-73.
[44]L. Guo, A. Roychoudhury, T. Wang. Accurately Choosing Execution Runs for Software Fault Localization. In Proceedings of the 15th International Conference on Compiler Construction(CC).2006:80-95.
[45]B. Dolan-Gavitt, T. Leek, M. Zhivich, J.T. Giffin, W. Lee. Virtuoso:Narrowing the Semantic Gap in Virtual Machine Introspection. In Proceedings of the 32nd IEEE Symposium on Security and Privacy(S&P).2011:297-312.
[46]L. Martignoni, R. Paleari, G.F. Roglia, D. Bruschi.Testing CPU Emulators. In Proceedings of the 18th International Symposium on Software Testing and Analysis(ISSTA).2009:261-272.
[2]McAfee Labs. McAfee Threats Report:Third Quarter 2010. from http://www.mcafee.com/us/resources/reports/rep-quarterly-threat-q3-2010.pdf.2010.
[3]McAfee Labs. McAfee Threats Report:Fourth Quarter 2020.from http://www.mcafee.com/us/resources/reports/rp-quarterly-threat-q4-2010.pdf.2011.
[4]Anubis:Analyzing Unknown Binaries.from http://anubis.seclab.tuwien.ac.at.2009.
[5]CWSandbox. from http://www.cwsandbox.org/.2009.
[6]Joebox:A Secure Sandbox Application for Win-dows. from http://www.joebox.org/.2009.
[7]Norman Sandbox. from http://www.norman.com/technology/norman_sandbox/.2009.
[8]ThreatExpert. from http://www.threatexpert.com/.2009.
[9]U. Bayer, C. Kruegel, E. Kirda. TT Analyze:A Tool for Analyzing Malware. In Proceedings of 15th European Institute for Computer Antivirus Research (EICAR),2006.
[10]P. Ferrie. Attacks on Virtual Machine Emulators. Technical report, Symantec Research White Paper.2006.
[11]P. Ferrie. Attacks on More Virtual Machines. Technical report, Symantec Research White Paper.2007.
[12]D. Quist, V.Smith. Computing, O. Detecting the Presence of Virtual Machines Using the Local Data Table.from http://www.offensivecomputing.net/files/active/0/vm.pdf.
[13]T. Raffetseder, C. Krugel, E. Kirda. Detecting System Emulators. In Proceedings of the 13th Information Security Conference(ISC).2007:1-18..
[14]X. Chen, J. Andersen, Z. Mao, M. Bailey, J. Nazario. Towards an Understanding of Anti-virtualization and Anti-debugging Behavior in Modern Malware. In Proceedings of the International Conference on Dependable Systems and Networks (DSN).2008:177-186.
[15]A. Vasudevan, R. Yerraballi. Cobra:Fine-grained Malware Analysis using Stealth Localized Executions. In Proceedings of the 27nd IEEE Symposium on Security and Privacy (S&P). 2006:264-279.
[16]A. Vasudevan,R. Yerraballi. Stealth Break-points. In Proceedings of the 21st Annual Computer Security Applications Conference (ACSAC).2005:381-392.
[17]A. Dinaburg, P. Royal, M. Sharif, W. Lee. Ether:Malware Analysis via Hardware Virtualization Extensions. In Proceedings of the 15th ACM Conference on Computer and Communications Security (CCS).2008:51-62.
[18]A.M. Nguyen, N. Schear, H. Jung, A. Godiyal, S.T. King, H.D. Nguyen. MAVMM: Lightweight and Purpose Built VMM for Malware Analysis. In:Proceedings of the 25st Annual Computer Security Applications Conference(ACSAC).2009:441-450.
[19]T. Garfinkel, K Adams, A. Warfield, J. Franklin. Compatibility is Not Transparency:VMM Detection Myths and Realities. In Proceedings of the 11th Workshop on Hot Topics in Operating Systems (HotOS-Ⅺ),2007:1-6.
[20]G.Pek, B. Bencsath, L. Buttyan. nEther:In-guest Detection of Out-of-the-guest Malware Analyzers. In Proceeding of the 4th European Workshop on System Security (EUROSEC).2011.
[21]D. Balzarotti, M. Cova, C. Karlberger, E. Kirda, C. Kruegel, G. Vigna. Efficient Detection of Split Personalities in Malware. In Proceedings of the 17th Annual Network and Distributed System Security Symposium(NDSS),2010.
[22]N.M. Johnson, J. Caballero, K.Z. Chen, S. McCamant, P. Poosankam, D. Reynaud, D. Song. Differential Slicing:Identifying Causal Execution Differences for Security Applications. In Proceedings of the 32nd IEEE Symposium on Security and Privacy(S&P).2011:347-362.
[23]M.G. Kang, H. Yin, S. Hanna, S. McCamant, D. Song. Emulating Emulation-Resistant Malware.In Proceedings of the 1st ACM Workshop on Virtual Machine Security (VMsec).2009:11-22.
[24]M. Lindorfer, C. Kolbitsch, P.M. Comparetti. Detecting Environment-Sensitive Malware.In Proceeding of the 14th International Symposium on Recent Advances in Intrusion Detection, MenloPark, California(RAID).2011:338-357.
[25]T.Liston, E.Skoudis. On the Cutting Edge:Thwarting Virtual Machine Detection.from http://handlers.sans.org/tliston/ThwartingVMDetection_Liston_Skoudis.pdf.2006.
[26]A. Moser, C. Kruegel,E. Kirda. Exploring Multiple Execution Paths for Malware Analysis. In Proceedings of the 28nd IEEE Symposium on Security and Privacy(S&P).2007:231-245.
[27]Andreas Moser, Christopher Kruegel, Engin Kirda. Limits of Static Analysis for Malware Detection. In Proceedings of the 23rd Annual Computer Security Applications Conference(ACSAC),2007:421-430.
[28]C. Linn, S. Debray. Obfuscation of Executable Code to Improve Resistance to Static Disassembly. In Proceedings of 10th ACM Conference on Computer and Communications Security(CCS).2003:290-299.
[29]N. Nethercote, J. Seward. Valgrind:A Framework for Heavyweight Dynamic Binary Instrumentation. In Proceedings of the 2007 Conference on Programming Language Design and Implementation (PLDI).2007:89-100.
[30]D. Brumley, C. Hartwig, Z. Liang, J. Newsome,P. Poosankam, D. Song, H. Yin. Automatically Identifying Trigger-based Behavior in Malware. In Proceedings of Botnet Detection.2008:65-88.
[31]J. Rutkowska. Red Pill... or How to Detect VMM Using (almost) One CPU Instruction. from http://invisiblethings.org/papers/redpill.html.2004.
[32]R. Paleari, L. Martignoni, G. F. Roglia, D. Bruschi. A Fistful of Red-pills:How to Automatically Generate Procedures to Detect CPU Emulators. In Proceedings of the 3rd USENIX Workshop on Offensive Technologies (WOOT).2009:2-2.
[33]T. Klein. ScoopyNG-The VMware detection tool. from http://www.trapkit.de/research/vmm/scoopyng/index.html.
[34]K. Kato. VMware Backdoor I/O Port from http://chichat.at.infoseek.co.jp/vmware/backdoor .html.
[35]余冲,王振兴,郭浩然,刘慧生.基于监控器时间开销的虚拟机发现方法.计算机工程,2009,35(22)：47-49.
[36]Intel Corporation. Intel 64 and IA-32 Architectures Software Developer's Manual[R/OL]. from http://www.intel.com/products/processor/manuals/.
[37]S. S. Muchnick. Advanced Compiler Design and Implementation. Morgan Kaufmann,1997.
[38]M. Weiser. Program Slicing. In Proceedings of IEEE Transactions on Software Engin eering(TSE).1984:352-357.
[39]B. Korel, J.W. Laski. Dynamic Program Slicing. In Proceedings of Information processing Letters(IPL).1988:155-163.
[40]B. Korel,J.W. Laski. Dynamic Slicing of Computer Programs. In Proceedings of Journal of Systems and Software(JSS).1990:187-195.
[41]H. Agrawal, J.R. Horgan, Dynamic Program Slicing. In Proceedings of the ACM SIGPLAN'90 Conference on Programming Language Design and Implementation (PLDI).1990:246-256.
[42]T. Ball, S. Horwitz. Slicing Programs with Arbitrary Control-flow. In Proceedings of the 1st International Workshop on Automated Debugging(AADEBUG).1993:206-222.
[43]H. Agrawal, R.A. DeMillo, E.H. Spafford. Dynamic Slicing in the Presence of Unconstrained Pointers. In Proceedings of the 4th Symposium on Testing, Analysis, and Verif ic ation(TAV).1991:60-73.
[44]L. Guo, A. Roychoudhury, T. Wang. Accurately Choosing Execution Runs for Software Fault Localization. In Proceedings of the 15th International Conference on Compiler Construction(CC).2006:80-95.
[45]B. Dolan-Gavitt, T. Leek, M. Zhivich, J.T. Giffin, W. Lee. Virtuoso:Narrowing the Semantic Gap in Virtual Machine Introspection. In Proceedings of the 32nd IEEE Symposium on Security and Privacy(S&P).2011:297-312.
[46]L. Martignoni, R. Paleari, G.F. Roglia, D. Bruschi.Testing CPU Emulators. In Proceedings of the 18th International Symposium on Software Testing and Analysis(ISSTA).2009:261-272.
© 2004-2018 中国地质图书馆版权所有 京ICP备05064691号 京公网安备11010802017129号 地址：北京市海淀区学院路29号 邮编：100083 电话：办公室：(+86 10)66554848；文献借阅、咨询服务、科技查新：66554700 |