VMAttack: Deobfuscating Virtualization-Based Packed Binaries

Kalysch A, Götzfried J, Müller T (2017)


Publication Language: English

Publication Type: Conference contribution, Conference Contribution

Publication year: 2017

Publisher: ACM

Edited Volumes: ACM International Conference Proceeding Series

Book Volume: Part F130521

Pages Range: 2:1--2:10

Conference Proceedings Title: 12th International Conference on Availability, Reliability and Security

Event location: Reggio Calabria, Italy

ISBN: 9781450352574

URI: https://dl.acm.org/citation.cfm?doid=3098954.3098995

DOI: 10.1145/3098954.3098995

Abstract

We present VMAttack, a deobfuscation tool for virtualization-packed binaries based on automated static and dynamic analysis, which offers a simplified view of the disassembly. VMAttack is implemented as a plug-in for IDA Pro and as such, integrates seamlessly with manual reverse engineering. The complexity of the disassembly view is notably reduced by analyzing the inner working principles of the VM layer of protected binaries. Using static analysis, complex bytecode sequences of the VM are mapped to easy-to-read pseudo-code instructions, based on an intermediate representation specifically designed for stack-based virtual machines. Using dynamic analysis, we identify structural components like the interpreter loop and compress instruction sequences by filtering out semantically redundant instructions of the execution trace. The integrated result, which rates both static and dynamic analysis's results, provides the reverse engineer with a deobfuscated disassembly that tolerates weaknesses of a single analysis technique. VMAttack is currently limited to stack-based virtual machines like VMProtect. We evaluated VMAttack using binaries obfuscated with VMProtect and achieved an average execution trace reduction of 89.86% for the dynamic and 96.67% for the combined static and dynamic analysis.

Authors with CRIS profile

How to cite

APA:

Kalysch, A., Götzfried, J., & Müller, T. (2017). VMAttack: Deobfuscating Virtualization-Based Packed Binaries. In 12th International Conference on Availability, Reliability and Security (pp. 2:1--2:10). Reggio Calabria, Italy: ACM.

MLA:

Kalysch, Anatoli, Johannes Götzfried, and Tilo Müller. "VMAttack: Deobfuscating Virtualization-Based Packed Binaries." Proceedings of the ARES'17, Reggio Calabria, Italy ACM, 2017. 2:1--2:10.

BibTeX: Download