Computer Science Colloquia
Thursday, September 26, 2013
Advisor: Jack Davidson
Attending Faculty: John Knight (Chair), Gabriel Robins; Ronald Williams, Minor Representative; and Yuan Gu, Irdeto Corporation
2:00 PM, Rice Hall, Rm. 242
PhD Dissertation Presentation
Software Protection via Composable Process-level Virtual Machines
Complex hardware/software systems are ubiquitous, affecting every aspect of
daily life. Software is integral to the normal functioning of critical systems such as power plants, financial systems, communication systems, modern medical systems and devices, and transportation systems, to name a few. Because of society's increasing reliance on these systems, it is of paramount importance that software perform as intended and not be subverted for malicious purposes. Consequently, techniques that thwart reverse engineering and tamper, (called tamper-resistance techniques), have become increasingly important as a means to hinder malicious exploitation of software in critical systems. Given the growing importance of preventing tampering with critical systems, research in this area has grown. Recently, software virtualization has been proposed as a suitable mechanism to impart tamper resistance to software applications. However, protections based on virtualization are not fully mature, which has led to successful attacks. This dissertation presents a detailed study of the application of low-overhead, process-level virtual machines (PVMs) to protect software applications from reverse engineering and tamper.
This research makes several contributions to the field of software protections. A formal model describing virtualization is presented. The model is useful in describing general-purpose computing systems and the applicability of virtualization in protecting applications. The dissertation also presents several novel tamper-resistance techniques that are based on process-level virtualization. For example, the research demonstrates that PVMs can be used to thwart static analysis of application binaries. It also presents new techniques that produce a shifting attack surface to the adversary. Such dynamic techniques make it difficult for the adversary to locate and identify crucial assets of the application. Each technique has been thoroughly evaluated in terms of performance overhead and protection.
The research also discloses serious vulnerabilities in current process-level virtual machines. Solutions to thwart such attacks that exploit such vulnerabilities are presented. Finally, the research describes a revolutionary protection technique to compose an application with multiple virtual machines, providing robust program protection. The ideas presented in this dissertation have been evaluated using current state-of-the-art attacks to gauge its effectiveness. The results of the investigations reveal that composable virtual machines are significantly more powerful in thwarting reverse engineering and software tamper than current protection techniques.