Date on Master's Thesis/Doctoral Dissertation

12-2007

Document Type

Doctoral Dissertation

Degree Name

Ph. D.

Department

Computer Engineering and Computer Science

Committee Chair

Graham, James H.

Author's Keywords

Kernel rootkits; Kernel analysis; Rootkit detection

Subject

Computer security

Abstract

Rootkits are stealthy, malicious software that allow an attacker to gain and maintain control of a system, attack other systems, destroy evidence, and decrease the chance of detection. Existing detection methods typically rely on a priori knowledge and operate by either (a) saving the system state before infection and comparing this information post infection, or (b) installing a detection program before infection. This dissertation focuses on detection using reduced a priori knowledge in the form of general knowledge of the statistical properties of broad classes of operating system/architecture pairs. Four new approaches to rootkit detection were implemented and evaluated. A general distribution model is employed against kernel rootkits utilizing the system call table modification attack. Using approaches from the field of outlier detection, this approach successfully detected four different rootkits, with no false positives. Scalability is, however, an issue with this approach. A second, normality-based approach was investigated for use against rootkits infecting systems via the system call table modification attack. This approach was partially successful, but did generate false positives in 0.35% of cases. The general distribution model was then applied to rootkits infecting systems via the system call target modification attack. This dataset is dramatically larger, including disassembled memory addresses from the entire kernel. Finally, a modified version of the normality based approach proved effective in detecting kernel rootkits infecting the kernel via the system call target modification attack. This approach capitalizes on the discovery that system calls are loaded into memory sequentially, with the higher level calls, which are more likely to be infected by kernel rootkits loaded first, and the lower level calls loaded later. In the single case evaluated, the enyelkm rootkit, neither false positives nor false positives were indicated. As a final evaluation, these techniques were applied to the Microsoft Windows operating systems. The Windows equivalent of the system call table, the system service descriptor table (SSDT), appears to be almost perfectly normally distributed. A Windows rootkit employing the system call table modification attack was detected using the general distribution and 'assumption of normality' models.

Download

Share

COinS