Malwares – Types & Defense Raghunathan Srinivasan Sept 25, 2007 CSE 466/598 Computer Systems Security

Malware  How to define malware? Over a broad sense, any malicious program  Types Viruses Trojans Rootkits Spyware

Virus  A program that can attach itself to another program Can replicate Encrypted  How to prevent them? Anti–virus???  How do they work No – real Answer

Types of Viruses [Evolution?]  Parasitic Viruses Also known as file infectors  Date / Logic bomb Michaelangelo, Sunday, Century  Macro Viruses Infect macro utility feature in word  Encrypted Virus cascade  Polymorphic Virus 1260  Stealth Virus

Encrypted Viruses  Viruses have certain patterns present in them Signatures  AV looks for these patterns in files  To avoid detection, the virus encrypts itself 1.Mov 2.Fetch 3.###$$$ 4.&&^^^^ Decryption engine Encrypted virus body

Encrypted virus  It is not possible to find out what the encrypted text is  So how to find if an encrypted entity is a virus? Look in previous slide  Next step – polymorphic viruses

Polymorphic  Can change form from infection to infection  There is a mutation engine present in the virus body  During run time – the virus loads the mutation engine  The ME changed the decryption routine  The virus changes form on every encryption Now the virus is difficult to spot

Detection  Creating random encryption – decryption routines is difficult See how many badly designed encryption algorithms are present  CSS  Hence encryption is weak, can be broken  Can this be reliable? No  Then what to do?

Detection  AV scanners use what is known as simulation  They create a virtual PC in the RAM  Load the program in the Virtual PC  The program executes, and shows its true behavior eventually You can read the following paper for further details  Understanding and Managing polymorphic viruses –  Google it, it’s a white paper by Symantec

A new trend in Virus  Viruses have become complex  Anti-Virus programs are running powerful engines game of cat and mouse  What further can virus writers do to prevent detection Go stealth  Install rootkits  Install portions of program in various other executables Disable detectors?

Disabling detectors  If you don’t have a defense mechanism, you cant escape infection  Kill all security processes Works, but a smart user can figure something Patch on the definitions Patch on the program policies  How does that help  Disable updates?

Examples of such viruses  SpamThru Locates existing AV in the machine Patches them to prevent updates Installs its own virus scanner  Why?  Beast Kills all existing security services Hooks on to winlogon.exe  What is winlogon.exe

Implications?  AV does not function  No method to detect the presence of viruses  How to solve this? Borrow some virus tricks Hide the AV Move the program code Hide files Hide Process name

Other Miscellaneous Malware  Worms Self replicating program Does not require host to replicate It uses the network to send copies of itself They use the bandwidth and harm the network  Viruses harm the computer (host)  Does worm not harm the PC? Not necessarily Worms for ATM’s  Slammer, Nachi

Trojan  USC Trojans?  People from the affair of Helen of Troy?  NO  Program that enters a system disguised as something else Never trust the gifts from Greeks (lesson learned from trojan war)  Trojan perhaps looks harmless Or useful  Allow installation Backdoors Rootkits

 Term derived from UNIX account ‘root’  Patches on to host kernel libraries, routines  Place hooks on API’s, OS services, Routines, etc  A good rootkit cannot be detected Does the statement sound too strong?

Shadow Walker  Designed to deceive in signature scanners That is how Anti-Virus and most Rootkit detectors work  Hides its presence in the system  It hooks on to the page table entries & the page fault handler  It flushes the TLB No page can be accessed bypassing the page fault handler initially

Shadow Walker ….  So how does that help A scanner attempts to read a page A fault is generated This causes a fetch The rootkit ensures that the scanenr never gets any access to infected pages

BluePill  Rootkit designed for Vista running on AMD pacifica technology Has special mode for VM executions  Allocate memory for a process More than required What does this do?  Rootkit writes on the paged drivers  When the drivers are loaded back, you have infected drivers in memory  Allows Vista to be moved in guest environment  Rootkit becomes a hypervisor

Bluepill.. contd  So what happens due to that?  The Vista OS becomes the guest, and is completely under the control of the rootkit. Any scanner working from within the OS can never see the rootkit.  Why? An OS process cannot have access to the layer below the OS So if we placed something below the OS, the OS cannot find out about it.

Scenario 1 Hardware OS Kernel Applications Application level malware – easy/slightly difficult to detect kernel level malware – very difficult to detect

Scenario 2 Hardware OS Kernel Applications VMM layer malware Not possible to detect from within the OS. Requires Hardware detection

Solutions  VM based rootkit detectors  Hardware based rootkit detectors

VMM based detector  Type I VMM [XEN]  The VMM runs on top of the hardware.  Root of trust mechanism  VMM checks the privileged VM  The PVM checks the SM  SM checks the other VM’s

VMM  The VMM runs 1 Privileged VM(VM0), and many other guest VM’s  The VMM checks the VM0 over periods of time Ensures the kernel of VM0 is not tampered with  VM0 runs the SM It contains the integrity values of SM, to detect tampering

VMM detector - contd  The SM can access the states of all applications running on all the Guest VM’s  Guest VM’s run OS’s that run user applications  So what has this achieved? Layered Software

The Trusted VM  What has to be done to penetrate the VMM layer Attack the applications Attack the guest OS Attack the Guest VM Finally attack the VMM  SM detects these before the final step

VMM layer  Is a micro kernel What is a micro kernel  Answer: Best left to OS classes Hence not a general purpose OS Does not execute third party software Due to this, it is secure  Too strong a statement?  Ok, has fewer vulnerabilities (due to less code)  Has fewer loopholes to exploit  Does not suffer from infected third party drivers

What does the VMM do?  Isolation between programs in an Operating System’s is a very difficult process  Many researches on it, fairly inconclusive  VMM provides isolation between the Guest VM’s  VMM also allows us to sandbox an OS and monitor it

VM0  Monitors the SM  It can also allow and prevent other VM’s from accessing certain memory locations  It can protect sections in memory  It can prevent other VM’s from accessing some I/O devices Why is this important?

SM  Checks the VM  Provides secure communication to User Why is this important?  The SM has access to the state of registers, memory and instructions being executed by each Guest Vm This helps to monitor the GVM’s

SM - contd  Checks the integrity values of Guest OS’s during boot Allows detection of boot sector infections, rootkits Can this help us detect VM based rootkits?  Checks kernel integrity, OS text section, interrupt vectors, etc

Last step  Can a rootkit impersonate a user Yes, at least it will attempt to do so  So how can this be prevented? The last module Secure I/O device Do you see the answer to a question regarding I/O device access 3 slides back?

Secure I/O  Provides a trusted mode of communication between user and VMM  It should be a separate device Why? Why cant it be a software channel

Why do we need secure I/O  Are human validations really true What happens if this step is not followed  A viral program can trick the guest OS into sending a message that an update was performed  Allows changing of integrity values  The malware gets certified by the SM

Hardware detectors  Separate hardware device  Attached to the PCI slots  Can be attached in other places also  Some implementations involve placing a co-processor on the motherboard

Hardware detectors  This is also a root of trust device  The hardware device runs an OS  Its resources and state are not accessible by host CPU/HW  It is capable of accessing the host’s memory  It can halt a system if required

Heirachical checking  Each level stored the integrity values of the level above it  The SecCore contains the integrity values of certain critical sections of the kernel

SecCore  The critical sections of the kernel is responsible for checking the rest of the kernel  It is also responsible for checking the applications  The kernel is responsible for maintaining the integrity of the User level programs

Advantages  The Coprocessor does not have to attest the entire OS  Keeps load low  It stores information only about a small space  Memory requirements low  Most of the checking is offloaded to the Host CPU

Problems  Many integrity values reside inside the kernel  Can be infected  Solution? Sign them  Digital Signatures

Thanks