1. CSC 482/582: Computer Security
Malware
CSC 482/582: Computer Security

2. Topics Types of Malware Self-Protection Mechanisms. Payloads.
Trojan Horses
Viruses
Worms
Backdoors
Rootkits
Self-Protection Mechanisms.
Payloads.
Malware Interactions.
Detecting Malware.
Defending against Malware.
The changing Malware environment.
CSC 482/582: Computer Security

3. Types of Malware Trojan Horse Virus Worm Backdoors
Tricks user into executing malicious code.
Virus
When run by user, copies self into other files.
Worm
Copies self from computer to computer.
Backdoors
Leaves opening for attacker to gain access.
Rootkits
Hides attacker activities from system administrators.
CSC 482/582: Computer Security

4. What about Spyware? Malware by any other name… Corporate malware.
Presents legal issues for anti-malware software.
CSC 482/582: Computer Security

5. Trojan Horse Attacker: Victim:
cat >ls
cp /bin/sh /tmp/.xxsh
chmod u+s,o+x /tmp/.xxsh
rm ./ls
ls $* ^D
Victim: ls
Program with both an overt and covert effect
Displays expected behavior when user executes.
Covert effect (executed with user’s privileges) violates security policy.
CSC 482/582: Computer Security

6. Virus Self-replicating code Virus Pseudocode:
Propagating (replicating) Trojan horse.
Inserts (possibly evolved) copy into other files.
Virus Pseudocode:
If spread condition then
Foreach target-file
if not infected then copy virus to target-file
Perform (malicious) action
Execute normal code
CSC 482/582: Computer Security

7. Types of Viruses Boot Sector Executable Dynamic Library
When system boots, code in boot sector executed.
Propagate by altering boot disk creation.
Uncommon today because of low use of boot floppies, but some Vista laptops shipped with one.
Executable
Infects executable programs (e.g., COM, EXE).
Executes when infected program is run.
Virus usually runs first, then runs original code.
Dynamic Library
Infected dynamicly linked libraries (DLLs.)
Executed when any program uses infected DLL.
CSC 482/582: Computer Security

8. Types of Viruses Device Driver Virtual Machine (.NET)
Infects loadable device driver.
Executes in kernel mode.
Virtual Machine (.NET)
Infects .NET MSIL binaries.
Portable: compiled to native code by CLR.
Archive Infectors
Inserts Trojan horse into ZIP files.
Uses social engineering techniques to get user to run.
CSC 482/582: Computer Security

9. Types of Viruses Macro Virus Infects embedded interpreted code.
Needs interpreter like sh, MS Word macro.
Can infect executables or data files
Executables must invoke appropriate interpreter.
Most modern data formats support some type of scripting, including
Microsoft Office
Windows Help files
HTML: VBScript, JScript
CSC 482/582: Computer Security

10. Infection Methods Overwriting Appending Prepending
Overwrites program code with virus.
Breaks infected program.
Appending
Append virus code to executable.
Insert JMP at beginning of executable.
Prepending
Insert virus code at beginning of executable.
Shift original code to follow virus.
CSC 482/582: Computer Security

11. Infection Methods Parasitic Cavity Fractionated Cavity
Inserts virus code at beginning of executable.
Shifts beginning of program to end of file.
Cavity
Insert virus code into unused blocks of file.
Insert JMP at beginning of executable.
Fractionated Cavity
Fragment virus; inject into multiple cavities.
Loader reads fragments into continuous memory.
CSC 482/582: Computer Security

12. Infection Methods Compressing Fragmenting Companion
Compresses executable to make space.
Inserts virus and decompression code.
Fragmenting
Dynamically fragment virus.
Insert fragments by overwriting or shifting code.
Fragments JMP/CALL each other.
Companion
Infects COM file of same name as EXE file.
Infects alternate data stream of Win32 file.
CSC 482/582: Computer Security

13. Worms Copies self from one computer to another
Self-replicating: No user action required unlike virus or Trojan horse programs.
Spreads via network protocols
ex: SMTP ( ), fingerd, MS SQL
CSC 482/582: Computer Security

14. History of Worms Morris Worm Nov 1988
Disabled most of Internet using multiple vectors.
Melissa
Mar 1999
MS Word macro virus spread via Outlook .
Code Red
Aug 2001
IIS Buffer overflow.
Code Green
Sep 2001
Removed Code Red II and patched vulnerability.
Slammer
Jan 2003
SQL Server worm infected entire Internet <1 hr.
Sobig
Jun 2003
Spam zombie botnet; RCI.
CSC 482/582: Computer Security

15. Worm Components Vector Propagation Engine Target Selection
Scanning Engine
Payload
CSC 482/582: Computer Security

16. Vector Software to gain access to target host. Common vectors:
Buffer overflow exploits.
Network file sharing, both NFS/SMB and P2P.
Social-engineering via or IM.
Weak passwords.
Parasitism: target backdoors and worm flaws.
CSC 482/582: Computer Security

17. Propagation Engine Transfers worm to host exploited by vector.
Small worms like Slammer included in vector.
Worm Propagation Methods:
FTP
HTTP
SMB
TFTP
CSC 482/582: Computer Security

18. Remote Control Interface
RCI allows creator to control infected hosts.
Many worms do not have a RCI.
May be a well-known backdoor program.
Common remote control features:
Start/stop infecting new targets.
Download new vectors.
Download new target selectors.
Download new payloads.
CSC 482/582: Computer Security

19. Target Selection Selecting targets for potential infection.
address harvesting
Address books.
Parse disk files.
Search news groups.
Network share enumeration
Check for filesystems shared with other systems.
Network scanning
Target hosts on current network and connected nets.
Randomized scanning of Internet space.
Web searching
Search Google for addresses or vulnerable software.
CSC 482/582: Computer Security

20. Scanning Engine Check targets for vulnerabilities.
If vector small, scanning can be skipped.
Scan for vulnerable services.
Like targeted nmap port scan.
OS Check
Check for correct OS for vector to work.
Version checking.
Check version of target software.
May customize vector based on information.
CSC 482/582: Computer Security

21. Morris Worm First Internet Worm: November 1988
Multi-architecture: Sun, VAX
Multi-vector
sendmail (debug backdoor)
fingerd (buffer overflow)
rsh (open .rhosts; password cracking)
CSC 482/582: Computer Security

22. Morris Worm Spreading algorithm Detection Avoidance
Local network topology: gateways, neighbors.
Used users’ .rhosts, .forward files.
Limited reinfection rate.
Detection Avoidance
Forged process listing as (sh).
Removed created files quickly after use.
CSC 482/582: Computer Security

23. Morris Worm Resource Requirements Problems Disk Space.
C compiler and linker.
Network connection to parent computer.
Problems
Didn’t limit re-infections.
Saturated CPU, network resources.
CSC 482/582: Computer Security

24. Malware Self-Protection
Anti-debugging
Detect/disable debuggers when used to analyze code.
Attack anti-malware tools
Disable anti-malware tools upon infection.
Kill processes or destroy/modify signatures.
API checksums
Avoid having UNIX/Win32 API calls in code.
Store checksums of API names and search for match.
Code obfuscation
Use unusual tricks and unused code to avoid dissassembly and prevent quick analysis of purpose.
Self-modifying code.
CSC 482/582: Computer Security

25. Self-Protection Compression Data encryption Embedding
Code looks almost random; size is smaller.
Use unusual executable packers to avoid analysis.
Data encryption
Encrypt strings, hostnames, IP addresses to avoid detection.
Embedding
Use multiple levels of executable packers like UPX.
Scanners have to understand and have time to parse and decompress each file format.
CSC 482/582: Computer Security

26. Self-Protection Entry-Point Obscuring Host morphing
Changing initial code or entry point easy to notice.
Alter program code to gain control randomly.
Host morphing
Alter host file during infection to prevent removal.
CSC 482/582: Computer Security

27. Self-Protection: Encryption
Encrypt all code except small decryptor.
Note that copy protected files will have similar decryptors to prevent analysis too.
Often uses multiple decryptors.
Change encryption key dynamically.
Random Decryption Algorithm (RDA)
Choose random key for encryption.
Brute force search for key to decrypt.
Slows VMs/debuggers used for analysis.
CSC 482/582: Computer Security

28. Self-Protection: Polymorphism
Alter malware code with each infection.
Cannot be detected by signature scanning.
May alter decryptor only or entire code.
Insert junk instructions that do nothing.
Fragment and rearrange order of code.
Alternate sets of instructions for the same task.
Ex: SUB -1 instead of ADD 1
Randomize names in macro viruses.
CSC 482/582: Computer Security

29. Case Study: Zmist EPO, encrypted, polymorphic virus. Code integration
Decompiles PE files to smallest elements.
Inserts virus randomly into existing code.
Rebuilds executable.
Polymorphic decryptor
Inserted as random fragments linked by JMPs.
Randomizes self with ETG engine.
CSC 482/582: Computer Security

30. Payloads Accidentally destructive. Nondestructive. Destructive.
Replication damages data due or exhausts system resources due to malware bugs.
Ex: Morris Worm reinfected hosts, using all CPU.
Nondestructive.
Displays message, graphics, sound, or open CD door.
Ex: Christma worm on IBM network in 1987.
Destructive.
Triggers randomly or on some event or machine type.
Deletes files or overwrites data.
Hardware destroyers: overwrite BIOS.
CSC 482/582: Computer Security

31. Payloads Denial of Service Data Theft Encryptors (ransomware) Spam
Sometimes accidental due to high network use.
Launch DDOS attack with all infected systems.
Data Theft
Phishing scams and spyware.
Encryptors (ransomware)
Encrypts user data.
Ex: One_Half encrypts disk; enables access while running.
Ex: AIDS Info: encrypts disk and holds for ransom.
Spam
Use network of infected systems to launder spam .
Ex: Sobig worm.
CSC 482/582: Computer Security

32. Malware Interactions What happens when a virus infects a worm?
Typically both propagate.
May use each other’s self-protection techniques.
What if anti-virus software removes a virus?
Likely leaves unknown virus/worm alone.
Partial removal can mutate the malware into a new form.
Competition and Parasitism
Malware may remove competing malware.
May exploit backdoors/RCI left by previous malware.
May infect competing malware, hijacking its propagation.
CSC 482/582: Computer Security

33. Theory of Malicious Code
Theorem 1: It is undecidable whether an arbitrary program contains a computer virus.
Proof:
Define virus v as TM program that copies v to other parts of the tape, while not overwriting any part of v.
Reduce to Halting Problem: T’ running code V’ reproduces V iff running T on V halts.
Theorem 2: It is undecidable whether an arbitrary program contains malicious logic.
CSC 482/582: Computer Security

34. Detecting Malware Signature-based Smart scanning Decryption
Look for known patterns in malicious code.
Defeated by polymorphic viruses.
Smart scanning
Skips junk instructions inserted by poly engines.
Skips whitespace/case changes in macro viruses.
Decryption
Brute-forces simple XOR-based encryption.
Checks decrypted text against small virus sig to decide whether has plaintext or not.
CSC 482/582: Computer Security

35. Detecting Malware Code Emulation Code Optimization.
Execute potential malware on VM.
Scan VM memory after certain # iterations.
Watch instructions for decryptor profile.
Code Optimization.
Optimize away junk instructions and odd techniques used by polymorphic viruses.
CSC 482/582: Computer Security

36. Detecting Malware Heuristics Neural Network Heuristics
Code execution starts in last section.
Suspicious code redirection.
Suspicious section ACLs or size.
Suspicious library routine imports.
Hard-coded pointers into OS kernel.
Neural Network Heuristics
IBM researchers trained neural net to recognize difficult polymorphic viruses.
Released in Symantec antivirus.
CSC 482/582: Computer Security

37. Detecting Malware Behavior-based Integrity Checking
Watch for known actions from malicious code.
Network access signature of worm.
Unexpected use of dangerous system calls.
Integrity Checking
Host-based Intrusion Detection System.
Record MAC, size, dates, ACL of files.
Periodically check for changes.
ex: Tripwire, AIDE, Osiris
CSC 482/582: Computer Security

38. Defences: Data vs. Code Separate data and instructions
Virus treats program as data
Writes self to file.
Virus treats program as instructions
Virus executes when program is run.
Solution: Treat all programs as data until trusted authority marks as executable.
Development difficult when compilers can’t produce executable code.
CSC 482/582: Computer Security

39. Defences: Information Flow
Limit Information Flow
Virus executes with user’s identity.
Soln: Limit information flow between users.
Set flow distance to be one for users A, B, C.
A creates virus (fd=0), B executes it (fd=1).
C cannot execute B’s infected program (fd=2).
Indirect virus spread limited.
How can we track information flow?
CSC 482/582: Computer Security

40. Defences: Least Privilege
Limit programs to least privilege needed
example: SELinux
Mail virus example
Virus arrives via .
Virus exploits bug in client to execute.
Virus saves self to file in Startup folder.
Virus infects Office documents.
How least privilege would stop
Mail application cannot create virus binaries.
Mail application cannot write to Startup folder.
Mail application cannot write to Office documents.
CSC 482/582: Computer Security

41. Defences: Sandboxes Execute code in protected sandbox or VM.
Virtual Browser Appliance
Linux guest running Firefox under VMWare.
Infections can only attack VM, not real host.
Reset VM to initial state if infected.
CSC 482/582: Computer Security

42. Defences: Anomaly Detection
Validate program actions with policy
Limit access to system calls.
Example: systrace.
Check statistical characteristics.
Programmer style.
Compare source code with object.
Statistics of write frequencies, program executions.
CSC 482/582: Computer Security

43. Defences: Counter-worms
Worm that removes other worms from net.
Nachi/Welchia
Multi-vector W32 worm
Nachi.A removes W32/Blaster worm
Nachi.B removes W32/MyDoom worm
Installed MSRPC DCOM patch to prevent future infections from Blaster.
Removes self after 2004.
Side-effects
Infected Diebold ATMs
Worm traffic DOSed Internet, esp Microsoft.
CSC 482/582: Computer Security

44. Fast Worms Slammer Worm Characteristics Attacked MS SQL servers.
Worm is single 404-bye UDP packet.
Random-scan (PRNG bugs limited.)
Limited by network bandwidth, not latency.
Observed scan rate of 26,000 hosts/second.
Infected 90% of vulnerable hosts in 10 min.
Too fast for humans to react.
Shutdown 13,000 Bank of America ATMs due to compromising db servers, heavy traffic.
CSC 482/582: Computer Security

45. Profitable Malware Sobig W32 worm using email/network share vectors.
Contains upgrade mechanism
Worm checked sites every few minutes.
When site valid, downloaded code.
Later variants could update upgrade server list.
Downloaded payload from upgrade mechanism
Key logger.
Wingate proxy server (for spam proxying.)
CSC 482/582: Computer Security

46. Profitable Malware Trojans Spyware and Adware
Backdoor.Lala transfers authentication cookies for eBay, PayPal, etc. to maker.
PWSteal.Bancos automates phishing by displaying fake web pages when browser goes to certain bank sites.
Spyware and Adware
More than ever using Trojan techniques.
Win32/Bube virus exploits IE flaw and acts as a virus infecting IE, then downloads adware.
CSC 482/582: Computer Security

47. Mobile Malware
2004: Cabir virus infecting Symbian OS mobile phones using Bluetooth appeared in June.
2005: Commwarrior-A worm spreads to Symbian series 60 phones via phone’s MMS.
Around a 1000 pieces of mobile malware exist.
For Blackberries and Palm Pilots too.
Expect more as smart phones become common.
MMS=multimedia messaging system
CSC 482/582: Computer Security

48. Offline Impact Davis-Besse nuclear power plant Seattle 911 system
Slammer infected Plant Process Computer and Safety Parameter Display System (Jan 2003.)
Analog backups unaffected.
Infected contractor’s network, then moved through T1 line that bypassed plant firewall.
Seattle 911 system
Slammer disabled computer systems.
Dispatchers reverted to manual systems.
2003 Blackout
Blaster infected First Energy systems.
CSC 482/582: Computer Security

49. Modern Malware is Stealthy: rootkit techniques common.
Targeted: targets smaller banks and countries, leverages current events:
January: Storm Worm appears via with subject “230 dead as storm batters Europe.”
February: Miami Dolphins Stadium site hacked before superbowl so that it would infect browsers with trojan that grabbed WoW data.
Blended: combine trojan, virus, worm features.
Web-based: use web for delivery and update.
Profit-driven: the goal is to make money.
CSC 482/582: Computer Security

50. References Ross Anderson, Security Engineering, Wiley, 2001.
Matt Bishop, Computer Security: Art and Science, Addison-Wesley, 2003.
William Cheswick, Steven Bellovin, and Avriel Rubin, Firewalls and Internet Security, 2nd edition, 2003.
Fred Cohen,
Simson Garfinkel, Gene Spafford, and Alan Schartz, Practical UNIX and Internet Security, 3rd edition, O’Reilly & Associates, 2003.
Alexander Gostev, “Malware Evolution: January - March 2005,” April
Elias Levy, “Crossover: Online Pests Plaguing the Offline World,” IEEE Security & Privacy, 2003.
Stuart McClure, Joel Scambray, George Kurtz, Hacking Exposed, 5th edition, McGraw-Hill, 2003.
Hilarie Orman, “The Morris Worm: A Fifteen-Year Perspective,” IEEE Security & Privacy, 2003
Cyrus Peikari and Anton Chuvakin, Security Warrior, O’Reilly & Associates, 2003.
Ed Skoudis, Counter Hack Reloaded, Prentice Hall, 2006.
Ed Skoudis and Lenny Zeltser, Malware: Fighting Malicious Code, Prentice Hall, 2003.
Staniford, Stuart, Paxson, Vern, and Weaver, Nicholas, ‘How to 0wn the Internet in Your Spare Time,” Proceedings of the 11th USENIX Security Symposium, 2002
Peter Szor, The Art of Computer Virus Research and Defense, Addison-Wesley, 2005.
Trend Micro, “1H2007 Threat Roundup,”
CSC 482/582: Computer Security