13/03/07 CENG334 Introduction to Operating Systems Erol Sahin Dept of Computer Eng. Middle East Technical University Ankara, TURKEY URL: Security Topics:

2 Goals of security in computer systems Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved Data confidentiality Secret data should remain secret Your bank account details Data integrity Unauthorized user should not be able to modify and change your data Your bank account goes from 1,000,000 YTL -> 0 in a second System availability Nobody should be able to disturb the system to make it unusable DOS (Denial of Service) attacks Exclusion of outsiders Outsiders can take control of other people’s computers converting them into zombies and use them to coordinate their attacks Sending spam or coordinating DOS attacks

3 Motivations of intruders and adversaries Casual prying by nontechnical users Setting all files readable Snooping by insiders Students, systems programmers and other technical personnel consider it a personal challenge to break the security Determined attempts to make money Attempts to get info to access bank accounts Commercial and military espionage Serious and well-funded attempt to steal programs, trade secrets

4 Cryptography Kerckhoff’s principle: The algorithms should be public and the secrecy should reside exclusively in the keys

5 Secret Key Cryptography One of the earliest types of cryptography Both encryption and decryption keys should remain secret Both the sender and the receiver should have possession of the keys The keys should be sent through a secure channel! Computation required for encryption and decryption are equal and manageable.

6 Monoaphabetic substitution ABCDEFGHIJKLMNOPQRSTUVWXYZ QWERTYUIOPASDFGHJKLZXCVBNM Plaintext Cipher text Encryption key

7 Monoaphabetic substitution QWERTYUIOPASDFGHJKLZXCVBNM KXVMCNOPHQRSZYIJADLEGWBUFT Plaintext Decryption key

8 Monoalphabetic substitution One of the earliest type of cryptography There are 26! = 4 X 10^26 possible keys But statistical properties of natural languages can be used to crack. In English, e is the most common letter If in the ciphertext, v is the most common letter, then it is likely to encrypt e. two-letter combinations can be used (th, in) Suffers from the same vulnerability

9 Public Key cryptography Distinct keys are used for encryption and decryption Encryption key is made public Everyone can see it and use it to encrypt their messages to send Decryption key is kept secret Only the receiver of the message can use this key to decrypt the message For a well-chosen encryption key, it is virtually impossible to discover the corresponding decryption key. The source of the symmetry: How much is: X ? Easy What is the square root of ? difficult

10 One-way functions y= f(x) such that for x, For a given x, it is easy to compute f(x) For a given f(x), it is difficult to compute x F() can mingle the bits in complex ways

11 Digital signatures Digital signatures used to sign s or other documents To detect that they are not modified by the received A hashing function that computes a value for a given document MD5

12 Watermarking Left: three zebras and a tree Right: three zebras and a tree and the complete text of five Shakespere plays Use the 8 th LSB of the pixels to code the compressed text To the bare eye, it is invisible

13 Protection mechanisms A computer system contains many “objects”, such as Files, directories, hardware,.. A file can be readable but not writable.. A domain is a set of (object, rights) pairs Right = read, write, execute … In UNIX, the domain of a process is defined by uid and guid (user and group id)

14 setuid How does the passwd program work When executed by the user, the process runs in the user’s domain Cannot modify the /etc/passwd file! Solution: passwd program has its setuid bit set, that allows it to run with root access Modify /etc/passwd file

15 Authentication Passwords Keep a list of user:encrypted_password pairs Used to be visible There is a function that encrypts passwords Not even the root can see your passwords But open to cracking through guessing: Get the password file, Guess passwords using different combinations Find the matches Physical objects Smart cards Biometrics Hands Eyes fingerprints

16 Insider attacks Logic bombs Check whether you are still on the payroll If so, do nothing, if not delete the filesystem Trap doors Add code to the login program such that it bypasses normal check

17 Login spoofing A user can write a program that looks exactly like the login screen The victim comes, enters his password The login sppofing program sends the password and quits The victim thinks it mistyped, and enters again to the actual login program

18 Exploiting code bugs Buffer overflow attacks You know this well ;)

19 Exploiting code bugs Format string attacks stems from the use of unfiltered user input as the format string parameter in certain C functions that perform formatting, such as printf(). A malicious user may use the %s and %x format tokens, among others, to print data from the stack or possibly other locations in memory. One may also write arbitrary data to arbitrary locations using the %n format token, which commands printf() and similar functions to write the number of bytes formatted to an address stored on the stack. Format string bugs most commonly appear when a programmer wishes to print a string containing user supplied data. The programmer may mistakenly write printf(buffer) instead of printf("%s", buffer). The first version interprets buffer as a format string, and parses any formatting instructions it may contain. The second version simply prints a string to the screen, as the programmer intended. Working example:

20 Other code bugs Return to libc attacks Pretty much all C programs link with libc Libc contains insecure functions such as strcpy() which copies an arbitrary string from any address to any other address Trick strcpy into copying the atackers program, called a shellcode, to the data segment and have it executed there Example: Integer overflow attacks For instance give large numbers a image size to a program Multiplication causes integer overflow Results in smaller buffer than needed Provides the ground for buffer overflow attack

21 Other code bugs Code injection attacks Getting the target progra to execute code without realizing it is doing so. > cp abc xyx > cp abc xyz; rm –rf /

22 Other code bugs Privilege escalation attacks The attacker tricks the system into giving it more access rights that it is entitled to Typically he tricks t into doing something that only root can do A daemon that allows user to schedule work to be done periodically. run as root so that it can access files from the user’s directory Has a directory in which it stores commands that are scheduled to run Users cannot write to this directory The attacker’s program set its working directory to the daemon’s directory. It crashes and forces a core dump. Core dumps are written into the working directory. Dumps can be written there since it is made by the system.

23 Malware Software that does bad things Trojan horses Hidden in a free program that is providing some useful function When called, it can do anything it wants Viruses Is a program that can reproduce itself by attaching its code to another program Can replicate through other programs Worms Replicates like viruses Can self-replicate

24 Viruses The lifetime: Usually written in assembler Inserted in a program or a file and distributed through filesharing, or downloads Free programs, pirated versions of commercial software Once installe, the infected program is executed. Companion viruses Does not actually infect a program When user types prog, MS-DOS looks for a program called prog.com or prog.exe Prog.com does not exist, but prog.exe is a useful executable provided on all the systems

25 Viruses Executable viruses Simplest version: overwrite an existing exeutable. Makes a readdir, and infects other executables Open the file and copy the virus code. Checks whether the file is already infected.

26 Viruses Memory-resident viruses Remains active in memory at the very top or the bottom of the address space. Can even change the memory bitmap to remain unnoticed. Typically captures one of the traps or interrupt vectors, such as the system call trap, and get executed at every system call. Can also do lots of spying.

27 Viruses Boot-sector viruses Overwrite the MBR such that it executes at every boot, before the booting of the OS. No OS, no protection: vulnerable state for the system Can also disguise itself as bad sectors. At boot time the virus copies itself to RAM, and directs the interrupt vectors to itself.

28 Viruses Device driver viruses Infect a driver and get loaded automatically by the OS when it gets called. This is why device drivers should run as user processes. Macro viruses Macros are programs that can be attached to data files such as Word or Excel. Source code viruses Change the c files such that #include Run_virus().

29 Spreading of viruses Typically through shared files through network s Downloads Browsing Sites that provide sexual content Sites that provide illegal download (warez’s) Sites that provide free programs Use checked software

30 Rootkits, and sony rootkit