1. CSC 482/582: Computer Security
Operating System Security
CSC 482/582: Computer Security

2. Example bash session with commands joined with ; and |
Example bash session with commands joined with ; and |. Explain how bash/PowerShell are important skills.
Explain how PATH works.
UID vs EUID, SETUID: use material from Chapter 9 of The Linux Programming Interface.
CSC 482/582: Computer Security

3. Topics What is an OS? Processes Memory management Filesystems
Virtual machines

4. A Computer Model
An operating system has to deal with the fact that a computer is made up of a CPU, random access memory (RAM), input/output (I/O) devices, and long-term storage.
RAM 1 2 3 4 5 6 7 8 9 .
Disk Drive
I/O
CPU

5. UNIX Family Tree (Simplified)
CSC 482/582: Computer Security

6. The Kernel Core of operating system. Always in RAM.
Layer between hardware and applications.

7. Linux Kernel Map
CSC 482/582: Computer Security

8. CPU Management Protection rings Windows/Linux ring use Changing rings
Lower number=higher privilege
Certain CPU instructions only available in lower rings.
Windows/Linux ring use
Kernel runs in ring 0.
User programs run in ring 3.
Changing rings
Interrupts change to ring 0.
Interrupts also changes location of current CPU instruction to address in kernel.

9. Multitasking Multitasking
Give each running program a “slice” of the CPU’s time.
The CPU is running so fast that to any user it appears that the computer is running all the programs simultaneously.
A program runs until one of the following occurs:
It has used up its entire time slice.
It asks the kernel to access a resource.
An interrupt occurs.

10. System Calls
User applications can’t access hardware directly as it requires privileged CPU instructions, and thus must ask kernel to access hardware via system calls.
System calls setup a data structure describing the request to make and then cause an interrupt. Examples:
File I/O: open, close, read, write
Request memory: brk
Creation process: fork
Running application: exec

11. Interrupts

12. What is a process? A process is a program in execution. Virtualization
Program code + dynamic execution context.
Virtualization
Processes provide virtual CPU + virtual memory.
Kernel refers to processes as tasks.

13. What is in a process? A process consists of: Process ID (PID)
Program code.
Address space.
Data.
Resources:
Open files
Network connections
If you run a program 3 times, you created 3 different processes.

14. Top: highest CPU processes

15. fork() and exec() model
fork() creates a new process
New PID
New address space
Same program code and data
exec() replaces code with that of a new program
$ ls
fork() creates a copy of the bash shell
exec() loads and runs the ls program
exit() terminates ls program

16. Process Creation and Termination

17. The Process Tree OS kernel creates first process init, PID 1.
All other processes created by init or by processes created by init via fork() and exec().
There init is the parent or greatn-grandparent of all processes..

18. Viewing the Process Tree with ps

19. Multitasking Processes

20. Memory Management OS manages physical RAM.
Gives each process a virtual address space.
On a 32-bit machine, 232 bytes=4GB maximum RAM
Process sees 3GB for itself.
1GB reserved for OS kernel.
By creating a page table for each process.
Memory is divided into pages of ~ 4KB each
Address divided into page number + offset.
Page table is a map from virtual pages to physical pages.
CPU uses page table to translate virtual addresses to physical addresses. Only the kernel can modify a page table.
A process cannot access memory of other processes since its page table does not contain mappings to their memory pages.

21. Virtual Address Translation

22. Virtual Memory OS gives each process 4GB
Most processes do not use that much RAM. Many page table entries are blank.
A single process cannot use more than 3GB (1GB reserved for OS kernel.)
All processes together may require more RAM than is physically available.
OS can map pages to the hard disk to handle that case.

23. Page Table Metadata Pages have permissions
Read
No execute (NX)
A page fault interrupt is generated by kernel when
Memory access attempted that would violate permissions.
Page is marked as not valid (not mapped to a physical page.)

24. Page Faults 1. Process requests virtual address not in memory,
causing a page fault.
2. Paging supervisor pages out an old block of physical memory.
3. Paging supervisor locates requested block
on the disk and brings it into RAM memory.
“read ”
“Page fault,
let me fix that.”
Blocks in
physical memory
Paging supervisor
External disk
old
new

25. Memory Layout of a Process

26. Input/Output
The input/output devices of a computer include things like its keyboard, mouse, video display, and network card, as well as other more optional devices, like a scanner, Wi-Fi interface, video camera, USB ports, etc.
Each such device is represented in an operating system using a device driver, which encapsulates the details of how interaction with that device should be done.
The application programmer interface (API), which the device drivers present to application programs, allows those programs to interact with those devices at a fairly high level, while the operating system does the “heavy lifting” of performing the low-level interactions that make such devices actually work.

27. Filesystems
A filesystem is an abstraction of how external storage of the computer is organized.
An OS can support multiple filesystems.
Examples: ext4fs, iso9660, YAFFS, etc.
Operating systems typically organize files hierarchically into folders, also called directories.
Each folder may contain files and/or subfolders.
Thus, a filesystem consists of a collection of nested folders that form a tree.
The topmost folder is the root of this tree and is also called the root folder.
Most filesystems provide access control for files.

28. File System Example

29. Virtual Machines
Virtual machine: Software that emulates a computer system so that another OS can run on top of the existing OS.
Benefits:
Hardware Efficiency
Portability
Security
Management
Public domain image from

30. Virtualization adds Hypervisor OS
In a VM, apps run on guest OS.
Guest OS runs on top of a hypervisor OS.

31. Each VM has own Guest OS Virtual Machines Physical Machine Linux BSD
W2k8
Virtual
Machines
Physical
Machine

32. Hypervisor Security Vulnerability consequences
Guest code execution with privilege
VM Escape (Host code execution)
Vendor
CVEs
KVM 32
QEMU 23
VirtualBox 9
VMware
126
Xen 86
Xen CVE
VBox CVE

33. Key Points An OS is a layer btw applications and hardware
Manages users, processes, and hardware resources.
A process is a program in execution
PID identifies process.
fork() creates a copy of a process.
exec() runs a new program into address space.
A process runs until
Its time slice expires.
It requests OS help via a system call.
An interrupt occurs.
Each process has its own virtual address space
Setup by kernel created page table.
CPU translates virtual to physical addresses via table.
Page fault occurs when page is mapped to disk (or does not exist.)

34. References Anderson, Security Engineering 2nd Edition, Wiley, 2008.
Bishop, Computer Security: Art and Science, Addison-Wesley, 2002.
Goodrich and Tammasia, Introduction to Computer Security, Pearson, 2011.
Sudhakar Govindavajhala and Andrew W. Appel, Using Memory Errors to Attack a Virtual Machine, July 2003.