Chapter 7 Securing Commercial Operating Systems

Chapter Overview Retrofitting Security into a Commercial OS History of Retrofitting Commercial OS's Commercial Era Microkernel Era UNIX Era – IX – Domain and Type Enforcement – Recent Unix Systems Summary

Retrofitting Security into a Commercial OS Requires reference Monitor Concept – Complete Mediation – Tamperproofing – Verifiability

Complete Mediation Challenges Identify all security-sensitive operations – Some embedded deep inside the kernel code. – Examples: Open Sockets Shared memory, etc. – Covert channel identification is usually not even attempted

Tamperproofing Challenges Obvious: place in ring 0, but Kernel is often updated. /dev/kmem, /proc, Sysfs, netlink sockets → Every root process must STILL be part of the UNIX TCB

Verification Challenges Musts: – Mediation is implemented correctly, but Mediation interface designed manually Implemented in unsafe languages – Policy enforces required security goals Large number of queries and processes. Complicate policies. – Reference monitor implementation is correct Rest of TCB is huge. – Rest of the TCB behaves correctly.

History of Retrofitting Commercial OS's Commercial Era – Emulate system on security kernel – Retrofit security into OS – → UNIX MLS Microkernel Era – Independent Server Processes → went to kernel – New security models addressing both confidentiality and integrity Unix Era – Composed solutions from the two eras with focus on system integrity.

Commercial Era Emulated Systems – Data Secure UNIX – KSOS KVM/370 – 25% Performance overhead VAX/VMS DEC/Sandia Labs: MLS Secure Xenix (IBM) Access control and auditing – Added Compatibility Retrofitted Unix services Hidden subdirectories – Polyinstantiated file systems – Trusted Path (Secure attention sequence) 1990 saw many secure Unix variants

Microkernel Era Goal: minimal size kernel emphasizing system abstractions; no emphasis on security per se. Source: Mach (1980's) – Trusted Mach (Tmach) – Distributed Trusted Mach (DTMach) – Distributed Trusted OS (DTOS) – Flask

Trusted Mach Built by Trusted Information Systems (TIS) Added MLS for files, memory. Aim was to provide function for other systems like Unix and Windows. (Single level)

Distributed Trusted Mach Secure Computing Corporation and NSA Hybrid access control model: – MLS labels for confidentiality – Type Enforcement labels for integrity (TE) Similar architecture to Tmach + servers for networking and general security policy server.

DTMach II DTMach = Mach + security server – Security server = reference monitor outside the kernel Each port access implies an authorization query For example, opening a file opens a port to the file server, etc. – Security server used both MLS and TE rules. TE rules: – code could only be modified by administrators – Limited code that could be executed There were limitations: – For example, there was an arbitrary send right...

Distributed Trusted OS (DTOS) AIM: True reference monitor in the Mach microkernel. Richer set of operations for ports than just send. Microkernel: – Managed labeling of subjects and kernel objects. – Mediated each kernel operation by querying security server. Focus on verifiability of microkernel and TCB.

Flask Fluke was a second generation microkernel developed at University of Utah, better than Mach. Flask = DTOS – Mach / Fluke More emphasis on TE.

UNIX Era By early 1990's, many Unices had MLS support. Search for adding integrity (very ad-hoc at this point). Cover two systems: – IX – DTE

IX AT&T prototype, enforces MLS and integrity. Includes a reference monitor over file access Mandatory access control policy providing both confidentiality and integrity protections. Care has been taken to prevent tampering in the TCB. Verification not a goal. MLS was high water mark, for files and processes. However processes could not go beyond a certain ceiling.

IX (2) Integrity was LoMac, with floors. Since levels are dynamic, each data transfer must be checked/mediated. No memory-mapped files. Trusted paths/pipes between processes (pex); a pex includes a label for the process at each end so that only that process may work with it.

An assured pipeline in IX

Domain and Type Enforcement Trusted Information Systems: Problem: protecting TCB from vulnerable root processes Runs on Tmach system, but reference monitor added to OSF/1.

DTE Policy Model Subject types are now called Domains, object types are still types. Each domain is a triple (access rights to objects, access rights to subjects in other domains (signals), entry point program) A domain describes how a process accesses files, signals other processes and creates processes. DTE Unix defines limited protection domains for root processes. Key point is “least privilege”. Domain transitions are limited and their execution is limited also. Labeled Networking.

Recent Unix Systems BSD variants – Trusted BSD MAC, auditing, authentication Reference monitor interface similar to LSM SEBSD is a version of SELinux for BSD – FreeBsd Jail – OpenBSD emphasizes correct coding and configuration Code separation Buffer overflow protection Least privilege configurations – NetBSD In-kernel authentication and verification of file execution Veriexec

Summary Retrofitting Security into a Commercial OS - Requirements and Challenges - History Commercial Era Microkernel Era Unix Era – recent Unix variants