Operating System Security Trent Jaeger The Pennsylvania State University SYNTHESIS LECTURES ON INFORMATION SECURITY, PRIVACY AND TRUST #1 Morgan &cLaypool publishers

Introduction Operating systems provide the fundamental mechanisms for securing computer processing. Since the 1960s, operating systems designers have explored how to build “secure” operating systems —operating systems whose mechanisms protect the system against a motivated adversary. Recently, the importance of ensuring such security has become a mainstream issue for all operating systems.

Three major tasks Operating systems must provide efficient resource mechanisms, Second, it is the operating system’s responsibility to switch among the processes fairly Third, access to resources should be controlled, such that one process cannot inadvertently or maliciously impact the execution of another.

This third task is the problem of ensuring the security of all processes run on the system. Ensuring the secure execution of all processes depends on the correct implementation of resource and scheduling mechanisms. Security becomes an issue because processes in modern computer systems interact in a variety of ways, and the sharing of data among users is a fundamental use of computer systems. First, the output of one process may be used by other processes. Second With the ubiquity of Internet-scale sharing mechanisms, such as e-mail, the web, and instant messaging, users may share anything with anyone in the world

The challenge in developing operating systems security is to design security mechanisms that protect process execution and their generated data in an environment with such complex interactions. The current state of operating systems security takes two forms: (1) constrained systems that can enforce security goals with a high degree of assurance and (2) general-purpose systems that can enforce limited security goals with a low to medium degree of assurance.

Security Goal A secure operating system provides security mechanisms that ensure that the system's security goals are enforced despite the threats faced by the system. Systems that provide a high degree of assurance in enforcement have been called secure systems, or even more frequently “trusted” Systems. However, it is also true that no system of modern complexity is completely secure. A security goal defines the operations that can be executed by a system while still preventing unauthorized access. Security goals describe how the system implements accesses to system resources that satisfy the following: 1. secrecy, 2. integrity, 3. and availability.

An example of an functional security goal is the principle of least privilege, which limits a process to only the set of operations necessary for its execution. To build any secure system requires that we consider how the system achieves its security goals under a set of threats (i.e., a threat model) and given a set of software, including the security mechanisms, that must be trusted(i.e., a trust model). TRUSTMODEL A system’s trust model defines the set of software and data upon which the system depends for correct enforcement of system security goals. For example, the operating system depends on a variety of programs to authenticate the identity of users (e.g., login and SSH).

of data that the user depends on. Threat Model A threat model defines a set of operations that an attacker may use to compromise a system. If an attacker is able to find a vulnerability in the system that provides access to secret information (i.e., violate secrecy goals) or permits the modification of information that subjects depend on (i.e.,violate integrity goals), then the attacker is said to have compromised the system. This threat model exposes a fundamental weakness in commercial operating systems they assume that all software running on behalf of a subject is trusted by that subject. This can result in the leakage of that user’s secrets and the modification of data that the user depends on.

Access Control An access enforcement mechanism authorizes requests from multiple subjects (e.g. users, processes, etc.) to perform operations (e.g., read, write, etc.) on objects (e.g., files, sockets, etc.). An operating system provides an access enforcement mechanism. Two fundamental concepts of access control: a protection system that defines the access control specification and a reference monitor that is the system’s access enforcement mechanism that enforces this specification.

Protection system A protection system consists of a protection state, which describes the operations that system subjects can perform on system objects, and a set of protection state operations, which enable modification of that state. A protection system enables the definition and management of a protection state. A protection state consists of the specific system subjects, the specific system objects, and the operations that those subjects can perform on those objects. The access matrix is used to define the protection domain of a process.

Mandatory protection system Problems with access matrix Untrusted processes can tamper with the protection system. A protection system that permits untrusted processes to modify the protection state is called a discretionary access control (DAC) system. Mandatory protection system A mandatory protection system is a protection system that can only be modified by trusted administrators via trusted software, consisting of the following state representations: A mandatory protection state is a protection state where subjects and objects are represented by labels where the state describes the operations that subject labels may take upon object labels; A labelling state for mapping processes and system resource objects to labels; A transition state that describes the legal ways that processes and system resource objects may be relabeled.

Mandatory access control A label is simply an abstract identifier—the assignment of permissions to a label defines its security semantics. Labels are tamperproof . Trusted administrators define the access matrix’s labels and set the operations that subjects of particular labels can perform on objects of particular labels. Such protection systems are mandatory access control (MAC) systems because the protection system is immutable to untrusted processes.

A reference monitor is the classical access enforcement mechanism A reference monitor is the classical access enforcement mechanism. It takes a request as input, and returns a binary response Indicating whether the request is authorized by the reference monitor’s access control policy. We identify three distinct components of a reference monitor: (1) its interface -The interface defines where the authorization module needs to be invoked to perform an authorization query to the protection state, a labeling query to the labeling state, or a transition query to the transition state. (2) its authorization module-determines the exact queries that are to be made to the policy store. ; (3) Its policy store-The policy store responds to authorization, labeling, and transition queries based on the protection system that it maintains..

Concluding Remarks A secure operating system is an operating system where its access enforcement satisfies the reference monitor concept The reference monitor concept defines the necessary and sufficient properties of any system that securely enforces a mandatory protection system, consisting of three guarantees: 1. Complete Mediation: The system ensures that its access enforcement mechanism mediates all security-sensitive operations. 2. Tamperproof: The system ensures that its access enforcement mechanism, including its protection system, cannot be modified by untrusted processes. 3. Verifiable:The access enforcement mechanism, including its protection system,“must be small enough to be subject to analysis and tests, the completeness of which can be assured” . That is, we must be able to prove that the system enforces its security goal correctly.