1. 1 setuid Demystified -- Examining the API of Security Operation in OS using Formal Models Hao Chen, David Wagner UC Berkeley Drew Dean SRI International

2. 2 Objective Understand the semantics of security operation API in OS precisely Applications –Using these system calls properly in programs –Verifying their documentations –Detecting inconsistency in OS kernels –Building security properties and checking them in programs automatically (e.g. by modelchecker)

3. 3 What is setuid Access control in Unix is based the User ID model Each process has 3 user Ids: –Real uid (ruid) –Effective uid (euid) –Saved uid (suid) Uid-setting system calls –setuid() seteuid() setreuid() setresuid()

4. 4 The setuid Mystery Uid-setting system calls are a semantic mess –Counter-intuitive semantics –Subtle differences among different calls –Incompatible semantics of the same call in different Unix systems (e.g.: Linux, Solaris, FreeBSD) –Incomplete, inaccurate, or even wrong documentation Reason: historical artifacts

5. 5 Solution: Formal Model Use a formal model to describe the user ID model Build an FSA where –The states describe the user IDs of a process –The transitions describe the semantics of the uid-setting system calls

6. 6 Determine the States of the FSA Each state is a tuple (ruid, euid, suid) The range of user ID values determines the number of states Example: –A process switches between a privileged user ID and an unprivileged ID –2 user IDs: 0(root), x(non-root) –8 states

7. 7 Problem: Difficult to Determine Transitions Large number of transitions. E.g. –Range of user ID values: {0, x} where x!=0 –Number of states: 8 –Number of transitions per state: setuid(uid): 2 transitions seteuid(euid): 2 transitions setreuid(ruid, euid): 4 transitions setresuid(ruid, euid, suid): 8 transitions –Total transitions: 8*(2+2+4+8)=128 A laborious, error-prone process

8. 8 Determine Transitions Automatically by Simulation Idea: Exhaustively make all system calls at each state For each state s=(ruid, euid, suid) where ruid, euid, suid  {0, uid 1, uid 2, …} For each system call c  {setuid(e), seteuid(e), setreuid(r,e), setresuid(r,e,s)} { Make the system call c in the state s Observe the ensuing state s’ Add the transition }

9. 9 FSA for setuid() in FreeBSD

10. 10 FSA for setuid() in Linux

11. 11 FSA for setreuid() in Linux

12. 12 FSA for setresuid() in Linux

13. 13 Benefits Correctness: the FSA reflects what programs experience Efficiency: the automatic method is portable to –Different Unix systems –Different kernel versions

14. 14 Application: Understanding the semantics of the system calls Find subtle semantic differences –Among different uid-setting system calls –Among the same system call on different Unix systems Find surprising, counter-intuitive semantics

15. 15 Application: Verifying Man Pages Incompete man page –Man page for setuid() in Linux fails to mention capabilities which affect how setuid() behaves Wrong man pages –FreeBSD 4.4: Unprivileged users may change the ruid to the euid and vice versa –Redhat Linux 7.2: The setgid function checks if the egid of the caller and if it is the superuser, …

16. 16 Application: Detecting Inconsistency in OS Kernel Linux has fsuid –Used for filesystem permission checking –Normally follows euid Invariant in Linux 2.4.18 ( kernel/sys.c ) –fsuid is 0 only if at least one of ruid, euid, suid is 0 Rationale –ensuring that an fsuid-unware cross-platform application can automatically drop root privilege in fsuid by dropping it in ruid, euid, suid

17. 17 Application: Detecting Inconsistency in OS Kernel (cont) A bug breaks the invariant: –The invariant is satisfied in setuid(), seteuid(), setreuid() –But it is broken in setresuid() We found the bug using the simulator The bug has been confirmed by Linus and Alan and will be fixed using our patch.

18. 18 Application: Checking Proper Usage of Syscalls in Programs Modelchecking security properties in programs –Model a program as a PDA –Intersect the PDA (program) with the FSA of uid-setting system calls to get a new PDA –Check reachability of risky states in the new PDA Can answer questions like: –Can a uid-setting system call fail in this program? –Can this program fail to drop privilege? –Which part of this program run with privilege? Result: Found known bugs in sendmail 8.10.1 and 8.12.0

19. 19 Conclusion Formal models are useful in –Understanding the APIs of security operations –Verifying their documentations –Detecting inconsistency in OS kernels –Checking proper usage of security-relevant APIs in programs