1. 1 Model Checking One Million Lines of C Code Hao Chen, UC Berkeley Drew Dean, SRI International David Wagner, UC Berkeley

2. 2 MOPS (MOdel checking Programs for Security properties) A static analysis tool that checks source programs for temporal safety properties. e.g. a setuid-root program must drop privilege before making risky system calls. Analysis –Pushdown model checking –Inter-procedural –Control flow centric

3. 3 The MOPS process Parser Model Checker C Program Safety Property CFG FSA Program satisifes safety property Error Traces FSA: finite state automaton CFG: control flow graph Treat the model checker as a black box for this talk

4. 4 Is software model checking ready for prime time? Can model checking be used by open source developers to find security vulnerabilities? Criteria for a successful tool –It is useful Can check many properties Can check diverse, widely-deployed programs Requires moderate computational resources –It is usable Can be used easily by non-tool developers Can generate comprehensible error reports

5. 5 Outline Experiment –Programs: 8 widely-deployed programs, with over 1 million LOC –Properties: 5 security-related properties Findings –More than a dozen vulnerabilities and weaknesses Usability improvements Conclusion

6. 6 Programs ProgramLines of Code (LOC) Apache HTTPD 2.0.40-21229K At 3.1.8-336K BIND 9.2.1-16279K OpenSSH 3.5p1-659K Postfix 1.1.11-1194K Samba 2.2.7a-7.9.0254K Sendmail 8.12.8-4222K VixieCron 3.0.1-744K Total1147K

7. 7 Security properties Drop privilege completely when needed Avoid stderr vulnerability Avoid race condition (TOCTTOU) Create chroot-jail safely –chdir(“/”) must follow chroot() immediately Create temporary files safely –Use only the safe function mkstemp() –Never reuse filename in mkstemp(filename)

8. 8 Property: drop privilege completely Setuid-root programs should drop root privilege completely –before executing an untrusted program via system(), popen(), execvp() and friends, or –when the program intends to do so Otherwise, the remaining privilege may be exploited by –the untrusted program that is executed –malicious code injected via buffer overrun attacks

9. 9 Vulnerability: fail to drop privilege completely seteuid(getuid()); setuid(getuid()); … execlp(askpass, askpass, msg, (char *) 0); … OpenSSH client (in readpass.c)

10. 10 What is wrong? R≠0, E=S=0 OpenSSH 3.5 on Linux R=E≠0, S=0 seteuid(getuid()) setuid(getuid()) R≠0, E=S=0 OpenSSH 3.5 on OpenBSD R=E≠0, S=0 R=E=S≠0 seteuid(getuid()) setuid(getuid()) R≠0, E=S=0 OpenSSH 2.5.2 on Linux R=E=S≠0 setuid(getuid())

11. 11 Potential Vulnerability Weaknesses –ssh: fails to drop privilege before executing a user program –ssh-keysign: fails to drop privilege before doing complex cryptographic operations A buffer overrun would allow the attacker to regain root privilege in euid.

12. 12 Property: drop privilege completely PackageLOCRunning Time # Error Traces Real BugsTotal Sendmail222K0:1200 Postfix94K0:1702 OpenSSH59K0:2328 Apache229K0:4514 BIND279K0:5301 At6K0:0500 Cron4K0:0500 Samba254K1:5305

13. 13 Vulnerability: stderr exploits in at attack.c at.c Code Standard File Descriptors stdinstdoutstderr close(1); close(2); execl(“at”, …); open(LFILE, O_WRONLY); fd = open(atfile, O_CREAT); ttyttytty tty ttyLFILE ttyLFILEatfile Rule: No setuid-root program may open a file for writing to stderr

14. 14 Property: stderr vulnerability PackageLOCRunning Time # Error Traces Real BugsTotal Sendmail222K14:1203 Postfix94K0:4601 OpenSSH59K0:5812 Apache229K0:1411 BIND279K0:0000 At6K0:0411 Cron4K0:0522 Samba254K0:5811

15. 15 Summary of Findings ProgramErrors (All Properties) RealTotal Apache HTTPD26 At17 BIND04 OpenSSH524 Postfix06 Samba28 Sendmail011 VixieCron34 Total1370

16. 16 Outline Experiment –Programs: 8 widely-deployed programs, with over 1 million LOC –Properties: 5 security-related properties Findings –More than a dozen vulnerabilities and weaknesses Usability improvements Conclusion

17. 17 Usability improvement 1: Make it really easy to run! Problems –Packages have different build processes –Tool has to be manually configured for each package Solution –Provide a script that integrates model checking into the build processes of packages automatically –Result: allow the user to run the tool as simple as mops –m setuid.fsa openssh-3.5p1-6.src.rpm

18. 18 Integrating MOPS into Software Build Processes 1st attempt: manually edit Makefiles –Too complicated; does not survive autoconf 2nd attempt: setenv GCC_EXEC_PREFIX to run MOPS instead of gcc –Build processes generate & run code 3rd attempt: build CFG & machine code –Dangling CFGs; links to object files broken 4th attempt: Put CFGs into ELF files –Solves all identified problems!

19. 19 Usability improvement 2: report comprehensible error messages Problem –One bug may trigger many error traces –The user has to review all the traces manually Criteria for good error trace reporting –Reporting one error trace per bug –Reporting shortest error traces

20. 20 Algorithm 1.Find the shortest error trace t and output it 2.Find the crucial statement s on t, i.e. the first statement that causes an error on t 3.Prune s from the program 4.If the program still has error traces, go to step 1

21. 21 Criteria for good tools: revisited It is useful –Can check many properties –Can check diverse, widely-deployed programs –Requires moderate computational resources It is usable –Can be used easily by non-tool developers –Can generate comprehensible error reports

22. 22 Conclusion Model checking is ready for prime time use by open source developers to find security vulnerabilities! We believe that our experience would transfer to other similar tools as well. Work in progress: check all 839 RPM packages in RedHat Linux 9