1. Application Communities Phase II Technical Progress, Instrumentation, System Design, Plans March 10, 2009

2. 2 Application Communities Phase 2 Instrumentation

3. 3 March 10, 2009Application Communities Phase 2 Instrumentation Firefox is run under PIN tool PIN allows us to insert call-outs into running executable Allows us to execute our code before or after executable’s code Can be targeted to specific system calls CreateFileW Can be targeted to general events Function calls Function returns

4. 4 March 10, 2009Application Communities Phase 2 Call Stack Instrumentation Determine stack at particular program points Gather valid stack traces during normal operation Detect invalid stack traces during attack

5. 5 March 10, 2009Application Communities Phase 2 Backtrace and Shadow Stack Shadow Stack Each function call and return is monitored Tricky bits: (Setjmp, longjmp) Maintains mirror of stack during execution Overhead associated with each call and return Stack accuracy is ensured Backtrace When system call is reached, walk back up stack to determine stack information Only system calls are instrumented so overhead is less Compiler optimizations can hide stack information

6. 6 March 10, 2009Application Communities Phase 2 Stack Trace Example funccall:[0] c:\AC2\firefox_bin\searchplugins\google.gif 54: CreateFileW[0x10760] C:\WINDOWS\system32\kernel32.dll [S] 53: CreateFileA[0x1a24] C:\WINDOWS\system32\kernel32.dll [S] 52: __PR_MD_OPEN[0x25920] c:\AC2\firefox_bin\nspr4.dll [S] 51: _PR_Open[0x1c8d0] c:\AC2\firefox_bin\nspr4.dll [S] 50: nsLocalFile::OpenNSPRFileDesc[0x39c90] c:\AC2\firefox_bin\xpcom.dll [S] 49: nsFileInputStream::Open[0xa2f0] c:\AC2\firefox_bin\components\necko.dll [S] 48: nsFileInputStream::Init[0xa440] c:\AC2\firefox_bin\components\necko.dll [S] 47: NS_NewLocalFileInputStream[0x7b750] c:\AC2\firefox_bin\components\necko.dll [S] 46: nsFileChannel::EnsureStream[0x7fe20] c:\AC2\firefox_bin\components\necko.dll [S] 45: nsFileChannel::AsyncOpen[0x80bb0] c:\AC2\firefox_bin\components\necko.dll [S] 44: imgLoader::LoadImage[0x5520] c:\AC2\firefox_bin\components\imglib2.dll [S] 43: nsContentUtils::LoadImage[0x138600] c:\AC2\firefox_bin\components\gklayout.dll [S] 42: nsImageBoxFrame::UpdateImage[0x10cdc0] c:\AC2\firefox_bin\components\gklayout.dll [S] 40: nsImageBoxFrame::AttributeChanged[0x10c790] c:\AC2\firefox_bin\components\gklayout.dll [S] 39: nsCSSFrameConstructor::AttributeChanged[0x8fcb0] c:\AC2\firefox_bin\components\gklayout.dll [S] 38: PresShell::AttributeChanged[0x334e0] c:\AC2\firefox_bin\components\gklayout.dll [S] 37: nsXULDocument::AttributeChanged[0x2c0f60] c:\AC2\firefox_bin\components\gklayout.dll [S] 36: nsXULElement::SetAttrAndNotify[0x3587d0] c:\AC2\firefox_bin\components\gklayout.dll [S] 35: nsXULElement::SetAttr[0x358500] c:\AC2\firefox_bin\components\gklayout.dll [S] 34: nsIContent::SetAttr[0x33230] c:\AC2\firefox_bin\components\gklayout.dll [S] 33: nsXBLPrototypeBinding::AttributeChanged[0x2ab860] c:\AC2\firefox_bin\components\gklayout.dll [S] 32: nsXBLBinding::AttributeChanged[0x2a8e10] c:\AC2\firefox_bin\components\gklayout.dll [S] 31: nsXULElement::SetAttrAndNotify[0x3587d0] c:\AC2\firefox_bin\components\gklayout.dll [S] 30: nsXULElement::SetAttr[0x358500] c:\AC2\firefox_bin\components\gklayout.dll [S] 29: nsIContent::SetAttr[0x33230] c:\AC2\firefox_bin\components\gklayout.dll [S] 28: nsXBLPrototypeBinding::AttributeChanged[0x2ab860] c:\AC2\firefox_bin\components\gklayout.dll [S] 27: nsXBLBinding::AttributeChanged[0x2a8e10] c:\AC2\firefox_bin\components\gklayout.dll [S] 26: nsXULElement::SetAttrAndNotify[0x3587d0] c:\AC2\firefox_bin\components\gklayout.dll [S] 24: nsXULElement::SetAttr[0x3550f0] c:\AC2\firefox_bin\components\gklayout.dll [S] 23: nsXULElement::SetAttribute[0x354fd0] c:\AC2\firefox_bin\components\gklayout.dll [S] 22: XPTC_InvokeByIndex[0x8e2b0] c:\AC2\firefox_bin\xpcom.dll [S] 9: nsJSContext::CallEventHandler[0x325830] c:\AC2\firefox_bin\components\gklayout.dll [S] 8: GlobalWindowImpl::RunTimeout[0x314310] c:\AC2\firefox_bin\components\gklayout.dll [S] 7: GlobalWindowImpl::TimerCallback[0x315180] c:\AC2\firefox_bin\components\gklayout.dll [S] 6: nsTimerImpl::Fire[0x6d050] c:\AC2\firefox_bin\xpcom.dll [S] 5: nsTimerManager::FireNextIdleTimer[0x6dad0] c:\AC2\firefox_bin\xpcom.dll [S] 4: nsAppShell::Run[0x14cd0] c:\AC2\firefox_bin\components\gkwidget.dll [S] 2: xre_main[0x2f30] c:\AC2\firefox_bin\firefox.exe [S] 0: mainCRTStartup[0x16d20] c:\AC2\firefox_bin\firefox.exe [S]

7. 7 March 10, 2009Application Communities Phase 2 Detect Exploit with Stack Traces Gather set of call stacks for a valid execution Confirm subsequent valid executions do not produce new stacks Gather set of call stacks when exploit is run Compare two sets to determine if exploit can be detected with stack traces only Exploit produced 8 different call stacks In this example, exploit writes file “c:\booom.bat”

8. 8 March 10, 2009Application Communities Phase 2 Exploit is Detected! funccall:[0] c:\booom.bat -------------------------------------- 43: CreateFileW[0x10760] C:\WINDOWS\system32\kernel32.dll [S] 42: CreateFileA[0x1a24] C:\WINDOWS\system32\kernel32.dll [S] 41: __PR_MD_OPEN[0x25920] c:\AC2\firefox_bin\nspr4.dll [S] 40: _PR_Open[0x1c8d0] c:\AC2\firefox_bin\nspr4.dll [S] 39: nsLocalFile::Create[0x39e20] c:\AC2\firefox_bin\xpcom.dll [S] 38: nsLocalFile::CreateUnique[0x42100] c:\AC2\firefox_bin\xpcom.dll [S] 37: XPTC_InvokeByIndex[0x8e2b0] c:\AC2\firefox_bin\xpcom.dll [S] 30: nsJSContext::EvaluateString[0x324b50] c:\AC2\firefox_bin\components\gklayout.dll [S] 29: nsJSThunk::EvaluateScript[0x3366c0] c:\AC2\firefox_bin\components\gklayout.dll [S] 28: nsJSChannel::InternalOpen[0x337ed0] c:\AC2\firefox_bin\components\gklayout.dll [S] 27: nsJSChannel::AsyncOpen[0x337eb0] c:\AC2\firefox_bin\components\gklayout.dll [S] 26: nsDocumentOpenInfo::Open[0x35cf0] c:\AC2\firefox_bin\components\docshell.dll [S] 25: nsURILoader::OpenURI[0x37d40] c:\AC2\firefox_bin\components\docshell.dll [S] 24: nsDocShell::DoChannelLoad[0x23740] c:\AC2\firefox_bin\components\docshell.dll [S] 23: nsDocShell::DoURILoad[0x226a0] c:\AC2\firefox_bin\components\docshell.dll [S] 22: nsDocShell::InternalLoad[0x21360] c:\AC2\firefox_bin\components\docshell.dll [S] 21: nsDocShell::LoadURI[0x12370] c:\AC2\firefox_bin\components\docshell.dll [S] 20: nsWindowWatcher::OpenWindowJS[0x2af0] c:\AC2\firefox_bin\components\embedcomponents.dll [S] 19: GlobalWindowImpl::OpenInternal[0x312b50] c:\AC2\firefox_bin\components\gklayout.dll [S] 18: GlobalWindowImpl::OpenDialog[0x30f080] c:\AC2\firefox_bin\components\gklayout.dll [S] 17: XPTC_InvokeByIndex[0x8e2b0] c:\AC2\firefox_bin\xpcom.dll [S] 9: nsJSContext::CallEventHandler[0x325830] c:\AC2\firefox_bin\components\gklayout.dll [S] 8: GlobalWindowImpl::RunTimeout[0x314310] c:\AC2\firefox_bin\components\gklayout.dll [S] 7: GlobalWindowImpl::TimerCallback[0x315180] c:\AC2\firefox_bin\components\gklayout.dll [S] 6: nsTimerImpl::Fire[0x6d050] c:\AC2\firefox_bin\xpcom.dll [S] 5: nsTimerManager::FireNextIdleTimer[0x6dad0] c:\AC2\firefox_bin\xpcom.dll [S] 4: nsAppShell::Run[0x14cd0] c:\AC2\firefox_bin\components\gkwidget.dll [S] 3:.text[0x2739] c:\AC2\firefox_bin\components\appshell.dll [U] 2: xre_main[0x2f30] c:\AC2\firefox_bin\firefox.exe [S] 1:.text[0x1005] c:\AC2\firefox_bin\firefox.exe [U] 0: mainCRTStartup[0x16d20] c:\AC2\firefox_bin\firefox.exe [S]

9. 9 March 10, 2009Application Communities Phase 2 Callstack Tree

10. 10 March 10, 2009Application Communities Phase 2 Conclusions: Instrumentation Able to gather stack traces by instrumenting executables Stack traces can detect exploits

11. 11 March 10, 2009Application Communities Phase 2 System Design

12. 12 March 10, 2009Application Communities Phase 2 AC2 Deployment approach ACServer RMI Registry Linux server Invariants Attacks App Records Patches ACClientService Windows Host N Windows Host 1 1. bind 2. Lookup (find server) Socket messages 3. registerService (give self as callback) RMI Socket Call ACServer and ACClientService are implemented in Java Firefox instrumentation Firefox instrumentation Keystore Truststore Keystore Truststore SSL handshake

13. 13 March 10, 2009Application Communities Phase 2 Design features Java implementation of ACServer and ACClientService for cross- platform deployment RMI is a high performance, secure middleware technology for a Java- based distributed application Point-to-point rather than message based Easy to deploy and debug High throughput Socket-based interface to instrumented applications for generality Standards-based (POSIX), platform neutral

14. 14 March 10, 2009Application Communities Phase 2 Security considerations Use Secure Sockets Layer (SSL)/Transport Layer Security (TLS) protocols (supported by Java 1.5) instead of Java Remote Method Protocol (JRMP) to secure communication between client and server javax.rmi.ssl.SslRMIClientSocketFactory and javax.rmi.ssl.SslRMIServerSocketFactory Will use keytool to generate key and trust stores Will pre-deploy security information ACClientService will be configured to accept communications only from the local host An outside attacker will not be able to hijack ACClientService unless the host is irretrievably compromised For connection between applications and ACClientService we will consider TCP which is completely portable and securable Named pipes, more efficient and easier to secure

15. 15 March 10, 2009Application Communities Phase 2 ACServer and ACClientService Interfaces RMI

16. 16 March 10, 2009Application Communities Phase 2 Application Startup Sequence Diagram

17. 17 March 10, 2009Application Communities Phase 2 RMI Transport Object Structure (1/2)

18. 18 March 10, 2009Application Communities Phase 2 RMI Transport Object Structure (2/2)

19. 19 March 10, 2009Application Communities Phase 2 Progress Summary And Plans

20. 20 March 10, 2009Application Communities Phase 2 Instrumentation Initial shadow stack on Unix and Windows using Pin Implemented using callouts Could inline for improved performance Initial backtrace on Unix and Windows using Pin Faster, but requires re-building the binary Compared the two approaches for testing Stack comparison tools for detecting exploits Visual tree representation of stack sets Evaluated Apache, insufficient number of in-scope exploits Reproduced several … Evaluated Firefox, many in-scope, explored details of many Investigated GUI-replay to recreate traces for learning Because many exploits require some user interaction Advanced Key and Mouse Recorder ($60/user, 5 users) http://mouse-recorder.macro-expert.com/recorder-features.htm Progress to date – Instrumentation & Exploits

21. 21 March 10, 2009Application Communities Phase 2 Progress to date – Implementation and Infrastructure AC2 designed as a distributed system ACServer interacts with AClientService nodes, deployed different workstations Each ACClientService node interacts with applications running on local host Designed the AC2 Environment so that it can be tested in isolation by running mock application instances that execute configurable attack scenarios Status: Scenario-driven simulated application can send messages to ACClientService, which passes messages to ACServer, receives responses from the server and transmits the responses to the simulated application.

22. 22 March 10, 2009Application Communities Phase 2 Plans – Infrastructure/Implementation Implement socket-based messages between applications and ACClientService Complete the design and implementation of the RMI transportable objects between ACServer and ACClientService Incrementally add functionality to ACServer Add detailed debug scenarios to thoroughly test the infrastructure with a covering set of use cases Integrate with instrumented Firefox Flesh out learning infrastructure – design & implementation. Create demonstration for next program review Attack on Firefox Response based on default repair to abnormal stack trace or context Prevent a system call and return error code Elaborate this demo by reporting invariant violation and receiving repairs