Why Security Testing Is Hard Herbert H. Thompson Presenter: Alicia Young

Introduction Software Testing good at verifying requirements Software Testing good at verifying requirements UML helps move from specification to test cases UML helps move from specification to test cases Several bugs routinely escape testing Several bugs routinely escape testing Not specification Violations Not specification Violations Would escape most automated testing Would escape most automated testing Examine Security bugs to discover why testing can be difficult Examine Security bugs to discover why testing can be difficult

Side-Effect Behavior Input A -> result B Input A -> result B What if Input A also resulted in C? What if Input A also resulted in C? Overt – unexpected dialog box appears Overt – unexpected dialog box appears Subtle – writing a file or opening a network port Subtle – writing a file or opening a network port RDISK utility for Windows RDISK utility for Windows Creates an emergency Repair Disk Creates an emergency Repair Disk Temporary file created with Universal Permissions Temporary file created with Universal Permissions During testing, product responds as specified During testing, product responds as specified

Intended vs. Implemented

The State of Security Testing Exploit Libraries (Librarian Method) Exploit Libraries (Librarian Method) New Products tested with only this library New Products tested with only this library Finds old vulnerabilities with no hope of finding anything new Finds old vulnerabilities with no hope of finding anything new Problem is…this strategy actually works! Problem is…this strategy actually works! Developers repeatedly make the same mistakes Developers repeatedly make the same mistakes Current software is really buggy Current software is really buggy Applications will eventually become immune to these test cases Applications will eventually become immune to these test cases

The Need for Techniques Test like detectives Test like detectives Past bugs teach us how vulnerabilities get into our applications Past bugs teach us how vulnerabilities get into our applications The key is to learn new techniques of finding bugs The key is to learn new techniques of finding bugs Four General Classes of testing techniques Four General Classes of testing techniques Dependencies Dependencies Unanticipated user input Unanticipated user input Techniques to expose Design Vulnerabilities Techniques to expose Design Vulnerabilities Techniques to expose implementation vulnerabilities Techniques to expose implementation vulnerabilities

Dependency Insecurities and Failures Software resides in co-dependent environment Software resides in co-dependent environment Two Security Concerns Two Security Concerns Application may inherit insecurities Application may inherit insecurities External security service resource may fail External security service resource may fail Internet Explorer’s Content Advisor Internet Explorer’s Content Advisor Content advisor password protects classes of sites Content advisor password protects classes of sites If the library fails to load, Internet explorer permits access to any previously blocked site If the library fails to load, Internet explorer permits access to any previously blocked site

Cause of Dependency Failures Severely under-applied inputs to software Severely under-applied inputs to software Error handling code gets little testing scrutiny Error handling code gets little testing scrutiny These types of failures need to be examined These types of failures need to be examined

Unanticipated User Input Inputs that cause undesirable side effects and require special testing Inputs that cause undesirable side effects and require special testing Reserved words Reserved words Escape characters Escape characters Long strings Long strings Boundary values Boundary values Most well known side-effect: Buffer Overflow Most well known side-effect: Buffer Overflow Input that can be interpreted as commands Input that can be interpreted as commands

Design Insecurities Many Security Vulnerabilities designed into application Many Security Vulnerabilities designed into application Seeing high-level impact on an application or host is difficult Seeing high-level impact on an application or host is difficult Test Instrumentation Test Instrumentation Many applications shipped with it Many applications shipped with it Bypassing security controls for ease of testing Bypassing security controls for ease of testing Ports left open Ports left open Insecure default values and configurations Insecure default values and configurations

Implementation Insecurities Perfect design means nothing if Implementation is flawed Perfect design means nothing if Implementation is flawed Man-in-the-middle attack Man-in-the-middle attack Attacker gets between time application checks security and when the application uses information Attacker gets between time application checks security and when the application uses information Xterm – can be exploited to allow a restricted user to append data to the password file Xterm – can be exploited to allow a restricted user to append data to the password file

Standard Bug-Severity Rankings Urgent Urgent System crash, Unrecoverable data loss, jeopardizes personnel System crash, Unrecoverable data loss, jeopardizes personnel High High Impairment of critical system functions and no work-around exists Impairment of critical system functions and no work-around exists Medium Medium Impairment of critical system functions and work-around exists Impairment of critical system functions and work-around exists Low Low Inconvenience, annoyance Inconvenience, annoyance None None None of the above or an enhancement None of the above or an enhancement

The Need For Tools Testers generally rewarded for both quantity and severity of bugs Testers generally rewarded for both quantity and severity of bugs Side-effect bugs may not get noticed or even dismissed by managers Side-effect bugs may not get noticed or even dismissed by managers Equipped with proper tools testers would notice odd behavior Equipped with proper tools testers would notice odd behavior Writing of a temporary file Writing of a temporary file Sending of extra network packets Sending of extra network packets

New Tools Regmon and Filemon – monitor application interactions with registry and file system Regmon and Filemon – monitor application interactions with registry and file system App-Sight – monitors environmental interactions App-Sight – monitors environmental interactions Holodeck – Fine grain control over interactions between application and environment Holodeck – Fine grain control over interactions between application and environment

Paper Analysis Quality Software is Secure Software Quality Software is Secure Software Important points made Important points made Better testing techniques Better testing techniques Better testing tools Better testing tools Design concerns Design concerns

Questions?