Closing the Gap: Analyzing the Limitations of Web Application Vulnerability Scanners David Shelly Randy Marchany Joseph Tront Virginia Polytechnic Institute and State University

Purpose 2 To analyze the limitations of web application scanners by using both a secure and insecure version of a custom-built web application Identify scanner weaknesses Improve scanner detection techniques Reduce false-negatives and false-positives

Problem > 60% of attacks seen on Internet aimed at web applications [SANS 2009] Web application vulnerability scanners are not capable of detecting all of the vulnerabilities and attack vectors that exist Numerous false-negatives and false- positives False sense of web application security Web application flaws remain unpatched

Relevant Vulnerabilities SQL Injection Cross-Site Scripting (XSS) Session Management Flaws

Web Application Scanners

Related Work Evaluation Applications WebMaven’s BuggyBank Foundstone’s Hacme Bank, Hacme Shipping, Hacme Travel Evaluation Techniques Scan publicly available web applications Benchmark system

Related Work Evaluation Applications WebMaven’s BuggyBank Foundstone’s Hacme Bank, Hacme Shipping, Hacme Travel Evaluation Techniques Scan publicly available web applications Benchmark system

Test Bed Design

Client-Side Features Regular users Submit, assess, purchase, and categorize products Password reminder Register new user Edit account settings Administrator users Add other administrators Approve/deny uploaded templates and item categorizations View validated templates

Server-Side Features Database Users table (10 regular, 1 administrator) Categories table (12 static, 1 custom) Categorize table (15 uncategorized, 5 categorized) Items table (10 available) Templates 5 validated item templates 5 awaiting approval

Vulnerabilities Implemented SQL Injection Form input (12) Cookie Variable (8) Session Management Predictable session ID (1) Unprotected variables (9) Cross-Site Scripting Reflected (10) Stored (6) DOM-based (1)

SQL Injection Example

XSS Injection Example

Session Management Example

Methodology CentOS 4.4 MySQL 4.1 Apache 2.0 PHP Most popular web server technologies:

Methodology Controlled benchmark environment: Controlled variables Web server technologies Independent variables Number of deliberate vulnerabilities Dependent variables Observed false positives and false negatives

Methodology Black-box and White-box Analysis: Black-box Analysis Perspective of client-side user Manipulates user supplied input White-box Analysis Source code is available Verify actual number of vulnerabilities

Testing Approach Phases: 1. Initialization 3. Classification 2. Execution4. Analysis

Initialization 1. Restore the MySQL database to its original state 2. Delete all client side and server side cookies 3. Restore the entire web server directory with a clean backup 4. Restart the web server

Execution 1. Configure the web application vulnerability scanner (primarily default settings) 2. Start Wireshark packet capturing 3. Execute the web application vulnerability scan 4. Stop Wireshark packet capturing and save the trace 5. Save the contents of the database, cookie files, and scanner results

Classification False Positives: 1. The scan result is due to an application robustness problem (error page, format exception, etc.) and not a vulnerability 2. Normal operation of the web application results in the same error/problem 3. Source code analysis determines scan result is incorrect 4. The results duplicate a vulnerability that has already been accounted for Note: Repeated input forms that are displayed recursively on a web page are not considered duplicate vulnerabilities if they are detected multiple times by a web application scanner (e.g. user comment boxes)

Analysis HTTP requests and responses Client and server cookies Database copies Scanner result reports

SQL Injection Results – Form Inputs

SQL Injection Results – Cookie Variables

XSS Injection Results - Reflected

XSS Injection Results - Stored

XSS Injection Results – DOM-based

Session Management Results – Predictable SID

Session Management Results – Insecure Cookie Variables

False Positives – SQL Injection

False Positives – XSS Injection

Analysis – SQL Injection False Negatives Unable to recognize reflected SQL error in response page Required fields not completed (ex: register and log-in forms need more than one input) Cookie variables not tested for injection Improvements Flag any error or warning response as suspected vulnerability (information disclosure) All possible combinations of form inputs should be tested Add cookie variables to list of parameters to be checked

Analysis – SQL Injection False Positives Duplicate entries reported Tested both GET and POST methods Multiple exploit strings for same vulnerability instance Blind SQL injection returned the same HTML page Improvements Only report the implemented HTTP method (unless both methods are supported) Verify vulnerability using multiple exploit scenarios, but only report SQL injection vulnerability once Blind SQL injection tests should use time delay statements

Analysis – XSS Injection False Negatives Required fields not completed Did not test all possible user supplied input (DOM variable, cookie variables, and uploadable files) Stored XSS detected through reflected techniques (relies on SQL error pages) Improvements Attempt every combination of form inputs Test all parameters (even search embedded scripts) Use unique injection strings and then crawl web application multiple times searching for string

Analysis – XSS Injection False Positives Duplicate entries reported Same exploit string with different set of parameters (causes request to appear different) Improvements Same as SQL injection Verify vulnerability using multiple exploit scenarios, but only report one XSS vulnerability per input form

Analysis – Session Management False Negatives Did not attempt to manipulate cookie to perform session hijacking (only tested for information disclosure) Most determined cookie not set as HttpOnly Only tested for session management vulnerabilities using information from initial connection Improvements Tamper with all unprotected session variables throughout the application (check if response page different) Analyze SIDs for strong algorithm

Analysis – Other Findings Possible explanations Overload of requests Server not producing proper response pages Scanners not handling all of server responses Failures in spidering techniques Scanners missing parameters and links while crawling Did not test initial log-in page Improvements Monitor current connection speed and throttle number of simultaneous connections Test all pages and variables within scope of web application

Conclusions Scanners detect simple forms of reflected XSS and SQL injection relatively well Work still needs to be done to detect non-traditional instances of these vulnerabilities Most false-positives observed from insecure version of web application Expected secure version to produce false-positives Did not test enough of the input parameters Better to use multiple web application scanners Combined results better than any one individually

Future Work Expand to include analysis of other web application technologies and server configurations Web 2.0 technologies (Ajax, Flash, Ruby, Python) Develop black-box web application scanner Using proposed techniques

Questions security.vt.edu