1. Closing the Gap: Analyzing the Limitations of Web Application Vulnerability Scanners David Shelly Randy Marchany Joseph Tront Virginia Polytechnic Institute and State University

2. Purpose 2 To analyze the limitations of web application scanners by using both a secure and insecure version of a custom-built web application Identify scanner weaknesses Improve scanner detection techniques Reduce false-negatives and false-positives

3. Problem > 60% of attacks seen on Internet aimed at web applications [SANS 2009] Web application vulnerability scanners are not capable of detecting all of the vulnerabilities and attack vectors that exist Numerous false-negatives and false- positives False sense of web application security Web application flaws remain unpatched

4. Relevant Vulnerabilities SQL Injection Cross-Site Scripting (XSS) Session Management Flaws

5. Web Application Scanners

6. Related Work Evaluation Applications WebMaven’s BuggyBank Foundstone’s Hacme Bank, Hacme Shipping, Hacme Travel Evaluation Techniques Scan publicly available web applications Benchmark system

7. Related Work Evaluation Applications WebMaven’s BuggyBank Foundstone’s Hacme Bank, Hacme Shipping, Hacme Travel Evaluation Techniques Scan publicly available web applications Benchmark system

8. Test Bed Design

9. Client-Side Features Regular users Submit, assess, purchase, and categorize products Password reminder Register new user Edit account settings Administrator users Add other administrators Approve/deny uploaded templates and item categorizations View validated templates

10. Server-Side Features Database Users table (10 regular, 1 administrator) Categories table (12 static, 1 custom) Categorize table (15 uncategorized, 5 categorized) Items table (10 available) Templates 5 validated item templates 5 awaiting approval

11. Vulnerabilities Implemented SQL Injection Form input (12) Cookie Variable (8) Session Management Predictable session ID (1) Unprotected variables (9) Cross-Site Scripting Reflected (10) Stored (6) DOM-based (1)

12. SQL Injection Example

13. XSS Injection Example

14. Session Management Example

17. Methodology CentOS 4.4 MySQL 4.1 Apache 2.0 PHP 4.3.9 Most popular web server technologies:

18. Methodology Controlled benchmark environment: Controlled variables Web server technologies Independent variables Number of deliberate vulnerabilities Dependent variables Observed false positives and false negatives

19. Methodology Black-box and White-box Analysis: Black-box Analysis Perspective of client-side user Manipulates user supplied input White-box Analysis Source code is available Verify actual number of vulnerabilities

20. Testing Approach Phases: 1. Initialization 3. Classification 2. Execution4. Analysis

21. Initialization 1. Restore the MySQL database to its original state 2. Delete all client side and server side cookies 3. Restore the entire web server directory with a clean backup 4. Restart the web server

22. Execution 1. Configure the web application vulnerability scanner (primarily default settings) 2. Start Wireshark packet capturing 3. Execute the web application vulnerability scan 4. Stop Wireshark packet capturing and save the trace 5. Save the contents of the database, cookie files, and scanner results

23. Classification False Positives: 1. The scan result is due to an application robustness problem (error page, format exception, etc.) and not a vulnerability 2. Normal operation of the web application results in the same error/problem 3. Source code analysis determines scan result is incorrect 4. The results duplicate a vulnerability that has already been accounted for Note: Repeated input forms that are displayed recursively on a web page are not considered duplicate vulnerabilities if they are detected multiple times by a web application scanner (e.g. user comment boxes)

24. Analysis HTTP requests and responses Client and server cookies Database copies Scanner result reports

25. SQL Injection Results – Form Inputs

26. SQL Injection Results – Cookie Variables

27. XSS Injection Results - Reflected

28. XSS Injection Results - Stored

29. XSS Injection Results – DOM-based

30. Session Management Results – Predictable SID

31. Session Management Results – Insecure Cookie Variables

32. False Positives – SQL Injection

33. False Positives – XSS Injection

34. Analysis – SQL Injection False Negatives Unable to recognize reflected SQL error in response page Required fields not completed (ex: register and log-in forms need more than one input) Cookie variables not tested for injection Improvements Flag any error or warning response as suspected vulnerability (information disclosure) All possible combinations of form inputs should be tested Add cookie variables to list of parameters to be checked

35. Analysis – SQL Injection False Positives Duplicate entries reported Tested both GET and POST methods Multiple exploit strings for same vulnerability instance Blind SQL injection returned the same HTML page Improvements Only report the implemented HTTP method (unless both methods are supported) Verify vulnerability using multiple exploit scenarios, but only report SQL injection vulnerability once Blind SQL injection tests should use time delay statements

36. Analysis – XSS Injection False Negatives Required fields not completed Did not test all possible user supplied input (DOM variable, cookie variables, and uploadable files) Stored XSS detected through reflected techniques (relies on SQL error pages) Improvements Attempt every combination of form inputs Test all parameters (even search embedded scripts) Use unique injection strings and then crawl web application multiple times searching for string

37. Analysis – XSS Injection False Positives Duplicate entries reported Same exploit string with different set of parameters (causes request to appear different) Improvements Same as SQL injection Verify vulnerability using multiple exploit scenarios, but only report one XSS vulnerability per input form

38. Analysis – Session Management False Negatives Did not attempt to manipulate cookie to perform session hijacking (only tested for information disclosure) Most determined cookie not set as HttpOnly Only tested for session management vulnerabilities using information from initial connection Improvements Tamper with all unprotected session variables throughout the application (check if response page different) Analyze SIDs for strong algorithm

39. Analysis – Other Findings Possible explanations Overload of requests Server not producing proper response pages Scanners not handling all of server responses Failures in spidering techniques Scanners missing parameters and links while crawling Did not test initial log-in page Improvements Monitor current connection speed and throttle number of simultaneous connections Test all pages and variables within scope of web application

40. Conclusions Scanners detect simple forms of reflected XSS and SQL injection relatively well Work still needs to be done to detect non-traditional instances of these vulnerabilities Most false-positives observed from insecure version of web application Expected secure version to produce false-positives Did not test enough of the input parameters Better to use multiple web application scanners Combined results better than any one individually

41. Future Work Expand to include analysis of other web application technologies and server configurations Web 2.0 technologies (Ajax, Flash, Ruby, Python) Develop black-box web application scanner Using proposed techniques

42. Questions security.vt.edu