Proposing SQL Statement Coverage Metrics Ben Smith Yonghee Shin Laurie Williams 1

Agenda Motivation and Objective Testing for Security SQL Injection Vulnerabilities/Attacks Terms and Metrics Case Study Limitations Future Work 2

3 US National Vulnerability Database All reported vulnerabilities tracked over the period 1995 – last year. increasing polynomially. 3

Cyber Vulnerabilities, cont’d Averages from NVD 2003 – 2007 [1] Input validation vulnerabilities comprise >50% of reported total SQL Injection Vulnerabilities comprise 11% of reported total Using the same data, we categorized any vulnerability appearing from the search string SQL injection, XSS and buffer overflow, and found that for the time period 03-07 they comprise more than half of all reported vulns. One way to reduce that number would be to mitigate or completely remove SQL injection attacks, which is what our research focuses on. These comprise 11% in that same time period. 4

Objective To propose coverage metrics which determine the adequacy of a test set’s ability to mitigate SQL injection attacks. 5

Security Testing [2] Application as Coded Missing or Incorrect Functionality Unintended Functionality Application as Coded Intended Functionality (Requirements Specification) 6

SQL Injection Attacks ‘ OR 1=1 -- $username = $_POST[‘username’]; $password = $_POST[‘password’]; $result = mysql_query( “select * from users where username = ‘$username’ AND password = ‘$password’”); $firstresult = mysql_fetch_array($result); $role = $firstresult[‘role’]; $_COOKIE[‘userrole’] = $role; $username = $_POST[‘username’]; $password = $_POST[‘password’]; $result = mysql_query( “select * from users where username = ‘’ OR 1=1 ---’ AND password = ‘$password’”); $firstresult = mysql_fetch_array($result); $role = $firstresult[‘role’]; $_COOKIE[‘userrole’] = $role ‘ OR 1=1 -- 7

Web Application Security Data Internet Query Query From this viewpoint, we arrive at a picture like this one. This model is admittedly simplistic, but it helps illustrate a point. The web, here on left, is where our user comes from. The web application stands in between the user and the data for many reasons, including computation and presentation, but also security. The web application must ensure that the user does not access or change any data on the right which the user is not allowed to, and also must ensure that the user can change any data on the right which the user is allowed to. These changes (or non-changes) happen through the query oval, which represents SQL queries issued to the RDBMS, and which we refer to as target statements. Our metrics ensure that each SQL statement is tested, because afterall, how can you be sure that a statement is secure if your tests do not even cover it? 8

Coverage Analysis Industry has been doing this sort of thing for a long time; this is a screen shot of a coverage report which represents the line coverage for each Java class after a JUnit test case execution has completed. We would like to see something similar except for target statements. 9

Theory Higher SQL Coverage Metrics  Higher System Security 10

Research Plan SQL Coverage Metrics Blacklist/Input Validation Testing 11

Terms and Metrics (1) Target Statement: a SQL statement which could cause a security problem when malicious input is used, for example: java.sql.Connection conn = factory.getConnection(); java.sql.PreparedStatement ps = conn.prepareStatement("UPDATE globalVariables SET value = ? WHERE name = ?;"); ps.setInt(1, value); ps.setString(2, name); java.sql.ResultSet rs = ps.executeQuery(); 12

Terms and Metrics (2) Input Variable: any variable in the server-side production code which is dynamically user-assigned and sent to the database management system. For example, java.sql.Connection conn = factory.getConnection(); java.sql.PreparedStatement ps = conn.prepareStatement("UPDATE globalVariables SET value = ? WHERE name = ?;"); ps.setInt(1, value); ps.setString(2, name); java.sql.ResultSet rs = ps.executeQuery(); 13

Terms and Metrics (3) Target Statement Coverage: the percentage of target statements in the production code executed/covered at least once by the test set. Input Variable Coverage: the percentage of input variables in the production code executed/covered at least once by the test set. 14

Case Study: iTrust [3] Role-based web healthcare application Developed by NC State students in a security testing class Aims to be HIPAA-compliant ~7700 SLOC, 143 production classes Tomcat + MySQL 84% line coverage 15

Case Study: Overview Manually instrument iTrust to record the first execution of each target statement Run the built-in unit tests to reveal the coverage result for both SQL statement coverage and input variable coverage Warning: Formative Research!! 16

Case Study: Results Coverage Metric Tested Total Result Target Statement 90 93 96.7% Input Variable 209 212 98.5% 17

A few more results… Three-tiered web applications SQL Tomcat Provided test sets Examples Hispacta IceScrum Care2x? 18

A few more results… iTrust IceScrum Hispacta Type of Tests Run Unit Unit + Integration Lines of Code 7,707 19,442 1,991 Production Classes 143 155 42 DAO Classes 20 10 4 SQL Statements 93 96 23 Covered by Tests 90 29 18 Coverage 96.7% 30.2% 78.3% SQL Input Variables 212 517 36 209 258 32 98.5% 49.9% 88.9% Line Coverage (EclEmma) 84.1% 9.7% 49.4% 19

Threats to Validity Human error Inaccurate analysis May not be feasible in other systems, languages or operational environments iTrust is highly testable, and well-tested 20

Future Work Empirical studies Automation Finer-grained metrics SQL coverage in other open source systems Linking SQL coverage to higher security levels Automation Dynamic Queries Identification Finer-grained metrics Use as platform for blacklist/whitelist testing 21

Very Future Work SQL Coverage Metrics Blacklist/Input Validation Testing A Framework 22

References [1] United States National Vulnerability Database. http://nvd.nist.gov [2] H. H. Thompson, "Why security testing is hard," Security & Privacy Magazine, IEEE, vol. 1, no. 4, pp. 83-86, 2003. [3] iTrust website. http://sourceforge.net/projects/iTrust 23

Questions? Feedback & Comments Welcome! 24

Dynamic Queries java.sql.Connection conn = factory.getConnection(); java.sql.PreparedStatement ps = null; if (user.isAdmin()) { ps = conn.prepareStatement(“select * from adminTable;"); } else ps = conn.prepareStatement(“select * from generalTable;”); ps.executeQuery(); 25

Case Study: Instrumentation java.sql.Connection conn = factory.getConnection(); java.sql.PreparedStatement ps = conn.prepareStatement("UPDATE globalVariables set SET VALUE = ? WHERE Name = ‘Timeout’;"); ps.setInt(1, mins); SQLMarker.mark(1, 1); java.sql.ResultSet rs = ps.executeQuery(); Sets a counter to 1 in the research database. 26