1. Proposing SQL Statement Coverage Metrics
Ben Smith
Yonghee Shin
Laurie Williams 1

2. Agenda Motivation and Objective Testing for Security
SQL Injection Vulnerabilities/Attacks
Terms and Metrics
Case Study
Limitations
Future Work 2

3. 3 US National Vulnerability Database
All reported vulnerabilities tracked over the period 1995 – last year.
increasing polynomially. 3

4. Cyber Vulnerabilities, cont’d
Averages from NVD 2003 – 2007 [1]
Input validation vulnerabilities comprise >50% of reported total
SQL Injection Vulnerabilities comprise 11% of reported total
Using the same data, we categorized any vulnerability appearing from the search string SQL injection, XSS and buffer overflow, and found that for the time period they comprise more than half of all reported vulns. One way to reduce that number would be to mitigate or completely remove SQL injection attacks, which is what our research focuses on. These comprise 11% in that same time period. 4

5. Objective
To propose coverage metrics which determine the adequacy of a test set’s ability to mitigate SQL injection attacks. 5

6. Security Testing [2] Application as Coded
Missing or Incorrect Functionality
Unintended Functionality
Application as Coded
Intended Functionality (Requirements Specification) 6

7. SQL Injection Attacks ‘ OR 1=1 -- $username = $_POST[‘username’];
$password = $_POST[‘password’];
$result = mysql_query(
“select * from users where username =
‘$username’ AND password = ‘$password’”);
$firstresult = mysql_fetch_array($result);
$role = $firstresult[‘role’];
$_COOKIE[‘userrole’] = $role;
$username = $_POST[‘username’];
$password = $_POST[‘password’];
$result = mysql_query(
“select * from users where username =
‘’ OR 1=1 ---’ AND password = ‘$password’”);
$firstresult = mysql_fetch_array($result);
$role = $firstresult[‘role’];
$_COOKIE[‘userrole’] = $role
‘ OR 1=1 -- 7

8. Web Application Security
Data
Internet
Query
Query
From this viewpoint, we arrive at a picture like this one. This model is admittedly simplistic, but it helps illustrate a point. The web, here on left, is where our user comes from. The web application stands in between the user and the data for many reasons, including computation and presentation, but also security. The web application must ensure that the user does not access or change any data on the right which the user is not allowed to, and also must ensure that the user can change any data on the right which the user is allowed to. These changes (or non-changes) happen through the query oval, which represents SQL queries issued to the RDBMS, and which we refer to as target statements. Our metrics ensure that each SQL statement is tested, because afterall, how can you be sure that a statement is secure if your tests do not even cover it? 8

9. Coverage Analysis
Industry has been doing this sort of thing for a long time; this is a screen shot of a coverage report which represents the line coverage for each Java class after a JUnit test case execution has completed. We would like to see something similar except for target statements. 9

10. Theory
Higher SQL Coverage Metrics 
Higher System Security 10

11. Research Plan SQL Coverage Metrics Blacklist/Input Validation Testing11

12. Terms and Metrics (1)
Target Statement: a SQL statement which could cause a security problem when malicious input is used, for example:
java.sql.Connection conn = factory.getConnection();
java.sql.PreparedStatement ps = conn.prepareStatement("UPDATE globalVariables SET value = ? WHERE name = ?;");
ps.setInt(1, value);
ps.setString(2, name);
java.sql.ResultSet rs = ps.executeQuery(); 12

13. Terms and Metrics (2)
Input Variable: any variable in the server-side production code which is dynamically user-assigned and sent to the database management system. For example,
java.sql.Connection conn = factory.getConnection();
java.sql.PreparedStatement ps = conn.prepareStatement("UPDATE globalVariables SET value = ? WHERE name = ?;");
ps.setInt(1, value);
ps.setString(2, name);
java.sql.ResultSet rs = ps.executeQuery(); 13

14. Terms and Metrics (3)
Target Statement Coverage: the percentage of target statements in the production code executed/covered at least once by the test set.
Input Variable Coverage: the percentage of input variables in the production code executed/covered at least once by the test set. 14

15. Case Study: iTrust [3] Role-based web healthcare application
Developed by NC State students in a security testing class
Aims to be HIPAA-compliant
~7700 SLOC, 143 production classes
Tomcat + MySQL
84% line coverage 15

16. Case Study: Overview
Manually instrument iTrust to record the first execution of each target statement
Run the built-in unit tests to reveal the coverage result for both SQL statement coverage and input variable coverage
Warning: Formative Research!! 16

17. Case Study: Results Coverage Metric Tested Total Result
Target Statement 90 93
96.7%
Input Variable
209
212
98.5% 17

18. A few more results… Three-tiered web applications SQL Tomcat
Provided test sets
Examples
Hispacta
IceScrum
Care2x? 18

19. A few more results… iTrust IceScrum Hispacta Type of Tests Run Unit
Unit + Integration
Lines of Code
7,707
19,442
1,991
Production Classes
143
155 42
DAO Classes 20 10 4
SQL Statements 93 96 23
Covered by Tests 90 29 18
Coverage
96.7%
30.2%
78.3%
SQL Input Variables
212
517 36
209
258 32
98.5%
49.9%
88.9%
Line Coverage (EclEmma)
84.1%
9.7%
49.4% 19

20. Threats to Validity Human error Inaccurate analysis
May not be feasible in other systems, languages or operational environments
iTrust is highly testable, and well-tested 20

21. Future Work Empirical studies Automation Finer-grained metrics
SQL coverage in other open source systems
Linking SQL coverage to higher security levels
Automation
Dynamic Queries
Identification
Finer-grained metrics
Use as platform for blacklist/whitelist testing 21

22. Very Future Work SQL Coverage Metrics
Blacklist/Input Validation Testing
A Framework 22

23. References
[1] United States National Vulnerability Database.
[2] H. H. Thompson, "Why security testing is hard," Security & Privacy Magazine, IEEE, vol. 1, no. 4, pp , 2003.
[3] iTrust website. 23

24. Questions?
Feedback & Comments Welcome! 24

25. Dynamic Queries java.sql.Connection conn = factory.getConnection();
java.sql.PreparedStatement ps = null;
if (user.isAdmin()) {
ps = conn.prepareStatement(“select * from adminTable;"); }
else
ps = conn.prepareStatement(“select * from generalTable;”);
ps.executeQuery(); 25

26. Case Study: Instrumentation
java.sql.Connection conn =
factory.getConnection();
java.sql.PreparedStatement ps =
conn.prepareStatement("UPDATE
globalVariables set SET VALUE = ? WHERE
Name = ‘Timeout’;");
ps.setInt(1, mins);
SQLMarker.mark(1, 1);
java.sql.ResultSet rs = ps.executeQuery();
Sets a counter to 1 in the research database. 26