1. Maureen Doyle, James Walden Northern Kentucky University Students: Grant Welch, Michael Whelan Acknowledgements: Dhanuja Kasturiratna

2. 1. Research Objective 2. Evolution of Web App Security 3. Security Resource Indicator 4. Vulnerability Type Analysis 5. Code Metrics 6. Analysis Limitations 7. Conclusion 10/12/2009 University of Kentucky2

3. Goal: Identify predictors for vulnerability density (VD) and change in VD for open source web applications. Research questions: 1.Can software security practices predict evolution of VD over time? 2.Can code size or complexity predict VD? 3.Can code change metrics predict VD? 10/12/2009 University of Kentucky3

4. Static Analysis ◦ Nagappan and Ball, ICSE 2005a ◦ Coverity Open Source Report 2009 ◦ Fortify Open Source Security Study 2008 Complexity and Change Metrics ◦ Nagappan and Ball, ICSE 2005b ◦ Nagappan, Ball, and Zeller, ICSE 2006 ◦ Shin and Williams, QoP 2008 10/12/2009 University of Kentucky4

5. Reported Vulnerabilities in NVD or OSVDNVD OSVD ◦ Coarse-grained time evolution. ◦ Difficult to correlate with revision. ◦ Undercounts actual vulnerabilities. Dynamic Analysis ◦ Expensive. ◦ False positives and negatives. ◦ Requires installation of application. Static Analysis ◦ Expensive. ◦ False positives and negatives. ◦ Static Analysis Vulnerability Density = vulns/kloc. 10/12/2009 University of Kentucky5

6. Selection process ◦ PHP web applications from freshmeat.net. ◦ Subversion repository with 100 weeks of revisions. Revisions ◦ One revision selected per week for analysis. Range of projects ◦ 14 projects met selection criteria. ◦ 5,800 to 388,000 lines of code (2008). ◦ Removing highest & lowest, SLOC range of 25-150 kloc. 10/12/2009 University of Kentucky6

7. Overall security improvement. ◦ first week average: 8.88 vulns/kloc ◦ final week average: 3.30 vulns/kloc Average SAVD high vs. Coverity’s 0.30 SAVD. ◦ Language differences: C/C++ vs. PHP. ◦ Vulnerability differences  buffer overflows vs XSS/SQL. No correlation with NVD vulnerabilities. ◦ NVD correlated with freshmeat popularity. 10/12/2009 University of Kentucky7

8. 10/12/2009 University of Kentucky8

9. 10/12/2009 University of Kentucky9

10. 10/12/2009 University of Kentucky10

11. Public security resources on project site ◦ URL for installation or configuration security ◦ Dedicated address to report security bugs ◦ Database of known security vulnerabilities ◦ Documentation of secure development practices Results ◦ Correlation of  = 0.67 (p < 0.05) with  SAVD 10/12/2009 University of Kentucky11

12. 10/12/2009 University of Kentucky12

13. 20062008 10/12/2009 University of Kentucky13

14. 10/12/2009 University of Kentucky14

15. 10/12/2009 University of Kentucky15

16. 10/12/2009 University of Kentucky16

17. Size measure ◦ Source Lines of Code (SLOC) Complexity measures ◦ Cyclomatic Complexity ◦ Nesting Complexity ◦ Maximum, average, total Change measures ◦ Churn = lines added + changed ◦ Lines deleted 10/12/2009 University of Kentucky17 1 2 1. do loop 2. stmt 3. end loop 3 CC = E – N + 2 P = 3 – 3 + 2*1

18. 10/12/2009 University of Kentucky18

19. 10/12/2009 University of Kentucky19

20. May not apply to apps that didn’t meet criteria ◦ Non-PHP applications ◦ No SVN repository with two years of history False positives ◦ 18.1% rate from two sample applications ◦ Coverity found a rate under 14% for their study SAVD will differ between static analysis tools 10/12/2009 University of Kentucky20

21. OS PHP web app security improved: ◦ 8.88 to 3.30 SAVD from 2008 to 2006. ◦ But 8 of 14 apps increased SAVD over period. SRI can indicate which apps will improve. No single code metric is predictive for SAVD. ◦ Complexity is an indicator for SAVD. ◦ Churn is not an indicator for SAVD. 10/12/2009 University of Kentucky21

22. Why does app security vary so much? ◦ Analyze security processes for each app. How do we validate SAVD measurement? ◦ NVD count correlates with popularity. Java web applications ◦ How does Java SAVD compare with PHP SAVD? ◦ How do trends compare between Java and PHP? ◦ More software metrics available for Java. 10/12/2009 University of Kentucky22

23. 10/12/2009 University of Kentucky23

24. 10/12/2009 University of Kentucky24

25. 10/12/2009 University of Kentucky25

26. 10/12/2009 University of Kentucky26

27. 10/12/2009 University of Kentucky27