MetriCon 2.0 Correlating Automated Static Analysis Alert Density to Reported Vulnerabilities in Sendmail Michael Gegick, Laurie Williams North Carolina State University 7 August 2007

2 Introducing Security Parallels Fault-prone component Likely to contain faults Failure-prone component Likely to have failures in field Component – any logical part of the software system [1] Make informed risk management decisions and prioritize redesign, inspection, and testing efforts on components. Reliability context (well-established) Security context (new) Vulnerability-prone component Likely to contain vulnerabilities Attack-prone component Likely to be exploited in the field [1] IEEE, "ANSI/IEEE Standard Glossary of Software Engineering Terminology (IEEE Std )," Los Alamitos, CA: IEEE Computer Society Press, 1990.

3 Early Reliability Metrics Static analysis –N. Nagappan and T. Ball, "Static Analysis Tools as Early Indicators of Pre-release Defect Density," in International Conference on Software Engineering, St. Louis, MO, 2005, pp –J. Zheng, L. Williams, W. Snipes, N. Nagappan, J. Hudepohl, and M. Vouk, "On the Value of Static Analysis Tools for Fault Detection," IEEE Transactions on Software Engineering, vol. 32, pp , Complexity metrics –J. Munson and T. Khoshgoftaar, "The Detection of Fault-Prone Programs," IEEE Transactions on Software Engineering, vol. 18, pp , –T. Khoshgoftaar and J. Munson, "Predicting Software Development Errors using Software Complexity Metrics," IEEE Journal on Selected Areas in Communications, vol. 8, pp , Historical (failure) –N. Nagappan, T. Ball, and A. Zeller, "Mining metrics to predict component failures," in International Conference on Software Engineering, Shanghai, China, –T. J. Ostrand, E. J. Weyuker, and R. M. Bell, "Where the bugs are," in International Symposium on Software Testing and Analysis, Boston, Massachusetts, 2004, pp Object-Oriented metrics –V. Basili, L. Briand, and W. Melo, "A Validation of Object Oriented Design Metrics as Quality Indicators," IEEE Transactions on Software Engineering, vol. 21, –Y. Zhou and L. Hareton, "Empirical Analysis of Object-Oriented Design Metrics for Predicting High and Low Severity Faults," IEEE Transactions on Software Engineering, vol. 32, no. 10, 2006, pp

4 Research Objective Build and validate models for predicting vulnerability- and attack-prone components based upon security- based automated static analyzer (ASA) alerts –Metric: ASA alert density and severity – early in the development phase –ASA cannot find all types of security vulnerabilities Are ASA alerts a good predictor? –Implementation bugs, design flaws, operational vulnerabilities –Software engineers plug the number of security alerts into the predictive models to determine which components are vulnerability- and attack- prone.

5 Building the Initial Predictive Model Generalized linear model (data are not normally distributed) Poisson distribution? mean number vulnerabilities in component estimated intercept estimated slope value of random variable – alert density of component

6 Feasibility Study Fortify Software’s Source Code Analyzer (SCA) Scanned ten releases of Sendmail – –996 total files scanned 21 potential vulnerabilities –Vulnerabilities reported in RELEASE_NOTES Nine vulnerabilities with known exploits

7 Feasibility Study – vulnerability-prone Poisson distribution –Models the response data Reported vulnerability Association between Hot alert density and number of vulnerabilities per reported per file –Positive slope  positive association between alerts and reported vulnerabilities –p-value  high significance in association Standard error  substantial overdispersion –Few data points Slopep-valueChi-Square /df Goodness- of- fit measure Standard error

8 Feasibility Study – attack-prone Poisson distribution –Models the response data Number of known exploits (nine) for a Sendmail file Association between Hot alert density and number of known exploits –Slope  positive association between alerts and exploits p-value  low significance –Standard error  substantial overdispersion »Few data points Slopep-valueChi-Square /df Goodness- of- fit measure Standard error

9 Questions Thank you!