This is a work of the U.S. Government and is not subject to copyright protection in the United States. The OWASP Foundation OWASP AppSec DC October A Taxonomy of Software Assurance Tools and the Security Bugs they Catch Michael Kass Information Technology Laboratory National Institute of Standards and Technology

2 OWASP AppSec DC 2005 SAMATE Goals:  A taxonomy of software assurance tools and their functions  A taxonomy of the software flaws that those tools find  A reference dataset (RDS) of tests to measure if SA tools correctly detect those flaws

3 OWASP AppSec DC 2005 Why a Tool Taxonomy?  Provide a common reference/classification of functions for evaluation of a tool’s effectiveness  A “first step” in defining a tool functional specification  Tool specification becomes basis for creating tool reference dataset/benchmarks  A common specification and reference dataset for comparing SA tools of the same class

4 OWASP AppSec DC 2005 Workshop SA tool classes focus group tool function taxonomy select tool class and functions functional specification test plan Tool Taxonomy and Follow-on Products test suite

5 OWASP AppSec DC 2005 Tool Survey and Taxonomy  ”External” Tools  Network Scanners  Web Application Scanners  Web Services Scanners  Dynamic Analysis Tools  “Internal” Tools  Software Requirements Verifiers  Software Design/Model Verifiers  Compilers  Static Source Code Scanners  Static Byte Code Scanners  Static Binary Code Scanners  Database Scanners Much of the information in the Software Security Tool Taxonomy is derived from the DISA Application Security Assessment Tool Survey, July 2004, to be published as a DISA STIG.

6 OWASP AppSec DC 2005 Network Scanners Remotely scan targeted machines, performing port scans, and probing for vulnerabilities known in operating systems and third party network software.  Functions - scan for known vulnerabilities in:  operating systems  applications  web servers  network devices  network protocols  RDS requirements:  networked testbed  customize-able deployment scenarios for vulnerability environment  test case documentation against a functional specification for general network scanner SA tools

7 OWASP AppSec DC 2005 Web Application Scanners A more “specialized” class of SA tool that focuses specifically on web applications only, and are not considered generalized network scanners.  Scan for vulnerabilities by:  simulating a web browser client  performing field manipulation/cookie poisoning  exploiting relative URL paths  injecting malicious data via form fields to exploit server  RDS requirements:  HTTP(S) server with applications having known security flaws and vulnerabilities  networked testbed  customize-able deployment environment for vulnerability scenarios  test case documentation against a functional specification for web application scanner SA tools

8 OWASP AppSec DC 2005 Web Services Scanner Functions A relatively new class of SA tool whose purpose is the analysis of web service applications. Web service scanners have functions of the following type:  Scan for vulnerabilities by:  functional test of SOAP server  test XML encryption, signatures and signature verification  test WS-Security  perform XML validation of received messages  RDS requirements:  a networked testbed of web services with known security flaws and vulnerabilities  networked testbed  customize-able deployment environment for vulnerability scenarios  test case documentation against a functional specification for web services scanner SA tools

9 OWASP AppSec DC 2005 Dynamic Analysis Tool Functions Dynamic analysis tools generate runtime vulnerability scenarios through the following functions:  Scan for vulnerabilities by:  performing file corruption  resource/network/system/interface fault injection  design attacks  implementation attacks  RDS requirements:  a testbed of applications with known security flaws and vulnerabilities  virtual environment (sandbox)  customize-able operating environment to create vulnerability scenarios  test case documentation against a functional specification for dynamic analysis SA tools

10 OWASP AppSec DC 2005 Software Requirements Verification Tool Functions Functions of some requirements verification tools that have been developed include determining whether requirements are: (from NASA’s Automated Requirements Measurement Tool )  Scan for vulnerabilities by verifying that application requirements are:  verifiable  complete  consistent  unambiguous  RDS requirements:  a corpus of application requirement documents that introduce security vulnerabilities into application design and/or implementation  test case documentation against a functional specification for requirements verification SA tools

11 OWASP AppSec DC 2005 Design/Model Verification Tool Functions Design/model verification tools provide are a vital “first line of defense“ in software assurance.  Identify potential flaws/vulnerabilities by performing:  application simulation  exhaustive verification  proof of design  RDS requirements:  a corpus of design documents that introduce security vulnerabilities into application design and/or implementation  test case documentation against a functional specification for design/model verification SA tools

12 OWASP AppSec DC 2005 Compilers Choosing “type safe” compilers, or extending the capability of some compilers can provide an additional level of software security (although not guaranteed). Some security functions of “extended compilers” include:  Identify potential flaws/vulnerabilities by injecting run-time code to identify:  buffer overflows  race conditions  un-initialised variables  RDS requirements:  a testbed of applications with known security flaws and vulnerabilities  A virtual test environment  customize-able operating environment to create vulnerability scenarios  test case documentation against functional specification for dynamic binary analysis SA tools

13 OWASP AppSec DC 2005 Source Code Scanners A relatively “new” class of SA tool that can perform data-flow, control-flow, inter-procedural analysis of code and identify software security flaws and potential vulnerabilities.  Some key functions include:  Identify security flaws and vulnerabilities  filter results based upon flaw type/priority  scale to large code sets  generate code metrics  RDS requirements:  a corpus of source code examples with known security flaws and vulnerabilities  test case documentation against functional specification for source code analysis SA tools

14 OWASP AppSec DC 2005 Static Bytecode Analysis Tools A step below “source code scanners’”, but above binary analysis tools, byte code analysis tools have similar functions to source code scanning tools.  Some key functions include:  Identify security flaws and vulnerabilities  filter results based upon flaw type/priority  scale to large code sets  RDS requirements:  a corpus of byte code examples with known security flaws and vulnerabilities  test case documentation against functional specification for static bytecode analysis tools

15 OWASP AppSec DC 2005 Static Binary Analysis Tools The “last line of defense” in SA tools. Performs similar SA functions as source code analysis tools, but does so at “runtime”.  Some key functions include:  perform boundary checks  identify memory leaks  identify unreachable code  identify race conditions  identify type mismatches  RDS requirements:  a corpus of binary executable examples with known security flaws and vulnerabilities  test case documentation against functional specification for static binary code analysis tools

16 OWASP AppSec DC 2005 Defining a Software Security Flaw Taxonomy Because of the nature of software development (new technologies, new languages and language features) a taxonomy of software security flaws will be a living and changing entity. Additional characteristics that must be considered include:  Program vulnerabilities are usually a combination of security flaws  Mutual flaw exclusion will be difficult to deal with (examples : authentication vs. logic flaw problem)  Some of the flaws in the taxonomy cannot be identified by tools today  Some flaws have never been seen in real world code… yet  Some flaws can be introduced at multiple points in the SDLC

17 OWASP AppSec DC 2005 A Taxonomy of Software Security Flaws and their Point of Introduction in the SDLC One of the first steps in comparing tools of the same class is to determine what software flaws all tools of that class should be able to identify. The following hierarchical classification of software flaw root causes is taken from Secure Software’s CLASP V1.1 Training Manual identify range and type errors Stack overflow – (requirements, design, implementation) Command injection (design, implementation) Double Free (implementation) identify environmental problems Resource exhaustion (design, implementation) identify synchronization and timing errors TOCTOU (time-of-check time-of-use) race condition (design, implementation) identify protocol errors Use of a broken/risky cryptographic algorithm (design) identify general logic errors  Performing a “chroot” without a “chdir” (implementation)

18 OWASP AppSec DC 2005 Reference Dataset The SAMATE team is creating a reference dataset of test cases to measure SA tool performance against the functions defined for that particular type of tool. Currently, we are focusing on source code scanning tools.  Existing source code material for a reference dataset include:  OWASP WebGoat  Foundstone HacmeBook and HacmeBank  CVE listing of known vulnerabilities  MIT Lincoln Lab Source Code Contribution (1000+ test cases)  Fortify Software Source Code Contribution (80+ test cases)  Both “In the Wild” and “Manufactured” code will be part of the reference dataset  Dataset will include representation of software flaws from the SA security flaw taxonomy  The SAMATE Reference Dataset will include testing artifacts from requirements, modelling and design, implementation and deployment phases of the SDLC

19 OWASP AppSec DC 2005 Ongoing Work  NIST is developing prototype reference dataset for SA tools, for review by SAMATE working group members  November 7,8 -- Workshop on Software Security Assurance Tools, Techniques, and Metrics, Long Beach, CA (  DAY #1 – Papers  DAY #2 - “Target Practice” for source code scanner tool developers against SAMATE SA Tool Reference Dataset  Continue work through SAMATE focus groups (online and at follow-on workshops) to refine:  a taxonomy of common functions for source code scanners  a common taxonomy of software security flaws  content of a reference dataset for testing SA tools