This is a work of the U.S. Government and is not subject to copyright protection in the United States. The OWASP Foundation OWASP AppSec DC October Developing a Reference Dataset to Accurately Evaluate SA Tools Paul E. Black Computer Scientist, NIST

OWASP AppSec DC Outline  Goals for a Reference Dataset  Content and Characteristics  Complications and Considerations  Tour of NIST’s Prototype RDS

OWASP AppSec DC Goals  Provide researchers, developers, and consumers with a set of known bugs, flaws, weaknesses, vulnerabilities.  This allows developers and researchers to test their methods and consumers to evaluate tools.  The reference dataset will encompass a wide variety of vulnerabilities, languages, platforms, etc. from all phases of the software life cycle.  The dataset is a long-term effort needing contributions from many sources.

OWASP AppSec DC How is a Reference Dataset Used?  Researchers  Is a new method “better” than existing methods? Faster? Finds more flaws? Fewer false alarms? Produces more reliable programs? In what instances?  Saves time of assembling test cases  Tool Developers  What flaws should be caught? What problems should be prevented in code?  Suggests direction for new development  End Users  Understand need for better tools and techniques  Save effort and improve thoroughness in evaluation  Confirm utility of methods and tools

OWASP AppSec DC Role in the NIST SAMATE Project  Surveys  Draw from researchers, developers, and users  Taxonomies  Use common taxonomy of flaws  Grouped by common taxonomy of tools and methods  Gaps and research agendas  Helps highlight what is needed  Studies to develop metrics  Well-characterized samples for study  Enable tool evaluations  Standard reference material for use in test plans

OWASP AppSec DC Outline  Goals for a Reference Dataset  Content and Characteristics  Complications and Considerations  Tour of NIST’s Prototype RDS

OWASP AppSec DC What’s in the Reference Dataset?  Samples of designs, source code, binaries, etc. with known flaws and vulnerabilities.  Corresponding samples with the problems fixed.  Metadata to label samples with  Contributor, source for binaries, remediation, etc.,  Location and type of flaw(s), compiler or platform where it occurs etc.,  Drivers, stubs, declarations, etc.,  Input demonstrating flaw, expected results, and  Comments  Test suites consisting of sets of samples

OWASP AppSec DC Characteristics of the Reference Dataset  Metadata for sample is separate from the sample itself.  Samples are “wild” (production), “synthetic” (written to test), and academic (from classes).  Samples are mostly small (10’s of lines), but some are large (1000’s or millions of lines).  Samples are “write-only”, but may be declared obsolete and superceded in test suite.

OWASP AppSec DC Samples for Static vs. Dynamic Analysis  Samples are mostly for static analysis.  Some executable samples, but …  Dynamic analysis is better done in a test bed.  Network applications must have controlled access. network applications.  Dynamic analysis may interfere with running systems.  Need to rebuild corrupted systems.  Hard to run legacy executables on new platforms.  Flawed executables should be auxotrophic: won’t run without a special file, dll, etc. saying Warning: Software has Known Vulnerability

OWASP AppSec DC Public Access  Submit a new sample.  Status is tentative until examined.  Add new “fixed” versions, inputs demonstrating flaw, etc.  Retrieve samples, metadata, and test suites.  Comment on a sample.  Report result of a run of a tool on a sample.

OWASP AppSec DC Outline  Goals for a Reference Dataset  Content and Characteristics  Complications and Considerations  Tour of NIST’s Prototype RDS

OWASP AppSec DC Complications and Considerations  What about new threats, languages, etc.?  Reference dataset will evolve  May be slightly easier to “port” samples than write from scratch  What if developers build to the test?  Use the error sample generator  These tests are only one consideration  How about malicious use, i.e., university for crackers?  These are known flaws  Limit access to high risk flaws?

OWASP AppSec DC Outline  Goals for a Reference Dataset  Content and Characteristics  Complications and Considerations  Tour of NIST’s Prototype RDS

OWASP AppSec DC NIST Prototype Reference Dataset (RDS)  Populating with material from  Fortify Software (80+ samples)  MIT Lincoln Lab (1000+ samples)  NIST (20 samples)  Other possible sources:  OWASP WebGoat  Foundstone HacmeBook and HacmeBank  CVE list of known vulnerabilities  Other tool developers

OWASP AppSec DC Home Page

OWASP AppSec DC Search RDS by Keyword

OWASP AppSec DC Result of Search

OWASP AppSec DC ID 7: gets is never safe for untrusted input

OWASP AppSec DC Code for gets1-bad.c

OWASP AppSec DC Contact to Participate  Paul E. Black Project Leader Software Diagnostics & Conformance Testing Division, Software Quality Group, Information Technology Laboratory, NIST