1. Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. The OWASP Foundation OWASP http://www.owasp.org Introducing the Web Application Security Scanner Evaluation Criteria Brian Shura Project Leader - WASSEC AppSec Consulting bshura73@gmail.com November 12, 2009

2. OWASP 2 Agenda  WASSEC Overview  Sections of the WASSEC  Advice for Conducting a Scanner Evaluation

3. OWASP 3 Definition of a Web Application Security Scanner  An automated tool to test web applications for common security problems.  Crawls a web application and locates application-layer vulnerabilities by:  Manipulating HTTP messages  Inspecting HTTP messages for suspicious attributes  Scanners are an important part of most application security programs.

4. OWASP 4 WASSEC - Background  Web application scanners are a complex class of tools.  Many scanners are available, both free and commercial tools.  Quality and usefulness varies greatly.  Wide variety of strong opinions on which scanners are “best”  No formal criteria for conducting a detailed evaluation  “Kick the tires a bit and go with your gut feel.”

5. OWASP 5 WASSEC Overview  Ongoing project promoted by the Web Application Security Consortium (WASC).  Goals  Provide scanner users with a reference that can be used to conduct a thorough scanner evaluation and make an informed decision.  Provide scanner developers with a list of capabilities to compare their tools against to help them create a roadmap of future enhancements

6. OWASP 6 WASSEC Project Contributors  Anurag Agarwal (Whitehat Security)  Vijay Agarwal (Foundstone)  Robert Auger (WASC)  Emilio Casbas (S21sec)  Leonardo Cavallari (NSRAV)  Matthieu Estrade (Bee Ware)  Romain Gaucher (Cigital, Inc.)  Jeremiah Grossman (Whitehat Security)  Robert Hansen (SecTheory)  Amit Klein  Chad Loder (Rapid7)  Ken Pfeil (WestLB AG)  Tyler Reguly (nCircle Network Security)  Ivan Ristic (Breach Security)  Ory Segal (IBM)  Sheeraj Shah (Blueinfy Solutions Pvt. Ltd.)  Chris Shiflett (OmniTI)  Brian Shura (AppSec Consulting) [Project Leader]  Tom Stripling (Security PS)  Chris Sullo (CIRT, Inc.)

7. OWASP 7 WASSEC Sections 1.Protocol Support 2.Authentication 3.Session Management 4.Crawling 5.Parsing 6.Testing 7.Command and Control 8.Reporting

8. OWASP 8 WASSEC Sections  Protocol Support  HTTP versions supported, SSL/TLS, proxy support, etc.  Authentication  Very important section to consider  What types of authentication are used by the application you’ll be scanning?  HTML form-based  HTTP negotiate (NTLM, Kerberos)  Single Sign-On  Login process that can’t be fully automated (OTP, CAPTCHA)

9. OWASP 9 WASSEC Sections  Session Management  For dynamic applications, establishing and maintaining a valid session is essential to achieving a thorough scan.  Does scanner properly establish a session at beginning of test?  Can scanner detect when its session is no longer valid and log back in?  What types of session tokens does the scanner support? (cookies, HTTP parameters, URL path)

10. OWASP 10 WASSEC Sections  Crawling  Scanner must first find the page in order to find the vulnerability!  How effectively does the scanner crawl your application?  The best scanners have many configuration options for fine-tuning the crawl.  Can the scanner automatically submit forms during the crawl?  How well does the scanner support redirects? AJAX?  Lots of details in this section for developers who would like to improve their scanners!

11. OWASP 11 WASSEC Sections  Parsing  Closely tied to crawling.  Scanner should be able to parse common web content types and extract information such as URLs, forms, parameters, etc.  Which content types are used by the applications you’ll be scanning?  HTML, Javascript, CSS  XML  Flash  ActiveX  Is it possible to customize the parser to handle special situations?  http://www.some.site/appEntry?param1^^value1::param2^^value 2...paramN^^valueN

12. OWASP 12 WASSEC Sections  Testing  Testing capabilities – a long list of security issues that a black-box scanner should be able to test for  Test configuration - it is often important to exclude certain pages, parameters, file extensions, etc. from testing.  Can test policies be created to run a subset of available tests against your application?  What type of support does the scanner provide for creating custom tests?

13. OWASP 13 WASSEC Sections  Command and Control  Does scanner allow you to view the real-time status of running scans?  Can scans be paused and resumed?  Can multiple scans be run simultaneously?  Does scanner support multiple users?  What type of control interfaces are provided?  CLI, client application with GUI, web-based interface…  Does the scanner have an API and the ability to integrate with common bug-tracking systems?

14. OWASP 14 WASSEC Sections  Reporting  What types of reports are available?  How useful is the information provided for each vulnerability?  Severity rating, CVSS score, remediation advice, etc.  Can reports be customized? For example:  Adding custom notes to vulnerabilities  Modifying severity levels  Marking issues as false positives  Adding company logo to report footer or header

15. OWASP 15 Advice for Conducting a Scanner Evaluation  Go through the WASSEC and choose which criteria are important to you.  In most cases you won’t care about every feature described in the WASSEC.  Add the relevant criteria to your evaluation spreadsheet and assign them weights.  If certain criteria are “must have”, mark them as such!

16. OWASP 16 Advice for Conducting a Scanner Evaluation  Factor in non-technical criteria such as:  Purchase cost  Ongoing support cost  Ease of use  Quality of documentation  Quality of technical support  Availability of training  Frequency of updates  Licensing restrictions  Usefulness of results

17. OWASP 17 Advice for Conducting a Scanner Evaluation  Decide which scanners will be in-scope for evaluation.  A thorough evaluation is a great learning experience but can be time-consuming.  Eliminate scanners that don’t meet your “must have” criteria.

18. OWASP 18 Advice for Conducting a Scanner Evaluation  Obtain latest version of each scanner that is in- scope for hands-on evaluation  For commercial scanners, contact vendor for free trial.  Don’t download old versions off of PirateBay!

19. OWASP 19 Advice for Conducting a Scanner Evaluation  Decide which applications will be scanned during evaluation. Some advice:  Avoid well-known vulnerable apps like WebGoat or HackMe Bank.  Chose a variety of technologies that represent what you’ll be scanning.  Java, ASP.Net, AJAX or Flash-heavy applications, etc.  Choose complex applications that will allow you to learn more.  Large # of dynamic pages, single sign-on, spanning multiple hostnames, multi-step sequences, multiple user roles, etc.

20. OWASP 20 Advice for Conducting a Scanner Evaluation  Prepare the applications for scanning  Best to use a non-Production environment  If scanning Production:  Back up the database first  Notify Production Support personnel of your plans  If you intend to publish your results, document the application setup so others can repeat it.

21. OWASP 21 Advice for Conducting a Scanner Evaluation  Multiple scans may be necessary using a variety of scan configurations.  “Point and shoot”  Login credentials and form-training  Monitor running scan to ensure it’s not just “spinning its wheels”.  For example, getting a 401 or “Session Timeout” response to every test…  Running scan through a proxy can help if you need to closely analyze activity, determine whether scanner is logging in successfully, etc. Scanner Target Website HTTP Proxy

22. OWASP 22 Advice for Conducting a Scanner Evaluation  Score each of the scanners on the criteria you’ve selected and add up the weighted scores.  For results, focus the amount of useful, actionable information provided by scanner, not just quantity.  Tally up the scores. You can now make an informed decision on which scanner to use, purchase, or recommend.

23. OWASP 23 Advice for Conducting a Scanner Evaluation

24. OWASP 24 For More Information View the WASSEC: http://webappsec.org/projects/wassec Join the project mailing list: wasc-wassec-subscribe@webappsec.org Contact project leader: bshura73@gmail.com