Gray, the New Black Gray-Box Web Vulnerability Testing Brian Chess Founder / Chief Scientist Fortify Software, an HP Company June 22, 2011

Todo Define gray-box testing Why black-box is insufficient What we built Examples Haters club

Definitions Black-box testing System-level tests No assumptions about implementation

Definitions White-box testing Examine implementation Test components in isolation

Definitions Gray-box testing System-level tests (like black-box) Examine implementation (like white-box)

The Software Security Game Objective Rules vs. Strategy Playing Field

OBJECTIVE: Protect everything OBJECTIVE: Exploit one vulnerability

Rules for the Defender 1.Don’t attack the attacker

Rules vs. Strategy Rules Don’t attack the attacker Strategy Emulate attacker’s techniques

Who wins? Technology Expertise

Who wins? Time Technology Expertise

Who wins? Technology Expertise Time

Changing the odds

The Defender’s Advantage Time Inside Access Technology Expertise

Prior Art 2005: Concolic testing: Sen, University of Illinois 2008: Microsoft SAGE: Godefroid, MSR 2008: Test Gen for Web Apps: Shay et al, U. Washington 2008: Accunetix: Accusensor

Access to the Software Allows for ‘Hybrid’ analysis Black-box ApproachWhite-box Approach

‘Hybrid’ Analysis Mostly Broken Correlation Engine

The ‘Real-Time Hybrid’ Approach Good Results Correlation Engine

Evolving to Integrated Analysis Application Real-time link Find More Fix Faster

Find More Reduce false negatives Automatic attack surface identification Understand effects of attacks Detect new types of vulnerabilities Privacy violation, Log Forging

Attack surface identification /login.jsp /pages/account.jsp /pages/balance.jsp /admin/admin.jsp File system Configuration-driven Programmatic

Understand effects of attacks /admin/admin.jsp ✗ Command Injection sysadmin$./sh ✔

Fix Faster Reduce False Positives Confirm vulnerabilities Provide Actionable Details Stack trace Line of code Collapse Duplicate Issues Tie to root cause

Reduce False Positives /admin/admin.jsp SQLi? ✔

Actionable Details /login.jsp

Collapse Duplicate Issues /login.jsp /pages/account.jsp /pages/balance.jsp 1Cross-Site Scripting23 1

JavaBB – Case Study Open Source Bulletin Board Additional Vulnerabilities Finds18 SQL Injection results Root cause analysis 18 SQL injection results have 1 root cause

Vulnerability Diagnosis Confirmed SQL Injection

Actionable Details Line of Code Parameters Stack Trace

Yazd – Case Study Open Source Forum Additional Attack Surface Discovers hidden ‘admin’ area 3 Additional Cross-Site Scripting results Root cause analysis Collapses 34 XSS into 24 root-cause vulnerabilities

Attack surface identification Hidden ‘admin’ area

Collapse Duplicate Issues

One More Case Study

Future Automated anti-anti automation

The Case Against “Hybrid” Hard to find attack surface with static analysis Static/dynamic correlation doesn’t work Doesn’t help with false positives / false negatives Nobody will run a software monitor (cheating!)

The Case for Gray-Box Testing Black-box is a losing game Find more Attack surface Vulnerability diagnosis Fix faster Root cause analysis Collapse duplicates

Gray, the New Black Gray-Box Web Vulnerability Testing Brian Chess Founder / Chief Scientist Fortify Software, an HP Company June 22, 2011