Blending Automated and Manual Testing Making Application Vulnerability Management Pay Dividends

My Background Dan Cornell, founder and CTO of Denim Group Software developer by background (Java,.NET, etc) OWASP San

My Background Steve Springett, Application Security Architect for Axway Software developer by background Leader of OWASP Dependency-Track Contributor to OWASP

Goal: Continuous Security Prerequisites – Standardization – Continuous Integration – Continuous Delivery Compliments – Continuous Acceptance

Standardization All projects use same build system All projects built the same way Automated onboarding for new projects Per-project build expertise not required

Metrics Artifacts Continuous Integration Continuous Integration Factory Source Code (SCM)

Deliverables Continuous Delivery Continuous Delivery Factory Artifacts

Security Metrics Continuous Security Continuous Security Factory Source Code (SCM) Deliverables

Automated Security Metrics Static Analysis Findings Dynamic Analysis Findings Component Analysis Findings Attack Surface Analysis Findings

Continuous Security Pipe Jenkins CI ThreadFix Defect Tracker SCM False Positive

Target Application

12 ThreadFix Accelerate Software Remediation ThreadFix is a software vulnerability aggregation and management system that helps organizations aggregate vulnerability data, generate virtual patches, and interact with software defect tracking systems.

ThreadFix Open Source (MPL) application vulnerability management platform Create a consolidated view of your applications and vulnerabilities Prioritize application risk decisions based on data Translate vulnerabilities to developers in the tools they are already using

ThreadFix Community Edition Main ThreadFix website: – General information, downloads ThreadFix GitHub site: – Code, issue tracking ThreadFix GitHub wiki: – Project documentation ThreadFix Google Group: – Community support, general discussion

Vulnerability Aggregation Automated Manual

Access to Vulnerability Data Tradeoffs – The more places the vulnerability data lives, the more likely a compromise – Withholding information from people who need it makes remediation more challenging

Managing All Vulnerability Data Manual activities – Penetration Testing – Code Reviews 3 rd Party Data Sources – Customer-performed Testing – External auditor-performed Results

SSVL and Manual Results SSVL Data Format: – SSVL Conversion Tool: –

RESTful API to Vulnerability Data Custom R&D Monitoring Dashboard Custom Dashboards

Key Performance Indicators Don’t go overboard – Use only what is needed Progress and velocity Per team comparison Min/max/avg time to close per severity By CWE

Lessons Learned Always automate static analysis Always automate attack surface analysis Always automate component analysis Always automate dynamic analysis Always perform manual dynamic analysis Use native tools & workflow for static analysis

Lessons Learned Provide as much visibility as possible – Varying degrees of detail – Multiple delivery vehicles Set clear pass/fail criteria for Security Bars – Provide custom dashboard to provide status and advanced warning

Additional Advice Automation is not better than manual – It’s faster and more efficient – Both are necessary Don’t forget manual assessments – Threat Modeling – Secure Design/Architecture and Code Review – Penetration Testing

Finally Vulnerabilities in CI / CD / CS Infrastructure – Threat Model – Secure Architecture Review – Patch Management – Configuration Management – Key Management – Always use TLS

Q & A