Blending Automated and Manual Testing Making Application Vulnerability Management Pay Dividends
My Background Dan Cornell, founder and CTO of Denim Group Software developer by background (Java,.NET, etc) OWASP San
My Background Steve Springett, Application Security Architect for Axway Software developer by background Leader of OWASP Dependency-Track Contributor to OWASP
Goal: Continuous Security Prerequisites – Standardization – Continuous Integration – Continuous Delivery Compliments – Continuous Acceptance
Standardization All projects use same build system All projects built the same way Automated onboarding for new projects Per-project build expertise not required
Metrics Artifacts Continuous Integration Continuous Integration Factory Source Code (SCM)
Deliverables Continuous Delivery Continuous Delivery Factory Artifacts
Security Metrics Continuous Security Continuous Security Factory Source Code (SCM) Deliverables
Automated Security Metrics Static Analysis Findings Dynamic Analysis Findings Component Analysis Findings Attack Surface Analysis Findings
Continuous Security Pipe Jenkins CI ThreadFix Defect Tracker SCM False Positive
Target Application
12 ThreadFix Accelerate Software Remediation ThreadFix is a software vulnerability aggregation and management system that helps organizations aggregate vulnerability data, generate virtual patches, and interact with software defect tracking systems.
ThreadFix Open Source (MPL) application vulnerability management platform Create a consolidated view of your applications and vulnerabilities Prioritize application risk decisions based on data Translate vulnerabilities to developers in the tools they are already using
ThreadFix Community Edition Main ThreadFix website: – General information, downloads ThreadFix GitHub site: – Code, issue tracking ThreadFix GitHub wiki: – Project documentation ThreadFix Google Group: – Community support, general discussion
Vulnerability Aggregation Automated Manual
Access to Vulnerability Data Tradeoffs – The more places the vulnerability data lives, the more likely a compromise – Withholding information from people who need it makes remediation more challenging
Managing All Vulnerability Data Manual activities – Penetration Testing – Code Reviews 3 rd Party Data Sources – Customer-performed Testing – External auditor-performed Results
SSVL and Manual Results SSVL Data Format: – SSVL Conversion Tool: –
RESTful API to Vulnerability Data Custom R&D Monitoring Dashboard Custom Dashboards
Key Performance Indicators Don’t go overboard – Use only what is needed Progress and velocity Per team comparison Min/max/avg time to close per severity By CWE
Lessons Learned Always automate static analysis Always automate attack surface analysis Always automate component analysis Always automate dynamic analysis Always perform manual dynamic analysis Use native tools & workflow for static analysis
Lessons Learned Provide as much visibility as possible – Varying degrees of detail – Multiple delivery vehicles Set clear pass/fail criteria for Security Bars – Provide custom dashboard to provide status and advanced warning
Additional Advice Automation is not better than manual – It’s faster and more efficient – Both are necessary Don’t forget manual assessments – Threat Modeling – Secure Design/Architecture and Code Review – Penetration Testing
Finally Vulnerabilities in CI / CD / CS Infrastructure – Threat Model – Secure Architecture Review – Patch Management – Configuration Management – Key Management – Always use TLS
Q & A