Testing Electronic Health Records Applications with a Security Test Pattern Developed Using Empirical Data Ben Smith Motivation Knowledge gap in software security testing expertise. Need vehicle to capture and disseminate knowledge about how to attack systems that novices can understand. Introduced and evaluated a pattern catalog of software security test patterns. Contributions Introduced the process for empirical development of test patterns using a grounded theory approach. Developed first six test patterns that target the CWE/SANS Top 25. Applied pattern catalog to 284 public requirements for EHR systems. Created 137 black box test cases, and ran these on five EHRs for 685 test executions. Thirty-seven percent (37%) or 253 of the tests revealed vulnerabilities. Different vulnerabilities than static analysis/automated penetration testing. Security Test Pattern Components Keywords Targeted Vulnerability Types CIA Properties Procedure Template Expected Results Template Example Procedure Example Expected Results User Study Conducted a study of 47 novices applying the six patterns on six requirements from the public requirements document. Created a consensus using a panel of seven experts. Novices make similar decisions about which patterns are applicable as experts do. Novices spent less than 18 minutes parsing the requirements and produced on average 15 tests. Novices reported that they thought the exercise would be useful for security. - still less text - make a handout with the pattern on it and maybe just have the component headers - “security testing knowledge transfer” Highlight the lack of expertise in the common developer/tester population. - remove the icons - show STPI with a pre-parsed requirement - indicate that we empirically-developed the patterns to target the CWE/SANS Top 25 - say what the top 25 actually are - remove the problem/object/approach/evaluation stuff and just have the highlights - make andy’s chart: “Requirement” -> a) actor (icon), b) action (icon), c) object (icon) -> test cases - make sure a list of all the patterns is still there - pattern -> “security test pattern” - highlight the fact that I evaluated it. - put the number of failures in addition to the percentage Make sure there is a tracing from the tool to the example pattern. http://securitytestpatterns.org