1. The role of the test organization in a Security Sensitive project
Ariel Cymberknoh
QA & Test Conference
Bilbao
October 2018

2. CSME - Converged Security & Manageability Engine
The role of the test organization in a Security Sensitive project
CSME - Converged Security & Manageability Engine
Micro Runtime
(JVM)
DAL Internal API
Ariel Cymberknoh – QA & Test conference – October 2018

3. The role of the test organization in a Security Sensitive project
Development
Validation/Testing
Security Assurance
Ariel Cymberknoh – QA & Test conference – October 2018

4. Security Transformation
The role of the test organization in a Security Sensitive project
Security Transformation
Recover
SW/FW update, Customer engagement
SDL = Security Development Lifecycle
Used and proposed by Microsoft*
Widely adopted industry standard
In a Glance:
S1  Architecture Review
S2  Design Review
S3  Code Review
S4  Security and Penetration testing
Protect
Advanced Mitigations (CET, Stack Canaries etc.)
Isolation, minimal Privilege, ARB
Detect
Advanced Code analysis, AFL, Advanced Flow reviews, External Reviews, Red Team
Secure Code
S0/S1/S2/S3/S4- SDL
Architecture and design changes
Org Decision to
Transform
Prioritize security, Mindset, accountability,
Practices, Award,
Training,
*Trade names are the ownership of the relevant companies
Ariel Cymberknoh – QA & Test conference – October 2018

5. The role of the test organization in a Security Sensitive project
Development
Validation/Testing
Security Assurance
Ariel Cymberknoh – QA & Test conference – October 2018

6. The role of the test organization in a Security Sensitive project
Development
Validation/Testing
Security Assurance
Difficult to recruit
Retention issues
 + CO$T
Ariel Cymberknoh – QA & Test conference – October 2018

7. The role of the test organization in a Security Sensitive project
Testing/Validation Teams:
Functional testing
Advanced testing:
Negative
Corner Cases
Stress/Reliability
Performance
Certification
Usability
Compatibility
Manual / Automated testing
Security Assurance Teams:
Security Research
Security Architecture
Architecture Review
Design Review
Code Review
Static Code Analysis tools
Manual Code review
Penetration Testing
Commercial Fuzzers
In house developed Fuzzers
“Standard” hacking techniques
Advanced hacking techniques
Ariel Cymberknoh – QA & Test conference – October 2018

8. The role of the test organization in a Security Sensitive project
Testing/Validation Teams:
Functional testing
Advanced testing:
Negative
Corner Cases
Stress/Reliability
Performance
Certification
Usability
Compatibility
Manual / Automated testing
Security Assurance Teams:
Security Research
Security Architecture
Architecture Review
Design Review
Code Review
Static Code Analysis tools
Manual Code review
Penetration Testing
Commercial Fuzzers
In house developed Fuzzers
“Standard” hacking techniques
Advanced hacking techniques
Ariel Cymberknoh – QA & Test conference – October 2018

9. The role of the test organization in a Security Sensitive project
Fuzzing or fuzz testing:
Is an automated software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program.
The program is then monitored for exceptions such as crashes, failing built-in code assertions, or potential memory leaks.
Typically, fuzzers are used to test programs that take structured inputs. This structure is specified, e.g., in a file format or protocol and distinguishes valid from invalid input.
An effective fuzzer generates semi-valid inputs that are "valid enough" in that they are not directly rejected by the parser, but do create unexpected behaviors deeper in the program and are "invalid enough" to expose corner cases that have not been properly dealt with.
For the purpose of security, input that crosses a trust boundary is often the most interesting.
Source: Wikipedia
Ariel Cymberknoh – QA & Test conference – October 2018

10. The role of the test organization in a Security Sensitive project
Fuzzing or fuzz testing:
Phase 1: Test Team only deals with the tools’ execution
Enable massive production execution
Phase 2: Test team participates in the tool’s development
Phase 3: Test team handles also the debugging and triage of issues found by the tools
Phase 4: Test team autonomously decides and selects which tools to use
Ariel Cymberknoh – QA & Test conference – October 2018

11. The role of the test organization in a Security Sensitive project
Validation engagement in Security activities:
Mindset, mindset, mindset!
Training
Identifying security champions within the sub-teams
1st phase: Security Assurance team keeps “complex” areas. 2nd phase: Validation team takes over
1st phase: Security Assurance team indicates “what” security testing to implement in new requirements. Validation develops the tests and execute. 2nd phase: Validation team takes over also the test strategy definition.
Utilize Existing test cases and modify them into security ones
Participate actively in security hackathons
Recognize the work of validation people contributing to security activities
Treat security bugs as top priority
Ariel Cymberknoh – QA & Test conference – October 2018

12. The role of the test organization in a Security Sensitive project
Summary:
Security Assurance teams are expensive, not scalable
Organizations need to know how to offload part of their tasks into the Design and the Test teams
Taking over Security tasks in validation teams takes time and should be done gradually
Need managerial commitment and a profound change of mindset!
Ariel Cymberknoh – QA & Test conference – October 2018

13. The role of the test organization in a Security Sensitive project
THANK YOU!
Ariel Cymberknoh – QA & Test conference – October 2018