1. Fujitsu Computer Systems Testing Considerations (Past experience, Methodology, Options…) CAMP F2F Redwood City, November 2012 Jacques Durand Fujitsu 1

2. Fujitsu Computer Systems Testing serves different purposes Validating a specification Verifying an implementation See: https://wiki.oasis-open.org/tab/TestingPolicy 2

3. Fujitsu Computer Systems Testing serves different purposes Validating a specification ambiguities, fuzziness gaps (unspecified features) inconsistencies / conflicts Verifying an implementation conformance interoperability: two+ implementations interoperable (or conforming artifacts portable?) maintenance: is the implementation still conforming/interoperable over time? (changes in config, etc.) See: https://wiki.oasis-open.org/tab/TestingPolicy 3

4. Fujitsu Computer Systems Fujitsu America, Inc. Test Case Test Case Test assertion Test assertion Specification Test assertions Test Case is addressing Actual Test suite Normative statement Normative statement Normative statements derived from Test assertions: a common test foundation Test Assertion: A “measurable/testable statement of behavior for an implementation” Blueprint for an executable test unit see: https://wiki.oasis-open.org/tab/TestabilityGuidehttps://wiki.oasis-open.org/tab/TestabilityGuide see: https://www.oasis- open.org/committees/download.php/44696/taguidelines-v1.0-wd02.pdf 4

5. Fujitsu Computer Systems Fujitsu America, Inc. Example: a specification on “widgets” [requirement 1023] “A widget MUST be of cuboid shape”. Here is a test assertion addressing this requirement: TA id: widget-TA100-1 Target: widget Normative Source: “widget specification”, requirement 1023 Predicate: [the widget] has six facets, and each one of these is of rectangular shape. Prescription Level: mandatory Worded: in knowledge of the testing ability understandable by a non-expert 5

6. Fujitsu Computer Systems Fujitsu America, Inc. An example TA (CIMI) - The normative source is an interpretation (summary) of several normative statements in the original specification - The “SUT” tag identifies the larger entity “responsible” for conformance, while the “target” is a sequence of messages 6

7. Fujitsu Computer Systems Fujitsu America, Inc. An example TA (SCA) - If the “prerequisite” is false, the “target” is not qualified for further evaluation of the “predicate”. - The tags here are just keywords used to classify this TA. 7

8. Fujitsu Computer Systems What is to be tested PaaS Provider Black-box testing: through the testing of concrete test “targets” being exchanges: artifacts, messages Major CAMP entities or roles Appl. developer App User#1 Appl. End-user Appl. developer Platform Administrator Messages Artifacts 8

9. Fujitsu Computer Systems Conformance aspects to verify: ( 1 ) Message & Artifact well-formedness Protocol aspects schema validation, HTTP headers… Request/Response basic consistency (2) Consistency between exchanges and what is currently deployed on the PaaS platform Right set of resources created, after operations? message responses consistent with platform state? 9

10. Fujitsu Computer Systems Benefits of Test Assertions (1): Improve specifications by “translating” normative requirements in precise implementation behaviors, discover : gaps / ambiguities / inconsistencies Align implementations ( before testing ) lead to a common interpretation of the spec. Facilitate Testing starting point for Test Suites 10

11. Fujitsu Computer Systems When to write Test Assertions ? POC work Test Suite developt Product dev Implementation work Technical work On Specification Standardization (WD  CS  OS) ? ? ? specification work Best for maximum benefit: while writing the spec 11

12. Fujitsu Computer Systems A look at 3 Testing Set-ups: ( all useful ) PaaS Provider (System Under Test) Test Driver - a single implementation of System Under Test (SUT) (CAMP) is being tested. - Primary goal: conformance assessment 1. Isolation Testing 12

13. Fujitsu Computer Systems A look at 3 Testing Set-ups: ( all useful ) 2. MatrixTesting App User#1 - a test scenario is repeated for each pair of implementations (or pair PaaS Consumer-Provider) - main goal: interoperability assessment App User#1 PaaS Provider A App User#1 App User#2 App User#1 App User#3 PaaS Provider B PaaS Provider C 13

14. Fujitsu Computer Systems A look at 3 Testing Set-ups: ( all useful ) PaaS Provider (System Under Test) Test Driver - concerns in-production systems - main goal: check for regressions - 2 variants: (a) driven tests (b) real exchange monitoring 3. Maintenance Testing (b) (a) App User#1 App User#1 In-production Provider 14

15. Fujitsu Computer Systems 15 Separating Test Operation from Test Analysis allows for reusing analysis test tool in various set-ups Phase 1: Test operations Test Analyzer Test assertions Phase 2: Test analysis Execution Trace Test Report Fujitsu America, Inc. SUT Test Driver SUT #1 SUT #2 SUT #3 users S UT In-production Provider Isolation testing Matrix testing Maintenance testing

16. Fujitsu Computer Systems Test Tools: overview Web Service ANALYZER MONITOR Interceptor Logger Message artifacts Example of 2-phase testing: WS-Interoperability Testing Execution report Test report Fujitsu America, Inc. Phase 1 Phase 2 Test SCENARIOS Client code 16

17. Fujitsu Computer Systems Benefits of Test Assertions (2): when implementing test suites Single Test Tool/Suite for conformance logic usable in all test set-ups Visibility of conformance logic in test suite ( blueprint for test cases ) in test report ( each report item refers to a T.A.) 17