1. Testing Web Services by XML Perturbation Joint research with Wuzhi Xu and Juan Luo Jeff Offutt Information & Software Engineering George Mason University Fairfax, VA USA www.ise.gmu.edu/~offutt/offutt@ise.gmu.edu Partially supported by NASA, Avaya Research Labs and NIST

2. © Jeff Offutt, 20052 Web Services A Web Service is a program that offers services over the Internet to other software programs –Internet-based –Uses SOAP and XML –Peer-to-peer communication Web service components can integrate dynamically, by finding other services during execution Web services transmit data that are formatted in XML

3. © Jeff Offutt, 20053 Web Service Architecture Web Services server Laptop PDA Work- station Cell phone Web-based internet servers clients Client-server server clients

4. © Jeff Offutt, 20054 Web Service Technologies WSDL Specification Specification Components Legacy System Wrapped Specification UDDI Registry Services Wrapped Applications SOAP / XML Points to URL SOAP / XML Publish Find Bind

5. © Jeff Offutt, 20055 XML Software components that pass data must agree on format, types, and organization Web services have unique requirements : –Very loose coupling and dynamic integration 1970s style P1P2 File File storage Un-documented format Data saved in binary mode Source not available 1980s style P1P2 File WM File storage Un-documented format Data saved as plain text Access through wrapper module Data hard to validate

6. © Jeff Offutt, 20056 XML Data is passed directly between components XML allows for self-documenting data 1980s style DTD P1 P2 Parser XML File P3 P1, P2 and P3 can see the format, contents, and structure of the data Data sharing is independent of type Format is easy to understand Grammars are defined in DTDs or Schemas

7. © Jeff Offutt, 20057 Testing Web Services 1.This research tests web services by focusing on interactions among the components 2.A formal model of the XML grammar is created 3.Previous research generated valid tests directly from the grammar ( TAVWEB 2004 [20] ) 4.This research focuses on invalid tests 5.The grammar is mutated, or perturbed 6.The perturbed grammar is used to generate new mutated XML messages 7.The mutated messages are used as test cases

8. © Jeff Offutt, 20058 XML Data Model An XML schema can be modeled as a tree T = (N, D, X, E, n r ) N is a finite set of elements and attribute nodes D is a finite set of built-in and derived data types X is a finite set of constraints E is a finite set of edges Edges are from N to N  D, plus a constraint n r is the root node

9. © Jeff Offutt, 20059 XML Data Model Example

10. © Jeff Offutt, 200510 XML Data Model

11. © Jeff Offutt, 200511 Perturbing Schemas This part of our project focuses on perturbing schemas to create invalid data –Results creating valid data in another paper Systematic perturbation of elements in tree model –Nodes (excluding the root node) –Datatypes –Edges Perturbation operators modify schema trees to create mutated schemas

12. © Jeff Offutt, 200512 Perturbation Operators Insert new node Delete node Insert a new node that connects a datatype Delete a node and its dataype Insert a new subtree Delete a subtree Change constraints on a node These operators are designed to mimic common XML errors

13. © Jeff Offutt, 200513 Test Criteria Delete Coverage (DC) –Delete each node, edge, subtree Insert Coverage (IC) –After deleting, insert new nodes, edges, subtrees Constraint Coverage (CC) –Uses change constraints operator and rules about XML “facets” to modify all constraints

14. © Jeff Offutt, 200514 XML Constraints – “Facets” Boundary Constraints Non-boundary Constraints maxOccursenumeration minOccursuse lengthfractionDigits maxExclusivepattern maxInclusivenillable maxLengthwhiteSpace minExclusiveunique minInclusive minLength totalDigits

15. © Jeff Offutt, 200515 Test Case Generation A test case is an XML message Tests are generated directly from mutated schemas Constraints are “violated” systematically –Values beyond the boundary values “maxLength=5”  “abcdef” –Values outside the non-boundary constraints “fractionDigits=2”  “456. 324” Multiple XML messages from the same schema Messages are invalid, so a valid response is an error –False positives : Messages that are accidentally valid

16. © Jeff Offutt, 200516 Test Case Generation – Example Original Schema (Partial) Mutants : value = “3” value = “1” Mutants : value = “100” value = “2000” XML from Original Schema 0-201-74095-8 37.95 2002 Mutant XML 1 0-201-74095-8 37.95 2002 505 Mutant XML 2 0-201-74095-8 37.95 2002 5 Mutant XML 3 0-201-74095-8 37.95 2002 101.00 Mutant XML 4 0-201-74095-8 37.95 2002 1001.00

17. © Jeff Offutt, 200517 Case Study 1 Small Web service created at GMU –Three components: Mars robot, Space station, ground control –Ground control is a three-tier web application Correct behavior is to have abnormal responses –Receiver cannot process the data, responds with a fault –Receiver has a runtime exception Results: DCICCCOriginalTotal XML Schemas233611463 XML Messages641035312232 Abnormal Response64125110190 Normal Response00421242 Only CC tests got normal responses

18. © Jeff Offutt, 200518 Case Study 2 From Web Services Interoperability Organization –Supply chain management –Seven XML schema –Three were requests and used for perturbation-based tests Mutated Schemas SchemaDCICCC Retailer14839 Warehouse6223 Manufacturer8323 XML Messages DCICCC 12971090 15486 14483 Twenty-two faults inserted into the program Seven faults found – all by CC tests

19. © Jeff Offutt, 200519 Analysis of Faults 14 faults not found 6 faults : Could not be found by external inputs –Unit testing 5 faults : Affected back-end log file –Observability … log-file was not seen 1 fault : Depends on inputs from the database –Controllability … tests depend on XML, not DB 2 faults : Required specific values that were not used All DC and IC tests were detected by program

20. © Jeff Offutt, 200520 Current and Future Work Improve invalid test generation –Focus on constraint tests –Expand mutation of constraints Automatic test generation Merge valid and invalid tests General problems –Dealing with observability –Dealing with controllability http://www.ise.gmu.edu/~offutt/http://mason.gmu.edu/~wxu2/