Testing Web Services by XML Perturbation Joint research with Wuzhi Xu and Juan Luo Jeff Offutt Information & Software Engineering George Mason University Fairfax, VA USA Partially supported by NASA, Avaya Research Labs and NIST

© Jeff Offutt, Web Services A Web Service is a program that offers services over the Internet to other software programs –Internet-based –Uses SOAP and XML –Peer-to-peer communication Web service components can integrate dynamically, by finding other services during execution Web services transmit data that are formatted in XML

© Jeff Offutt, Web Service Architecture Web Services server Laptop PDA Work- station Cell phone Web-based internet servers clients Client-server server clients

© Jeff Offutt, Web Service Technologies WSDL Specification Specification Components Legacy System Wrapped Specification UDDI Registry Services Wrapped Applications SOAP / XML Points to URL SOAP / XML Publish Find Bind

© Jeff Offutt, XML Software components that pass data must agree on format, types, and organization Web services have unique requirements : –Very loose coupling and dynamic integration 1970s style P1P2 File File storage Un-documented format Data saved in binary mode Source not available 1980s style P1P2 File WM File storage Un-documented format Data saved as plain text Access through wrapper module Data hard to validate

© Jeff Offutt, XML Data is passed directly between components XML allows for self-documenting data 1980s style DTD P1 P2 Parser XML File P3 P1, P2 and P3 can see the format, contents, and structure of the data Data sharing is independent of type Format is easy to understand Grammars are defined in DTDs or Schemas

© Jeff Offutt, Testing Web Services 1.This research tests web services by focusing on interactions among the components 2.A formal model of the XML grammar is created 3.Previous research generated valid tests directly from the grammar ( TAVWEB 2004 [20] ) 4.This research focuses on invalid tests 5.The grammar is mutated, or perturbed 6.The perturbed grammar is used to generate new mutated XML messages 7.The mutated messages are used as test cases

© Jeff Offutt, XML Data Model An XML schema can be modeled as a tree T = (N, D, X, E, n r ) N is a finite set of elements and attribute nodes D is a finite set of built-in and derived data types X is a finite set of constraints E is a finite set of edges Edges are from N to N  D, plus a constraint n r is the root node

© Jeff Offutt, XML Data Model Example

© Jeff Offutt, XML Data Model

© Jeff Offutt, Perturbing Schemas This part of our project focuses on perturbing schemas to create invalid data –Results creating valid data in another paper Systematic perturbation of elements in tree model –Nodes (excluding the root node) –Datatypes –Edges Perturbation operators modify schema trees to create mutated schemas

© Jeff Offutt, Perturbation Operators Insert new node Delete node Insert a new node that connects a datatype Delete a node and its dataype Insert a new subtree Delete a subtree Change constraints on a node These operators are designed to mimic common XML errors

© Jeff Offutt, Test Criteria Delete Coverage (DC) –Delete each node, edge, subtree Insert Coverage (IC) –After deleting, insert new nodes, edges, subtrees Constraint Coverage (CC) –Uses change constraints operator and rules about XML “facets” to modify all constraints

© Jeff Offutt, XML Constraints – “Facets” Boundary Constraints Non-boundary Constraints maxOccursenumeration minOccursuse lengthfractionDigits maxExclusivepattern maxInclusivenillable maxLengthwhiteSpace minExclusiveunique minInclusive minLength totalDigits

© Jeff Offutt, Test Case Generation A test case is an XML message Tests are generated directly from mutated schemas Constraints are “violated” systematically –Values beyond the boundary values “maxLength=5”  “abcdef” –Values outside the non-boundary constraints “fractionDigits=2”  “ ” Multiple XML messages from the same schema Messages are invalid, so a valid response is an error –False positives : Messages that are accidentally valid

© Jeff Offutt, Test Case Generation – Example Original Schema (Partial) Mutants : value = “3” value = “1” Mutants : value = “100” value = “2000” XML from Original Schema Mutant XML Mutant XML Mutant XML Mutant XML

© Jeff Offutt, Case Study 1 Small Web service created at GMU –Three components: Mars robot, Space station, ground control –Ground control is a three-tier web application Correct behavior is to have abnormal responses –Receiver cannot process the data, responds with a fault –Receiver has a runtime exception Results: DCICCCOriginalTotal XML Schemas XML Messages Abnormal Response Normal Response Only CC tests got normal responses

© Jeff Offutt, Case Study 2 From Web Services Interoperability Organization –Supply chain management –Seven XML schema –Three were requests and used for perturbation-based tests Mutated Schemas SchemaDCICCC Retailer14839 Warehouse6223 Manufacturer8323 XML Messages DCICCC Twenty-two faults inserted into the program Seven faults found – all by CC tests

© Jeff Offutt, Analysis of Faults 14 faults not found 6 faults : Could not be found by external inputs –Unit testing 5 faults : Affected back-end log file –Observability … log-file was not seen 1 fault : Depends on inputs from the database –Controllability … tests depend on XML, not DB 2 faults : Required specific values that were not used All DC and IC tests were detected by program

© Jeff Offutt, Current and Future Work Improve invalid test generation –Focus on constraint tests –Expand mutation of constraints Automatic test generation Merge valid and invalid tests General problems –Dealing with observability –Dealing with controllability