Formal Testing of Web Content using TTCN-3 By Robert Probert, Bernard Stepien, Pulei Xiong University of Ottawa
Motivation Web Applications (WA) are the most widespread target domain of software development in computer history. WA are error prone because mostly because anyone can develop them and they are mostly distributed. Testing of WAs: manual or using dedicated tools or robots.
Current state of research on formal Web content testing XML based formalism Xiaoping Jia and Hongmin Liu, Rigorous and Automatic Testing of Web Applications Sahil Thaker, Robust Testing for Web Applications
Web content testing elements Hyperlink checking, including recursive Text fragment checking Protocol checking Table checking Form checking Formatting checking Mirror sites content checking Version checking (languages,etc…) Invalid HTML syntax checking …
Specific needs of web content testing TTCN has been originally designed for telecommunication protocols. In Telecom, every single data element is relevant and needs to be coded and decoded to and from the abstract layer. In Web application, only part of the content is testable and needs to be extracted because most of the web page content is either too large or unpredictable. Thus, in web content testing, we test only invariants.
DePaul U Formalism BNF TestSpec ::= TestSuite* TestSuite ::= TestCase* TestCase ::= TestStep + TestStep ::= Name Condition opt RequestSpec ResponseSpec TestStep* RequestSpec ::= URL RecursiveSpec opt HearderSpec* ParameterSpec* VariableDecl* ResponseSpec ::= StatusCode ContentType HearderSpec* VariableDecl* Predicate + RecursiveSpec ::= Depth Domain opt HeaderSpec ::= Name CDATA optional opt ParameterSpec ::= Name CDATA optional opt Condition ::= Predicate Predicate ::= not Predicate | Predicate and predicate | Predicate or Predicate | Predicate implies Predicate | ( forall VariableDecl + Predicate ) | ( exists VariableDecl + Predicate ) | SimplePredicate SimplePredicate ::= MatchPred | ContainPred | ComparePred MatchPred ::= SelectExp MatchOp ( RegExp | CDATA ) MatchOp ::= equals | contains | startsWith | endsWith ComparisonPred ::= Exp ComparisonOp Exp Exp ::= NumberExp | DateExp ComparisonOp ::= == | != | | >= ContainPred ::= HrefSpec | FormSpec HrefSpec ::= URL CDATA opt VariableDecl opt FormSpec ::= ActionSpec opt MethodSpec opt VariableDecl ::= Name SelectExp
Text Fragment checking DePaul U formalism <match op="contains" regexp="false" select="/html/body" value="Undergraduate Degree"/> <match op="contains" regexp="false" select="/html/body" value="Bachelor Degree"/> <match op="contains" regexp="true" select="/html/body” value="[M|m]aster [D|d]egree"/>
Mapping DePaul U Formalism to TTCN-3 Testsuite -> testsuite Testcase -> testcase Teststep -> function Request -> send Response -> receive Match -> built-in TTCN-3 matching mech. Regexp -> pattern Value -> template
Text Fragment checking TTCN-3 types and templates template charstring program_page_web_url := " type record content_response_type { charstring status, charstring content } template content_response_type content_response_pred :={ status := "HTTP/ OK", content := pattern "*<BODY * [Undergraduate Degree | Bachelor Degree] * [[M|m]aster [D|d]egree] * *" }
Text Fragment checking TTCN-3 test case testcase Content_check_pred_TC() runs on MTCType system SystemType { var verdicttype theOverallVerdict; map(mtc:web_port, system:system_web_port); theOverallVerdict := CheckContent_pred(program_page_web_url, content_response_pred); setverdict(theOverallVerdict); }
Text Fragment checking TTCN-3 test logic function CheckContent_pred(charstring theURLLink, content_response_type theContent_response) runs on MTCType return verdicttype { var charstring theBadResponse; var verdicttype theFinalVerdict; web_port.send(theURLLink); alt { [] web_port.receive(theContent_response){ theFinalVerdict := pass; } [] web_port.receive(charstring:?) -> value theBadResponse { log("Bad Response: " & theBadResponse); theFinalVerdict := fail; } return theFinalVerdict; }
Contains concept and TTCN-3 pattern The ‘and’ operator does not imply sequence. Permutations of values must be specified in the TTCN-3 template using pattern. Combinatorics explosion when the number of text fragments increases (np). TTCN-3 should implement a concept of “contains” with logical expressions
Evaluation of TTCN-3 advantages with text fragment checking DePaul U has the contains concept. TTCN-3 has better structuring of data sent or received. TTCN-3 has the concept of alternative to structure exceptions, etc… TTCN-3 is more flexible because new functionalities can be added both at the abstract and adapter level.
Recursive hyperlink checking DePaul formalism <request url=" recursive="true", recursivedepth="3"/>
Recursive hyperlink checking TTCN-3 testcase Link_Check_TC() runs on MTCType system SystemType {... theOverallVerdict := CheckChildLink(main_page_web_url, 1); setverdict(theOverallVerdict); } Solution: Test case invokes a recursive function and appropriate list type definitions. type record of charstring HrefListType; type record hyperlink_response_type { charstring status, HrefListType hrefList } template hyperlink_response_type hyperlink_response :={ status := "HTTP/ OK", hrefList := ? }
Recursive hyperlink checking TTCN-3 recursive function function CheckChildLink(charstring theURLLink, integer theDepth) runs on MTCType return verdicttype { web_port.send(theURLLink); web_port.receive(hyperlink_response) -> value theHyperlink_response; if(theDepth <= maxDepth) { var integer theNewDepth; var HrefListType theResponseHrefList :=theHyperlink_response.hrefList; var integer numOfLinks := sizeof(theResponseHrefList); var integer i; theNewDepth := theDepth + 1; for(i:=0; i < numOfLinks; i:=i+1) { oneVerdict := CheckChildLink(theResponseHrefList[i], theNewDepth); if(oneVerdict == fail) { theFinalVerdict := fail;} } return theFinalVerdict; }
Evaluation of TTCN-3 advantages with recursive link checking TTCN-3 is more powerful because it allows more filtering on the recursion. For example, one would only recurse through the links to other web pages internal to the same company and ignore external links. TTCN-3 allows more sophisticated recursion control than simple depth limits. DePaul formalism is more compact but also more limited.
Quantified predicate checking DePaul U Formalism <match op="startswith" select="$l" regexp="false" value="
Mapping to TTCN-3 Forall -> TTCN-3 for loop with bounds StartsWith -> TTCN-3 substring Select -> done in TTCN-3 CODEC
Quantified predicate checking TTCN-3 testcase Forall_protocol_check_TC(…) runs on MTCType system SystemType { web_port.send(theURLLink); web_port.receive(hyperlink_response) -> value theHyperlink_response; all_match := true; for(i:=0; i < numOfLinks; i:=i+1) { var HrefListType theResponseHrefList := theHyperlink_response.hrefList; if(substr(theResponseHrefList[i],0,theLength) != theProtocol) { all_match := false; log("link: " & theResponseHrefList[i] & " is not of protocol " & theProtocol); } if(all_match) { setverdict(pass); }else { setverdict(fail); } }
Remarks about the TTCN-3 solution Matching is not achieved via the inherent TTCN- 3 matching mechanism, but instead using traditional programming languages features (substr). This is mainly because it is impossible to match a list of unknown length. There is no parametric solution. Two different select values would require a dedicated definition of types, templates and test cases for each select values.
Our Recommendation parametric CoDec type record myRecord { integer field_1, charstring field_2 codecparameter, charstring field_3 } template myRecord myTemplate := { field_1 := 5, field_2 := “href”, field_3 := “abcd” } Some fields act as parameters passed to the CODEC rather than matching values coming from the CODEC
Sahil Thaker WebDeveloppers ltd Form testing formalism
Form testing in TTCN-3 type definitions type record FormResponseType { charstring status, FormType form } type record FormType { charstring name, charstring method, charstring action, InputListType inputList } type record InputType { charstring name, charstring type_input, integer size_input, charstring value_input } type record of InputType InputListType ;
Form testing in TTCN-3 template definitions template FormType theRegistrationForm_template := { name := "subscription_form", method := "POST", action := " inputList := { {name :="firstname", type_input := "text", size_input := 20, value_input := ""}, {name :="lastname", type_input := "text", size_input := 35, value_input := ""}, {name :="city", type_input := "text", size_input := 40, value_input := ""}, {name :="send_me_ s", type_input := "checkbox", size_input := 0, value_input := ""}, {name :="wine_and_cheese", type_input := "checkbox", size_input := 0, value_input := ""}, {name :="", type_input := "submit", size_input := 0, value_input := "send"}, {name :="", type_input := "reset", size_input := 0, value_input := "Oooops!_Let_me_try_again!"} }
Form testing in TTCN-3 parametric testcase definition testcase Form_Check_TC(charstring theURLLink, FormResponseType theFormResponse) runs on MTCType system SystemType { … web_port.send(theURLLink); alt { [] web_port.receive(theFormResponse) -> value theFormResponseTypeValue { log("the form " & theFormResponseTypeValue.form.name & " conforms to specification"); setverdict(pass); } [] web_port.receive(FormResponseType:?) { log("in the catch all receive !"); setverdict(fail); }
Form testing in TTCN-3 test control definition control { execute (Form_Check_TC(form_page_web_url, form_response(theRegistrationForm_template))); execute (Form_Check_TC(another_web_page_url, form_response(another_form_template))) } Maximizing re-usability using: Parametric test cases Parametric data templates
Advantages of TTCN-3 Parametric test cases Parametric templates Structuring at all levels, types, templates, test cases, functions Concept of control Test configuration specification
Conclusions TTCN-3 is adequate for web content testing. There is no need to create new formalism dedicated to web content testing. Instead, it would be more beneficial to use TTCN-3 to develop generic web content testing tools mostly because of TTCN-3 parametric features.