1. Paul Ammann & Jeff Offutt http://www.cs.gmu.edu/~offutt/softwaretest/
Introduction to Software Testing Chapter 9.1 & 9.5 Grammar-based Testing
Paul Ammann & Jeff Offutt
This version modified for our undergrad class (SWE 437).
It includes material from section 9.5.

2. Using Syntax to Generate Tests
Lots of software artifacts follow strict syntax rules
The syntax is often expressed as a grammar
Commonly BNF
Syntactic descriptions can come from many sources
Programs
Integration elements
Design documents
Input descriptions
Tests are created with two general goals
Cover the syntax in some way
Violate the syntax (invalid tests)
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

3. The set of allowable inputs to software
Input Space Grammars
Input Space
The set of allowable inputs to software
The input space can be described in many ways
User manuals
Unix man pages
Method signature / Collection of method preconditions
A language
Most input spaces can be described as grammars
Grammars are usually not provided, but creating them is a valuable service by the tester
Testers often find errors when creating the grammar
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

4. Using Input Grammars Software should reject or handle invalid data
Programs often do this incorrectly
Some programs (rashly) assume all input data is correct
Even if it works today …
What about after maintenance changes ?
What if the component is reused in a new program ?
Consequences can be severe …
The database can be corrupted
Users are not satisfied
Most security vulnerabilities are from unhandled exceptions … from invalid data
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

5. Deciding if input values can be processed by the software
Validating Inputs
Input Validation
Deciding if input values can be processed by the software
Before processing inputs, good programs check that the inputs are valid
How should a program recognize invalid inputs ?
What should a program do with invalid inputs ?
If the input space is described as a grammar, a parser can check automatically
This is very rare
It is easy to write input checkers—but also easy to make mistakes
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

6. Representing Input Domains
Desired inputs (goal domain)
Described inputs (specified domain)
Accepted inputs (implemented domain)
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

7. † More details are on : http://www.merriampark.com/anatomycc.htm
Example Input Domains
Goal domains are often irregular
Goal domain for credit cards†
First digit is the Major Industry Identifier
First 6 digits and length specify the issuer
Final digit is a “check digit”
Other digits identify a specific account
Common specified domain
First digit is in { 3, 4, 5, 6 } (travel and banking)
Length is between 13 and 16
Common implemented domain
All digits are numeric
† More details are on :
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

8. Representing Input Domains
goal domain
specified domain
This region is a rich source of software failures
implemented domain
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

9. Input Space BNF Grammars (9.5.1)
Input spaces can be expressed in many forms
Grammars are common
We will look at three grammar-based languages
Regular expressions
BNF grammars
XML and Schema
All are similar and can be used in different contexts
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

10. Initial Regular Expression
Regular Expressions
Consider bank software that processes a sequence of deposits and debits
Inputs
deposit 5306 $4.30
debit 0343 $4.14
deposit 5306 $7.29
Initial Regular Expression
(deposit account amount | debit account amount) *
Ready
deposit
debit
FSM to represent the grammar
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

11. BNF Grammar for Bank Example
Grammars are more expressive than regular expressions–they can capture more details
bank ::= action*
action ::= dep | deb
dep ::= “deposit” account amount
deb ::= “debit” account amount
account ::= digit4
amount ::= “$” digit+ “.” digit2
digit ::= “0” | “1” | “2” | “3” | “4” | “5” | “6” |
“7” | “8” | “9”
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

12. FSM for Bank Grammar
deposit
debit
digit
“.” $
Derive tests by systematically replacing each non-terminal with a production
If the tester designs the grammar from informal input descriptions, do it early
In time to improve the design
You will almost always find mistakes and omissions
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

13. eXtensible Markup Language (XML)
Software components that pass data must agree on format, types, and organization
Web applications have unique requirements :
Very loose coupling and dynamic integration
Data passed directly between components
XML allows data to be self-documenting
Schema C1 C2
Parser
XML
File C3
Components 1, 2, and 3 can see the data format, contents, and structure
Data sharing is independent of type
Format is easy to understand
Grammars are defined in schemas
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

14. schemas define input spaces for software components
XML for Book Example
<books>
<book>
<ISBN> </ISBN>
<title>The Art of Software Testing</title>
<author>Glen Myers</author>
<publisher>Wiley</publisher>
<price>50.00</price>
<year>1979</year>
</book>
</books>
XML messages are defined by grammars
Schemas and DTDs
Schemas can define types
Schemas include “facets,” which refine the grammar
schemas define input spaces for software components
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

15. XML Constraints – “Facets”
Boundary Constraints
Non-boundary Constraints
maxOccurs
enumeration
minOccurs
use
length
fractionDigits
maxExclusive
pattern
maxInclusive
nillable
maxLength
whiteSpace
minExclusive
unique
minInclusive
minLength
totalDigits
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

16. Using Grammars to Design Tests
This form of testing allows us to focus on interactions among the components
Web services depend on XML
The grammar is used to create valid as well as invalid tests
The grammar is mutated
The mutated grammar is used to generate new XML messages
The XML messages are test cases
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

17. Book Grammar – Schema <xs:element name = “books”>
<xs:complexType>
<xs:sequence>
<xs:element name = “book” maxOccurs = “unbounded”>
<xs:element name = “ISBN” type = “isbnType” minOccurs = “0”/>
<xs:element name = “author” type = “xs:string”/>
<xs:element name = “title” type = “xs:string”/>
<xs:element name = “publisher” type = “xs:string”/>
<xs:element name = “price” type = “priceType”/>
<xs:element name = “year” type = “yearType”/>
</xs:sequence>
</xs:complexType>
</xs:element>
Built-in types
<xs:simpleType name = “priceType”>
<xs:restriction base = “xs:decimal”>
<xs:fractionDigits value = “2” />
<xs:maxInclusive value = “ ” />
</xs:restriction>
</xs:simpleType>
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

18. Generating Tests Valid tests Invalid tests
Generate tests as XML messages by deriving strings from grammar
Take every production at least once
Take choices … “maxOccurs = “unbounded” means use 0, 1 and more than 1
Invalid tests
Mutate the grammar in structured ways
Create XML messages that are “almost” valid
This explores the gray space on the previous slide
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

19. Grammar-based Coverage Criteria
The most common and straightforward criteria use every terminal and every production at least once
Terminal Symbol Coverage (TSC) : TR contains each terminal symbol t in the grammar G.
Production Coverage (PDC) : TR contains each production p in the grammar G.
PDC subsumes TSC
Grammars and graphs are interchangeable
PDC is equivalent to EC, TSC is equivalent to NC
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

20. Generating Tests for Books
The only choice in the books grammar is based on “minOccurs”
Production coverage (PDC) requires two tests
ISBN is present
ISBN is not present
The facets are used to generate values that are valid
We also want values that are not valid …
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

21. Mutating Input Grammars (9.5.2)
Modify an existing grammar, then generate strings that are “close to correct”
Each mutated string is a test
Possible mutations :
Replace nonterminal symbols with other nonterminal symbols
Replace terminal symbols with other terminal symbols
Delete terminal and nonterminal symbols
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

22. New Bank Tests from Mutation
Nonterminal Replacement
dep ::= “deposit” account amount
dep ::= “deposit” amount amount
dep ::= “deposit” account digit
deposit $ $
deposit
Terminal Replacement
amount ::= “$” digit+ “.” digit2
amount ::= “.” digit+ “.” digit2
amount ::= “$” digit+ “$” digit2
amount ::= “$” digit+ “1” digit2
deposit
deposit $1500$00
deposit $
Terminal and Nonterminal Deletion
dep ::= “deposit” account amount
dep ::= account amount
dep ::= “deposit” amount
dep ::= “deposit” account
4400 $
deposit $
deposit 4400
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

23. Test Case Generation – Example
Original Schema (Partial)
<xs:simpleType name = “priceType”>
<xs:restriction base = “xs:decimal”>
<xs:fractionDigits value = “2” />
<xs:maxInclusive value = “ ” />
</xs:restriction>
</xs:simpleType>
Mutants : value = “3”
value = “1”
Mutants : value = “100”
value = “2000”
XML from Original Schema
<books>
<book>
<ISBN> </ISBN>
<price>37.95</price>
<year>2002</year>
</book>
Mutant XML 1
<books>
<book>
<ISBN> </ISBN>
<price> </price>
<year>2002</year>
</book>
Mutant XML 2
<books>
<book>
<ISBN> </ISBN>
<price>37.95</price>
<year>2002</year>
</book>
Mutant XML 3
<books>
<book>
<ISBN> </ISBN>
<price> </price>
<year>2002</year>
</book>
Mutant XML 4
<books>
<book>
<ISBN> </ISBN>
<price> </price>
<year>2002</year>
</book>
505 5
99.00
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt

24. Input Space Grammars Summary
This application of mutation is fairly new
Automated tools do not exist
Can be used by hand in an “ad-hoc” manner to get effective tests
Applications to special-purpose grammars very promising
XML
SQL
HTML
Introduction to Software Testing, edition 2 (Ch 9)
© Ammann & Offutt