1. SE 3800 Note 5 Beyond Functional Testing
Dr. Rob Hasker
Dr. Brad Dennis
SE 3800 Note 5 Beyond Functional Testing

2. Reviewing CI Setup So what do the steps do? Key concept: search path
Windows: for each command, run through PATH to find first folder containing specified command (an executable with the same name)
Variables: strings that can be retrieved from the environment on demand
JDK_PATH, JAVA_HOME: so java computer and runtime knows where libraries are located
Values of variables retrieved from “owner process” when create new ones – variables are inherited
System variables: shared by all users, recorded by OS when it starts

3. More on CI Setup DOS prompt: variables referred to by %X%
Eg: open DOS prompt and enter the command
echo %JAVA_PATH%
Unix uses $JAVA_PATH, hence some of the typos
Much of the setup has to do with getting the proper tools in your path (javac, ant, junit libraries)
Windows services: background jobs

4. Reliability Systems Engineering:
Linear System: response to complex input can be described as a sum of responses to simpler inputs
Eg: Wave propagation
Nonlinear examples: AC power flow, almost any complex system
Reliability for linear systems: product of reliability of each component
A: 90% Reliable
B: 90%
C: 90%
Time
Once upon a time: seen as constricting to force every build to pass tests – need branching
Reliability: .93 = .73

5. Reliability in Software
Car that works 73% of the time: find a shop or dealer
How many components in software?
How can we achieve high reliability for SW?

6. Review – what is Integration, system, acceptance testing?
What is performance testing?
How do you do it?
What can you learn from this type of testing?
What can't you learn from this type of testing?
Stress test
Answers what?
What do you want to happen?
Goal: graceful degradation - bend but don't break

7. More testing methods How do you test security? Websites:
Database injection, cross-site scripting
Fuzz checkers – inserting random data
How to test usability?
In general, how to test non-functional requirements?
Must identify testable component!
Generally, this is very hard: need to try to break system, but attacker has far more time than testers!

8. Testing strategy
Why might we not be able to run every test every time?
How should we prioritize tests?
Regression set minimization: identifying a subset that would reveal most bugs
Research question!
Should be able to run all unit tests, majority of system tests
Less frequently: performance, other tests
Maybe these are just run once per day

9. Qualities of good tests
Thorough – coverage
Start with statement coverage
Repeatable
Generate small data sets that have predictable outputs
Test runs should not depend on each other
Understandable
Maintainable
Easy to identify failed case

10. Testing
What about “grandma testing”?
“grandma testing”: sexist, agist

11. Testing What about “grandma testing”? Sexist? Ageist?
CES 2009: hall for products for retirees
“grandma testing”: sexist, agist
"Local grandmother finally uses printer"

12. Testing What about “grandma testing”? Sexist? Ageist?
CES 2009: hall for products for retirees
“grandma testing”: sexist, agist
"Local grandmother finally uses printer"
Is grandma-using-the-computer the
modern equivalent of racist jokes?

13. Testing What about “grandma testing”? Sexist? Ageist?
Fuzz testing tools
“grandma testing”: sexist, agist

14. Testing What about “grandma testing”? What if need large data sets?
Sexist? Ageist?
Fuzz testing tools
What if need large data sets?
Test oracles
Version matching
“grandma testing”: sexist, agist

15. Testing What about “grandma testing”? What if need large data sets?
Sexist? Ageist?
Fuzz testing tools
What if need large data sets?
Test oracles
Version matching
Key: differentiate between robustness, correctness
“grandma testing”: sexist, agist

16. System Testing Focus: functional testing
that is, are there errors in features?
In some sense, this is the only test that matters
tests all functionality seen by customer
why shouldn't we do all testing at this level?
harder to test deeply
harder to know how to fix system if test fails
Acceptance testing: system testing w/ focus on common scenarios

17. Functional/unit testing
How is functional testing different from unit testing?
Scope: unit test exercises pieces, functional is end-to-end
How are the the same?
Written by developer organization
Likely to use mocks to isolate components

18. Acceptance/functional testing
How does acceptance testing differ from functional testing?
Functional:
Internal, verify features
Acceptance:
External, validate requirements
Rare to use mocks – only for very low probability/high impact errors that must be tested
Production environment (as close as possible)

19. The Real Differences Lies in What You Are Asking
Does this feature meet the customer requirement?
Acceptance Test.
Does this feature work?
Functional Test.
Does this feature work with the database?
Integration Test.
Does this method do what I want it to do?
Unit Test.

20. What does a manual test look like?
Step
Req
Pass Conditions
Pass?
1. Select the Biology program.
UIR-2
System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52
P / F
2. Double-click on Class# 1330
UIR-1
System includes Class# 1330 in schedule at bottom
3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01)
UIR-9
System displays Class# 1331 with a pink background 4.
All sections listed between #1311 and #1331 have a white background
5. Select the GENENG program.
System displays general engineering courses 6.
GENENG 1030 sections have a pink background all other sections of GENGENG 1030 have a white background
P / F P / F 7.
All sections of GENGENG 1000 have a white background

21. What does a manual test look like?
Step
Req
Pass Conditions
Pass?
1. Select the Biology program.
UIR-2
System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52
P / F
2. Double-click on Class# 1330
UIR-1
System includes Class# 1330 in schedule at bottom
3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01)
UIR-9
System displays Class# 1331 with a pink background 4.
All sections listed between #1311 and #1331 have a white background
5. Select the GENENG program.
System displays general engineering courses 6.
GENENG 1030 sections have a pink background all other sections of GENGENG 1030 have a white background
P / F P / F 7.
All sections of GENGENG 1000 have a white background
Must document passes manually
Lots of detail
Maintenance issue when steps depend on each other

22. Who should write acceptance tests?
Obvious answer: client
Probably manual tests
Not much help to development team
Alternative: behavior-driven development
Functional, acceptance tests written up front
Simple model: acceptance criteria for each story = collection of system-level tests

23. Getting clients to write tests
Why can’t clients write our tests?
Barrier: testing language
Solution: Cucumber
Testing language: specify start conditions, expected results
Supports tables when have several examples
Developer: provides hooks defining how the tests are executed

24. Client-written tests Wins: Problems
Test documents what system supposed to do in a verifiable manner!
Expected behavior is very concrete
Customer gets to control behavior from start
Problems
Extra time to write tests – if client not writing tests, probably should use other methods
But then client may not really want system!
Must maintain tests
But manual tests need maintenance, too

25. Acceptance tests in https://cucumber.io/
Supports Ruby, Java, .NET, C++, all web applications
Underlying language: Gherkin

26. Steps: 1. describe behavior
Feature: Addition In order to avoid silly mistakes As a math-challenged person I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add Then the result should be 120 on the screen

27. Steps: 1. describe behavior
Feature: Addition In order to avoid silly mistakes As a math-challenged person I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add Then the result should be 120 on the screen
These actually all have
the same effect.

28. 2. Step definition in Ruby
Given /I have entered (.*) into the calculator/ do |n| calculator = Calculator.new calculator.push(n.to_i) end
Run and watch it fail:

29. 2. Write code to make it pass
class Calculator def ||= << n end
Run see step pass:

30. 3. Repeat until “green like a cuke”

31. Behavior-driven development
Basic model: test-driven development
Great for starting, probably not for regular work
Especially friendly to people who are learning about syntax

32. Gherkin Syntax Each feature described in single .feature file
3 elements:
Feature: a name – free form syntax
Description: documentation
Any line not starting w/ a keyword
Scenario: concrete example

33. Gherkin Syntax Keywords: Given, when, then, and, but: steps Background
Feature
Scenario outline
Scenario
Examples
Feature: Refund item
Sales associates required by law to refund purchases
Scenario: Jeff returns a faulty microwave
Given Jeff has bought a microwave for $100
And he has a receipt # don’t assume this is true!
When he returns the microwave
Then Jeff should be refunded $100

34. Gherkin Syntax Keywords: Given, when, then, and, but: steps Background
Feature
Scenario outline
Scenario
Examples
Documentation
- no keyword,
multiline
Feature: Refund item
Sales associates required by law to refund purchases
Scenario: Jeff returns a faulty microwave
Given Jeff has bought a microwave for $100
And he has a receipt # don’t assume this is true!
When he returns the microwave
Then Jeff should be refunded $100
On same line
Can also comment with #

35. Gherkin Syntax Keywords: Given, when, then, and, but: steps Background
Feature
Scenario outline
Scenario
Examples
Feature: Refund item
Sales associates required by law to refund purchases
Scenario: Jeff returns a faulty microwave
Given Jeff has bought a microwave for $100
And he has a receipt # don’t assume this is true!
When he returns the microwave
Then Jeff should be refunded $100
No distinctions between these
Chose for readability
Single when (or break up)
Then: compare actual to expected

36. Gherkin Syntax Keywords: Steps which are assumed before each scenario
Given, when, then, and, but: steps
Background
Feature
Scenario outline
Scenario
Examples
Background:
Given a $100 microwave was sold on
And today is
Steps which are assumed before each scenario

37. Scenario outline Parameterize values, specify with table:
Scenario Outline: feeding a suckler cow
Given the cow weighs <weight> kg
When we calculate the feeding requirements
Then the energy should be <energy> MJ
And the protein should be <protein> kg
Examples:
| weight | energy | protein |
| | | |
| | | |
| | | |
| | | |

38. Scenario outline Parameterize values, specify with table:
Scenario Outline: feeding a suckler cow
Given the cow weighs <weight> kg
When we calculate the feeding requirements
Then the energy should be <energy> MJ
And the protein should be <protein> kg
Examples:
| weight | energy | protein |
| | | |
| | | |
| | | |
| | | |
One or more examples sections
Required header row
‘|’ is a key symbol

39. Implementing tests C++ code implementing this step: Scenario:
Given I have 48 cukes in my belly
C++ code implementing this step:
Given(“^I have (\\d+) cukes in my belly$”) {
REGEX_PARAM(int, cukes);
USING_CONTEXT(MyAppCtx, context);
std::cout << “Cukes: “ << cukes; }

40. Implementing tests C++ code implementing this step: Capture group
Scenario:
Given I have 48 cukes in my belly
Capture group
Usually regular expression
d+: one or more digits
C++ code implementing this step:
Matches and, then, etc.
Given(“^I have (\\d+) cukes in my belly$”) {
REGEX_PARAM(int, cukes);
USING_CONTEXT(MyAppCtx, context);
std::cout << “Cukes: “ << cukes; }
Declaration
Boiler plate

41. Key: regular expressions
See n regex.html
Flexible tool for capturing words with specific patterns
Will generally use several simple cases:
\d+, \w+
^, $: anchor to start and end
backslash to escape special characters

42. Review Unit vs. functional vs. acceptance testing
Testing non-functional requirements: performance, security, etc.
Behavior-driven development with Cucumber
Next class period: exercise on writing tests in Cucumber