SE 3800 Note 5 Beyond Functional Testing Dr. Rob Hasker Dr. Brad Dennis SE 3800 Note 5 Beyond Functional Testing

Reviewing CI Setup So what do the steps do? Key concept: search path Windows: for each command, run through PATH to find first folder containing specified command (an executable with the same name) Variables: strings that can be retrieved from the environment on demand JDK_PATH, JAVA_HOME: so java computer and runtime knows where libraries are located Values of variables retrieved from “owner process” when create new ones – variables are inherited System variables: shared by all users, recorded by OS when it starts

More on CI Setup DOS prompt: variables referred to by %X% Eg: open DOS prompt and enter the command echo %JAVA_PATH% Unix uses $JAVA_PATH, hence some of the typos Much of the setup has to do with getting the proper tools in your path (javac, ant, junit libraries) Windows services: background jobs

Reliability Systems Engineering: Linear System: response to complex input can be described as a sum of responses to simpler inputs Eg: Wave propagation Nonlinear examples: AC power flow, almost any complex system Reliability for linear systems: product of reliability of each component A: 90% Reliable B: 90% C: 90% Time Once upon a time: seen as constricting to force every build to pass tests – need branching Reliability: .93 = .73

Reliability in Software Car that works 73% of the time: find a shop or dealer How many components in software? How can we achieve high reliability for SW?

Review – what is Integration, system, acceptance testing? What is performance testing? How do you do it? What can you learn from this type of testing? What can't you learn from this type of testing? Stress test Answers what? What do you want to happen? Goal: graceful degradation - bend but don't break

More testing methods How do you test security? Websites: Database injection, cross-site scripting Fuzz checkers – inserting random data How to test usability? In general, how to test non-functional requirements? Must identify testable component! Generally, this is very hard: need to try to break system, but attacker has far more time than testers!

Testing strategy Why might we not be able to run every test every time? How should we prioritize tests? Regression set minimization: identifying a subset that would reveal most bugs Research question! Should be able to run all unit tests, majority of system tests Less frequently: performance, other tests Maybe these are just run once per day

Qualities of good tests Thorough – coverage Start with statement coverage Repeatable Generate small data sets that have predictable outputs Test runs should not depend on each other Understandable Maintainable Easy to identify failed case

Testing What about “grandma testing”? “grandma testing”: sexist, agist

Testing What about “grandma testing”? Sexist? Ageist? CES 2009: hall for products for retirees “grandma testing”: sexist, agist "Local grandmother finally uses printer"

Testing What about “grandma testing”? Sexist? Ageist? CES 2009: hall for products for retirees “grandma testing”: sexist, agist "Local grandmother finally uses printer" Is grandma-using-the-computer the modern equivalent of racist jokes?

Testing What about “grandma testing”? Sexist? Ageist? Fuzz testing tools “grandma testing”: sexist, agist

Testing What about “grandma testing”? What if need large data sets? Sexist? Ageist? Fuzz testing tools What if need large data sets? Test oracles Version matching “grandma testing”: sexist, agist

Testing What about “grandma testing”? What if need large data sets? Sexist? Ageist? Fuzz testing tools What if need large data sets? Test oracles Version matching Key: differentiate between robustness, correctness “grandma testing”: sexist, agist

System Testing Focus: functional testing that is, are there errors in features? In some sense, this is the only test that matters tests all functionality seen by customer why shouldn't we do all testing at this level? harder to test deeply harder to know how to fix system if test fails Acceptance testing: system testing w/ focus on common scenarios

Functional/unit testing How is functional testing different from unit testing? Scope: unit test exercises pieces, functional is end-to-end How are the the same? Written by developer organization Likely to use mocks to isolate components

Acceptance/functional testing How does acceptance testing differ from functional testing? Functional: Internal, verify features Acceptance: External, validate requirements Rare to use mocks – only for very low probability/high impact errors that must be tested Production environment (as close as possible)

The Real Differences Lies in What You Are Asking Does this feature meet the customer requirement? Acceptance Test. Does this feature work? Functional Test. Does this feature work with the database? Integration Test. Does this method do what I want it to do? Unit Test.

What does a manual test look like? Step Req Pass Conditions Pass? 1. Select the Biology program. UIR-2 System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52 P / F 2. Double-click on Class# 1330 UIR-1 System includes Class# 1330 in schedule at bottom 3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01) UIR-9 System displays Class# 1331 with a pink background 4. All sections listed between #1311 and #1331 have a white background 5. Select the GENENG program. System displays general engineering courses 6. GENENG 1030 sections 01-07 have a pink background all other sections of GENGENG 1030 have a white background P / F P / F 7. All sections of GENGENG 1000 have a white background

What does a manual test look like? Step Req Pass Conditions Pass? 1. Select the Biology program. UIR-2 System displays biology classes w/ first class BIOLOGY 1150, Section 01, Title GENERAL BIOLOGY, Instructor Block, Anna, Filled/Seats 52/53, Class# 1311, Credits 5, Meets BOE 0221 MWF 8:00-8:52 P / F 2. Double-click on Class# 1330 UIR-1 System includes Class# 1330 in schedule at bottom 3. Scroll down to Class# 1331 (BIOLOGY 1650, Section 01) UIR-9 System displays Class# 1331 with a pink background 4. All sections listed between #1311 and #1331 have a white background 5. Select the GENENG program. System displays general engineering courses 6. GENENG 1030 sections 01-07 have a pink background all other sections of GENGENG 1030 have a white background P / F P / F 7. All sections of GENGENG 1000 have a white background Must document passes manually Lots of detail Maintenance issue when steps depend on each other

Who should write acceptance tests? Obvious answer: client Probably manual tests Not much help to development team Alternative: behavior-driven development Functional, acceptance tests written up front Simple model: acceptance criteria for each story = collection of system-level tests

Getting clients to write tests Why can’t clients write our tests? Barrier: testing language Solution: Cucumber Testing language: specify start conditions, expected results Supports tables when have several examples Developer: provides hooks defining how the tests are executed

Client-written tests Wins: Problems Test documents what system supposed to do in a verifiable manner! Expected behavior is very concrete Customer gets to control behavior from start Problems Extra time to write tests – if client not writing tests, probably should use other methods But then client may not really want system! Must maintain tests But manual tests need maintenance, too

Acceptance tests in https://cucumber.io/ Supports Ruby, Java, .NET, C++, all web applications Underlying language: Gherkin

Steps: 1. describe behavior Feature: Addition In order to avoid silly mistakes As a math-challenged person I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add Then the result should be 120 on the screen

Steps: 1. describe behavior Feature: Addition In order to avoid silly mistakes As a math-challenged person I want to be told the sum of two numbers Scenario: Add two numbers Given I have entered 50 into the calculator And I have entered 70 into the calculator When I press add Then the result should be 120 on the screen These actually all have the same effect.

2. Step definition in Ruby Given /I have entered (.*) into the calculator/ do |n| calculator = Calculator.new calculator.push(n.to_i) end Run and watch it fail:

2. Write code to make it pass class Calculator def push(n) @args ||= [] @args << n end Run see step pass:

3. Repeat until “green like a cuke”

Behavior-driven development Basic model: test-driven development Great for starting, probably not for regular work Especially friendly to people who are learning about syntax

Gherkin Syntax Each feature described in single .feature file 3 elements: Feature: a name – free form syntax Description: documentation Any line not starting w/ a keyword Scenario: concrete example

Gherkin Syntax Keywords: Given, when, then, and, but: steps Background Feature Scenario outline Scenario Examples Feature: Refund item Sales associates required by law to refund purchases Scenario: Jeff returns a faulty microwave Given Jeff has bought a microwave for $100 And he has a receipt # don’t assume this is true! When he returns the microwave Then Jeff should be refunded $100

Gherkin Syntax Keywords: Given, when, then, and, but: steps Background Feature Scenario outline Scenario Examples Documentation - no keyword, multiline Feature: Refund item Sales associates required by law to refund purchases Scenario: Jeff returns a faulty microwave Given Jeff has bought a microwave for $100 And he has a receipt # don’t assume this is true! When he returns the microwave Then Jeff should be refunded $100 On same line Can also comment with #

Gherkin Syntax Keywords: Given, when, then, and, but: steps Background Feature Scenario outline Scenario Examples Feature: Refund item Sales associates required by law to refund purchases Scenario: Jeff returns a faulty microwave Given Jeff has bought a microwave for $100 And he has a receipt # don’t assume this is true! When he returns the microwave Then Jeff should be refunded $100 No distinctions between these Chose for readability Single when (or break up) Then: compare actual to expected

Gherkin Syntax Keywords: Steps which are assumed before each scenario Given, when, then, and, but: steps Background Feature Scenario outline Scenario Examples Background: Given a $100 microwave was sold on 2015-08-26 And today is 2015-08-26 Steps which are assumed before each scenario

Scenario outline Parameterize values, specify with table: Scenario Outline: feeding a suckler cow Given the cow weighs <weight> kg When we calculate the feeding requirements Then the energy should be <energy> MJ And the protein should be <protein> kg Examples: | weight | energy | protein | | 450 | 26500 | 215 | | 500 | 29500 | 245 | | 575 | 31500 | 255 | | 600 | 37000 | 305 |

Scenario outline Parameterize values, specify with table: Scenario Outline: feeding a suckler cow Given the cow weighs <weight> kg When we calculate the feeding requirements Then the energy should be <energy> MJ And the protein should be <protein> kg Examples: | weight | energy | protein | | 450 | 26500 | 215 | | 500 | 29500 | 245 | | 575 | 31500 | 255 | | 600 | 37000 | 305 | One or more examples sections Required header row ‘|’ is a key symbol

Implementing tests C++ code implementing this step: Scenario: Given I have 48 cukes in my belly C++ code implementing this step: Given(“^I have (\\d+) cukes in my belly$”) { REGEX_PARAM(int, cukes); USING_CONTEXT(MyAppCtx, context); std::cout << “Cukes: “ << cukes; }

Implementing tests C++ code implementing this step: Capture group Scenario: Given I have 48 cukes in my belly Capture group Usually regular expression d+: one or more digits C++ code implementing this step: Matches and, then, etc. Given(“^I have (\\d+) cukes in my belly$”) { REGEX_PARAM(int, cukes); USING_CONTEXT(MyAppCtx, context); std::cout << “Cukes: “ << cukes; } Declaration Boiler plate

Key: regular expressions See n05-3800-regex.html Flexible tool for capturing words with specific patterns Will generally use several simple cases: \d+, \w+ ^, $: anchor to start and end backslash to escape special characters

Review Unit vs. functional vs. acceptance testing Testing non-functional requirements: performance, security, etc. Behavior-driven development with Cucumber Next class period: exercise on writing tests in Cucumber