Hands on Experiment Design Foutse Khomh, Queen’s University, Canada Massimiliano Di Penta, University of Sannio, Italy

Use of acceptance test during change tasks Usage of natural language to specify (change) requirements: current state of the practice highly inaccurate error prone “Agile movement” advocates [Melnik et alt. 2004]: Acceptance test cases constitute an expressive form of documentation Acceptance test cases are “talking” representation of the requirements We focus on evaluating whether Fit acceptance tests are useful in maintenance/evolution tasks

Framework for Integrated Test The most well-known open source implementation of the table-based acceptance testing approach. A table represents a set of test cases. Fit lets: customers/analysts write “executable” acceptance tests using simple HTML tables (Fit tables). developers write “Fixtures” to link the test cases with the actual system itself. to compare (Test Runner) these test cases with actual values, returned by the system, and highlights the results with colors. Red for failures Green for passed tests.

“The complete picture” Text Reqs. Fit Table Fixture Analyst (and customer) (i, o) System i o’ Developer o ≠ o’ Test Runner Output Fit Table O = expected output O’ = actual output failure

Examples of Fit Tables Core Fit tables: Column Fit tables for testing calculations Action Fit tables for testing user interfaces or workflows; Row Fit tables to check the result of a query (some) other Fit tables: Summary Fit tables Html Fit tables Command line Fit tables … fit_tests.DiscountStructure small bags beveragediscount total price() 2Coffee Tea Coffee12.1 5Tea03.1 7Coffee Tea04.34 fit.ActionFixture startfit_tests.VerifySupply entertype of productCoffee check number of small bug remained 10 enternumber of box5 pressbuying boxes check number of small bug remained 310 Column Fit table Action Fit table 1 box = 60 small bags small bag price = 0.62

FitNesse A collaborative testing and documentation tool. It supports Java (eclipse plug-in),.Net, C++ It Combines Fit with a Wiki Web for writing “natural language requirements” + Fit tables. It provides a simple way to run tests (Fit tables) and suits. It supports sub Wikis for managing multiple projects.

Text only vs. Text + Fit Text onlyText + Fit Change Requirement: Change price of Boxes The vendor of boxes of beverages changed his selling policy. Each five bought boxes (of the same type) one is added as a gift. Change Requirement: Change price of Boxes The vendor of boxes of beverages changed his selling policy. Each five bought boxes (of the same type) one is added as a gift. fit.ActionFixture startfit_tests.VerifySupply entertype of productCoffee check number of small bug remained 310 enternumber of box11 pressbuying boxes check number of small bug remained 960 checkcash account504 What to compare

Where do we start? We have an idea/conjecture about a cause effect relation We have a theory Thus we can formulate a hypothesis And to test it.. we run an experiment! The planning steps follows...

Planning: steps Context selection Definition Hypothesis formulation Variable selection Selection of subjects Experiment design Instrumentation Validity evaluation Experiment design Study planning

Context Selection Select the type of study to perform: survey, case study, and experiment. In this training we focus on experiment (more 10 of the lecture slides) Identify the objects and subjects of the study (more page of the lecture slides)

Hypothesis formulation Formulate the null hypothesis. (more page of the lecture slides)

Variables Selection Dependant variables Used to measure the effect of treatments Derived from hypotheses (more page of the lecture slides)

Variables Selection  Independant variables Variables we can control and modify The choice depends on the domain knowledge One independent variable is the main factor of our experiment Other independent variables are the co-factors (more page of the lecture slides)

Experiment Design Is the set of treatment tests Combinations of treatments, subjects and objects Defines how tests are organized and executed Based on the formulated hypotheses Experimental design is based on three principles 1. Randomization 2. Blocking 3. Balancing (more page of the lecture slides)

Validity evaluation Answer the following crucial questions : To what extent are our results valid? They should be at least valid for the population of interest Then, if we could generalize… Mitigates the following threats to validity 1. Conclusion validity (C) 2. Internal validity (I) 3. Construct validity (S) 4. External validity (E) Prioritize threats to validity: Many threats are conflictual (more page of the lecture slides)

Summary What to produce: Experiment definition Context selection Hypotheses formulation Identification of dependent and independent variables Experiment design Experiment procedure and identification of the required material

Operation After having designed an experiment we need to execute it We are in touch with subjects for the first time Besides the pre-experiment briefing and training Although the design and plan are perfect, everything depends on the operation If something goes wrong in a couple of hours we could waste months of work… 

Experiment operation: steps Preparation Experiment design Execution Data validation Experiment data Experiment operation

Preparation Obtain consent from participants Do not disseminate sensitive data Reward in some way participation Before the experiment, it is advisable to show a short presentation explaining the experiment and its objectives Introduce the objects and the instrumentation Describe in detail the steps of the experiment (more page of the lecture slides)

Execution Different ways to execute an experiment Online You can actually monitor the experiment Offline Distribute the task via and wait for results Run a post-experiment questionnaire to understand whether anything went wrong with clarity of objectives material, time available, tasks and to collect some qualitative information. (more page of the lecture slides)

Data validation Once the experiment has been completed, we need to do a consistency check on the collected data Were treatments correctly applied? Did subjects understand the provided forms? Did subjects correctly fill the forms? Remove subjects that Did not participate to the experiment Exhibited a weird behavior (e.g. did not pay attention to the task) Try to have a quick look at data as soon as possible At least using descriptive statistics

Replication Replication is essential in experimentation You should document your experiments so that others can Repeat the experiment with different subjects Replicate your statistical analysis Look at the non-replicable case of the cold fusion! (more page of the lecture slides)