1. Software Engineering Experimentation
An Experimental Framework
Jeff Offutt
Used with permission by Jane Hayes

2. Experimental Framework
Victor Basili, Richard Selby and David Hutchens, “Experimentation in Software Engineering”, IEEE Transactions on Software Engineering, SE-12(7):733–743, July 1986
A little old but still relevant and helpful
Three objectives
Framework for performing experiments
Classify and discuss experiments
Problem areas and lessons learned
© Jeff Offutt,

3. Framework Four categories
Definition – Problem definition and hypothesis
Planning – Design
Operation – Conduct test
Interpretation – Hypothesis test, disseminate
© Jeff Offutt,

4. Framework – 1. Definition
Motivation
(why)
Object
(what)
Purpose
Perspective
(who)
Domain
(which)
Scope
Understand
Assess
Manage
Engineer
Learn
Improve
Validate
Assure
Product
Process
Model
Metric
Theory
Characterize
Evaluate
Predict
Motivate
Developer
Modifier
Maintainer
Project manager
Corp. manager
Customer
User
Researcher
Programmer
Program /
project
Single project
Multi-project
Replicated project
Blocked subject-
© Jeff Offutt,

5. Framework – 2. Planning Design (samples, control) Criteria
(what to measure)
Measurement
(how to measure)
Experimental designs
Incomplete block
Completely random
Randomized block
Fractional factorial
Multivariate analysis
Correlation
Factor analysis
Regression
Statistical models
Non-parametric
Sampling
Direct reflections of cost / quality
Cost
Errors
Changes
Reliability
Correctness
Indirect reflections of cost / quality
Data coupling
Information visibility
Programmer comprehension
Execution coverage
Size
Complexity
Metric definition
Goal-question-metric
Factor-criteria-metric
Metric validation
Data collection
Automatability
Form design and test
Objective vs. subjective
Level of measurement
Nominal / classificatory
Ordinal / ranking
Interval
Ratio
© Jeff Offutt,

6. Framework – 3. Operation Preparation Execution Analysis Pilot study
Data collection
Data validation
Quantitative vs. qualitative
Preliminary data analysis
Plots and histograms
Model assumptions
Primary data analysis
Model application
© Jeff Offutt,

7. Framework – 4. Interpretation
Interpretation context
Extrapolation
Impact
Statistical framework
Study purpose
Field of research
Sample representativeness
Visibility
Replication
Application
© Jeff Offutt,

8. Framework Example Definition
Definition Element
Example
Motivation
To improve unit testing
Object
characterize and evaluate
Purpose
the process of functional & structural testing
Perspective
from the perspective of the developer
Domain: programmer
as they are applied by experienced programmers
Domain: program
to unit-size software
Scope
in a blocked subject-project study
© Jeff Offutt,