1. Software Engineering An Introduction Experimentation in Jianyun Zhou
Today We will talk about this book: Experimentation in software engineering. It give an introduction to the empirical software engineering and particularly the application and the use of experimentation.
Jianyun Zhou
Dept. of Computer Science
NTNU, 4th Oct 2002

2. Experimentation in software engineering
Focus
The application of empirical studies, in particular experimentation, in software engineering: as one way to evaluating new methods and techniques.
The focus of this book is about the application of empirical studies,...
4/5/2019
Experimentation in software engineering

3. Experimentation in software engineering
Content
Chapter 1-3: Introduction
Chapter 4-9: Experiment process
Chapter 10-14: Examples and exercises
Appendices: Statistical tables and process overview
This book contains fourteen chapters totally, chapter one to nine will be presented in these two lectures, they will be divided into two parts: introduction part and experimentation part.
4/5/2019
Experimentation in software engineering

4. Introduction part: Outline
Why empirical studies in software engineering?
Fitting empirical studies to software engineering
Empirical strategies
Research environments in software engineering
Measurement theory
At first, we will talk about the introduction part, it contains five points:
at first, why empirical studies is needed in software engineering? And how to fit the empirical studies to the software engineering context.
Then we will look at some general empirical strategies, and how they can be used in the different software engineering research environments.
At last we will introduce the measurement theory.
4/5/2019
Experimentation in software engineering

5. Why empirical studies in software engineering?
To have control of the developed software
advocacy research: new methods based on marketing and conviction
evaluate new methods and tools before using them
To turning software engineering into a science
put forward hypotheses
hypothesis testing through empirical studies
More empirical studies is needed to be conducted in software engineering.
Why empirical studies is needed?
At first, it is a way to have control of the software we developed.
Over the years, software engineering has been driven by so-called advocacy research, which means that we have invented and introduced new methods and techniques over the years based on marketing and conviction rather than scientific results. So we develop the software using these methods, we will have no control.
Using empirical studies, we can evaluate the new methods or tools before we introduce them, use them, so the software developed can be controlled to a certain extent.
Empirical studies is also a way to turn SE to a science. In science, we often build a theory in this way: at first, we put forward a hypothesis, then we made experiments to observe the phenomena, based on the observed phenomena we can either reject or accept the hypothesis.
Experiments and empirical studies are thus important to test the hypothesis to be correct or wrong. But it is certainly not a most often used research method in SE today, so more empirical studies are required to be conducted in this field.
4/5/2019
Experimentation in software engineering

6. Fitting empirical studies in SE
Software engineering context
Resources
Produce idea
Software product
Software process
This figure gives a simple view of the software process. The input to this process include an product idea and some resources, for example,people. The output from a software process is the software product.
But the software process is not as simple as showed in this figure, much activities are included in this. It can be performed in a long period and can be very complex. So,for the companies, in order to improve the software product, lower the cost. they want also to improve the software process.
Empirical studies can be used in this context to improve the software process.
Application: improve software process
4/5/2019
Experimentation in software engineering

7. Experimentation in software engineering
Improvement process
Two activities:
assessment of the software process
identify suitable areas for improvement (problem)
identify improvement proposals
evaluation of a software process improvement proposal
It is necessary to evaluate the proposal before making any major changes
through empirical studies
In order to improve a software process, we often need to perform two activities.
One is the assessment of the software process. Several models in SE are found to do this assessment. Through these models, we can found where the improvement is needed, and further give some improvement proposals.
The next step is the need to evaluate the improvement proposals before to make any changes to the software process. Because the process is a human-based activity, it is not possible to build a prototype for such activities, the only real evaluation of a process improvement proposal is to have people to use it. so Empirical studies is very suitable in this context and can be used to evaluate different improvement proposals.
4/5/2019
Experimentation in software engineering

8. Experimentation in software engineering
Empirical strategies
Three major strategies:
Survey
Case study
Experiment
The fourth strategy:
Post-mortem analysis (PMA)
Using the experiences gained from projects within the organisation to learn. It is performed in retrospect of a project
In the book, three major strategies are introduced for empirical software engineering, they are…
In addition, in some other literatures, a fourth strategy is often mentioned, it is Post-mortem analysis, which means that we use the experiences gained from projects to learn. It is performed in the retrospect of a project. Compared with other strategies, The main advantage of this strategy is the cost is rather low.
4/5/2019
Experimentation in software engineering

9. Experimentation in software engineering
Survey
An investigation performed before a tool or technique takes into use, or when been in use for a while
Primary means to gather data:
interviews
questionnaires
Provide no control
This strategy provides little control about the execution.
4/5/2019
Experimentation in software engineering

10. Experimentation in software engineering
Case study
Case studies are used for monitoring projects or activities.
Data is collected for a specific purpose throughout the study.
The level of control is low.
The level of control is also low in the case study. The experimenter have no control for the execution of the studied project.
4/5/2019
Experimentation in software engineering

11. Experimentation in software engineering
Experiments are normally done in a laboratory environment.
Data is collected by performing experiment.
Having good control over the situation
We will learn more about this strategy in the next part.
4/5/2019
Experimentation in software engineering

12. Research environments in SE
How to use the strategies when evaluating software process changes?
Three research environments
Desktop
the change proposal is evaluated off-line
no people involved in applying methods or tools
suitable to conduct surveys
Laboratory
the change proposal is evaluated in a laboratory setting
an experiment is conducted
Development projects
the change proposal is evaluated in a real development situation (observed on-line)
case studies are more appropriate
We have introduced three main strategies in empirical studies, the questions now is how to use them in the different research environments in SE?
There are three research environments found in the software engineering: …
In the desktop environment, … so survey is the suitable strategy ti be used in this environment.
4/5/2019
Experimentation in software engineering

13. Research environments and strategies
Development
projects
Case study
High risk
Laboratory
Experiment
Desktop
Survey
Low risk
The picture shows the placement of the three different research environment, and indicate the increased risks.
For example, in order to try out a new design method in a realistic environment, we may apply it in a real development project, this is, of course, more risks compared with desktop and experiments because it will affect the quality of the delivered product.
4/5/2019
Experimentation in software engineering

14. Experimentation in software engineering
Measurement theory
Measurement is a central part in empirical studies.
“You cannot control what you cannot measure.”
measure both inputs and outputs
Definition
A measure is the number or symbol assigned to an entity in order to characterise an attribute of the entity
Measurement is a mapping from the empirical world to the formal, relational word, i.e. providing a measure
Metrics
Attributes to be measured, how to measure (scale), etc. e.g. LOC (lines of code)
Now we will give a brief introduction to measurement theory because measurement is a very important part in the empirical studies and in experimentation. You can not control something if you can not measure.
Empirical studies are often used to investigate the effects of some input to the studied object. So there are two things we want to measure, they are input and the effect (that is output).
Measurement and measure can be defined as:
For example, we can measure the length of a table and map it to a number denoted by meters, or inches. This number will be a measure.
The term metrics is often used in SE, it is used to denote something to be measured, for example, LOC, the lines of code.
4/5/2019
Experimentation in software engineering

15. Experimentation in software engineering
Scale and scale type
Scale: the different ways to map an attribute to a measure
Scale types are related to statistical analysis
Nominal scale:
only map attribute to a name or symbol, least powerful
e.g. classification: IS, SU, DB…
Ordinal scale:
ranks after an ordering criterion, more powerful than nominal
e.g. grades: poor (1), medium (2), good (3)
Ratio scale:
most powerful
e.g. length: 1.89m
Another important aspect in measurement theory is scale. Scales is the different ways to measure the same thing. For example, if we want to measure the length of an object, we can measure it in meters, centimetres or inches. Each of them is a scale.
Scale types classify scales into different types, three most common scale types used in software engineering are: …
One typical nominal scale is classification. For example, we can classify the students in computer science into different groups, and measure one student as “SE”, and measure another student as “IS”.
Grade, 100% or
4/5/2019
Experimentation in software engineering

16. Classification of measures
Objective and subjective measures
an objective measure means no judgment in the measurement value, e.g. LOC
a subjective measure is made by person through judgment, e.g. personnel skill
Direct and indirect measures
a direct measure involves no other measurements,e.g. LOC
a indirect measure is derived from other measurements,e.g. defect rate = #defects/LOC
Objective: lines of code.
subjective: personnel skill
direct: lines of code
indirect: defect density - number of defects divided by the number of lines of code
4/5/2019
Experimentation in software engineering

17. Measurements in software engineering
Three classes objects are of interest
Process
e.g. testing: effort (I), cost (E)
Product
e.g. code: size (I), reliability (E)
Resources
e.g. personnel: age (I), productivity (E)
Internal and external attributes (Table 3 p.29)
The objects that are of interest in SE can be divided into three different classes:
process describe which activities are needed to produce software, an object belonging to this class is like, testing: for testing, we maybe want to measure cost or effort.
Product is the result from a process.
Resources are the objects needed for a process activity.
4/5/2019
Experimentation in software engineering

18. Experimentation part (outline)
Experimentation basics
Experimentation principles
Terminology
Experiment process
Now we will talk about the experimentation part, it contains …, and terminology, that is, the concepts used in experimentation. And the experiment process.
4/5/2019
Experimentation in software engineering

19. Experimentation basics
Experiments are controlled studies often to compare one thing with another.
They include a formal hypothesis and statistical tests.
A hypothesis means that we have an idea of, for example, a relationship, which are able to stated formally.
The main objective of an experiment is mostly to evaluate a hypothesis or relation
4/5/2019
Experimentation in software engineering

20. Experiment principles
Experiment objective
Theory
cause-effect
Cause
construct
Effect
construct
construct
Observation
treatment-
outcome
The algorithm is probabilistic. Works if round trip times are shortB
construct
Treatment
Outcome
Independent variable
Dependent variable
Experiment operation
4/5/2019
Experimentation in software engineering

21. Terminology in experimentation
Variables
Independent variables
All the variables in a process that are manipulates and controlled
E.g. design method, personnel experience, tool support
Dependent variables (or response variables)
The variables we want to study to see the effect of the changes in independent variables
Often only one dependent variable in an experiment
E.g. efficiency, productivity
Factors
One or more changing independent variables, e.g. design method
Treatment (conditions)
One particular value of a factor, e.g. design =OO method or FO method
Object,subject and test
Subjects (persons, students) apply treatments to object (program, document)
Each test is a combination of subject, treatment and object, e.g. student A uses OO method to develop the program N.
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

22. Illustration of experiment
Treatment
Experiment
design
Process
Dependent
variable
Independent
variables
Independent variables
With fixed level
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

23. Experimentation in software engineering
Experiment process
Experiment
Idea
Experiment process
Experiment
definition
Experiment
planning
Experiment
operation
Analysis &
interpretation
The algorithm is probabilistic. Works if round trip times are shortB
Presentation
& package
Conclusions
4/5/2019
Experimentation in software engineering

24. Definition phase
Define
experiment
Experiment
definition
Experiment
idea
The purpose of the definition phase is to define the goals of an experiment
according to a defined framwork (Goal definition template).
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

25. Goal definition template
The goal template:
Object of the study: the entity that is studied in the experiment, e.g. methods, models, processes, final products
Purpose: the intention of the experiment, e.g. evaluation
Quality of focus: the primary effect under study, e.g. reliability, cost
Perspective: from which viewpoint to interpret the results, e.g. customer
Context: the environment to run the experiment
Single object study; Multi-object variation study; Multi-test within object study; Blocked subject-object study
By answering these questions, it is a good way to get the experiment definition
Analyze <Object(s) of study>
For the purpose of <purpose>
With respect to <Quality focus>
From the point view of the <Perspective>
In the context of <Context>
Analyze <Object(s) of study>
For the purpose of <purpose>
With respect to <Quality focus>
From the point view of the <Perspective>
In the context of <Context>
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

26. Experimentation in software engineering
An example definition
Analyze the PBR and checklist techniques
For the purpose of evaluation
With respect to effectiveness and efficiency
From the point of view of the researcher
In the context of students reading requirements documents
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

27. Experimentation in software engineering
Experiment process
Experiment
Idea
Experiment process
Experiment
definition
Experiment
planning
Experiment
operation
Analysis &
interpretation
The algorithm is probabilistic. Works if round trip times are shortB
Presentation
& package
Conclusions
4/5/2019
Experimentation in software engineering

28. Experiment planning – phase overview
definition
Experiment planning
Context
selection
Hypothesis
formulation
Variables
selection
Selection of
subjects
Experiment
design
Instrumen-
tation
The algorithm is probabilistic. Works if round trip times are shortB
Validity
evaluation
Experiment
design
4/5/2019
Experimentation in software engineering

29. Experimentation in software engineering
Context selection
On-line or off-line
Students or professional
Toy size or real problems
Specific or general
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

30. Hypothesis formulation
An hypothesis is a specific statement of prediction.
Two hypothesis have to be formulated
A null hypothesis, H0 : the other possible outcomes
An alternative hypothesis, H1: the one you support
The objective is to reject the null hypothesis with a certain significance
If H0 cannot be rejected, no conclusion can be drawn?
Hypothesis testing is the basis for statistical analyze of en experiment
Risks in hypothesis testing:
Type-I-error: P(type-I-error) =P(reject H0 | H0 true) – significance level
Type-II-error: P(type-II-error) =P(not reject H0 | H0 false)
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

31. Experimentation in software engineering
Planning:design
Variables selection
Factors: controllable, changeable, have effect on dependent variable
Dependent variable: often one, effected by treatments
Simultaneously or in reverse order
Subjects: representative
Design principles:
Randomization: the allocation of subjects, objects and the order
Blocking: “reducing noses”
Balancing: same number of subjects in groups
Design types presented
One factor with two treatments; One factor with more than two treatments; Two factor with two treatments; More than two factor each with two treatments;
Different design types will use different test method.
4/5/2019
Experimentation in software engineering

32. Experimentation in software engineering
Instrumentation
The overall goal of the instrumentation is to provide means for performing the experiment and to monitor it, without affecting the control of the experiment
The instruments for an experiment are of three types,
Objects
E.g. specification or code documents
Guidelines
To guide the participants in the experiment
Measurements instruments
Data collection via manual forms or in interviews
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

33. Experimentation in software engineering
Validity evaluation 4 3 3 1 2
The algorithm is probabilistic. Works if round trip times are shortB
Conclusion validity: treatment to outcome (“right” analysis)
Internal validity: treatment causes outcome (“right” measures)
Construct validity: theory to observation (“right” metrics)
External validity: generalization (“right” context)
4/5/2019
Experimentation in software engineering

34. Experimentation in software engineering
Validity
Conclusion validity
Is there a relationship between the two variables?
Internal validity
Assuming that there is a relationship in this study, is the relationship a causal one?
Construct validity
Assuming that there is a causal relationship in this study, can we claim that the treatments reflected well our cause construct and that the outcome reflected well our idea of the effect construct ?
External validity
Assuming that there is a causal relationship in this study between the constructs of the cause and the effect, can we generalize this effect to other persons, places or times?
The algorithm is probabilistic. Works if round trip times are shortB
4/5/2019
Experimentation in software engineering

35. Experimentation in software engineering
Experiment process
Experiment
Idea
Experiment process
Experiment
definition
Experiment
planning
Experiment
operation
Analysis &
interpretation
The algorithm is probabilistic. Works if round trip times are shortB
Presentation
& package
Conclusions
4/5/2019
Experimentation in software engineering

36. Experimentation in software engineering
Experiment operation
Experiment operation
Experiment
Preparation
design
Execution
Data
validation
The algorithm is probabilistic. Works if round trip times are shortB
Experiment
data
Preparation: subjects are chosen and forms etc. are prepared
Execution: subjects perform their tasks according to different treatments and data is collected
Data validation: the collected data is validated
4/5/2019
Experimentation in software engineering

37. Experimentation in software engineering
Experiment process
Experiment
Idea
Experiment process
Experiment
definition
Experiment
planning
Experiment
operation
Analysis &
interpretation
The algorithm is probabilistic. Works if round trip times are shortB
Presentation
& package
Conclusions
4/5/2019
Experimentation in software engineering

38. Analysis and interpretation
Experiment
Descriptive
Data
statistics
Data set
reduction
Hypothesis
testing
The algorithm is probabilistic. Works if round trip times are shortB
Conclusions
4/5/2019
Experimentation in software engineering

39. Descriptive statistics and data set reduction
The goal is to get a feeling for how data is distributed
Descriptive statistics characterize the data by
measures of central tendency: mean, median, mode etc.
measures of dispersion: variance, range, relative frequency etc.
measures of dependency: covariance etc.
graphical visualization: scatter plot, box plot, histogram etc.
The scale of the measurement restricts the type of statistics to use
Data set reduction removes abnormal or false data points (outliners) and reduces the data set to a set of valid data points
4/5/2019
Experimentation in software engineering

40. Experimentation in software engineering
Hypothesis testing
Principle
The objective is to see the possibility to reject the null hypothesis
If the null hypothesis is not rejected, nothing can be said from the experiment, while if it is rejected, it can be stated that the null hypothesis is false with a significance (α).
α = P(type-I-error) =P(reject H0 | H0 true)
The different types of tests are related to the different design types.
Parametric tests
based on a model that involves a specific distribution
Parameters be measured in ration scale
Non-parametric tests
Based on a model with very general conditions
May be applied to nominal and ordinal scales
4/5/2019
Experimentation in software engineering

41. Experimentation in software engineering
Experiment process
Experiment
Idea
Experiment process
Experiment
definition
Experiment
planning
Experiment
operation
Analysis &
interpretation
The algorithm is probabilistic. Works if round trip times are shortB
Presentation
& package
Conclusions
4/5/2019
Experimentation in software engineering

42. Presentation and packaging
It is essential not to forget important aspects or necessary information, needed to enable others to replicate or take advantage of the experiment, and knowledge gained through the experiment
Report outline:
Introduction
Problem statement
Experiment planning
Experiment operation
Data analysis
Interpretation of results
Discussion and conclusions
Appendix
4/5/2019
Experimentation in software engineering

43. Experimentation in software engineering
Rest of the books
Chapter 10: Literature survey
references for some published software engineering experiments
Chapter 11: An example of process
an example to illustrate the experiment process
Chapter 12: Experiment example
how to report an experiment in a paper
Chapter 13: Exercises and data
Appendices: Statistical tables and process overview
4/5/2019
Experimentation in software engineering