1. Eike Send, send@inf.fu-berlin.de 1 Seminar „Komponenten“ Empiricism in Computer Science Eike Send Freie Universität Berlin, Institut für Informatik http://www.inf.fu-berlin.de/inst/ag-se/ Part 1 Definition: Empiricism Definition: Experiment Quantative vs Qualitive Data Analysis Part 2 State of Empiricism in CS Object Oriented vs Procedural Sample Experiment

2. Eike Send, send@inf.fu-berlin.de 2 Part 1: Empiricism in general

3. Eike Send, send@inf.fu-berlin.de 3 Gaining knowledge How do we gain knowledge?  by intuition (direct insight)  from some authority (tradition, teacher, book etc.)  by rational thought (reasoning, deduction)  by direct observation  via the scientific method No method can make totally sure that the information is correct All of these methods can lead to a correct result The scientific method will most likely do  It will also be the most convincing one What is the scientific method though?

4. Eike Send, send@inf.fu-berlin.de 4 Definition: Empiricism Empiricism (greek εμπειρισμός, from empirical, latin experientia - the experience) [...] is generally regarded as being at the heart of the modern scientific method, that our theories should be based on our observations of the world rather than on intuition or faith

5. Eike Send, send@inf.fu-berlin.de 5 The scientific method Empiricism is:  based on observation  opposed to: being based on theoretical considerations intuition random selection The purpose is typically making some kind of decision:  Selecting among solution candidates  Understanding priorities  Deciding yes or no

6. Eike Send, send@inf.fu-berlin.de 6 Falsification and Karl Popper Falsification:  Falsification is the contradiction of hypotheses oder theories through empirical statements (for example by observation or experiment)  Note: Even if a theory is wrong, it might be that it predicts the results of an experiment correctly This view of science was suggested by Karl Popper (1904-1994)  It is the prevalent scientific paradigm today  In this view theories cannot bedirectly confirmed, only refuted  If a theory cannot be refuted for a long time, it will gradually beaccepted as confirmed

7. Eike Send, send@inf.fu-berlin.de 7 Different kinds of experiments Empirical research can be conducted in different ways, Tichy has identified seven methods:  Case study  Field experiment  Controlled experiment  Survey  Meta study  Simulation  Benchmark The scientifically most credible method is the controlled experiment / formal experiment

8. Eike Send, send@inf.fu-berlin.de 8 Quantitative vs. qualitative A word on other aspects to discriminate research:  Quantitative research is the numerical representation of observations for describing phenomena  Qualitative research describes how individuals and groups view and understand the world and construct meaning out of their experiences It is essentially narrative-oriented  It is a good idea to combine both: Quantitative: How much time is spent on a type of work? Qualitative: On what activities is it spent?

9. Eike Send, send@inf.fu-berlin.de 9 Terminology of experiments Part 1 The terminology of a formal experiment in Computer Science (CS):  Treatments are activities, methods or tools to be compared e.g. object oriented programming vs. procedural tech.  Control is used to help isolate the effects of a treatment e.g. using a limited inheritance depth or NOT  Trials are individual test runs, using only one treatment  Subjects are the people applying the treatment

10. Eike Send, send@inf.fu-berlin.de 10 Terminology of experiments Part 2  Dependent variables are those factors that are expected to change or differ as a result of applying a treatment e.g. the time it takes to find an error  Independent variables are those variables that may influence the application of a treatment and thus indirectly the result e.g. the subjects in a study  Error is the failure of two identically treated experimental units to yield identical results

11. Eike Send, send@inf.fu-berlin.de 11 The Formal Experiment But what is a formal experiment?  A controlled empirical investigation into some phenomenon with A clearly stated hypothesis and A random assignment of subjects to different treatments

12. Eike Send, send@inf.fu-berlin.de 12 Example of an experiment In 1990 Henry and Humphrey compared an object oriented language with a procedural language:  Treatments: C, objective C  Control: Using either one or the other  Subjects: Twenty students  Dependent variables: maintenance time, error counts, change counts, programmer's subjective impression  Independent variables: subject (student identifier), group (A or B), language, modification task We'll get back to this experiment later on

13. Eike Send, send@inf.fu-berlin.de 13 Six steps of an Experiment (in CS) Conception: Deciding what we wish to learn more about  Define the goals of the experiment  State clearly and precisely the objective of the study Design: Translate the objective into a formal hypothesis  Re-express the goal as a hypotheses The hypothesis is a tentative theory or supposition that we think explains the behaviour we want to explore Often there will be more than one or two hypotheses The null hypotheses states that there is no difference between the treatments The alternative hypotheses states that there is a significant difference Preparation: Make the subjects and the environment ready. If possible, a pilot study of the experiment should be conducted.

14. Eike Send, send@inf.fu-berlin.de 14 Six steps of an Experiment (in CS) Part 2 Execution Analysis: This phase consists of two parts.  All the measurements taken must be reviewed in order to ensure that they are valid and useful.  Analyse of the sets of data according to usual statistical principles. Dissemination and Decision-making: Document the experimental materials and conclusions in a way that will allow others to replicate and confirm the conclusions in a similar setting. The results can be used to  Support decisions about software development and maintenance  Suggest improvements to development environments  Perform similar experiments with different subjects or dependent variables

15. Eike Send, send@inf.fu-berlin.de 15 Data Analysis

16. Eike Send, send@inf.fu-berlin.de 16 Data analysis There are several different kinds of general goals when analysing data:  Exploring something: You do not know in advance what to expect in the data. You try to get an overview of the data you have and to find interesting structure in the data  Measuring something: You know exactly what aspect of an object you are interested in  Modelling something for explanation: You want to describe the mechanism that has produced the data  Modelling something for prediction: You consider your data to be examples. You want to find out how to predict output values given the input values  Comparing something: You have two or more “things” and want to compare them with respect to one or more attributes.

17. Eike Send, send@inf.fu-berlin.de 17 Data analysis Part 2 Data analysis has to support the primary quality attributes of the empirical study overall:  Credibility  Relevance In order to gain credebility the data has to be  Correct: Neither mis-collected nor mis-processed and we trust the analysis (and hence its results)  Illustrative: Easy to understand what the results say and how they came to be, given the data. The analysis makes us understand the data itself  Informative: The analysis reports results that are relevant and helpful for answering the study question

18. Eike Send, send@inf.fu-berlin.de 18 Four steps for data analysis Prechelt provides four steps to carry out a proper data analysis:  Make data available E.g. making it machine readable  Validate data Compare it with expected results Check for impossible or unlikely values  Explore data Get an overview of the whole data set Look at individual variables Look at pairs of variables Quick-check specific expectations  Perform analysis: measure, model, or compare

19. Eike Send, send@inf.fu-berlin.de 19 Part 2: The state of Empiricism in Computer Science

20. Eike Send, send@inf.fu-berlin.de 20 The State of Empiricism in Computer Science “Large parts of CS may not meet standards long established in the natural and engineering sciences” Knowledge is based on “intuitive feelings and anecdotal evidence, than on empirical and quantitative evidence” “Students of computer science are not educated as scientists. They are trained as programmers” “The state of experimentation in software engineering is poor. [...] On the whole, we consider this situation as unacceptable, even alarming” “A survey of over 400 recent research articles suggests that computer scientists publish relatively few papers with experimentally validated results.” Experts developed a consensus “that favoured fostering a culture closer to the traditional sciences as far as experimentation is concerned.”

21. Eike Send, send@inf.fu-berlin.de 21 Theory, Construction, Empiricism The modern sciences consist of:  Theory  Construction  Empiricism Computer science is a mixture of the purely theoretical mathematics and the mostly constructive but also theoretical and empirical electrical engineering All three are essential for successful work Empiricism is slowly gaining ground

22. Eike Send, send@inf.fu-berlin.de 22 Example: Meta Study

23. Eike Send, send@inf.fu-berlin.de 23 Meta study: Object oriented vs. procedural Deligiannis et al. have conducted a meta study:  The question was: “What have studies found out about object oriented technology?”  They took 27 studies into account of which 18 met their requirements of a formal experiment or were case studies  10 of these were on the question “Is there a significant advantage of object oriented technology over procedural technology?”  The other studies were on the subject of inheritance, design principles, design patterns or inspection techniques

24. Eike Send, send@inf.fu-berlin.de 24 Evaluation framework For the evaluation of the experiments they used a framework provided by Wohlin et al. which separates the analysis into the following parts:  Motivation  Hypotheses  Variables  Participants  Experiment design  Results and interpretation  Critique The main critique was that most papers merely used students as participants, because in OOT expertise is important More critique: Designs were flawed or too simple or the setups lacked a documentation of the domain. In the results of the 10 experiments on OOT vs. PT they found out that only 2 could make out an advantage of OOT

25. Eike Send, send@inf.fu-berlin.de 25 Example: OOT vs. PT Example: Study of Object Oriented Technology vs. Procedural Technology

26. Eike Send, send@inf.fu-berlin.de 26 Summary: Henry and Humphrey One of the experiments which could make out a difference was the earlier mentioned study by Henry and Humphrey Motivation Lack of scientific evidence for OOT superiority Hypotheses  alternative H 0 : It is easier to maintain OO programs than structured programs,  H 1 : OO programmers take less time to perform a maintenance task,  H 2 : OO maintenance requires fewer changes to the code,  H 3 : OO programmers perceive the changes as conceptually easier, and  H 4 : OO programmers make fewer errors during the maintenance task

27. Eike Send, send@inf.fu-berlin.de 27 Summary: Henry & Humphrey Part 2 Variables  There were four independent variables: Subject (students) Group (A or B) Programming language Modification task  The dependent variables were: Maintenance times Error counts Change counts Programmers subjective impression Participants “Twenty students participated.”

28. Eike Send, send@inf.fu-berlin.de 28 Summary: Henry & Humphrey Part 3 Experiment design Using a counterbalancing procedure the subjects were to perform enhancement maintenance tasks. The subjects performed each task twice, once in C and once in Objective C and completed a post-experimental questionnaire. Results and interpretation “This experiment supports the hypothesis that subjects produce more maintainable code with an OO language than with a PO language.” Critique The counting of maintenance time relied solely “upon the accuracy of subjects reporting minutes of thinking time” and it is also noted “that inexperienced subjects were used and design documentation was not provided.”

29. Eike Send, send@inf.fu-berlin.de 29 Summary Empiricism:  Is good for gaining valid and credible knowledge  Is an important component of scientific work  Works through falsification  Can use different methods  Can be quantitative or qualitative or both There is a formal way to carry out an experiment which uses a formal terminology Data analysis can have different goals using different methods Empiricism is a part of computer science Currently empiricism in CS does not meet scientific standards There is some empirical research being conducted, although it still needs to be validated

30. Eike Send, send@inf.fu-berlin.de 30 Thank you!