Magne Jørgensen Myths, Over- simplifications and Unconfirmed Claims in Software Engineering Magne Jørgensen

The paper clip was invented by a Norwegian

Short men are more aggressive (The Napoleon complex)

Most communication is non-verbal

There were/is a software crisis (page 13 of their 1994-report): “We then called and mailed a number of confidential surveys to a random sample of top IT executives, asking them to share failure stories.”

Difficult to remove a myth Years of critique may, however, have had an small effect (from the CHAOS Report – 2013):

45% of features of “traditional projects” are never used (source: The Standish Group, XP 2002) No-one seems to know (and the Standish Group does not tell) anything about this study! Why do so many believe (and use) this non-interpretable, non-validated claim? They benefit from it (agile community) + confirmation bias (we all know at least one instance that fit the claim)

14% Waterfall and 42% of Agile projects are successful (source: The Standish Group, The Chaos Manifesto 2012) Successful = “On time, on schedule and with specified functionality” Can you spot a serious error of this comparison?

There is an increase in cost of removing errors in later phases

The magic number 7 ± 2

The number one in the stink parade …

The ease of creating myths: Are risk-willing or risk-averse developers better? Study design: Research evidence + Self-generated argument. Question: Based on your experience, do you think that risk-willing programmers are better than risk-averse programmers? 1 (totally agree) – 5 (No difference) - 10 (totally disagree) Neutral group: Average 5.0 Group A: Group B: Initially Average 3.3 Debriefing Average 2: weeks later Average 3: 3.5 Initially Average 5.4 Debriefing Average 2: weeks later Average 3: 4.9

How to convince software engineers Context: Assume that a test course provider claims: ”The course will lead to substantial increase in test efficiency and quality for most participants.” How likely do you think this claim is true, given [reduced explanation of options]: A: No other information B: Supporting claims from reference clients C: Supporting study conducted by the course provider D: Convincing explanation (but no empirical evidence) E: Supporting experience from a colleague (It helped him) F: Supporting scientific study completed at a renowned university G: Own experience (It helped me)

Results not very promising for evidence-based software engineering A: No other information B: Support from reference clients C: Supporting study conducted by the course provider D: Convincing explanation (but no empirical evidence) E: Supporting experience from a colleague (It helped him) F: Supporting scientific study completed at a renowned university G: Own experience (It helped me)

“I see it when I believe it” vs “I believe it when I see it” Design: –Data sets with randomly set performance data comparing “traditional” and “agile” methods. –Survey of each developer’s belief in agile methods Question: How much do you, based on the data set, agree in: “Use of agile methods has caused a better performance when looking at the combination of productivity and user satisfaction.” Result: –Previous belief in agile determined what they saw in the random data

When making a decision or choice, the world is no more the same (Dan Gilbert)

Elements of more evidence-based practise (Myth busting in software engineering) 1.Find out what is meant by the claim –Is it possible to falsify evaluate the claim? –How general is it meant to be? 2.Prepare for evaluation of validity of claim and argumentation –Be aware of your confirmation bias tendency, especially when you agree –Consider what you would consider as valid evidence to support the claim. –Check for vested interest 3.Collect/create and evaluate evidence –Research-based evidence –Practice-based evidence –Locally created, experimental evidence 4.Synthesize evidence and conclude (if possible)

Evidence-based software engineering (EBSE) The main steps of EBSE are as follows: 1.Convert a relevant problem or need for information into an answerable question. 2.Search the literature and practice-based experience for the best available evidence to answer the question. 3.Critically appraise the evidence for its validity, impact, and applicability. 4.Integrate the appraised evidence with practical experience and the client's values and circumstances to make decisions about practice. 5.Evaluate performance in comparison with previous performance and seek ways to improve it. NB: EBSE is NOT the same as systematic literature review! Tore Dybå, Barbara Kitchenham and Magne Jørgensen, Evidence-based Software Engineering for Practitioners, IEEE Software, Vol. 22, No. 1, Jan-Feb 2005.

EXPERIENCE FROM TEACHING EVIDENCE-BASED SOFTWARE ENGINEERING

Background University courses with students and with software professionals –30 hours lecturing –Supervised project report writing with self-selected “problem formulation” Short courses/seminars with software professionals In total, I’v taught several hundreds of students and software professionals EBSE.

Example of what they learn: A structure of analyzing claims, argumentation and evidence (Toulmin’s model) Data Claim Backing Warrant Qualifier Reservation

Any positive impact from teaching Evidence-Based Software Engineering? Positive impact: –They get more critical towards claims made by gurus and in textbooks, especially when there are “vested interest” –The get better and asking what claims really mean –They discover the strength of collecting and evaluating evidence Challenges: –Evaluating evidence and argumentation is new to them. Only a few (20%?) develop good evaluation skills, although all seem to improve through the courses. –Unlike evidence-based medicine, EBSE cannot rely on research-based evidence only. They need to be trained in collecting practice-based evidence.

Challenge: Lack of evidence Empirical research is frequently sparse and not very relevant! –… and the quality is not impressive either. A need for training in collecting practice-based evidence –Very much the same method as for evaluating research A need for training in designing and collecting own evidence –This is more new ground, but the benefit from training in experimental method, use of representative “trials” etc. is very promising. –On the long term, this type of evidence (local, highly relevant, collected in the context it will be used) is perhaps the one with most potential of changing the software practices. NB: The problems with this kind of learning/improvement in other frameworks (CMM-I, PSP, PDCA, GQM, …) do not defend very much optimism. An advantage of EBSE may be the focus on posing a meaningful problems and the combination of different sources of evidence.

What I tried to say … The software engineering discipline is filled with myths, fashion, over- simplifications and unconfirmed claims. –Being aware of the mechanisms may make us more rational The software engineering industry waste much energy due to lack of collecting evidence on what works and what not. –Example: We change from one technology/method to the next without knowing much about whether it will improve anything As researchers/university employees we should try –Train software professionals in EBSE, including how to become myth busters –Provide (synthesized) evidence that is relevant and of good quality. –Write books and guidelines that are evidence-based

Coffee dehydrates your body

BONUS MATERIAL

IMPACT OF VESTED INTEREST IN SOFTWARE ENGINEERING STUDIES (A BRIEF DISTRACTION) 35 PUBLISHED COMPARISONS OF REGRESSION AND ANALOGY- BASED EFFORT ESTIMATION MODELS

Regression-based cost estimation model better Study Effect size = MMRE_analogy – MMRE_regression Analogy-based models better

Regression-based cost estimation model better Study Effect size = MMRE_analogy – MMRE_regression Development of own analogy-based model (vested interests) Regression models better