1 14.Sep.2010 Systematic Literary Reviews: Experience and Advice Systematic Literary Reviews: experience and challenges ISERN discussion session 14 Sept. 2010, Bolzano, Italy Reidar Conradi 1), Mohammad Ali Babar 2) 1) NTNU, NO-7491, Trondheim, Norway 2) ITU, Copenhagen, Denmark

X-discipllnarity in OSS; GoOpen, 20 Apr Table of contents 0. Some challenges of SRLs 1.What is so special about SE and ESE/EBSE ? 2.Availability and Utility of current SLRs 3.Utility of SLRs for Agile incl. Pair Programming 4.SLR: status, challenges, future advice 5.Future: OSS and networked development 6.References

X-discipllnarity in OSS; GoOpen, 20 Apr Some challenges of SRL Summary of challenges: What is currently known about the strengths and limitations of systematic reviews for software engineering? How rigorous and relevant are the systematic reviews reported so far? What are the challenges of the adoption of this methodology in software engineering? What are the improvements proposed for maturing this methodology? What is the role of systematic reviews in software engineering research and practice in the coming ten years? How can ISERN members contribute to the evidential assessment of the rigor, effectiveness and relevance of systematic reviews in software engineering? What kinds of Methods are required to ensure the rigor and relevance of systematic reviews for software practice? What can be done to address the most common challenges (such as forming optimal search strategy, assessing primary studies, and synthesizing the evidence) involved in adopting systematic reviews in software engineering research and practice.

X-discipllnarity in OSS; GoOpen, 20 Apr What is so special about SE and ESE/EBSE? Still looking for the silver bullet; cf. Conradi’s own “odyssee”: PLs, ADT/OO, PEs, SE DBs, SCM, Process, Quality/SPI, Reuse/CBSE and gradually open source software (OSS) -- plus ESE/EBSE. SE is both “hard” and “soft“ – so tough to improve predictibly: still 5+-2 SLOCs per person-hour for large software; rule. Five trends: –Massive global reuse/CBSE by OSS - half million software components available –New cooperation and innovation processes around OSS –Flood of mobile/media applications w/ fancy and fast-moving technologies –Some new SE technologies: Agile methods, SOA, cloud computing, … –Empirical track: what works how, where and why – takes time w/ narrow focus – so are we not studying the most relevant issues?

X-discipllnarity in OSS; GoOpen, 20 Apr Don’t forget relevance: PITL vs. PITS Relevance vs. rigor: PITL vs. PITS, case studies vs. experiments ? Ex. top-10 project killers are PITL-related (Boehm89, Jones02, Charette05): –Goals/commitment –Project management –Communication –Bad requirements –Changing requirements –Estimation, tracking –Product quality –Technical personnel –Development technology –Insufficient software reuse So try mix agile + OSS + mobile … – and less experiments?

X-discipllnarity in OSS; GoOpen, 20 Apr The EBSE challenge How to synthesize relevant experience in a meta-analysis? How to find and retrieve relevant experience – via data mining and semantic searches e.g. by Google or Case-Based reasoning. Costly, but effort not reported! Ex. Medline portal receives 2000 new journal articles per day! Industry: how to motivate for an empirical approach. And yet much turbulence in companies and their contact persons. COMMENTS: Status of available SLRs….. Comments on SLR on Agile methods (Dybå&al.2007). Detailed comments on SLR with a statistical meta-analysis of Pair Programming (Hannay&al.2009). Recommendations.

X-discipllnarity in OSS; GoOpen, 20 Apr The utility of existing SLRs, v1 Controlled Experiments (Sjøberg2005): –Found 121 such, or 2% of all empirical studies, –20 repeated experiments: 7 of 11 had same results internally, but only 2 of 9 had same results externally (unclear context factors?). Quasi-experiments (Kampenes20xx): need randomized subjects. Effect size (Dybå200x): often too small to avoid Type-II errors, i.e. need a β between 60 and 80. Theory use (Hannay20xx): marginal, as most experiments have their own agenda, and it is hardly not “feasible” for an experiment to be repeated N > 100 times. Cf. 3p physics papers with 150 co-authors, and similar in medicine with over 100,000 subjects. => Merge above insights in a shared recommendation!! Means that future EBSE experiments must be more closely coordinated – but by whom and how?

X-discipllnarity in OSS; GoOpen, 20 Apr The utility of existing SLRs, v2 Estimation of effort/schedule (Jørgensen20xx): stable 30% overruns in the last 30 years. Domain expert will usually beat a general and formal tool. Knowledge management (Dingsøyr&Bjørnson20xx): not covering actual use. Agile methods (Dybå&Dingsøyr2007): see later. PP meta-analysis (Hannay&al.2009): see later. Open Source Software (Hauge&al.2010): Industrial OSS adoption. I.e. a Scandinavian discipline??

X-discipllnarity in OSS; GoOpen, 20 Apr The utility of existing SLRs, v3 From the Univ. Keele portal: ++??add more Design inspections (Runeson20xx): generally more effective than unit testing. Testing (Juristo20xx): Unit testing showing diverse results. Software Architecture (Travassos20xx):... Requirements engineering (20xx):......

X-discipllnarity in OSS; GoOpen, 20 Apr Some quasi SLRs, v4 SCM: –SCM overview (Mili and Mili and Mili) –Version models (Conradi&Westfechtel2007) – ICSE IMPACT paper on SCM (Estublier&al.2007) Reuse: –HP?? (Wayne Lim) –Industrial reuse (Mohagheghi&Conradi2008) Inspections and testing: –ICSE IMPACT paper on defect discovery (Biffl&al.2007) – Grady’s book?? – Gilb&Graham book??

X-discipllnarity in OSS; GoOpen, 20 Apr More conventional SLRs, v5 ICSE: SE roadmap book (2000), FOSE roadmap book (2007). IMPACT papers ESE community: laws: Endres/Rombach Dagstuhl seminars 2007 (Basili et al.) Tutorials, keynotes, textbooks, industrial seminars Make comparison table of some of this?

X-discipllnarity in OSS; GoOpen, 20 Apr How to find more SLRs: what queries work? Repeat for i=1..n over the entire Google universe (not limited to ACM, IEEE, …) the following query: Software and SE-subarena_i and (Literature Review or Review or Overview or Survey of or Lessons learned or Experience With or …). Where SE_subarena_i:, engineering, requirements engineering, design, architecture, coding, implementation, programming, testing, project management, configuration management, … But much manual post-processing of the query results!

X-discipllnarity in OSS; GoOpen, 20 Apr Two SLRs on Agile Methods Agile manifesto: improvisation and quick changes, not plan-based and stiff waterfall regime. Agile technologies: XP (12 techniques incl. PP and TDD), Scrum, EVO, DSDM, rapid prototyping, … Much overselling, with an anti-metrics culture. Recent challenges with global, distributed/mobile teamwork. (Dybå&Dingsøyr2007): SLR of 33 primary research papers on Agile technologies. (Hannay&al.2009): Meta-analysis of 18 primary research papers on PP.

X-discipllnarity in OSS; GoOpen, 20 Apr SLR on Agile Methods (Dybå&Dingsøyr2007): SLR w/ no clear conclusion of XP benefits, but with perceived positive attitudes. StudyContextProductivity of XP vs. traditional Quality of XP vs. traditional S07 (Dalcher05), experiment 4 diff. life-cycle models, stud., 1 y, 15 teams, t.size: > 13.1 LOC/h for XP, but XP gives 3.5x more code No recorded (n SLR) S10 (Ilieva04), case study Two similar projects, prof., 900h, t.size: > 5.4 LOC/h for XP, mostly in 1st of 3 releases 13% fewer defect reports by XP S14 (Layman04), case study Reimpl.proj., prof., 3.5 months, w/ 3 of 10 prev. team-memb > 340 LOC/month for XP Much better pre/post- delivery quality/defects by XP S15 (Macias03), experiment 10 XP vs. 10 trad. teams, stud., 1 semester, t.size: 4-5 No difference in productivity No difference in quality/defects S32 (Wellington05), experiment One XP and one trad. team, stud.,1 semester, t.size: 16 Trad. team: 78% more code than XP XP team delivered much better quality

X-discipllnarity in OSS; GoOpen, 20 Apr Reflections on the two Agile SLRs In Agile: Project management and requirements engineering most importent – not PITS-inspired factors like productivity or defect density. In first SLR on Agile: 3 XP (PP?) experiments in In second SLR on PP: 18 other experiments, 9 from , 9 from

X-discipllnarity in OSS; GoOpen, 20 Apr SLR on Pair Programming (PP): intro PP: One of the 12 methods in XP, with roots in generous teamwork elsewhere – cf. book by (Weinberg1971). So, what do we know on PP: either less effort, shorter duration, better “quality”– or just happier people with more shared knowledge and experience, learning from another. Moderate effects on programming-related factors: +- 15%. (Hannay&al.2009): A statistical meta-analysis of PP experiments in 18 primary research papers. Technical emphasis: quality, duration, effort – not longitudinal studies of social/cognitive factors to promote learning.

X-discipllnarity in OSS; GoOpen, 20 Apr ”3.5 SLR must consider ”Project- triangle” – three competing variables or aspects Result: Functionality / Quality ”Project- Triangle” Effort Duration/ deadline

X-discipllnarity in OSS; GoOpen, 20 Apr SLRs on Pair Programming (PP) – in detail No common experiment organization or treatment wrt.: Goals and objectives. Team composition: software expertise (student vs. practitioner, programming skills), soloists vs. pairs, application task, use of external reviewers, … Software technology: Eclipse; UML; Java, C, C++ Randomized vs. self-chosen team composition. And here: no social/cognitive aspects studied. Tree major dependent variables: –Quality (hmm?) – see next slides –Duration (hours) – elapsed time –Effort (person-hours) – often just named hours –No volume / productivity measures (SLOC, FP, …), calculated by tools.

X-discipllnarity in OSS; GoOpen, 20 Apr SLRs on PP – in detail Study Quality (14 non-empty slots) Duration (11 non-empty slots) Effort 11 non-empty slots) P1 Arisholm et al. (2007) QDE P2 Baheti et al. (2002) QD- P3 Canfora et al. (2005) - Q missingDE P4 Canfora et al. (2007) QDE P5 Domino et al. (2007) Q (not elsewhere)-- P6 Heiberg et al. (2003) QD- P7 Madeyski (2006) Q (not elsewhere) -- P8 Madeyski (2007) Q (not elsewhere) -- P9 Müller (2005) QDE P10 Müller (2006) QDE P11 Nawrocki & Wojciechowski (2001) - Q missingDE P12 Nosek (1998) QDE P13 Phongpaibul & Boehm (2006) Q-E P14 Phongpaibul & Boehm (2007) Q-E P15 Rostaher & Hericko (2002) - Q missingD (only mentioned here)- P16 Vanhanen & Lassenius (2005) ++ reduce col. height?? - Q missingE (only mentioned here) P17 Williams et al. (2002) Q (not elsewhere)-- P18 Xu & Rajlich (2006) QDE

X-discipllnarity in OSS; GoOpen, 20 Apr SLRs on PP – the Quality results Study Quality (14 non-empty slots) Effort 11 non-empty slots) P1 Arisholm et al. (2007) Q P10 Müller (2006) Q P2 Baheti et al. (2002) Q P11 Nawrocki & Wojciechowski (2001) - Q missing P3 Canfora et al. (2005) - Q missing P12 Nosek (1998) Q P4 Canfora et al. (2007) Q P13 Phongpaibul & Boehm (2006) Q P5 Domino et al. (2007) Q (not elsewhere) P14 Phongpaibul & Boehm (2007) Q P6 Heiberg et al. (2003) Q P15 Rostaher & Hericko (2002) - Q missing P7 Madeyski (2006) Q (not elsewhere) P16 Vanhanen & Lassenius (2005) ++ reduce col. height?? - Q missing P8 Madeyski (2007) Q (not elsewhere) P17 Williams et al. (2002) Q (not elsewhere) P9 Müller (2005) Q P18 Xu & Rajlich (2006) Q

X-discipllnarity in OSS; GoOpen, 20 Apr The future: how to consider OSS? OSS: changes the whole paradigm of software development and associated economic patterns and interactions – by cooperative, distributed innovation. Norwegian software-intensive companies and public institutions must undergo this economic and cultural revolution: –novel Innovation Models: new products and services –novel Business Models: make money on these Need professional partnership communities to establish and evolve their needed software, itself being OSS. Move away from “unpredictable” volonteers working for free. OSS: adopted by private and public policy makers: e.g., IBM and Sun Microsystems, IKT-Norge, …, Skattedirektoratet, KS, … St.meld. 17 ( ) by Norwegian Government recommends OSS and open standards. competence center. Also (NFR).

X-discipllnarity in OSS; GoOpen, 20 Apr From linear to networked paradigm Before: Now:

X-discipllnarity in OSS; GoOpen, 20 Apr Networked development actors Emerging network paradigm for OSS will affect: Long-term vs. short-term planning, Cross-company cooperation and dependence: (re)negotiation vs. dictate, Control mechanisms: many risky parts, (Open) Innovation: new combinations, spin-offs, … Legal: IPRs and licensing, Economical: new business models, Marketing: close to development, => Requirement Engineering reborn! See [Ayala09] [Hauge10] [Lindman09].

X-discipllnarity in OSS; GoOpen, 20 Apr OSS Ecosystem for support - by Friprog-.no

X-discipllnarity in OSS; GoOpen, 20 Apr Summary of X-disciplinary aspects in OSS Situated selection of promising components, pragmatic approach, later auto-generated ontologies for functionality properties. New ways of cooperation and sharing: new work organization, very interesting paradigm – “The world is flat”, global cooperation. Ex. Cooperate on shared, domain-specific middleware. Open Innovation model: altruism gives the highest long-term profit, by combining several open technologies into new products and services [Browning08] [Chesbrough03] [Chesbrough03a] [Chesbrough06]. Business model: and making money of this - proven to work! –Apache: open and shared source supported by a cooperative foundation (“IBM”); separate payable services. –eZ: dual model with free previous version, payable current version plus services and support. 25

X-discipllnarity in OSS; GoOpen, 20 Apr References (1) [Ayala09] C. Ayala, Ø. Hauge, R. Conradi, X. Franch, et al.: ”Challenges Using the OSS Component Marketplace in the Industry”, Proc. OSS'09. [Bowker99] Geoffrey C. Bowker and Susan Leigh Star: ”Sorting Things Out - Classification and Its Consequences”, MIT Press, Boston, USA, Oct. 1999, 389 pages, ISBN [Browning08] L. D. Browning, Alf Steinar Sætre, K. Stephens and J.-O. Sørnes: ”Information and Communication Technologies in Action: Linking Theory and Narratives of Practice”. New York: Routledge, [Chesbrough03] Henry W. Chesbrough: OPEN Innovation: The new imperative for creating and profiting from technology, Harvard Business School Press, [Chesbrough03a] Henry W. Chesbrough: “The era of open innovation”, MIT Sloan Management Review, 44(3):35-41 (2003). [Chesbrough06] Henry W. Chesbrough: OPEN Business Models – How to Thrive in the New Innovation Landscape, Harvard Business School Press, [Hauge10] Ø. Hauge, D. Cruzes, R. Conradi et al.:"Risks and Risk Mitigation in Open Source Software Adoption", Proc. OSS'2010. [Lindman09] Juho Lindman et al.: “Beyond the Business Model: Incentives for Organizations to Publish Software Source Code”, Proc. OSS'2009.