Software Engineering Research Approaches zBalancing theory and praxis zHow engineering research differs from scientific research zThe role of empirical studies zModels for SE research

The need to link research with practice zWhy after 25 years of SE has SE research failed to influence industrial practice and the quality of resulting software? zPotts argues that this failure is caused by treating research and its application by industry as separate, sequential activities. zWhat he calls the research-then-transfer approach. The solution he proposes is the industry-as-laboratory approach.. Colin Potts, Software Engineering Research Revisited, IEEE Software, September 1993

Research-then-Transfer Incremental Refinement of research solutions Problem evolves invisibly to the research community Problem V1 Wide gulf bridged by indirect, anecdotal knowledge Research Solution V1 Technology transfer Gap bridged by hard, but frequently inappropriate technology Problem V2 Problem V3 Problem V4 Research Solution V2 Research Solution V3 Research Solution V4

Research-then-Transfer Problems zBoth research and practice evolve separately zMatch between current problems in industry and research solutions is haphazard zNo winners

Disadvantages of Research-then- Transfer zResearch problems described and understood in terms of solution technology - whatever is current research fashion. Connection to practice is tenuous. zConcentration is on technical refinement of research solution - OK but lacks industrial need as focus, so effort may be misplaced. zEvaluation is difficult as research solutions may use technology that is not commonly used in industry zDelay in evaluation means problem researchers are solving has often evolved through changes in business practice, technology etc. zTransfer is difficult because industry has little basis for confidence in proposed research solution.

Industry-as-Laboratory Approach to SE research Problem V2 Problem V1 Problem V3 Problem V4 Research Solution V1 Research Solution V2 Research Solution V3 Research Solution V4

Advantages of Industry-as- Laboratory Approach zStronger connection at start because knowledge of problem is acquired from the real practitioners in industry, often industrial partners in a research consortium. zConnection is strengthened by practitioners and researchers constantly interacting to develop the solution z Early evaluation and usage by industry lessens the Technology Transfer Gap. zReliance on Empirical Research yshift from solution-driven SE to problem-focused SE ysolve problems that really do matter to practitioners

Early SEI industrial survey research zWhat a SEI survey* learned from industry: yThere was a thin spread of domain knowledge in most projects yCustomer requirements were extremely volatile. zThese findings point towards research combining work on requirements engineering with reuse - instead of the approach of researching these topics by separate SE research communities - as is still found today! * From ‘A field study of the Software Development Process for Large Systems’, CACM, November 1988.

Further Results from Potts et al Early 90s Survey 23 software development organizations (during ). (Survey focused on Requirements Modeling process) zRequirements were invented not elicited. zMost development is maintenance. zMost specification is incremental. zDomain knowledge is important. zThere is a gulf between the developer and user zUser-interface requirements continually change. zThere is a preference for office-automation tools over CASE tools to support development. I.e. developers found using a WP + DB more useful than any CASE tools.

Industry-as-Laboratory emphasizes Real Case Studies Advantages of case studies over studying problems in research lab. zScale and complexity - small, simple (even simplistic) cases avoided - these often bear little relation to real problems. zUnpredictability - assumptions thrown out as researchers learn more about real problems zDynamism - a ‘real’ case study is more vital than a textbook account The real-world complications of industrial case studies are more likely to throw up representative problems and phenomena than research laboratory examples influenced by the researchers’ preconceptions.

Need to consider Human/Social Context in SE research zNot all solutions in software engineering are solely technical. zThere is a need to examine organizational, social and cognitive factors systematically as well. zMany problems are “people problems”, and require “people-orientated” solutions.

Theoretical SE research zWhile there is still a place for innovative, purely speculative research in Software Engineering, research which studies real problems in partnership with industry needs to be given a higher profile. zThese various forms of research ideally complement one another. zNeither is particularly successful if it ignores the other. zToo industrially focused research may lack adequate theory! zAcademically focused research may miss the practice!

Research models for SE zProblem highlighted by Glass*: Most SE Research in 1990s was Advocacy Research. Better research models needed. zThe software crisis provided the platform on which most 90s research was founded. zSE Research ignored practice, for the most part; lack of practical application and evaluation were gapping holes in most SE research. zAppropriate research models for SE are needed. * Robert Glass, The Software -Research Crisis, IEEE Software, November 1994

Methods underlying Models zScientific method zEngineering method zEmpirical method zAnalytical method From W.R.Adrion, Research Methodology in Software Engineering, ACM SE Notes, Jan. 1993

Scientific method Observe real world Propose a model or theory of some real world phenomena Measure and analyze above Validate hypotheses of the model or theory If possible, repeat

Engineering method Observe existing solutions Propose better solutions Build or develop better solution Measure, analyze, and evaluate Repeat until no further improvements are possible

Empirical method Propose a model Develop statistical or other basis for the model Apply to case studies Measure and analyze Validate and then repeat

Analytical method Propose a formal theory or set of axioms Develop a theory Derive results If possible, compare with empirical observations Refine theory if necessary

Need to move away from purely analytical method zThe analytical method was the most widely used in mid-90s SE research, but the others need to be considered and may be more appropriate in some SE research. zGood research practice combines elements on all these approaches.

4 important phases for any SE research project (Glass) zInformational phase - Gather or aggregate information via yreflection yliterature survey ypeople/organization survey ycase studies zPropositional phase - Propose and build hypothesis, method or algorithm, model, theory or solution zAnalytical phase - Analyze and explore proposal leading to demonstration and/or formulation of principle or theory zEvaluation phase - Evaluate proposal or analytic findings by means of experimentation (controlled) or observation (uncontrolled, such as case study or protocol analysis) leading to a substantiated model, principle, or theory.

Software Engineering Research Approaches zThe Industry-as-Laboratory approach links theory and praxis zEngineering research aims to improve existing processes and/or products zEmpirical studies are needed to validate Software Engineering research zModels for SE research need to shift from the analytic to empirical.