Software Engineering Experimentation An Experimental Framework Jeff Offutt http://www.cs.gmu.edu/~offutt/ Used with permission by Jane Hayes http://www.cs.uky.edu/~hayes

Experimental Framework Victor Basili, Richard Selby and David Hutchens, “Experimentation in Software Engineering”, IEEE Transactions on Software Engineering, SE-12(7):733–743, July 1986 A little old but still relevant and helpful Three objectives Framework for performing experiments Classify and discuss experiments Problem areas and lessons learned © Jeff Offutt, 2005-2012

Framework Four categories Definition – Problem definition and hypothesis Planning – Design Operation – Conduct test Interpretation – Hypothesis test, disseminate © Jeff Offutt, 2005-2012

Framework – 1. Definition Motivation (why) Object (what) Purpose Perspective (who) Domain (which) Scope Understand Assess Manage Engineer Learn Improve Validate Assure Product Process Model Metric Theory Characterize Evaluate Predict Motivate Developer Modifier Maintainer Project manager Corp. manager Customer User Researcher Programmer Program / project Single project Multi-project Replicated project Blocked subject- © Jeff Offutt, 2005-2012

Framework – 2. Planning Design (samples, control) Criteria (what to measure) Measurement (how to measure) Experimental designs Incomplete block Completely random Randomized block Fractional factorial Multivariate analysis Correlation Factor analysis Regression Statistical models Non-parametric Sampling Direct reflections of cost / quality Cost Errors Changes Reliability Correctness Indirect reflections of cost / quality Data coupling Information visibility Programmer comprehension Execution coverage Size Complexity Metric definition Goal-question-metric Factor-criteria-metric Metric validation Data collection Automatability Form design and test Objective vs. subjective Level of measurement Nominal / classificatory Ordinal / ranking Interval Ratio © Jeff Offutt, 2005-2012

Framework – 3. Operation Preparation Execution Analysis Pilot study Data collection Data validation Quantitative vs. qualitative Preliminary data analysis Plots and histograms Model assumptions Primary data analysis Model application © Jeff Offutt, 2005-2012

Framework – 4. Interpretation Interpretation context Extrapolation Impact Statistical framework Study purpose Field of research Sample representativeness Visibility Replication Application © Jeff Offutt, 2005-2012

Framework Example Definition Definition Element Example Motivation To improve unit testing Object characterize and evaluate Purpose the process of functional & structural testing Perspective from the perspective of the developer Domain: programmer as they are applied by experienced programmers Domain: program to unit-size software Scope in a blocked subject-project study © Jeff Offutt, 2005-2012