Empirically Assessing End User Software Engineering Techniques Gregg Rothermel Department of Computer Science and Engineering University of Nebraska -- Lincoln

Questions Addressed How can we use empirical studies to better understand issues/approaches in end user SE? What are some of the problems empiricists working on end-user SE face? What are some of the opportunities for software engineering researchers working in this area?

Outline Background on empirical approaches Empiricism in the end-user SE context Problems for empiricism in end-user SE Conclusion

Outline Background on empirical approaches Empiricism in the end-user SE context Problems for empiricism in end-user SE Conclusion

Empirical Approaches: Types Survey – interviews or questionnaires Controlled Experiment - in the laboratory, involves manipulation of variables Case Study - observational, often in-situ

Empirical Approaches: Surveys Pose questions via interviews or questionnaires Process: select variables and choose sample, frame questions that relate to variables, collect data, analyze and generalize from data Uses: descriptive (assert characteristics), explanatory (assess why), exploratory (pre-study) Resource: E. Babbie, Survey Research Methods, Wadsworth, 1990

Empirical Approaches: Controlled Experiments Manipulate independent variables and measure effects on dependent variables Requires randomization over subjects and objects (partial exception: quasi-experiments) Relies on controlled environment (fix or sample over factors not being manipulated) Often involves a baseline (control group) Supports use of statistical analyses Resource: Wohlin et al., Experimentation in Software Engineering, Kluwer, 2000

Empirical Approaches: Case Studies Study a phenomenon (process, technique, device) in a specific setting Can involve comparisons between projects Less control, randomization, and replicability Easier to plan than controlled experiments Uses include larger investigations such as longitudinal or industrial Resource: R. K. Yin, Case Study Research Design and Methods, Sage Publications, 1994

Empirical Approaches: Comparison

Outline Background on empirical studies Empiricism in the end-user SE context Problems for empiricism Conclusion

Three Aspects of Empiricism 1.Studies of EUSE (and SE) have two focal points –The ability of end users to use devices/processes –The devices and processes themselves 2.Evaluation and design of devices and processes are intertwined: –Summative evaluation helps us assess them –Formative evaluation helps us design them 3.We need families of empirical studies: –To generalize results –Studies inform and motivate further studies

Domain Analyses Think-Aloud, Formative Case Studies, Surveys Controlled Experiments Controlled Experiments Summative Case Studies Exploratory, Theory Dev. Hypothesis Testing Generalization Building Empirical Knowledge through Families of Studies user environment, device

Domain Analyses Think-Aloud, Formative Case Studies, Surveys Controlled Experiments Controlled Experiments Summative Case Studies Exploratory, Theory Dev. Hypothesis Testing Generalization Building Empirical Knowledge through Families of Studies user environment, device

Empirical Studies in WEUSE Papers Surveys -Scaffidi et al.: usage of abstraction, programming practices -Miller et al.: how users generate names for form fields -Segal: needs/characteristics of professional end user developers -Sutcliffe: costs/benefits perceived by users of a web- based content mgmt. system Domain analysis –Elbaum et al.: fault types in Matlab programs Controlled experiments –Fisher et al.: infrastructure support for spreadsheet studies

Cell turns more blue (more “tested”). Testing also flows upstream, marking other affected cells too. Example: What You See is What You Test (WYSIWYT) At any time, user can check off correct value.

Domain Analyses Think-Aloud, Formative Case Studies, Surveys Controlled Experiments Controlled Experiments Summative Case Studies Exploratory, Theory Dev. Hypothesis Testing Generalization Building Empirical Knowledge of End User SE through Families of Studies user environment, device

Study 1: Effectiveness of DU- adequate test suites (TOSEM 1/01) RQ: Can DU-adequate test suites detect faults more effectively than other types of test suites? Compared DU-adequate vs randomly generated suites of the same size, for ability to detect various seeded faults, across 8 spreadsheets Result: DU-adequate suites were significantly better than random at detecting faults

Domain Analyses Think-Aloud, Formative Case Studies, Surveys Controlled Experiments Controlled Experiments Summative Case Studies Exploratory, Theory Dev. Hypothesis Testing Generalization Building Empirical Knowledge of End User SE through Families of Studies user environment, device

RQs: Are WYSIWYT users more (effective, efficient) than Ad-Hoc? Compared two groups of users, one using WYSIWYT, one not, each on two spreadsheet validation tasks Participants drawn from Undergraduate Computer Science classes Participants using WYSIWYT were significantly better at creating DU-adequate suites, with less redundancy in testing Study 2: Usefulness of WYSIWYT (ICSE 6/00)

Domain Analyses Think-Aloud, Formative Case Studies, Surveys Controlled Experiments Controlled Experiments Summative Case Studies Exploratory, Theory Dev. Hypothesis Testing Generalization Building Empirical Knowledge of End User SE through Families of Studies user environment, device

Study 3: Usefulness of WYSIWYT with End Users (ICSM 11/01) RQs: Are WYSIWYT users more (accurate, active at testing) than Ad-Hoc? Compared two groups of users, one using WYSIWYT, one not, each on two spreadsheet modification tasks Participants drawn from Undergraduate Business classes Participants using WYSIWYT were more accurate in making modifications, and did more testing

User can enter assertions System can figure out more assertions User can enter assertions Study 4: Using Assertions (ICSE 5/03)

Domain Analyses Think-Aloud, Formative Case Studies, Surveys Controlled Experiments Controlled Experiments Summative Case Studies Exploratory, Theory Dev. Hypothesis Testing Generalization Building Empirical Knowledge of End User SE through Families of Studies user environment, device

RQs: will end users use assertions and do they understand the devices Observed persons as they worked with Forms/3 spreadsheets with assertion facilities provided Study 4: Using Assertions (ICSE 5/03)

There’s got to be something wrong with the formula! Study 4: Using Assertions (ICSE 5/03)

Outline Background on empirical studies Empiricism in the end-user SE context Problems for empiricism in end-user SE Conclusion

Problems for Empiricism in EUSE Threats to validity – factors that limit our ability to draw valid conclusions –External: ability to generalize –Internal: ability to correctly infer connections between dependent and independent variables –Construct: ability of dependent variable to capture the effect being measured –Conclusion: ability to apply statistical tests

External Validity Subjects (participants) aren’t representative Programs (objects) aren’t representative Environments aren’t representative Problems are trivial or atypical

Internal Validity Learning effects, expectation bias, … Non-homogeneity among groups (different in experience, training, motivation) Devices or measurement tools faulty Timings are affected by external events The act of observing can change behavior (of users, certainly, but also of artifacts)

Construct Validity Lines of code may not adequately represent amount of work done Test coverage may not be a valid surrogate for fault detection ability Successful generation of values doesn’t guarantee successful use of values Self-grading may not provide an accurate measure of confidence

Conclusion Validity Small sample sizes Populations don’t meet requirements for use of statistical tests Data distributions don’t meet requirements for use of statistical tests

Other Problems Cost of experimentation Difficulty of finding suitable subjects Difficulty of finding suitable objects Difficulty of getting the design right

Outline Background on empirical studies Empiricism in the end-user SE context Problems for empiricism in end-user SE Conclusion

Questions Addressed How can we use empirical studies to better understand issues/approaches in end user SE? –Via families of appropriate studies, using feedback and replication What are some of the problems empiricists working on end-user SE face? –Threats to validity, many particular to this area –Costs, and issues for experiment design/setup What are some of the opportunities for software engineering researchers working in this area? –Myriad, given the range of study types applicable –Better still with collaboration

Empirically Assessing End User Software Engineering Techniques Gregg Rothermel Department of Computer Science and Engineering University of Nebraska -- Lincoln