Towards Common Standards for Studies of Software Engineering Tools and Tool Features Timothy C. Lethbridge University of Ottawa

Premise: It is desirable to guide researchers studying SE tools Proposal: Create an inventory of practices to guide such studies Researchers could then create papers that would be  More comparable  More easily reviewable  More indexable

Types of Evaluation Commonly Found in Tools Papers  a) None - just a description  b) Includes rationale  c) Demonstration of adoption  d) Anecdotes and lessons learned  e) Informal studies - includes descriptive stats  f) Formal experiments involving students  g) Formal experiments involving practitioners Case studies papers:  Some combination of b-e Experimental papers:  f and g  but beware of overconfidence in results Papers of type e, f and g would benefit from following certain consistency patterns to facilitate comparability

Inventory of Measures. The following are purely examples that might be found in such an inventory  M1. Time taken to perform a given task.  M2. Amount of a given task completed correctly in a fixed time. The fixed time might depend on the task.  M3. Errors made in a given task  M4. Subjective answers on a scale to specific questions: (Questions to be listed in the inventory)

Inventory of study types ST1. Usability evaluation of a specific feature or tool implementation.  Help ensure that results from other study types are not confounded purely by poor usability. Provides evidence for these research questions:  Q1a To what extent is the feature or tool usable? Measures: M1, M2 and M3 (compared against a threshold).  Q1b What usability defects are present and which ones should be repaired? (qualitative).

Study types - continued ST2. Comparison of a small number of different feature implementations, each providing roughly the same functionality. Provides evidence for these research questions:  Q2a What is the best user interface for a certain feature? Measures: M1, M2, M3, M4 (measured separately for each implementation)  Q2b What comments do users have about each implementation? (qualitative)

Study types - continued ST3. Comparison of two alternative feature sets that achieve roughly the same goal, but in different ways. Provides evidence for these research questions:  Q3 What is the 'best' functionality for a certain task? Measures: M1, M2, M3, M4  Measured separately for each feature set

Study types - continued ST4.Comparison of presence and absence of a feature (or of a small feature set) in a tool Provides evidence for these research questions:  Q4a Is the feature worth including in a final tool set? Measures: M1, M2, M3 (measured separately for a tool with presence or absence of the features)  Q4b What benefits are provided by the feature? (qualitative)

Study types - continued ST5. Determination of which specific combinations of features are most useful as the context varies Provides evidence for these research questions:  Q5 Which features should be available in a given tool so the tool can be used in a variety of contexts? Measures: M1, M2, M3, M4a, M4c  Measured as the feature sets and contexts are varied in different combinations

Study types - continued ST6 Comparison of entire tools  Incorporating sets of features  Less abstract than ST3 Provides evidence for these research questions:  Q6 Which of several tools is best used for a given task? Measures: M1, M2, M3, M4  Measured separately for each tool