Chapter 12 Evaluating Products, Processes, and Resources

Review  Software engineering provides a large variety of methods and tools for use by software developers throughout the software life-cycle.  What are some of them?

Review of S.E. Methods and Tools 1.Project Management 2.Requirements Engineering 3.Architectural Design 4.Implementation 5.Testing 6.Maintenance

Discussion Question  In a given situation, how do developers decide which method or tool is the most appropriate to use?

Discussion Question  How can developers tell if a change to the development process actually results in improvement?

Discussion Question  How can developers evaluate the effectiveness and efficiency of what they are already doing, so that they can tell if a change to the development process actually results in improvement?

Discussion Question  Once a change has been made, how can developers demonstrate that the products, processes and resources have the desired characteristics (such as improved quality)?

Chapter 12 Contents  Approaches to Evaluation  Selecting an Evaluation Technique  Assessment vs. Prediction  Evaluating Products  Evaluating Processes  Evaluating Resources

Approaches to Evaluation  Measure key aspects of product, processes, and resources and use this information to  Determine whether we have met goals for productivity, performance, quality, and other desire attributes

Four Categories of Evaluation Techniques  Feature analysis rate and rank attributes  Surveys document relationships and outcomes  Case studies compare with situation with another  Formal experiments rigorous, controlled investigation

Feature Analysis Example:  Buying a Project Scheduling Tool List five key attributes that the tool should have Identify three possible tools and rate each on the criterion Examine the scores, creating a total score based on the importance of each criterion Based on the score, select the tool with the highest score

Buying a Project Scheduling Tool FeaturesTool 1: MS Project Tool 2: Visio Tool 3: Rational Rose Importance Good user interface4543 Easy to learn5555 Consistency checking5313 Report Generation4442 Cost5455 Score857773

Surveys  Retrospective  Record data to determine how project participants reacted to a particular method, tool, or technique Compare data with other similar situations

Case Studies  Compares one situation with another Projects chosen for case study must be typical within the organization or company  Identify key factors that may affect an activity’s outcome and then document them

Case Studies  Involve sequence of steps: conception hypothesis setting design preparation execution analysis dissemination decision making

Case Study Types  Sister projects each is typical and has similar values for the independent variables  Baseline compare single project to organizational norm  Random selection partition single project into parts

Formal Experiments  Control variables  Uses methods to reduce bias and eliminate confounding factors  Often replicated

Evaluation Steps  Setting the hypothesis deciding what we wish to investigate, expressed as a hypothesis we want to test  Maintaining control over variables: identify variables that can affect the hypothesis, and decide how much control we have over the variables  Making investigation meaningful: the result of a formal experiment is more generalizable, while a case study or survey only applies to certain organization

Short answer question  How does feature analysis, surveys, case studies, and controlled experiments  differ?

Selecting An Evaluation Technique  Formal experiments: research in the small  Case studies: research in the typical  Surveys: research in the large

Key Selection Factors  Level of control over the variables  Degree to which the task can be isolated from the rest of the development process  Degree to which we can replicate the basic situation

Short answer question  Under which circumstances are feature analysis, surveys, case studies, and controlled experiments  appropriate?

Real-world example  Hypothesis: Extreme programming is more effective than the waterfall model for student capstone S.E. projects  Variables that can affect outcome?  Evaluation Technique?

Assessment vs. Prediction  Assessment system examines an existing entity by characterizing it numerically  Prediction system predicts characteristic of a future entity; involves a model with associated prediction procedures deterministic prediction (we always get the same output for an input) stochastic prediction (output varies probabilistically)