1. Chapter 12 Evaluating Products, Processes, and Resources

2. Review  Software engineering provides a large variety of methods and tools for use by software developers throughout the software life-cycle.  What are some of them?

3. Review of S.E. Methods and Tools 1.Project Management 2.Requirements Engineering 3.Architectural Design 4.Implementation 5.Testing 6.Maintenance

4. Discussion Question  In a given situation, how do developers decide which method or tool is the most appropriate to use?

5. Discussion Question  How can developers tell if a change to the development process actually results in improvement?

6. Discussion Question  How can developers evaluate the effectiveness and efficiency of what they are already doing, so that they can tell if a change to the development process actually results in improvement?

7. Discussion Question  Once a change has been made, how can developers demonstrate that the products, processes and resources have the desired characteristics (such as improved quality)?

8. Chapter 12 Contents  Approaches to Evaluation  Selecting an Evaluation Technique  Assessment vs. Prediction  Evaluating Products  Evaluating Processes  Evaluating Resources

9. Approaches to Evaluation  Measure key aspects of product, processes, and resources and use this information to  Determine whether we have met goals for productivity, performance, quality, and other desire attributes

10. Four Categories of Evaluation Techniques  Feature analysis rate and rank attributes  Surveys document relationships and outcomes  Case studies compare with situation with another  Formal experiments rigorous, controlled investigation

11. Feature Analysis Example:  Buying a Project Scheduling Tool List five key attributes that the tool should have Identify three possible tools and rate each on the criterion Examine the scores, creating a total score based on the importance of each criterion Based on the score, select the tool with the highest score

12. Buying a Project Scheduling Tool FeaturesTool 1: MS Project Tool 2: Visio Tool 3: Rational Rose Importance Good user interface4543 Easy to learn5555 Consistency checking5313 Report Generation4442 Cost5455 Score857773

13. Surveys  Retrospective  Record data to determine how project participants reacted to a particular method, tool, or technique Compare data with other similar situations

14. Case Studies  Compares one situation with another Projects chosen for case study must be typical within the organization or company  Identify key factors that may affect an activity’s outcome and then document them

15. Case Studies  Involve sequence of steps: conception hypothesis setting design preparation execution analysis dissemination decision making

16. Case Study Types  Sister projects each is typical and has similar values for the independent variables  Baseline compare single project to organizational norm  Random selection partition single project into parts

17. Formal Experiments  Control variables  Uses methods to reduce bias and eliminate confounding factors  Often replicated

18. Evaluation Steps  Setting the hypothesis deciding what we wish to investigate, expressed as a hypothesis we want to test  Maintaining control over variables: identify variables that can affect the hypothesis, and decide how much control we have over the variables  Making investigation meaningful: the result of a formal experiment is more generalizable, while a case study or survey only applies to certain organization

19. Short answer question  How does feature analysis, surveys, case studies, and controlled experiments  differ?

20. Selecting An Evaluation Technique  Formal experiments: research in the small  Case studies: research in the typical  Surveys: research in the large

21. Key Selection Factors  Level of control over the variables  Degree to which the task can be isolated from the rest of the development process  Degree to which we can replicate the basic situation

22. Short answer question  Under which circumstances are feature analysis, surveys, case studies, and controlled experiments  appropriate?

23. Real-world example  Hypothesis: Extreme programming is more effective than the waterfall model for student capstone S.E. projects  Variables that can affect outcome?  Evaluation Technique?

24. Assessment vs. Prediction  Assessment system examines an existing entity by characterizing it numerically  Prediction system predicts characteristic of a future entity; involves a model with associated prediction procedures deterministic prediction (we always get the same output for an input) stochastic prediction (output varies probabilistically)