1 Experience-Based Approaches to Process Improvement © Otto Vinter / Quality Week 2000 Experience-Based Approaches to Process Improvement - Otto Vinter Project Manager Software Process Improvements, DELTA Software Engineering, Tel: , Fax: & Software Engineering Mentor Tel/Fax: , Mobile:

2 Experience-Based Approaches to Process Improvement © Otto Vinter / Normative Models for SPI CMM BOOTSTRAP SPICE (ISO15504) Defined comprehensive process Levels of maturity (capability) Key process areas (KPAs) for each level Assessed by certified assessors

3 Experience-Based Approaches to Process Improvement © Otto Vinter / Fundamentals in Normative Models Software Process Assessment & Improvement (The BOOTSTRAP Approach) ISBN

4 Experience-Based Approaches to Process Improvement © Otto Vinter / Problems with Normative Models Abstract model Assessment by external body Costs money Organisational focus Points out KPAs to be improved Little help on precisely what to do Raises a lot of expectations

5 Experience-Based Approaches to Process Improvement © Otto Vinter / Alternative Approaches to SPI Experience-based improvement actions Analyse problems from previous projects to extract knowledge on frequently occurring problems Change the development process through the use of an optimum set of methods and tools available to prevent these problems from reappearing Measure the impact of the changes in a real-life development project Diffuse the results to the rest of the organisation

6 Experience-Based Approaches to Process Improvement © Otto Vinter / Characteristics of Alternative Approaches Experience-based No specific model Hot-spot driven Focus on prevention One issue at a time (incremental) Piloting at project level Evolve rather than define (feed-back) Fits CMM level 1-2 cultures (where most of us are)

7 Experience-Based Approaches to Process Improvement © Otto Vinter / Examples of Alternative Approaches Analysis of defects progress reports, etc. Structured/Selective interviews project managers project members customers, etc. Goal-Question-Metric paradigm (GQM) Some frameworks for alternative approaches: Experience Factory (V. Basili e.a.) Product Process Dependency Models (PROFES: but primarily: you must find your own way

8 Experience-Based Approaches to Process Improvement © Otto Vinter / Brüel & Kjær - Sound & Vibration Measurement TransducerSignal conditioning Digital conversion Signal processing Post processing Output Brüel & Kjær Conditioning Amplifier ISO 9001 CERTIFIED

9 Experience-Based Approaches to Process Improvement © Otto Vinter / Defect Analysis from Error Logs Definitions Bugs are anything between serious defects and suggestions for improvements Problem reporting starts in the integration phase Error Logs Analysed Embedded and PC Windows development projects Project sizes app. 5-7 person years In total app bugs analyzed in detail Problem reports covered a period until 18 months after first release

10 Experience-Based Approaches to Process Improvement © Otto Vinter / Problem Report Distribution over Time

11 Experience-Based Approaches to Process Improvement © Otto Vinter / Defect Analysis Technique Interview sessions 1-2 developers and 1-2 process consultants app. 5 minutes / bug Bug Categorisation based on a bug taxonomy by Boris Beizer: -Boris Beizer: Software Testing Techniques, Van Nostrand Reinhold, 1990 comprehensive set of bug categories and statistics Capture Subjective Information on the Bugs where and how the bug occurred quality factor (reliability, usability, functionality...) complexity of bug (correction cost) what could prevent the bug

12 Experience-Based Approaches to Process Improvement © Otto Vinter / The Beizer Bug Taxonomy 1. Requirements and Features 2. Functionality as Implemented 3. Structural Bugs 4. Data 5. Implementation (standards violation, and documentation) 6. Integration 7. System and Software Architecture 8. Test Definition or Execution Bugs 9. Other Bugs, Unspecified Each category detailed to a depth of up to 4 levels

13 Experience-Based Approaches to Process Improvement © Otto Vinter / Problem Report Categorization Category

14 Experience-Based Approaches to Process Improvement © Otto Vinter / 1st Action: The Prevention of Errors through Experience-driven Test Efforts (PET) Results of the analysis of error logs no special bug class dominates embedded software development requirements problems, and requirements related problems, are the prime bug cause (>36%) problems due to lack of systematic unit testing is the second largest bug cause (22%) Action: Improve Testing Processes static analysis -source code complexity, data flow anomalies, coding standards dynamic analysis -code coverage by test cases, cross references code to test cases Funded by CEC. Final Report:

15 Experience-Based Approaches to Process Improvement © Otto Vinter / Static Flowgraph (McCabe = 10)

16 Experience-Based Approaches to Process Improvement © Otto Vinter / Static Flowgraph (McCabe = 20)

17 Experience-Based Approaches to Process Improvement © Otto Vinter / Test Coverage Strategy Test coverage must take into consideration perceived complexity -based on a visual inspection criticality -product and project issues developer experience -domain, product, and unit bug and change request history degree and type of reuse Test cases for dynamic analysis are selected by normal testing techniques equivalence partitioning, boundary value analysis etc.

18 Experience-Based Approaches to Process Improvement © Otto Vinter / Results of the Improved Testing Process 46% Improvement in Testing Efficiency Removing static bugs Increasing unit branch coverage to > 85% 75% Reduction in Production-Release Bugs Compared to Trial-Release 70% Requirements Bugs in Production-Release Next Action: Improve Requirements Engineering

19 Experience-Based Approaches to Process Improvement © Otto Vinter / 2nd Action: A Methodology for Preventing Requirements Issues from Becoming Defects (PRIDE) Results of the analysis of error logs Requirements related bugs 51% Usability issues dominate 64% External software (3rd party & MS products) 28% Action: Introduce requirements techniques Use Situations (Scenarios) -Relate demands to use situations. Describe tasks for each scenario. Usability Test, Daily Tasks, Navigational Prototype -Check that the users are able to use the system for daily tasks, based on a navigational prototype of the user interface. Funded by CEC. Final Report:

20 Experience-Based Approaches to Process Improvement © Otto Vinter / Scenarios “The defining property of a scenario is that it projects a concrete description of activity that the user engages in when performing a specific task, a description sufficiently detailed so that design implications can be inferred and reasoned about” John M. Carroll, Scenario-Based Design: Envisioning Work and Technology in System Development, Wiley Many names for the same concept: scenarios use situations work situations work setting

21 Experience-Based Approaches to Process Improvement © Otto Vinter / Road Test Scenario Road tests are done in the car when it is driving on special test roads. The purpose of the recordings is to identify noise sources, comparing them to earlier measurements, and eventually removing the noise through changes to the car design. The engineer will record noises from various parts of the car when it is driving at various speeds, when it is turning, when it is breaking, etc. The microphone will have to be mounted at various places not accessible from the drivers seat. Usually, the engineer has a plan for what to measure, but circumstances may change so that he has to do something different and later find out what he actually did and which sounds relate to what. Back at the lab the sounds will be analyzed by the engineer himself, or - in many cases - someone else.

22 Experience-Based Approaches to Process Improvement © Otto Vinter / Usability Test Environment PC V Experiment Leader User Log Keeper Tape Recorder Microphone

23 Experience-Based Approaches to Process Improvement © Otto Vinter / Results of the 2nd Improvement Action Product is selling steadily more than twice as many copies compared to a similar product developed for a similar domain by the same team Usability is exceptional in the market Users’ interaction with the product was totally changed as a result of the early usability tests Almost 3 times increase in productivity for the development of the user interface 27% reduction in problem reports

24 Experience-Based Approaches to Process Improvement © Otto Vinter / Improvement Approach Based on Project Manager Perceived Problems Interview project managers 1 project manager and 3 process consultants app. 1 hour per interview process-oriented interview guide Analyze problems raised classify the problems according to software process present the issues at a project manager workshop establish consensus on a few major issues let each manager select an issue to improve Monitor development projects introduce and train the team on the selected issue coach, and support regularly and on request collect experience and results

25 Experience-Based Approaches to Process Improvement © Otto Vinter / Improvement Approach Based on Project Manager Perceived Problems Major Issues iterative software development model requirements project monitoring (estimation, progress evaluation, etc.) project conclusion (test, configuration management, etc.) Improvement Actions: diffuse and adopt the test and requirements techniques define new iterative development models improve project monitoring Results: improvement actions were performed with enthusiasm project related actions completed successfully

26 Experience-Based Approaches to Process Improvement © Otto Vinter / 3rd Action: Iterative Software Development Model Demo Prototype Functionally Complete Model Specification Spec. Test Design Test Code Design Test Spec. Customer Req. Spec. Validation Release Vers. 1.0 Final Customer Validation Intermediate Customer Validation Pre-Project StabilizationPartial Implementation Full Implementation Specification and user interface mock-up Released Product Customer Views: Product Views: Cycles: Phases: SpecificationDevelopmentOperation Development Views:

27 Experience-Based Approaches to Process Improvement © Otto Vinter / Iterative Software Development Model Types of Iteration user validation (usability test) exploration (e.g. performance) proof-of-concept (e.g. architecture) well-defined functionality (increment) Duration time-boxed -less than 6 weeks, with 2 weeks spacing full increments -app. 3 months, with 2 weeks of stabilization Essential for Success user validation stabilization

28 Experience-Based Approaches to Process Improvement © Otto Vinter / Results of the 3rd Improvement Action drive and motivation of team increased dramatically turbulence within and around team decreased control over project progress increased requirements creep was kept under control resources used more efficiently serious problems were uncovered early quality improved step-by-step

29 Experience-Based Approaches to Process Improvement © Otto Vinter / Comparison of Recommendations RecommendationDefect 1 st Bootstrap Project Mgr. 2 nd Bootstrap Analysis Assessment Interviews Assessment Development Model - iterations x x x  - risk management x Requirements x x x  Project Monitoring - estimation x x x - time & resource usage x x - monitor progress x x x Project Conclusion - configuration mgmt. x x x - testing x x x (  ) - release criteria x x Reuse x Process Descriptions x 

30 Experience-Based Approaches to Process Improvement © Otto Vinter / Problem Diagnosis Results Defect analysis from error logs has established a basic process for testing and release has improved our requirements process and products Improvements from interviews successful diffusion and adoption of previous improvement actions established a new software development model Impacts on maturity development model, test, and requirement issues are no longer on the Bootstrap recommendation list

31 Experience-Based Approaches to Process Improvement © Otto Vinter / In Conclusion Problem diagnosis approach is a simple and effective way to find problems in the software development process good starting point for process improvement programmes in companies step-wise improvements with quick wins changes assessment recommendations However, normative models are needed comprehensive framework -KPAs are important, levels are not established assessments -effect of improvements