1. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 1 Standards & Assessments CMMI, ISO 9000, TL9000 Sources: ASQ CSQE Primer Introduction to CMMI CMMI Distilled

2. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 2 August 4 Class CMMI Introduction & Configuration Management Appraisal ISO 9000/TL-9000 Due today (31-July): Cycle 2 Design & Code, hand off to System Tester System Test Plan Inspected & Baselined Project notebook updates including inspection records, meeting minutes, etc.

3. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 3 Topics Audits & Assessments CMM / CMMI & SCAMPI ISO 9000: ISO 9001:2000, ISO 9000-3:1997, TickIT Q9000, TL9000

4. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 4 Capability Maturity Model (CMM) Created in 1987 by Software Engineering Institute (SEI) 5 level model based on proficiency in Key Process Areas (KPAs) Migrating to Capability Maturity Model Integration (CMMI) Three source models: –CMM for Software –Systems Engineering Capability model –Integrated Product Development CMM CMMI v1.1

5. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 5 What is it?

6. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 6 Why Would I want one? Required –Contractual –Senior Management Decree (e.g. ROI of 7 to 1) Sales Tool Want to improve

7. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 7 Schedule Example

8. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 8 Process Capability Ability of a process to produce planned results Predictable Measureable

9. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 9 Process Models CMMI is model based Model = structured collection of elements that describes characteristics of effective processes Process Area = cluster of related practices that when performed collectively, satisfy a set of goals considered important for making significant improvement in that area Processes selected are those proven by experience to be effective (i.e. best practices, practical knowledge from previous endeavors) Notes: A process area is not a process A model is not a process models show what to do, not how to do it! Philosophy “All models are wrong, some are useful” – George Box

10. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 10 CMMI Models Model Options: Software Engineering (SW) Systems Engineering + Software Engineering (SE/SW) Systems Engineering + Software Engineering + Integrated Process & Product Development (SE/SW/IPPD) … + Supplier Sourcing (SE/SW/IPPD/SS) Representation Options: Staged (Maturity Levels) Migration from CMM to CMMI Continuous (Capability Levels) Migration from EIA/IS-731 to CMMI Recommended order for process improvements, but not prescribed …

11. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 11 Levels Zero – Ad Hoc One – Doing it (in Continuous, Ad Hoc in Staged) Two – Process performed for individual projects Three – Process focus at organizational level Four – Projects and processes are quantitatively managed Five – Projects and processes being optimized based on performance data & results

12. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 12 Representations Revisited Continuous Model –25 Process Areas each assessed at level 0-5 Configuration Mgmt = capability level 3 Risk Mgmt = not done (capability level 0) Requirements Mgmt = capability level 2 –Result can be presented as a Kiviat chart Staged Model –25 Process Areas assigned to each of 4 Maturity Levels (see next slide) –Result is a grade (1-5)

13. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 13 Staged Representation Maturity Levels (MLx) 1 Initial Process unpredictable, poorly controlled & reactive 2 Managed Process characterized for project & often reactive 3 Defined Process characterized by organization is proactive 4 Quantitatively Managed Process measured & controlled 5 Optimizing Focus on process improvement Project Organization Project Organization

14. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 14 CMMI Levels Level 5 Optimizing Statistical evidence of process effectiveness Continuous process improvement focus Rigorous defect causal analysis & defect prevention Level 4 Managed Productivity & quality are measured Process operates within acceptable limits Predictable process & product quality Level 3 Defined Documented managerial / engineering process Organizational software process focus Project activities under control, quality tracked Level 2 Repeatable Disciplined management control process Project management policies & procedures Project activities under control, quality tracked Level 1 Initial Absence of sound management practices Software process is ad hoc, unstable Software development results are unpredictable

15. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 15 Staged Representation Process Area Mapping to Maturity Levels 5. Optimizing Continuous Process Improvement Organizational Innovation & Deployment Causal Analysis & Resolution 4. Quantitatively Managed Quantitative Management Organizational Process Performance Quantitative Project Management 3. Defined Process Standardization Requirements Development Technical Solution Product Integration Verification Validation Organizational Process Focus Organizational Process Definition Organizational Training Risk Management Decision Analysis & Resolution 2. Managed Basic Project Management Requirements Management Project Planning Project Monitoring & Control Supplier Agreement Management Measurement & Analysis Process & Product Quality Assurance Configuration Management 1. Initial None

16. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 16 Continuous Representation Process Areas Process Management –Organizational Process Focus (OPF-3) –Organizational Process Definition (OPD-3) –Organizational Training (OT-3) –Organizational Process Performance (OPP-4) –Organizational Innovation & Deployment (OID-5) Project Management –Project Planning (PP-2) –Project Monitoring & Control (PMC-2) –Supplier Agreement Management (SAM-2) –Integrated Project Management (IPM-3) –Risk Management (RSKM-3) –Integrated Teaming (IT-3) –Integrated Supplier Management (ISM-3) –Quantitative Project Management (QPM-4) Engineering –Requirements Management (REQM-2) –Requirements Development (RD-3) –Technical Solution (TS-3) –Product Integration (PI-3) –Verification (VER-3) –Validation (VAL-3) Support –Configuration Management (CM-2) –Process & Product Quality Assurance (PPQA – 2) –Measurement and Analysis (MA-2) –Decision Analysis and Resolution (DAR-3) –Organizational Environment for Integration (OEI-3) –Causal Analysis and Resolution (CAR-5)

17. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 17 CMMI Assessment Cheat Sheet Institutionalization – Ingrained Way of Doing Business that an organization follows routinely as part of its corporate culture Specific Goals – Required model component that describes the unique characteristics that must be present to satisfy the process area Specific Practice – Expected model component that is considered important to achieving the associated specific goal. The specific practices describe the activities expected to result in achievement of the specific goals of a process area. (In continuous representation – every specific practice (SP) is associated with a CL, in staged – all SPs are treated equally) Generic Goal – Required model component that describes the characteristics that must be present to satisfy the institutionalization of the processes that implement a process area Generic Practice – Expected model component that is considered important in achieving the associated generic goal. The generic practices describe the activities that are expected to result in achievement of the generic goal and contribute to the institutionalization of the processes associated with a process area.

18. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 18 CMMI Assessment Cheat Sheet (continued) Managed Process: –Performed process planned & executed in accordance with policy –Employs skilled people –Adequate resources –Produces controlled outputs –Involves relevant stake holders –Monitored, controlled & reviewed –Evaluated for adherence to process description Defined Process: –Managed process tailored from the organizational standard processes –Maintained process description –Contributes work products, measures & other process info to organizational process assets Performed Process –Accomplishes needed work to produce work products –Specific goals of the process area are satisfied Establish & Maintain –Includes documentation & usage: Planned Documented & Used

19. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 19 Configuration Management (CM) Assessment

20. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 20 DeMarco & Lister on Process Organizations driving to be SEI Level 5 (at least level N+1) Standards are good, but … Most success centered around standard interfaces Mandating a “best practice” is a bad practice Process improvement is good, but process improvement programs aren’t Competent people improve processes all the time (pride, growth, etc.) Formal process improvement moves responsibility from the individual to the organization Process improvement programs focus on process rather than product (making a poor product efficiently is often worse than making a good product poorly) Focus on process “level” tends to make organizations risk averse “The projects most worth doing are the ones that will move you down one full level on your process scale!”

21. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 21 Break

22. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 22 Quality Standard Rationale Customers want & need assessments of supplier quality Means: Individually audit (i.e. qualify) vendor: Specific products Processes (e.g. manufacturing, design & development, support) Alternative: Common Quality Assurance standards & audits

23. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 23 Major Audit Types First Party Audit Within own company (aka internal audit) Used to measure own performance, strengths & weaknesses against internally established procedures & systems Second Party Audit Performed by customer on their supplier (aka external audit) Third Party Audit Outside, independent auditor contracted to audit on behalf of company or a supplier (e.g. ISO 9000 registration audit) Assessments (e.g. SCAMPI) Similar to first party audit, but typically performed by external assessors

24. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 24 Other Audit Types System Audit – examination of bigger picture of organization &/or project Typical cross organizational, cross process & cross product Process Audit – verify inputs, actions & outputs in accordance with defined requirements (e.g. software inspections) Product Audit – final product or service for “fitness for use” Customer oriented Compliance Audit Regulatory – audit to government regulations Management – audit to organizational rules, effectiveness & conformance Quality – systematic & independent of quality activities vs. established procedures

25. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 25 ISO 9001:2000 Objective Provide confidence that vendor can produce quality products Assumptions: good practices will produce good products Standard for assessing organization’s Quality Management System (QMS) –Processes –Activities –Behaviors –Training But, ISO focuses on Quality Assurance not Quality Control ISO-9001 certification does not guarantee quality products!

26. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 26 Tenants of ISO 9001 1)Say what you do 2)Do what you say 3)Prove it!

27. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 27 ISO 9000 Audits Customers write requirements for current ISO-9001 certification into purchasing contracts Organizations apply for 3 rd party audit, end result is ISO-9001 certification ISO International Accreditation Forum (IAF) board Audits national accreditation boards (i.e. one board each nation) Who register individual registrars (e.g. Lloyd’s, DNV) Who audit organization internal auditors (e.g. Lucent Optical Networking) & spot check Who audit design, development, manufacturing & support teams within the organization

28. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 28 ISO Alphabet Soup ISO 9000:2000 Overall framework, fundamentals of quality management systems & terminology ISO 9001:2000 Requirements for quality management systems (qms) & what is required to demonstrate compliance ISO 90003 2004 (previously 9000-3) Guidelines for the application of ISO 9001:2000 to computer software ISO 19011 Guidelines for auditing quality and environmental management systems

29. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 29 What is wrong with ISO 9001? Vendors ISO-9001 certified, but quality still elusive! No visibility into supplier quality levels Not getting quality levels they wanted Solution: TL9000 (Quest forum, telecommunications) QS9000 (automotive)

30. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 30 TL9000 ISO on steroids Wholly subsumes ISO 9001-2000 Requires vendors prove they are actually improving Metrics focused on cost drivers of service providers: Know vendor is measuring Visibility into quality improvement results

31. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 31 TL9000 Top Management Requirements Monitor & improve customer satisfaction Set long & short term objectives for organization effectiveness Set targets for TL9000 product performance metrics Use an explicit life-cycle model Establish a quality improvement program Periodic management review of quality system

32. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 32 TL9000 Metrics Cross-discipline metrics –# of problem reports –Problem report fix response time –Overdue problem report fix responsiveness –On-time delivery Hardware & Software measurements –System Outages Hardware measurements –Return rates Software measurements –Software installation & release application aborts –Corrective patch quality –Feature patch quality –Software update quality

33. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 33 TL9000 Common Audit Questions Do you know how to find your Quality Policy, QMS and the processes you should be using for your work? Do you know your organization’s product delivery & improvement goals and what you must do to support them? Do you know what skills you should have? Do you know what you have to do to approve/baseline/finalize your documents, designs & code? Do you know how to store & find records of reviews, inspections, key decisions, etc.? Do you know what to do if a problem is found with the product or process? Do you know your organization’s performance with respect to customer satisfaction, quality of delivered products & process execution?

34. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 34 TL9000 Sample Requirements Planning –Must have methods for estimating & tracking –Determine where you will do reviews & tests –Risk management plans, customer, user & supplier involvement in reviews & evaluation Software Outputs –Requires architecture, detailed designs, code & user documentation –Each design thread must be reviewed at some point prior to integration or system test Software Testing –All testing must have test plans; test process must be documented –Plans must include test cases with inputs, output & test success criteria –Plans must include types of testing, requirements traceability, coverage definition & measurement, test environment, defect handling, et.al. –Integration testing specifically required

35. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 35 Team Project Postmortem Tracking process improvements during project Process Improvement Proposals (PIP) Port-Mortem Areas to consider Better personal practices Improved tools Process changes

36. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 36 Postmortem process Team discussion of project data Review & critique of roles

37. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 37 Postmortem process Review Process Data Review of cycle data including SUMP & SUMQ forms Examine data on team & team member activities & accomplishments Identify where process worked & where it didn’t Quality Review Analysis of team’s defect data Actual performance vs. plan Lessons learned Opportunities for improvement Problems to be corrected in future PIP forms for all improvement suggestions Role Evaluations What worked? Problems? Improvement areas? Improvement goals for next cycle / project?

38. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 38 Cycle Report Table of contents Summary Role Reports Leadership – leadership perspective Motivational & commitment issues, meeting facilitation, req’d instructor support Development Effectiveness of development strategy, design & implementation issues Planning Team’s performance vs. plan, improvements to planning process Quality / Process Process discipline, adherence, documentation, PIPs & analysis, inspections Cross-team system testing planning & execution Support Facilities, CM & Change Control, change activity data & change handling, ITL Engineer Reports – individual assessments

39. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 39 Role Evaluations & Peer Forms Consider & fill out PEER forms Ratings (1-5) on work, team & project performance, roles & team members Additional role evaluations suggestions Constructive feedback Discuss behaviors or product, not person

40. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 40 Project Notebook Updated Requirements & Design documents Conceptual Design, SRS, SDS, System Test Plan, User Documentation* Updated Process descriptions Baseline processes, continuous process improvement, CM Tracking forms ITL, LOGD, Inspection forms, LOGTEST Planning & actual performance Team Task, Schedule, SUMP, SUMQ, SUMS, SUMTASK, CCR*

41. 7/31/2007SE 652 - 2007_7_31_CMMI_Software_Quality.ppt 41 August 4 Class CMMI Introduction & Configuration Management Appraisal ISO 9000/TL-9000 Due July 31: Cycle 2 Design & Code, hand off to System Tester System Test Plan Inspected & Baselined Project notebook updates including inspection records, meeting minutes, etc. Deliverables for August 7 Project Postmortem (cycle report) Cycle 2 presentations Peer Feedback forms Completed project notebooks Cycle Exit Completed project (source, documents & all quality records)