1 Lecture 4.3a: Metrics Overview (SEF Ch 14) Dr. John MacCarthy UMBC CMSC 615 Fall, 2006

2 (Program) Metrics in Management (Program) Metrics are (quantitative) measurements collected for the purpose of determining project progress and overall condition Metrics are used to manage risk (technical, cost, and schedule) Metrics are collected and reported periodically (to management and the customer) Metrics are measured over time so that changes (trends) can be observed Technical Management requires three types of Program Metrics: Product Metrics: track the development of the product Earned Value: tracks conformance to the planned schedule and costs Management Process Metrics: track management activities DoD Policy requires the identification and tracking of TPMs

3 Product Metrics [1] Product Metrics: Demonstrate progress toward meeting customer requirements Three Types: Operational Performance Life-cycle Suitability Affordability Technical Performance Measurements (TPMs): A set of (5-20) key product performance metrics selected for tracking and periodic reporting to management. Derived from MOPs Generally allocated to CIs Metric Hierarchy: Measures of Effectiveness (MOEs): high-level measures of a system’s ability to achieve operational mission success (See CDD KPPs) Measures of Suitability (MOSs): high-level measures of the extent to which the system integrates will into the operational environment. They generally include supportability, human interface compatibility, and maintainability Measures of Performance (MOPs): high-level physical or functional attributes of the system that relate (trace) to the MOEs and MOSs.

4 Product Metrics [2] Suitability Metrics: Degree to which a system can be placed satisfactorily in field use: Availability, Reliability, Maintainability (RAM) Usage Rates Supportability & Logistics Human Factors, Manpower, Training, & Safety Interoperability, compatibility, transportability Environmental Impact Robustness, Survivability Producibility, Testability, Simplicity … TPMs often include suitability metrics MTBF MTTR Affordability Metrics: LCC drivers: (Development Costs) Production Costs Operational Costs Maintenance Costs Disposal Costs

5 Earned Value (14.2) Earned Value is the standard metric for tracking cost and schedule progress and risk See Separate Briefing on Earned Value

6 Process Metrics (14.3) Management Process Metrics: measurements taken to track process of developing, building and introducing the systems: Examples include: Availability of resources (personnel) Number of new hires Items Completed Customer Satisfaction Trouble Tickets Lines of Code (developed) Complexity ECPs baselined Time to approve documents Number of High Risks Etc. These are important for CMMI

7 Summary Points (14.4) Technical Management requires three types of metrics Product Metrics Earned Value Management Process Metrics Metrics should be reported on a periodic basis The measurement, evaluation, control, and reporting of metrics should be planned to assure proper measurement, evaluation and dissemination TPMs are performance-based product metrics (key technical parameters) that are used to track progress toward meeting the program’s operational requirements Use of TPMs is required by DoDI

8 Technical Performance Measurement (S14-A) [1] See separate TPM briefing