Measures of Effectiveness for Supportability Lesson 3 Measures of Effectiveness for Supportability

Topic 1: Introduction What’s in it for me? Measures of Effectiveness (MOE) for Supportability are objective measures of a system’s ability to meet user needs and achieve the right performance and sustainment outcomes over the Life Cycle. As the system matures, the objectives identified in the early phases of the Life Cycle, are transformed into outcomes through their meeting of Key Performance Parameter (KPP) and Key System Attributes (KSA), and other Technical Performance Measurement (TPM) parameters. MOEs serve as inputs to the Affordable System Operational Effectiveness (ASOE) Model to drive its trade-off processes. Within that context, MOEs serve the Program Management, Systems Engineering, Test & Evaluation and Life Cycle Logistics Communities as the gauges by which a system’s Operational Effectiveness, Suitability and Affordability are tracked and evaluated.

Life Cycle Management Framework Where Are You Life Cycle Management Framework Where Are You? What Influence Do You Have? Technology Maturation & Risk Reduction  Understand and Communicate User Sustainment Needs Design, Test, and Redesign using Sustainment Metrics Monitor Performance (Program Progress, Technical Performance Metrics, Process Metrics)

MOEs enable the Closed-loop Systems Engineering process. MOE Life Cycle Role Requirements to Design MOEs enable the Closed-loop Systems Engineering process. Sustainment to Disposal Design to Fielding Fielding to Sustainment

Affordable System Operational Effectiveness Model MOEs have the greatest influence in Design Effectiveness. MOEs

Technology Maturation & Risk Reduction (TMRR) Phase Adapted from SAE GEIA-HB-0007

Logistics Product Data Technology Maturation & Risk Reduction (TMRR) Phase Milestone B Activities Update Logistics Product Data Objectives Support System Eval & Trade-off Criterion Technology Trade-offs Trade-off Impacts ID Objectives Conduct Assess Analysis Output

MOE Lesson Approach This lesson approach emphasizes the LCL’s role in selecting MOEs and translating metrics into achievable design and Sustainment outcomes. Understand MOE Select MOEs and translate them into achievable outcomes Monitor Sustainment metrics and report deviations

TLO: Analyze Measures of Effectiveness to ensure a supportable design Topic Objectives Topic 1: Introduction Welcome Where Are You? What Influence Do You Have? Topic 2: Understand Measures of Effectiveness Distinguish the relationships between Measures of Effectiveness, Key Performance Parameters/Key System Attributes, Measures of Performance, Measures of Suitability, and Technical Performance Measures Analyze the JCIDS Sustainment Measures of Effectiveness and their maturity over the system life cycle Topic 3: Select Measures of Effectiveness for Supportability Analyze candidates for Supportability Technical Performance Measures Analyze achievable Sustainment candidates for promotion to capability and contractual design documents Topic 4: Translate Measures of Effectiveness into Achievable Outcomes Establish evaluation criteria for each Sustainment metric to validate design performance Topic 5: Validate a Supportable Design Monitor Sustainment metrics to assure the system meets Supportability design criteria Topic 6: Summary

Topic 2: Understand Measures of Effectiveness

Classify MOEs into Categories Align MOEs to Milestones MOE Set Up 1.1 1.2 Classify MOEs into Categories Align MOEs to Milestones Our planning considers two main areas: 1.1 Classify MOEs – Categorize and describe MOEs in terms of KPP/KSAs, performance, suitability, and use 1.2 Align MOEs – Describe MOEs from a milestone and artifact perspective

Set Up – Classify MOEs into Categories MOE MS A User Need MOE KPP MOEs: Are defined by users Define success in the operational environment Focus on capabilities

Set Up – Classify MOEs into Categories KPP MS A User Need MOE KPP KSA Sustainment KPP, KSA, and Metric KPP – Availability KSA – Operation & Support Cost KSA – Reliability Materiel Availability (AM) Supportability Process Efficiency # of Operational End Items/Population Operational Availability (AO) Uptime/(Uptime + Downtime) Downtime is (total time – uptime) Reliability (R) Reliability Mean Time between Failures Operation & Support Cost (O&S Cost) Maintenance Spares Fuel Support Mean Down Time (MDT) Maintainability Support Facilities Producibility Metric – Mean Down Time Metric KPPs: Belong to the program Are required for all ACAT I programs Note: ACAT II programs and below with materiel solutions, shall include the Sustainment KPP or sponsor defined Sustainment metrics. ∆ R ∆ AM ∆ AO ∆ $ ∆ MDT

Set Up – Classify MOEs into Categories MOP, MOS, and TPM MS A User Need MOE KPP KSA Metric MOP E.g.: Speed Payload TPM E.g.: MTBF MTTR MOS E.g.: Reliability Maintainability

Set Up – Classify MOEs into Categories TPM MTBF MTTR TPMs: Assess the design process Determine compliance to performance Planned Profile Mean Time between Failure (hours) Lower Limit Upper Limit Variance Achieved to Date MSA TD EMD PD

Set Up – Classify MOEs into Categories Inputs and Outputs User Needs/Requirements AoA ICD CDD Inputs TPM Candidates Process Analyze Sustainment TPM candidates and their attributes Establish evaluation criteria Refine Create/Refine Consult/Refine TPMs Availability KPP (AM and AO) Reliability KSA O&S Cost KSA MDT Weight Speed CDD CPD Outputs

Set Up – Classify MOEs into Categories Attributes for Good Measures/Metrics Good measures/metrics include the following attributes: Have value to team members or are an attribute essential to customer satisfaction Show a trend Are clearly defined Use data that is cost-effective to collect $

Set Up – Classify MOEs into Categories One way to classify MOEs is by using the following categories: CDD Product Development Program Management Process Management Product Support Arrangement

Set Up – Classify MOEs into Categories Product Development CDD Product Development Examples Operational Availability (AO) Weight budget MTBF Speed Range Objective Threshold 1 2 3 4 5 6 7 8 9 Component Number MTBF Trend All good

Set Up – Classify MOEs into Categories Program Management Program Management Examples Schedule performance index and variance Cost performance index and variance Manpower (planned vs. actual) Risk assessment tracking Earned Value Deliveries Variance Schedule $’s in Millions Cost $10.00 $0.00 ($10.00) ($20.00) ($30.00) Cumulative Dollar Variances

Set Up – Classify MOEs into Categories Process Management Process Management Examples Product Support Arrangement Number and cost of: Requirement changes Engineering change proposals Test failures Defect rates Cycle time Defects per 100 units 30 25 20 15 10 5 Time

Set Up – Align MOEs to Milestones The Life Cycle and Artifacts Technology Maturation & Risk Reduction  1 Understand and Communicate User Sustainment Needs ICD CDD CPD Update RAM-C Rationale Report Project Management Plan SEP TEMP 2 Design, Test, and Redesign using Sustainment Metrics LCSP 3 Monitor Performance (Program Progress, Technical Performance Metrics, Process Metrics) Affordable Effective Suitable

Set Up – Align MOEs to Milestones MOE Maturity and the Life Cycle Technology Maturation & Risk Reduction  User Need Traceability MOP Speed Payload MOS Reliability Maintainability KPP/KSA AO AM Reliability O&S Cost TPM MTBF MTTR Power Range Endurance Payload Refine TPMs T&E MOEs ICD CDD CPD MOEs mature with the life cycle and are traceable in artifacts.

Topic 3: Select Measures of Effectiveness for Supportability

MOE Analyze – TPM Candidates and Attributes Our analysis consists of three main areas: 2.1 Analyze Candidates – Determine best TPMs for a supportable design 2.2 Analyze Achievable TPM Attributes– Describe TPM relationship to program capability document maturity 2.3 Establish Criteria– Describe Sustainment TPM role in design performance 2.1 2.2 2.3 Analyze TPM Candidates Analyze Sustainment TPM Attributes Establish Evaluation Criteria

Analyze – TPM Candidates and Attributes ASOE Model and TPMs The ASOE Model has four distinct intersections representing program performance and Sustainment objectives. Balancing these priorities requires using TPMs in a series of trade-offs. Technical Performance Supportability Process Efficiency Life Cycle Cost Functions Reliability Production Total Ownership Cost Capabilities Maintainability Maintenance Support Features Logistics Operations

Analyze – TPM Candidates ASOE and Product Development 1 2 3 4 Capability Concept Description TPMs Be on station for ‘n’ hour duration Supportability Reliability/Maintainability MTBF and MTBM Respond to operational tasking within ‘n’ hours Process Efficiency Operational response time Time in hours to start mission 1 2 1 3

Analyze – TPM Candidates Product Development and Logistics Product Data Strike-Talon (Indenture Level A) LCN A LCN ACA COM/NAV System (Indenture Level B) LCN ACAABA Assembly ST UAV SATCOM Control (Indenture Level C)

Analyze – TPM Candidates ASOE and Product Development (cont.) 1 2 3 4 Capability Concept Description TPMs Weigh no more than ‘n’ pounds Supportability Support Features PDR Weight/CDR Weight Carry and deploy payload within weight and loiter parameters Support Features/Reliability IOT&E Weight/MTBF/Mission endurance time 1 2 1 2

Analyze – TPM Candidates ASOE and Program Management 1 2 3 4 Capability Concept Description TPMs Deliver Supportability analyses by xQFYyy Process Efficiency Delivery schedule Schedule variance Deploy ‘n’ prototypes within budget Process Efficiency/ Life Cycle Cost Cost Cost variance Earned Value 1 3 1 4

Analyze – TPM Candidates ASOE and Process Management 1 2 3 4 Capability Concept Description TPMs Deploy ‘n’ prototypes with ≤ ‘m’ requirement changes Process Efficiency/ Life Cycle Cost Quality/Cost # Requirement changes per phase/Cost per requirement change Deploy ‘n’ full rate production units with ≤ ‘m’ defects/unit Process Efficiency Production quality Defect rate 1 3 4 1 3

Analyze – Sustainment TPM Attributes Thresholds and Objectives KPPs/KSAs are expressed using threshold and objective values defined in the draft CDD: Threshold is the absolute minimum performance the customer will accept Objective is the performance that the customer really wants In this example, anything above the objective is questionable. Objective Planned Profile Threshold Mean Time between Failure (hours) Variance Achieved to Date In this example, the utility of anything below threshold is questionable. Planned Profile

Analyze – Sustainment TPM Attributes Thresholds and Objectives (cont.) Drive Reliability up to optimum level Drive Sustainment cycle time down to optimum level

Analyze – Sustainment TPM Attributes TPM Promotion to Program Documents Product TPM Candidates Table B. (U) STRIKE TALON UCAS Suitability Attributes Attribute Parameter Development Threshold Development Objective Operational Availability AO, AI, AM > 0.75 at IOT&E > 0.80 at IOC plus 2 years > 0.90 Reliability Mean Time Between Operational Mission Failure (MTBOMF) (Hours) ≥ 40 ≥ 90 Maintainability Mean Corrective Maintenance Time Operational Mission Failure (MCMTOMF) ≤ 3.0 ≤ 2.0 Diagnostics Mean Flight Hours Between False Alarms (MFHBFA) >300 hours >2000 hours Applicable product TPMs are promoted to program capability development and capability production documents (CDD & CPD).

Program Management TPM Candidates Analyze – Sustainment TPM Attributes TPM Promotion to Program Management Plan Program Management TPM Candidates Applicable Program Management TPMs are promoted to the Program Management Plan (PMP).

Analyze – Sustainment TPM Attributes TPM Promotion to Contracts Process TPM Candidates Development/Production Contracts Applicable Process TPMs are promoted to Program Contractual Documents. Threshold Objective

Topic 4: Translate Measures of Effectiveness into Achievable Outcomes

MOE Analyze – Establish Evaluation Criteria Our analysis consists of three main areas: 2.1 Analyze Candidates – Determine best TPMs for a supportable design 2.2 Analyze Achievable TPM Attributes – Describe TPM relationship to program capability document maturity 2.3 Establish Criteria – Describe Sustainment TPM role in design performance 2.1 2.2 2.3   Analyze TPM Candidates Analyze Sustainment TPM Attributes Establish Evaluation Criteria

Analyze – Establish Evaluation Criteria TPM Role in Design Performance Technical Performance Measures UAV Weight (kg) 53 52 51 50 49 48 47 46 45 Threshold Objective TPMs that are mission critical and cost sensitive are used as evaluation criteria to measure progress across major life cycle milestones and Supportability reviews.

Analyze – Sustainment TPMs TPM Maturity Technology Maturation & Risk Reduction  Understand and Communicate User Sustainment Needs ICD CDD CPD Update RAM-C Rationale Report Project Management Plan SEP TEMP Design, Test, and Redesign Using Sustainment Metrics LCSP TPMs grow in number and mature as the life cycle progresses.

Topic 5: Validate a Supportable Design

Monitor Sustainment Metrics MOE Report Findings Our analysis findings fall into two broad categories: 3.1 Monitor Sustainment Metrics – Use a surveillance process to collect and update Sustainment data 3.2 Report Sustainment Outcomes – Qualify Sustainment TPMs that validate a supportable design 3.1 3.2 Monitor Sustainment Metrics Report Deviations

Report Findings – Monitor Sustainment Metrics Surveillance Process Failure Reporting, Analysis, and Corrective Action System LCLs PSM SE

Report Findings – Report Deviations Communication Structure All IPS Elements involve MOEs/TPMs and are reported to all IPTs. Sustainment Metrics

Report Findings – Report Deviations Design Reviews Did the metric meet threshold? For example, at CDR (i.e., Post-Milestone B), the TPM/metric being analyzed should be between the threshold and objective values. 53 52 51 50 49 48 47 46 45 Technical Performance Measures UAV Weight (kg) Threshold Objective

Is a given MTBF TPM on the correct trajectory to meet the requisite Supportability reviews? 20 minutes Individual grade Independently access Blackboard and select the Exercise link in the Lesson 3: Measures of Effectiveness for Supportability folder to answer two questions.

Lesson Exercise Debrief Question 1 Graph Upper Limit Planned Profile Mean Time between Failure (hours) Achieved to Date Lower Limit

Lesson Exercise Debrief Question 2 Graph 44 43 42 41 40 39 38 37 36 35 34 Planned Profile Objective Threshold Mean Time between Failure (hrs. x 100) Achieved to Date

Topic 6: Summary

Takeaways MOEs for Supportability are: User defined Matured through the life cycle Used to measure product, program, and process performance Translated by the program into KPPs KPPs/KSAs are MOEs that serve as important criteria to assess whether the design meets user requirements TPM/metrics help users determine whether their needs are met. Programs use TPMs/metrics to evaluate design maturity and progress. An MOE is a mission-oriented qualitative or quantitative measure of operational success closely related to the mission objective or operation being evaluated.

TLO: Analyze Measures of Effectiveness to ensure a supportable design Topic Objectives Topic 1: Introduction Welcome Where Are You? What Influence Do You Have? Topic 2: Understand Measures of Effectiveness Distinguish the relationships between Measures of Effectiveness, Key Performance Parameters/Key System Attributes, Measures of Performance, Measures of Suitability, and Technical Performance Measures Analyze the JCIDS Sustainment Measures of Effectiveness and their maturity over the system life cycle Topic 3: Select Measures of Effectiveness for Supportability Analyze candidates for Supportability Technical Performance Measures Analyze achievable Sustainment candidates for promotion to capability and contractual design documents Topic 4: Translate Measures of Effectiveness into Achievable Outcomes Establish evaluation criteria for each Sustainment metric to validate design performance Topic 5: Validate a Supportable Design Monitor Sustainment metrics to assure the system meets Supportability design criteria Topic 6: Summary