Download presentation
Presentation is loading. Please wait.
1
Measures of Effectiveness for Supportability
Lesson 3 Measures of Effectiveness for Supportability
2
Topic 1: Introduction What’s in it for me?
Measures of Effectiveness (MOE) for Supportability are objective measures of a system’s ability to meet user needs and achieve the right performance and sustainment outcomes over the Life Cycle. As the system matures, the objectives identified in the early phases of the Life Cycle, are transformed into outcomes through their meeting of Key Performance Parameter (KPP) and Key System Attributes (KSA), and other Technical Performance Measurement (TPM) parameters. MOEs serve as inputs to the Affordable System Operational Effectiveness (ASOE) Model to drive its trade-off processes. Within that context, MOEs serve the Program Management, Systems Engineering, Test & Evaluation and Life Cycle Logistics Communities as the gauges by which a system’s Operational Effectiveness, Suitability and Affordability are tracked and evaluated.
3
Life Cycle Management Framework Where Are You
Life Cycle Management Framework Where Are You? What Influence Do You Have? Technology Maturation & Risk Reduction Understand and Communicate User Sustainment Needs Design, Test, and Redesign using Sustainment Metrics Monitor Performance (Program Progress, Technical Performance Metrics, Process Metrics)
4
MOEs enable the Closed-loop Systems Engineering process.
MOE Life Cycle Role Requirements to Design MOEs enable the Closed-loop Systems Engineering process. Sustainment to Disposal Design to Fielding Fielding to Sustainment
5
Affordable System Operational Effectiveness Model
MOEs have the greatest influence in Design Effectiveness. MOEs
6
Technology Maturation & Risk Reduction (TMRR) Phase
Adapted from SAE GEIA-HB-0007
7
Logistics Product Data
Technology Maturation & Risk Reduction (TMRR) Phase Milestone B Activities Update Logistics Product Data Objectives Support System Eval & Trade-off Criterion Technology Trade-offs Trade-off Impacts ID Objectives Conduct Assess Analysis Output
8
MOE Lesson Approach This lesson approach emphasizes the LCL’s role in selecting MOEs and translating metrics into achievable design and Sustainment outcomes. Understand MOE Select MOEs and translate them into achievable outcomes Monitor Sustainment metrics and report deviations
9
TLO: Analyze Measures of Effectiveness to ensure a supportable design
Topic Objectives Topic 1: Introduction Welcome Where Are You? What Influence Do You Have? Topic 2: Understand Measures of Effectiveness Distinguish the relationships between Measures of Effectiveness, Key Performance Parameters/Key System Attributes, Measures of Performance, Measures of Suitability, and Technical Performance Measures Analyze the JCIDS Sustainment Measures of Effectiveness and their maturity over the system life cycle Topic 3: Select Measures of Effectiveness for Supportability Analyze candidates for Supportability Technical Performance Measures Analyze achievable Sustainment candidates for promotion to capability and contractual design documents Topic 4: Translate Measures of Effectiveness into Achievable Outcomes Establish evaluation criteria for each Sustainment metric to validate design performance Topic 5: Validate a Supportable Design Monitor Sustainment metrics to assure the system meets Supportability design criteria Topic 6: Summary
10
Topic 2: Understand Measures of Effectiveness
11
Classify MOEs into Categories Align MOEs to Milestones
MOE Set Up 1.1 1.2 Classify MOEs into Categories Align MOEs to Milestones Our planning considers two main areas: 1.1 Classify MOEs – Categorize and describe MOEs in terms of KPP/KSAs, performance, suitability, and use 1.2 Align MOEs – Describe MOEs from a milestone and artifact perspective
12
Set Up – Classify MOEs into Categories MOE
MS A User Need MOE KPP MOEs: Are defined by users Define success in the operational environment Focus on capabilities
13
Set Up – Classify MOEs into Categories KPP
MS A User Need MOE KPP KSA Sustainment KPP, KSA, and Metric KPP – Availability KSA – Operation & Support Cost KSA – Reliability Materiel Availability (AM) Supportability Process Efficiency # of Operational End Items/Population Operational Availability (AO) Uptime/(Uptime + Downtime) Downtime is (total time – uptime) Reliability (R) Reliability Mean Time between Failures Operation & Support Cost (O&S Cost) Maintenance Spares Fuel Support Mean Down Time (MDT) Maintainability Support Facilities Producibility Metric – Mean Down Time Metric KPPs: Belong to the program Are required for all ACAT I programs Note: ACAT II programs and below with materiel solutions, shall include the Sustainment KPP or sponsor defined Sustainment metrics. ∆ R ∆ AM ∆ AO ∆ $ ∆ MDT
14
Set Up – Classify MOEs into Categories MOP, MOS, and TPM
MS A User Need MOE KPP KSA Metric MOP E.g.: Speed Payload TPM E.g.: MTBF MTTR MOS E.g.: Reliability Maintainability
15
Set Up – Classify MOEs into Categories TPM
MTBF MTTR TPMs: Assess the design process Determine compliance to performance Planned Profile Mean Time between Failure (hours) Lower Limit Upper Limit Variance Achieved to Date MSA TD EMD PD
16
Set Up – Classify MOEs into Categories Inputs and Outputs
User Needs/Requirements AoA ICD CDD Inputs TPM Candidates Process Analyze Sustainment TPM candidates and their attributes Establish evaluation criteria Refine Create/Refine Consult/Refine TPMs Availability KPP (AM and AO) Reliability KSA O&S Cost KSA MDT Weight Speed CDD CPD Outputs
17
Set Up – Classify MOEs into Categories Attributes for Good Measures/Metrics
Good measures/metrics include the following attributes: Have value to team members or are an attribute essential to customer satisfaction Show a trend Are clearly defined Use data that is cost-effective to collect $
18
Set Up – Classify MOEs into Categories
One way to classify MOEs is by using the following categories: CDD Product Development Program Management Process Management Product Support Arrangement
19
Set Up – Classify MOEs into Categories Product Development
CDD Product Development Examples Operational Availability (AO) Weight budget MTBF Speed Range Objective Threshold Component Number MTBF Trend All good
20
Set Up – Classify MOEs into Categories Program Management
Program Management Examples Schedule performance index and variance Cost performance index and variance Manpower (planned vs. actual) Risk assessment tracking Earned Value Deliveries Variance Schedule $’s in Millions Cost $10.00 $0.00 ($10.00) ($20.00) ($30.00) Cumulative Dollar Variances
21
Set Up – Classify MOEs into Categories Process Management
Process Management Examples Product Support Arrangement Number and cost of: Requirement changes Engineering change proposals Test failures Defect rates Cycle time Defects per 100 units 30 25 20 15 10 5 Time
22
Set Up – Align MOEs to Milestones The Life Cycle and Artifacts
Technology Maturation & Risk Reduction 1 Understand and Communicate User Sustainment Needs ICD CDD CPD Update RAM-C Rationale Report Project Management Plan SEP TEMP 2 Design, Test, and Redesign using Sustainment Metrics LCSP 3 Monitor Performance (Program Progress, Technical Performance Metrics, Process Metrics) Affordable Effective Suitable
23
Set Up – Align MOEs to Milestones MOE Maturity and the Life Cycle
Technology Maturation & Risk Reduction User Need Traceability MOP Speed Payload MOS Reliability Maintainability KPP/KSA AO AM Reliability O&S Cost TPM MTBF MTTR Power Range Endurance Payload Refine TPMs T&E MOEs ICD CDD CPD MOEs mature with the life cycle and are traceable in artifacts.
24
Topic 3: Select Measures of Effectiveness for Supportability
25
MOE Analyze – TPM Candidates and Attributes
Our analysis consists of three main areas: 2.1 Analyze Candidates – Determine best TPMs for a supportable design 2.2 Analyze Achievable TPM Attributes– Describe TPM relationship to program capability document maturity 2.3 Establish Criteria– Describe Sustainment TPM role in design performance 2.1 2.2 2.3 Analyze TPM Candidates Analyze Sustainment TPM Attributes Establish Evaluation Criteria
26
Analyze – TPM Candidates and Attributes ASOE Model and TPMs
The ASOE Model has four distinct intersections representing program performance and Sustainment objectives. Balancing these priorities requires using TPMs in a series of trade-offs. Technical Performance Supportability Process Efficiency Life Cycle Cost Functions Reliability Production Total Ownership Cost Capabilities Maintainability Maintenance Support Features Logistics Operations
27
Analyze – TPM Candidates ASOE and Product Development
1 2 3 4 Capability Concept Description TPMs Be on station for ‘n’ hour duration Supportability Reliability/Maintainability MTBF and MTBM Respond to operational tasking within ‘n’ hours Process Efficiency Operational response time Time in hours to start mission 1 2 1 3
28
Analyze – TPM Candidates Product Development and Logistics Product Data
Strike-Talon (Indenture Level A) LCN A LCN ACA COM/NAV System (Indenture Level B) LCN ACAABA Assembly ST UAV SATCOM Control (Indenture Level C)
29
Analyze – TPM Candidates ASOE and Product Development (cont.)
1 2 3 4 Capability Concept Description TPMs Weigh no more than ‘n’ pounds Supportability Support Features PDR Weight/CDR Weight Carry and deploy payload within weight and loiter parameters Support Features/Reliability IOT&E Weight/MTBF/Mission endurance time 1 2 1 2
30
Analyze – TPM Candidates ASOE and Program Management
1 2 3 4 Capability Concept Description TPMs Deliver Supportability analyses by xQFYyy Process Efficiency Delivery schedule Schedule variance Deploy ‘n’ prototypes within budget Process Efficiency/ Life Cycle Cost Cost Cost variance Earned Value 1 3 1 4
31
Analyze – TPM Candidates ASOE and Process Management
1 2 3 4 Capability Concept Description TPMs Deploy ‘n’ prototypes with ≤ ‘m’ requirement changes Process Efficiency/ Life Cycle Cost Quality/Cost # Requirement changes per phase/Cost per requirement change Deploy ‘n’ full rate production units with ≤ ‘m’ defects/unit Process Efficiency Production quality Defect rate 1 3 4 1 3
32
Analyze – Sustainment TPM Attributes Thresholds and Objectives
KPPs/KSAs are expressed using threshold and objective values defined in the draft CDD: Threshold is the absolute minimum performance the customer will accept Objective is the performance that the customer really wants In this example, anything above the objective is questionable. Objective Planned Profile Threshold Mean Time between Failure (hours) Variance Achieved to Date In this example, the utility of anything below threshold is questionable. Planned Profile
33
Analyze – Sustainment TPM Attributes Thresholds and Objectives (cont.)
Drive Reliability up to optimum level Drive Sustainment cycle time down to optimum level
34
Analyze – Sustainment TPM Attributes TPM Promotion to Program Documents
Product TPM Candidates Table B. (U) STRIKE TALON UCAS Suitability Attributes Attribute Parameter Development Threshold Development Objective Operational Availability AO, AI, AM > 0.75 at IOT&E > 0.80 at IOC plus 2 years > 0.90 Reliability Mean Time Between Operational Mission Failure (MTBOMF) (Hours) ≥ 40 ≥ 90 Maintainability Mean Corrective Maintenance Time Operational Mission Failure (MCMTOMF) ≤ 3.0 ≤ 2.0 Diagnostics Mean Flight Hours Between False Alarms (MFHBFA) >300 hours >2000 hours Applicable product TPMs are promoted to program capability development and capability production documents (CDD & CPD).
35
Program Management TPM Candidates
Analyze – Sustainment TPM Attributes TPM Promotion to Program Management Plan Program Management TPM Candidates Applicable Program Management TPMs are promoted to the Program Management Plan (PMP).
36
Analyze – Sustainment TPM Attributes TPM Promotion to Contracts
Process TPM Candidates Development/Production Contracts Applicable Process TPMs are promoted to Program Contractual Documents. Threshold Objective
37
Topic 4: Translate Measures of Effectiveness into Achievable Outcomes
38
MOE Analyze – Establish Evaluation Criteria
Our analysis consists of three main areas: 2.1 Analyze Candidates – Determine best TPMs for a supportable design 2.2 Analyze Achievable TPM Attributes – Describe TPM relationship to program capability document maturity 2.3 Establish Criteria – Describe Sustainment TPM role in design performance 2.1 2.2 2.3 Analyze TPM Candidates Analyze Sustainment TPM Attributes Establish Evaluation Criteria
39
Analyze – Establish Evaluation Criteria TPM Role in Design Performance
Technical Performance Measures UAV Weight (kg) 53 52 51 50 49 48 47 46 45 Threshold Objective TPMs that are mission critical and cost sensitive are used as evaluation criteria to measure progress across major life cycle milestones and Supportability reviews.
40
Analyze – Sustainment TPMs TPM Maturity
Technology Maturation & Risk Reduction Understand and Communicate User Sustainment Needs ICD CDD CPD Update RAM-C Rationale Report Project Management Plan SEP TEMP Design, Test, and Redesign Using Sustainment Metrics LCSP TPMs grow in number and mature as the life cycle progresses.
41
Topic 5: Validate a Supportable Design
42
Monitor Sustainment Metrics
MOE Report Findings Our analysis findings fall into two broad categories: 3.1 Monitor Sustainment Metrics – Use a surveillance process to collect and update Sustainment data 3.2 Report Sustainment Outcomes – Qualify Sustainment TPMs that validate a supportable design 3.1 3.2 Monitor Sustainment Metrics Report Deviations
43
Report Findings – Monitor Sustainment Metrics Surveillance Process
Failure Reporting, Analysis, and Corrective Action System LCLs PSM SE
44
Report Findings – Report Deviations Communication Structure
All IPS Elements involve MOEs/TPMs and are reported to all IPTs. Sustainment Metrics
45
Report Findings – Report Deviations Design Reviews
Did the metric meet threshold? For example, at CDR (i.e., Post-Milestone B), the TPM/metric being analyzed should be between the threshold and objective values. 53 52 51 50 49 48 47 46 45 Technical Performance Measures UAV Weight (kg) Threshold Objective
46
Is a given MTBF TPM on the correct trajectory to meet the requisite Supportability reviews?
20 minutes Individual grade Independently access Blackboard and select the Exercise link in the Lesson 3: Measures of Effectiveness for Supportability folder to answer two questions.
47
Lesson Exercise Debrief Question 1 Graph
Upper Limit Planned Profile Mean Time between Failure (hours) Achieved to Date Lower Limit
48
Lesson Exercise Debrief Question 2 Graph
44 43 42 41 40 39 38 37 36 35 34 Planned Profile Objective Threshold Mean Time between Failure (hrs. x 100) Achieved to Date
49
Topic 6: Summary
50
Takeaways MOEs for Supportability are:
User defined Matured through the life cycle Used to measure product, program, and process performance Translated by the program into KPPs KPPs/KSAs are MOEs that serve as important criteria to assess whether the design meets user requirements TPM/metrics help users determine whether their needs are met. Programs use TPMs/metrics to evaluate design maturity and progress. An MOE is a mission-oriented qualitative or quantitative measure of operational success closely related to the mission objective or operation being evaluated.
51
TLO: Analyze Measures of Effectiveness to ensure a supportable design
Topic Objectives Topic 1: Introduction Welcome Where Are You? What Influence Do You Have? Topic 2: Understand Measures of Effectiveness Distinguish the relationships between Measures of Effectiveness, Key Performance Parameters/Key System Attributes, Measures of Performance, Measures of Suitability, and Technical Performance Measures Analyze the JCIDS Sustainment Measures of Effectiveness and their maturity over the system life cycle Topic 3: Select Measures of Effectiveness for Supportability Analyze candidates for Supportability Technical Performance Measures Analyze achievable Sustainment candidates for promotion to capability and contractual design documents Topic 4: Translate Measures of Effectiveness into Achievable Outcomes Establish evaluation criteria for each Sustainment metric to validate design performance Topic 5: Validate a Supportable Design Monitor Sustainment metrics to assure the system meets Supportability design criteria Topic 6: Summary
Similar presentations
