Advancing the knowledge of systems engineering Toward an Ontology for Measuring Systems Engineering Return on Investment Advancing the knowledge of systems engineering Eric Honour +1 (850) 479-1985 ehonour@hcode.com Dr. Ricardo Valerdi +1 (617) 253-8583 rvalerdi@mit.edu Toward Ontology for SE-ROI Version

Topics SE-ROI Project “Ontology” concept COSYSMO work toward ontology Categorization from current standards Future directions Toward Ontology for SE-ROI

Systems Engineering Return on Investment Summary of the SE-ROI Project Toward Ontology for SE-ROI

“System Thinking” Design Heuristic Claim of SE Better systems engineering leads to Better system quality/value Lower cost Shorter schedule SYSTEM DESIGN DETAIL PRODUCTION INTEGRATION TEST Traditional Design Time Risk Saved Time/ Cost “System Thinking” Design Time Risk Toward Ontology for SE-ROI

Value of SE – 2004 Results Problems/challenges: Value = 1.0 if program met cost/schedule goals Each dot is one program, with sizes between $1M and $6.5B The upper left chart shows that the data supports the original hypothesis, that cost and schedule performance are improved (up to a point) by the use of more SE effort. (The vertical axis on this chart is the inverse average of cost and schedule overruns.) The lower right chart is an independent, subjective check on the hypothesis, based on the subjective inputs of respondents. Their evaluation of “comparative success” on a scale of 0-10 also shows higher project success with greater SE effort. Problems/challenges: Quantitative data on SE not available in program databases All data points were subjective Detailed structure not available Greater SE Effort led to better cost/schedule compliance and better predictability Source: SECOE 01-03 INCOSE 2003 Toward Ontology for SE-ROI Measurable Systems Engineering

SE-ROI Project Interviews Just-completed programs Desired Results Key PM/SE/Admin Translate program data into project structure Desired Results Statistical correlation of SE methods with program success. Leading SE indicators that can be used during a program. Identification of good SE practices under different conditions. Program characterization Program success data SE data (hours, quality, methods) Statistical correlation Toward Ontology for SE-ROI

What is this word and how does it relate to systems engineering? “Ontology” Concept What is this word and how does it relate to systems engineering? Toward Ontology for SE-ROI

Current State of SE Definition Fragmented by domain opinions Military – DOD/MOD Space - NASA/ESA Commercial products Aircraft Automobiles Nuclear waste Process engineering Tool vendors Etc. Etc. Etc. Fragmented by discipline opinions Technical leaders System architects System analysts Requirements engineers Operations analysts Design engineers Fragmented by standards ANSI/EIA-632 IEEE-1220 ISO-15288 CMMI MIL-STD-499C Toward Ontology for SE-ROI

Ontology “…a branch of metaphysics concerned with the nature and relations of being” aesthetics interfaces functions structure inputs components methods outputs categories understanding POSIWID – The Purpose of a Systems Is What It Does Jack Ring The purpose of systems engineering is different in the eyes of different people, because they perceive different actions/results from SE Toward Ontology for SE-ROI

Purpose of this Paper Explore the variety of what people see in SE Formulate some general categories Interpret historical SE effort data Provide a structure for the data-gathering in the SE-ROI project. Toward Ontology for SE-ROI

COSYSMO work toward ontology An exploration of ontology as performed in the COSYSMO project Toward Ontology for SE-ROI

COSYSMO Effort Profile How is Systems Engineering effort distributed over time? Phase Conceptualize Develop Operational Test & Eval Transition to Operation %Effort (STDEV) 23 (12) 36 (16) 27 (13) 14 (9) Toward Ontology for SE-ROI

Effort Distribution Across ANSI/EIA 632 Fundamental Processes Average Standard Deviation Acquisition & Supply 7% 3.5 Technical Management 17% 4.5 System Design 30% 6.1 Product Realization 15% 8.7 Technical Evaluation 31% Toward Ontology for SE-ROI

Systems Engineering Effort Profile Toward Ontology for SE-ROI

Categorization from Current Standards The start of an ontology, by identifying the widely-accepted categories. Toward Ontology for SE-ROI

Categories in the Standards Mission/Purpose Definition Requirements Management System Architecting System Implementation Technical Analysis Technical Management/Leadership Verification & Validation CMMI ANSI/EIA-632 MIL-STD-499C IEEE-1220 ISO-15288 Colored boxes on following slides show the terminology used by each standard Toward Ontology for SE-ROI

Mission/Purpose Definition Define the mission or purpose of the new/changed system. Typically described in the language of the system users rather than in technical language CMMI Develop customer requirements ANSI/EIA-632 Not included MIL-STD-499C Not included IEEE-1220 Define customer expectations ISO-15288 Stakeholder needs definition Toward Ontology for SE-ROI

Requirements Management Creation and management of requirements Efforts to define, analyze, validate, and manage the requirements CMMI Requirements development Requirements mgmt ANSI/EIA-632 System design Requirements definition MIL-STD-499C System requirements analysis and validation IEEE-1220 Requirements analysis ISO-15288 Requirements analysis Toward Ontology for SE-ROI

System Architecting Define the system in terms of its component elements and their relationships Diagrams that depict the system, its environment, components, and relationships CMMI Technical solution ANSI/EIA-632 System design Solution definition MIL-STD-499C System product technical requirements analysis and validation Design or physical solution representation IEEE-1220 Synthesis ISO-15288 Architectural design System life cycle mgmt Toward Ontology for SE-ROI

System Implementation Development/completion of the system Specific system-level efforts in the standards are system integration and transition to use ANSI/EIA-632 Product realization Implementation Transition to use CMMI Product integration MIL-STD-499C Not included IEEE-1220 Not included ISO-15288 Implementation Integration Transition Toward Ontology for SE-ROI

Technical Analysis System-level technical analysis Assessment of system performance Trade-off analysis of alternatives CMMI Measurement and analysis ANSI/EIA-632 Technical evaluation System analysis MIL-STD-499C Functional analysis, allocations and validation Assessments of system effectiveness, cost, schedule, and risk Tradeoff analyses IEEE-1220 Functional analysis Requirements trade studies and assessments Functional trade studies and assessments Design trade studies and assessments ISO-15288 Requirements analysis Toward Ontology for SE-ROI

Technical Management/Leadership Guiding the engineering teams involved in system design programs Size/complexity of teams demands leadership CMMI Project planning Project monitoring & control Supplier agreement mgmt Process/product quality assur. Configuration mgmt Integrated project mgmt Decision analysis/resolution Quantitative project mgmt Risk mgmt ANSI/EIA-632 Technical Mgmt Planning Assessment Control MIL-STD-499C Planning Monitoring Decision making, control, and baseline maintenance Risk mgmt Baseline change control Interface mgmt Data mgmt Subcontract mgmt Technical reviews/audits IEEE-1220 Technical mgmt Track analysis data Track requirements and design changes Track performance Track product metrics Update specifications Update architectures Update plans Maintain database ISO-15288 Planning, Assessment, Control Decision mgmt Config mgmt Acquisition, Supply Resource mgmt Risk mgmt Toward Ontology for SE-ROI

Verification & Validation Verification: comparison of the system with its requirements through objective evidence. Validation: comparison of the system or requirements with the intended mission ANSI/EIA-632 Technical Evaluation Requirements validation System verification End products validation CMMI Verification Validation MIL-STD-499C Design or physical solution verification and validation IEEE-1220 Requirement verification Functional verification Design verification ISO-15288 Verification Validation Quality mgmt Toward Ontology for SE-ROI

Future Directions Where is SE-ROI going? Toward Ontology for SE-ROI

Project Advisory Group Group of interested people/organizations Communicate via web, telecon, meetings Help define the data organization Build public interest in the project Provide access to real programs View interim (protected) data as it develops Current members come from: AF Institute of Technology Northrop Grumman DOD Office of Secy of Defense DRS Johns Hopkins Univ MIT The Mitre Corp NAVAIR Raytheon Rand Corporation Rafael Systems & Software Consortium Univ of South Australia USN Chief Engineer For information, see http://www.hcode.com/seroi/ Toward Ontology for SE-ROI

Summary Systems engineering current state of knowledge is fragmented Broadly-accepted ontology is needed SE-ROI project needs categorization now Structure the data to be correlated Discover leading SE indicators Identify SE best practices and methods Categorization across standards helps develop the needed ontology Toward Ontology for SE-ROI

Questions? Eric Honour +1 (850) 479-1985 ehonour@hcode.com Dr. Ricardo Valerdi +1 (617) 253-8583 rvalerdi@mit.edu For information, see http://www.hcode.com/seroi/ Toward Ontology for SE-ROI

Survey of SE-ROI Knowledge 1992 Gruhl (NASA) Project Definition – NASA Program definition 10-15% reduces cost overruns 1990 Ancona/ Caldwell Boundary Management Study Boundary management averages 14%; more is better 2000 Miller (MIT) Large Engineering Projects Study Programs value cost over schedule over tech Leadership important 1995 Franz (Boeing) Impact of Systems Engineering on Quality and Schedule Better SE led to significant cost reduction 2003 Barker (IBM) Systems Engineering Effectiveness Better SE reduced parametric costs by 30% 2004 Kludze (NASA) Impact of Systems Engineering on Complex Systems General belief that SE improves program cost 2004 Honour (SECOE) Value of Systems Engineering Greater SE 10-15% reduces cost/schedule overruns Toward Ontology for SE-ROI

Next Steps Define interview data sheets Use this categorization Identify and interview trial projects Obtain initial data Evaluate the interview data sheets Identify and interview projects Ongoing effort for 2-3 years Perform statistical correlation work Ongoing effort for duration of project Interim reports to participating organizations Final report expected 2009 Toward Ontology for SE-ROI