The Statistical Analysis of the Impact of Adequate Testing on the Defense Systems Development Zaw P. Tun GWU, EMSE PhD Candidate Dr. Shahram Sarkani, Ph.D.,

Slides:



Advertisements
Similar presentations
The Robert Gordon University School of Engineering Dr. Mohamed Amish
Advertisements

Model Based System Engineering Approach to Managing Requirements in a Multi-Agency/Multi-Stakeholder Environment of Science and Technology/Research and.
Enhancing Data Quality of Distributive Trade Statistics Workshop for African countries on the Implementation of International Recommendations for Distributive.
Mitigating Risk of Out-of-Specification Results During Stability Testing of Biopharmaceutical Products Jeff Gardner Principal Consultant 36 th Annual Midwest.
© 2005 The MITRE Corporation. All rights reserved The Telemetry Spectrum Crunch: An Economist’s Perspective ITC 2005 Las Vegas, NV Carolyn A. Kahn The.
Small Projects & Tailoring Using the PPA 1:15 – 2:15 Teresa Kinley, OPHPR With Panelists: Susan Wilkin, NCCDPHP Andy Autry, NCBDDD Carol Waller, NCEH/ATSDR.
Statistics for Managers Using Microsoft® Excel 5th Edition
Chapter 5 Time Series Analysis
Multivariate Data Analysis Chapter 4 – Multiple Regression.
STRATEGIES FOR RESEARCH Approaching the Paper Assignment.
Recent Trends in DoD Systems and Software Engineering Processes Bruce Amato Acting Deputy Director, Software Engineering and Systems Assurance Office of.
Systems Engineering Management
DITSCAP Phase 2 - Verification Pramod Jampala Christopher Swenson.
Correlational Designs
Applied Business Forecasting and Planning
Software Verification and Validation (V&V) By Roger U. Fujii Presented by Donovan Faustino.
Pre-Project Planning Lessons from the Construction Industry Institute Construction Industry Institute Michael Davis, P. Eng, PMP Ontario Power Generation.
Factor Analysis Psy 524 Ainsworth.
Factors that Associated with Stress in Nursing Faculty in Thailand
Chapter 13: Inference in Regression
Research Methodology Lecture No :16
9 Closing the Project Teaching Strategies
Chapter 7: The 30 elements of systems engineering
Literature Review and Parts of Proposal
U.S. Department of Energy Project Management: Communicating Progress – Celebrating Success Paul Bosco, PE, PMP, CFM, LEED-AP Director, Office of Procurement.
Suwimon Vongsingthong Department of Information Technology and Management, Krirk University, Thailand
Software Inspection A basic tool for defect removal A basic tool for defect removal Urgent need for QA and removal can be supported by inspection Urgent.
Introduction Financial Statement Analysis Prepared By: Anuj Bhatia, Professor, Shah Tuition Classes Ph
1. IASC Operational Guidance on Coordinated Assessments (session 05) Information in Disasters Workshop Tanoa Plaza Hotel, Suva, Fiji June
GBA IT Project Management Final Project - Establishment of a Project Management Management Office 10 July, 2003.
Life Cycle Cost Minimization & the Impact of Early Implementation of Human Systems Integration Requirements Liana Algarín, Ph.D. Candidate, The George.
L 1 Chapter 12 Correlational Designs EDUC 640 Dr. William M. Bauer.
U.S. International Travel Statistics OECD Working Party on International Trade in Goods and Trade in Services Statistics Robert E. Yuskavage.
United States Department of Agriculture Food Safety and Inspection Service 1 National Advisory Committee on Meat and Poultry Inspection August 8-9, 2007.
Comprehensive Educator Effectiveness: New Guidance and Models Presentation for the Virginia Association of School Superintendents Annual Conference Patty.
Patterns of Event Causality Suggest More Effective Corrective Actions Abstract: The Occurrence Reporting and Processing System (ORPS) has used a consistent.
1 Chapter 12 Simple Linear Regression. 2 Chapter Outline  Simple Linear Regression Model  Least Squares Method  Coefficient of Determination  Model.
1 GAO’s Use of DAMIR October 31, 2007 David Best Assistant Director
What grounded theory is not
6. Evaluation of measuring tools: validity Psychometrics. 2012/13. Group A (English)
Measurement Models: Exploratory and Confirmatory Factor Analysis James G. Anderson, Ph.D. Purdue University.
IMPACT OF QUALITY ASSURANCE SYSTEM IN EFFECTIVENESS OF VOCATIONAL EDUCATION IN ALBANIA IMPACT OF QUALITY ASSURANCE SYSTEM IN EFFECTIVENESS OF VOCATIONAL.
1 1 Nunn-McCurdy Legislation/Program Impacts Della McPhail Chief Financial Officer 308 ARSW DISTRIBUTION STATEMENT A: Approved for public release; distribution.
Business Forecasting 1 Dr. Mohammed Alahmed (011) Dr. Mohammed Alahmed.
Week 101 ANOVA F Test in Multiple Regression In multiple regression, the ANOVA F test is designed to test the following hypothesis: This test aims to assess.
CHAPTER OVERVIEW Say Hello to Inferential Statistics The Idea of Statistical Significance Significance Versus Meaningfulness Meta-analysis.
Criteria for selection of a data collection instrument. 1.Practicality of the instrument: -Concerns its cost and appropriateness for the study population.
Time Phasing Development and Production Chapter 16.
Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov February 16, 2011.
Life Cycle Cost Savings by Improving Reliability Dr. Charles E. McQueary Director, Operational Test and Evaluation January 15, 2009.
Recent work on revisions in the UK Robin Youll Director Short Term Output Indicators Division Office for National Statistics United Kingdom.
1 Lecture 2.4a: SEF SE Planning and the SEP (SEF Ch 16) Dr. John MacCarthy UMBC CMSC 615 Fall, 2006.
Approaches to quantitative data analysis Lara Traeger, PhD Methods in Supportive Oncology Research.
Applying Cost Improvement Theory to the Production of Military Space Systems— Where Do We Go From Here? Paul Killingsworth SCEA National Conference June,
Overview of Types of Measures Margaret Kasimatis, PhD VP for Academic Planning & Effectiveness.
Logistic Regression: Regression with a Binary Dependent Variable.
Chapter 8 Introducing Inferential Statistics.
Effort Estimation Models for Contract Cost Proposal Evaluation
October 31, 2007 David Best Assistant Director
Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov March 23, 2011.
Quantitative methods Lecture 1.
Milestone A to Milestone B Requirements Management Activities
CJT 765: Structural Equation Modeling
The Systems Engineering Context
Evaluation of measuring tools: validity
Self Introduction Dr. Sou Veasna
Project Management Process Groups
Brown County Financial Decision and Support Model
Chapter 6 Logistic Regression: Regression with a Binary Dependent Variable Copyright © 2010 Pearson Education, Inc., publishing as Prentice-Hall.
Lack of Confidence Interval Reporting in Dermatology: A Call to Action
Presentation transcript:

The Statistical Analysis of the Impact of Adequate Testing on the Defense Systems Development Zaw P. Tun GWU, EMSE PhD Candidate Dr. Shahram Sarkani, Ph.D., P.E. Dr. Thomas Mazzuchi, D.Sc. May 1, 2015

Overview  Purpose  Background  Focus Area  Data Extraction  Source of Data  Proposed Methodology  Conclusion  Future Work

Introduction  Purpose Analyze the cost growth trend to determine if adequate testing plays an essential role in successful development of new systems in the DOD acquisition environment  Background DOD is managing $1.68 trillion worth of acquisition programs DOD’s acquisition costs has increased by $135 billion over the past 2 years for 98 major defense acquisition programs GAO found that most of DOD’s portfolio total cost growth in past 2 years occurred in the DOD’s largest programs, which are all in production –This means programs experience significant problems and changes well after the programs and their costs should have stabilized

Focus Areas  GAO has continuously stated that in order to mitigate risks of uncovering problems in the later phases of the development cycle, knowledge based testing should be conducted at key junctures, specifically around the Engineering and Manufacturing Development (EMD) phase Source: GAO (2011), Defense Acquisition: Assessments of Selected Weapon Programs, GAO SP

Data Extraction Mission Need Statement Between MS-B and MS-C Appropriate TRL level (GAO knowledge point 1) PDR (GAO knowledge point 1) CDR (GAO knowledge point 2) Minimum design changes after this point Manufacturing process should be stabilized before entering Production phase (GAO knowledge point 3) Should not have significant cost growth at this point Beyond MS-C RDT&E cost growth should be minimum Data form SAR, GAO, and DOT&E Annual Report, etc. Significant RDT&E cost growth in the Production and Development Phase is the indicator of insufficient T&E activities in earlier phases of the acquisition

Primary Source of Data  Selected Acquisition Reports (SARs) RAND conducted several research studies based on SAR data and identified numerous “disadvantages" that an analyst must be aware of in order to maintain the validity of the data. Most prevalent problems are: o Failure of some programs to use a consistent baseline cost estimate o Exclusion of some significant elements of cost o Exclusion of certain classes of major programs (e.g., special access programs) o Constantly changing preparation guidelines o Inconsistent interpretation of preparation guidelines across programs o Unknown and variable funding levels for program risk o Cost sharing in joint programs o Reporting of effects of cost changes rather than their root causes Using SARs provides key “advantages” o SARs conform to a strict reporting format, providing consistency to the data o Those who create SAR reports receive annual SAR training, which adds to the consistency of the data o SARs are presented to Congress, so the level of scrutiny that SARs receive in the review process bolsters both the consistency and accuracy of the documents. Databases in general contain inaccuracies, but a database built from SAR data arguably withstands scrutiny better than most

Supplementary Data Sources  DOT&E Annual Reports Current Activities Cost data Issues Milestone dates Recommendations  GAO Reports Technology, Design, and Production Maturity Cost Data Milestone Data Issues

Conceptual Model Performance Indicators

Proposed Methodology  Data Extraction Base year RDT&E dollars from SAR chosen for analysis, since these dollars exclude estimated inflationary effects This format facilitates conversion of the various base years of individual estimates into a single base year, making possible easy comparison across programs  Data Analysis Relationship between early adequate testing and program success –RDT&E cost growth Prior/Post MS-C  Schedule Slip and Unit Cost Growth –Performance indicators  Schedule Slip and Unit Cost Growth

Using SEM for Data Analysis  Powerful multivariate techniques – Specialized version of other analysis methods  Advantages over Multiple Regression SEM programs provide overall tests of model fit and individual parameter estimate tests simultaneously Regression coefficients, means, and variances may be compared simultaneously, even across multiple between subjects groups Measurement and confirmatory factor analysis models can be used to purge errors, making estimated relationships among latent variables less contaminated by measurement error Ability to fit non-standard models, including flexible handling of longitudinal data, databases with auto correlated error structures (time series analysis), and databases with non-normally distributed variables and incomplete data

An Example of Cost Growth Due to Inadequate Early Testing Compare Baseline and Current Status(Cost and Schedule) RDT&E Cost increase in Production Phase indicate that more problems discovers during that phase It is an indicator that program did not perform adequate early testing GAO Early Knowledge Based Testing Activities

 Recently DOT&E conducted a systematic review of recent 67major acquisition programs that experienced delays 56 programs (or 84 percent) had performance problems in testing (either DT, OT, or both)  56 programs were categorized and cost growth were analyzed and compared with other programs in the same category  Preliminary results shows that programs with inadequate testing in the earlier phase of the development has higher cost growth Preliminary Results Average % Growthn Aircraft Helicopter Ship Missile Satellite Land Vehicle C3I Inadequate Testing Cost Growth Other Cost Growth Aircraft244.94%114.64% Missile116.26%77.92%  Preliminary data analysis shows that aircraft acquisition programs has highest average percentage of total acquisition cost growth compared to others programs.

Conclusion  Knowledge based testing is essential in detecting problems early in the acquisition phase  Program success depends on the knowledge obtained during earlier tests in the development of the system  Structural Equation Modeling (SEM) is a powerful statistical technique to analyze program success factors from the T&E perspective.

Future Work  Complete Data collection and data base construction  Refine Performance Indicators and Conceptual Model based on further research  Perform data analysis using Structural Equation Modeling (SEM) to test model and find the correlation between program success and “Performance Indicators”

Contact Information Zaw P. Tun (407) Shahram Sarkani, Ph.D., P.E (888) Thomas A. Mazzuchi, D.Sc. (202)

Backup

 References Arena, M., Leonard, R., Murray, S., & Younossi, O. (2006). Historical Cost Growth of Completed Weapon System Programs. Santa Monica CA: RAND GAO (2011). Defense Acquisition: Assessments of Selected Weapon Programs, GAO SP GAO (2011). WEAPONS ACQUISITION REFORM :Actions Needed to Address Systems Engineering and Developmental Testing Challenges, GAO Hough, Paul G. (1992). Pitfalls in Calculating Cost Growth from Selected Acquisition Reports. Santa Monica CA: RAND Jarvaise, J., Drenzer, J., & Norton, D. (1996). The Defense System Cost Performance Database: Cost Growth Analysis Using Selected Acquisition Reports. Santa Monica CA: RAND Sipple, V. (2002). ESTIMATING ENGINEERING COST RISK USING LOGISTIC AND MULTIPLE REGRESSION, Master Thesis, AFIT. The office of the Director, Operational Test and Evaluation (DOT&E) (2011). FY 2011 Annual Report