Lockheed Martin Aeronautics Company Copyright ©2007 Lockheed Martin Corporation. All Rights Reserved. Empirical Study of Software Quality and Reliability Michael F. Siok, PE Lockheed Martin Aeronautics Company P.O. Box 748, MZ 8604 Fort Worth, TX Tel: (817) High Assurance Systems Engineering Conference IEEE HASE November 2007 Jeff Tian, Ph.D., PE Southern Methodist University Dallas, TX USA Tel: (214) FAX: (817)

Lockheed Martin Aeronautics Company 2 © 2007 Lockheed Martin Corporation Avionics Software Development  How much software process is enough ?  How much software is enough ?  Benchmarks ?  In-house Best Practice ? Cost too Much Takes too long Metrics, metrics, metrics Good Software = f (productivity, reliability, quality)

Lockheed Martin Aeronautics Company 3 © 2007 Lockheed Martin Corporation Aircraft Software at LM Aero... Radar Mission Computer Gyros Accelerometers Flight Control Rudder(s) Electro- Optical Global Positioning Inertial Stabilizers Ailerons ControlsDisplays Weapons External Stores Stick and Throttle Engine(s) Characteristics Large Complex Systems Decades Lifespan Frequent Software Updates Mix of Computation Types Computational Displays Logic/State Machine Signal Processing Feedback Control Hard & Soft Real-Time Severe Computing Resource Constraints COTS requirements Legacy reuse

Lockheed Martin Aeronautics Company 4 © 2007 Lockheed Martin Corporation Aircraft Software at LM Aero... Embedded Code size ~2 MSLOC & climbing fast Hundreds of object instances Some COTS Products: F-16, C-130, U-2, F-22, JSF, support equipment, others....

Lockheed Martin Aeronautics Company 5 © 2007 Lockheed Martin Corporation Background (Cont’d)  Avionics Software Managers want to know...  How are projects performing, individually and collectively?  Is OO better than SA/SD?  Does programming language make a difference?  Productivity, reliability, quality... How are we doing?  Issues...  Measuring success  Benchmarking  How to improve  Use statistics to provide some answers

Lockheed Martin Aeronautics Company 6 © 2007 Lockheed Martin Corporation Topics  Background  Avionics Software Project Data  Data Analysis  Wrap up and Next Steps Empirical Study of Embedded Software Quality and Productivity By Michael F. Siok and Jeff Tian IEEE HASE 2007 Paper:

Lockheed Martin Aeronautics Company 7 © 2007 Lockheed Martin Corporation Avionics Software Project Data  39 software projects carefully chosen for study  Project Application Domain (a.k.a. project complexity)  Project Size  Software Development Methodology  Programming Language used  Metrics Framework  Size  Cost  Schedule  TP & Quality

Lockheed Martin Aeronautics Company 8 © 2007 Lockheed Martin Corporation Avionics Software Project Data (Cont’d)  Metrics Framework provides reference to capture metric and scope  27 metrics available from all projects and normalized, where appropriate  Comparable  Individually  In Aggregate

Lockheed Martin Aeronautics Company 9 © 2007 Lockheed Martin Corporation Data Analysis -- Approach  Study metrics data using Descriptive Statistics  Individual Metrics  Metrics in aggregate  Perform Hypothesis Tests to answer management questions  Domain Separation  OO –vs- SA/SD  Size –vs- Reliability  Size –vs- Productivity  Cost –vs- Reliability  Cost –vs- Productivity  Language –vs- Productivity  Summarize & Report Results

Lockheed Martin Aeronautics Company 10 © 2007 Lockheed Martin Corporation Data Analysis (Cont’d) -- Descriptive Statistics  Used to discover interesting characteristics about each metric in dataset

Lockheed Martin Aeronautics Company 11 © 2007 Lockheed Martin Corporation Data Analysis (Cont’d) -- Hypothesis Tests  Hypothesis Testing  Since data not normally distributed, used non-parametric tests to accept or reject hypotheses  H 0 – Data compared is from same population  H 1 – Data compared is from different populations Wilcoxon Rank Sum Test

Lockheed Martin Aeronautics Company 12 © 2007 Lockheed Martin Corporation Data Analysis (Cont’d) -- Reporting  Hypothesis Testing (Continued)  Conducted on metrics, results collected, summarized  Majority rules policy on Accept/Reject test results  Reject = difference at.05 level of significance  Accept = no difference

Lockheed Martin Aeronautics Company 13 © 2007 Lockheed Martin Corporation Wrap Up and Next Steps  Avionics software organization needed methodology to  Assess project performance  Assess project performance relative to other similar projects  Identify and act on opportunities for improvement  Software project data was difficult to acquire  Data actually very easy to get  Projects had to demonstrate selected process controls  Managed variability in metrics  Projects had to submit data to company metrics repository and use it  Projects had to validate data in company repository  Analysis method fairly simple, straight-forward  Descriptive Statistics to study metrics behaviors  Hypothesis Testing  Summary reporting to capture analysis results for action

Lockheed Martin Aeronautics Company 14 © 2007 Lockheed Martin Corporation Wrap Up and Next Steps (Cont’d)

Lockheed Martin Aeronautics Company 15 © 2007 Lockheed Martin Corporation Wrap Up and Next Steps (Cont’d)  Statistical Testing did not uncover a clear ‘best all around project’  Want to identify well-rounded best-in-class project(s)  Project demonstrated best in cost, schedule, performance, and quality  Use Data Envelopment Analysis (DEA) as benchmarking method to identify best in class software projects  Non-parametric analysis method  Establishes Multivariate production efficiency frontier  Statistical Analysis coupled with DEA will provide repeatable methodology to study & assess company software project data  To understand software project and organizational performance  To identify best performing software projects  To clearly identify practical software process & product improvement opportunities to better the business practice of software development

Lockheed Martin Aeronautics Company 16 © 2007 Lockheed Martin Corporation Points of Contact Michael F. Siok, PE Lockheed Martin Aeronautics Company P.O. Box 748, MZ 8604 Fort Worth, TX Tel: (817) Jeff Tian, Ph.D., PE Southern Methodist University Dallas, TX USA 76 Tel: (214) FAX: (817)