1 Independent Verification and Validation Current Status, Challenges, and Research Opportunities Dan McCaugherty IV&V Program Manager Titan Systems Corporation 5 September, 2001

2 Agenda Where we are today Developer challenges observed by IV&V IV&V challenges Opportunities for R&D

3 Where we are today

4 IV&V Gaining Acceptance IV&V is gaining acceptance as a mechanism for mitigating software risk IV&V is naturally evolving to also include programmatic assessments such as cost/schedule performance analysis Recent NASA IV&V policy presents an opportunity for wider application and understanding of IV&V

5 IT Project Success Factors Gartner Institute study identified the following essential areas as critical to IT project success 1 –Risk Management –Communications –Resource Management –Lessons Learned Additional areas based on professional experience –Project planning –Requirements!!! Projected that implementation of sound project management approach emphasizing these key areas can reduce number of failed projects by 30% 1 1 GartnerGroup

6 IT Project Success Rates Odds for Success 1&2 –> 80% of IT projects are late, over budget, lack essential functionality, or are never delivered –28% of IT projects just flat fail representing more than $81 Billion in losses –Of 15,000 data migration projects started in 1999, as many as 88 percent either overran or failed Leading causes for project failure 1 –poor project planning –failure to manage scope and change –failure to manage risk 1 GartnerGroup 2 The Standish Group

7 Developer Challenges Observed by IV&V

8 Incremental Delivery Definition “Build a Little, Test a Little” approach can avoid major development failures, but creates other challenges, such as: –Maintaining end state specifications concurrent with increment specifications –Implementing the right requirements in the right builds to meet testable increment specific capabilities –Understanding the cost/schedule effects of pushing requirements into later builds

9 Early Definition of Interfaces and Key Architectural Concepts Late interface or architecture changes or problems create significant re-work or are unmanageable Distributed computing stresses system engineering efforts for early definition of –API’s, remote procedures, and reused objects –Redundancy Management, –Fault Detection, Isolation, and Recovery –Exception handling

10 Use of New Technologies A great way to attract and keep highly skilled professionals Majority of focus on what the technology can do, not enough focus on what it can not do or was never designed to do Limited skill base for new technologies If COTS, may not have future support

11 Measuring Progress or Quality Measurable definitions for “Done” are needed. Metrics efforts and collection mechanisms need to be part of the process from day one Optimism (a good thing) must be tempered with facts

12 IV&V Challenges

13 IV&V Technical Challenges Adapting IV&V methods to developer environments Understanding system dynamic behavior Convincing a project that the process is too risky Validating Non-Development-Items Reverse engineering incompletely specified systems

14 IV&V Managerial and Business Challenges Demonstrating that selected IV&V tasks and scope are appropriate for the Project’s acceptable levels of risk Appropriate IV&V integration into project Demonstrating IV&V ROI Accounting for IV&V data needs within developer contracts.

15 Opportunities for R&D

16 Opportunities for R&D Research IV&V ROI –Past case study 3 shows 1.25 < IV&V ROI < 1.82 –Recommend a study of many projects with varying amounts and types of IV&V –Correlate overall project success to IV&V effort types Continue development of code reverse engineering tools and interface V&V tools –Software and Interface Analysis Tool (SIAT) –Interface Management System (IMS) Develop Reconfigurable engineering simulation for Space Craft –Improve early “validation” analyses 3 Rogers

17 Opportunities for R&D Develop a guidebook identifying sets of IV&V tasks for certain types of systems and acceptable levels of mission risk. –Research history of IV&V method application and results Develop a guidebook for developer managers on how to realize benefits of IV&V. –Contracts and agreements –Integration of IV&V into different types of projects –Reporting and issue resolution

18 Summary

19 Summary There are many software challenges/risks out there –Many can be managed through use of accepted engineering practices IV&V can help mitigate risk if used/applied correctly We need to better quantify the effects of IV&V and the ROI Recommended areas of R&D –Research IV&V ROI, effects of IV&V use –Develop tools that are needed today –Develop guidance for future IV&V'ers and IV&V users.