1. 1PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 Dr. Forrest Shull, FCMD Kurt Woodham, L-3 Communications OSMA SAS 08 Infusion of Perspective-Based Inspections for NASA IV&V (Technical Briefing)

2. 2PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Introduction to Perspective-Based Inspection (PBI)  Tailoring of PBI for the IV&V Context  Evaluation Plan Outline

3. 3PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  A long history of research & application shows that structured human inspection is one of the most cost-effective practices for achieving quality software  “Cost savings rule” – Cost to find & fix software defects is about 100x more expensive after delivery than in early lifecycle phases, for certain types of defects.  IBM: 117:1 between code and use  Toshiba: 137:1 between pre- and post-shipment  Data Analysis Center for Software: 100:1  “Inspection effectiveness rule” – Reviews and inspections find over 50% of the defects in an artifact, regardless of the lifecycle phase applied.  50-70% across many companies (Laitenberger)  64% on large projects at Harris GCSD (Elliott)  60% in PSP design/code reviews (Roy)  50-95%, rising with increased discipline (O’Neill)  … many others Software Inspection

4. 4PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 Planning Preparation Defect Report Form Meeting Follow- through Software Artifact Planning Form Defect Correction Form 1 2 3 4 organizer inspector moderator inspectors author Corrected Software Artifact Software Inspection Defect Collection Form Roles Activities Products Improving Inspections Software Reading Techniques: Procedural guidance incorporating tested best practices for individual preparation

5. 5PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  A procedural technique incorporating three best practices: 1.Giving each reviewer a particular and unique focus (or perspective) on the document under review  Base perspectives on document stakeholders to help understand necessary participants, relate inspection tasks to development skills 2.Making individual review of a document an active (rather than passive) undertaking 3.Articulating the quality aspects of interest Perspective-Based Inspection (PBI)

6. 6PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  History  2001-2003: SARP research funding  Refined basic approach, implemented on NASA projects and in classrooms, measured results  2004: Tech infusion with Flight Software Branch / GSFC  Goal: Produce new Branch standards  Success metric: Based on experiences on pilot(s), how much of the recommended process is adopted by Branch?  2004: Tech infusion with United Space Alliance / JSC  Goal: Reduce defect slippage over current increment  Success metric: Show reduced defect slippage on PB-inspected artifacts  2007-2009: SARP funding for improved general inspection planning  Other Results and Benefits  Improved defect detection effectiveness substantially (NASA & elsewhere)  Helped re-invigorate inspection practices  Forms the basis of industrial and academic training Perspective-Based Inspection (PBI)

7. 7PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Pathfinder: Infuse PBI into the IV&V practices being applied to JWST  JWST Project/Developer artifact review  JWST IV&V System Reference Model (SRM) validation  Technology Provider…  Adapts PBI to IV&V domain and JWST-specific context  Administers training  Facilitates initial reviews  Monitors efficacy of PBI and documents results/recommendations  JWST IV&V Team…  Receives training  Conducts reviews in-line with IV&V execution  Provides ancillary feedback to Technology Provider (minimal disruption)  IV&V Facility Research…  Monitors progress and determines next-step expansion based on JWST pathfinder Adopting PBI for IV&V

8. 8PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 JWST System Overview Integrated Instruments Mid-Infrared Instrument (MIRI) Near-Infrared Spectrograph (NIRSpec) Near-Infrared Camera (NIRCam) Fine Guidance Sensor (FGS) Launch in 2013 Collaboration between NASA, ESA, CSA Managed by GSFC NGST – prime contractor STScI – operator after launch

9. 9PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 JWST Observatory Diagram Image credit: http://www.jwst.nasa.gov/observatory.html

10. 10PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 JWST IV&V Overview  IV&V Scope  Observatory and Ground System segments  SRM-based approach starting with top-level user needs flowed systematically down through segment, elements, subsystem, and component levels  Specific focus areas driven by IV&V criticality/risk analysis with close coordination with JWST project  WBS driven  SLP 09-1 Rev J (http://www.nasa.gov/centers/ivv/ims/slps/index.html)http://www.nasa.gov/centers/ivv/ims/slps/index.html  Defines validation and verification tasking to be performed at various levels of system decomposition Emphasis on early validation of requirements with respect to system needs, goals, and objectives in light of three validation questions Validated SRM plays pivotal role in validation and verification activities throughout IV&V life-cycle

11. 11PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Motivation:  Focus the responsibilities of each inspector  Minimize overlap among inspector responsibilities  Maximize union of defects found Document with defects High overlap Low coverage (Ad hoc) Low overlap High coverage (PBI) Adopting PBI for IV&V: Perspectives

12. 12PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Tailoring required for new domain, new issues, new artifacts Adopting PBI for IV&V: Perspectives Needs, goals, objectives elucidation Analysis val./ver. target Model construction System Ref Model Model validation Do sources adequately define user needs? Is model complete, correct, consistent, and testable? Do behaviors defined in val./ver. targets map to equivalent definition in model?

13. 13PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Interpretation of “defect”  Avoid focusing on correctness as the one and only quality property  Any issue that would cause a problem for a stakeholder downstream. Adopting PBI for IV&V: Defect Types

14. 14PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 Adopting PBI for IV&V: Defect Types  Other sources of defect categories:  Checklists and defect categories from previous work  Experienced inspectors  What is it that the most experienced people are looking for?  Can get from: Asking them! Noting recurring questions at inspection meetings  Defect/discrepancy report databases  What types of problems seem to be common?  What types of problems take the longest to close?

15. 15PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Reading through a document one page at a time is not the most effective approach.  In PBI every inspector receives a scenario to guide his/her work.  Based on your perspective…  Imagine you are creating some other artifact you need as part of your job…  E.g. for tester perspective, think about building a test plan  Then read through the document as you normally would to get the necessary information.  Inspectors have questions about quality to answer while they follow the scenario. Adopting PBI for IV&V: Active Review

16. 16PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Initial perspectives for this domain:  Modeler  Uses requirements & existing models to create the next iteration  Quality foci: Appropriate level of detail in requirements; completeness and clarity of requirements & models; consistency and correctness of models  Domain expert  Assesses whether model accurately and usefully captures domain  Quality foci: Correctness of requirements (independent of models); identification of stakeholders and evaluation of usability from their POV; checking flow of control & use of reusable components  Quality assurance  Assesses whether models and system can be validated properly  Quality foci: Handling of exception cases / unexpected system conditions; robustness of system; testability Adopting PBI for IV&V: Summary

17. 17PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 Infusion Plan  Understand team-specific quality concerns  Understand likely perspectives for the team  Refine set of perspectives and define procedures for each. E.g.,  What quality concerns map to which perspectives  What is a feasible/effective process for checking those concerns  Finalize procedures and provide training  Analyze ongoing inspection results (and update procedures if necessary)  Monitor downstream defect profile  Write final report Status: Reviewed sample models & processes; Constructed draft PBI techniques; Created draft training course

18. 18PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Evaluation  Success metric: Effectiveness of PB inspection in comparison to past inspections  Current JWST approach  Project documents  Issues written against missing/incorrect/incomplete/inconsistent content in source artifacts  SRM Reviews  IV&V team maintains SRM change logs, action item lists, and other sources for the infusion team to evaluate relative effectiveness  IV&V team documents discrepancy when comparing SRM against val/ver target  Strategy is to maintain current approach (issue reports, review records), but to track how PBI assisted in identifying the items  Likely to involve follow-up conversation/interviews to assess results and collect ancillary observations from the IV&V team  Will balance this with objective to minimize impact to IV&V activities Infusion Plan: Path Forward

19. 19PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162  Use measurement results to further optimize, e.g.,  Consistently missing a certain type of defect?  Add a new defect type and associated questions to relevant perspectives  Missing a whole set of defects?  Consider whether a new perspective should be added.  Do we never find defects of type x? Are there any perspectives that don’t seem to add much that the other perspectives didn’t find?  Consider deleting the relevant defect type or perspective.  Do we consistently face a lot of problems of type x?  Add more reviewers using relevant perspectives Future Considerations

20. 20PBI_SAS_08_Tech_ShullSeptember 2008MAC-T IVV-08-162 Forrest Shull fshull@fc-md.umd.edu 301-403-8970 Contact information Kurt Woodham Kurt.Woodham@L-3com.com 757-644-5807