1. Robust Requirements Tracing Via Internet Tech:Improving an IV&V Technique SAS 2004July 20, 2004 Alex Dekhtyar Jane Hayes Senthil Sundaram Ganapathy Chidambaram Sarah Howard Department of Computer Science University of Kentucky

2. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

3. Who Is Who Sponsor NASA IV&V Center, Fairmont, WV Principal Investigators: Alexander Dekhtyar Jane Hayes Ph. D. Student: Senthil Karthekian Sundaram* M.S. Student: Sarah Howard Past Undergraduate Students: James Osborne* Rijo Jose Thozhal Subcontractor: SAIC * Supported by the NASA grant

4. The Problem How can we automate tracing requirments during IV&V? Relevance to NASA  Alleviate work of NASA IV&V analysts  Improve quality of IV&V for NASA software Importance/Benefits Importance/Benefits Improve analyst productivity on one of the most time-consuming IV&V tasks

5. Approach Use Information Retrieval Techniques for Requirements Tracing Build RETRO (REquirements TRacing On-target) Evaluate performance  TF-IDF, Thesaurus, Probabilistic IR, LSI  Analyst Feedback  Metrics  Special-purpose requirments tracing tool  Standalone version  Integrated with SAIC’s SuperTracePlus  MODIS, LOFAR, CM-1 datasets

6. representation Approach: IR for Requirements Tracing Matching algorithm 1 2 3 Design Document 1. 2. 3. Analyst Requirements Document Yes No Feedback

7. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

8. Methods TF - IDF TF = Term Frequency IDF = Inverse Document Frequency (rare terms) Latent Semantic Indexing (LSI) term x document => “factor” x document #”factors” << # terms Enhancements: Thesaurus Feedback Processing Filtering

9. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

10. Metrics N - number of low-level requirements; M - number of high-level requirments; Hits - number of correct candidate links Strikes - number of false positives Misses - number of missed links

11. Metrics N - number of low-level requirements; M - number of high-level requirments; Hits - number of correct candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses

12. Metrics N - number of low-level requirements; M - number of high-level requirments; Hits - number of correct candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses Selectivity = Hits + Strikes M * N

13. Metrics N - number of low-level requirements; M - number of high-level requirements; Hits - number of correct candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses Selectivity = Hits + Strikes M * N AvgH = average relevance of Hits AvgS = average relevance of Strikes DiffR = AvgH - AvgS

14. Metrics N - number of low-level requirements; M - number of high-level requirments; Hits - number of correct candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses Selectivity = Hits + Strikes M * N AvgH = average relevance of Hits AvgS = average relevance of Strikes DiffR = AvgH - AvgS Lag(Hit) = # Strikes for high-level requirement with Higher relevance Lag = average Lag(Hit) over all Hits

15. Metrics N - number of low-level requirements; M - number of high-level requirments; Hits - number of correct candidate links Strikes - number of false positives Misses - number of missed links Precision = Hits Hits + Strikes Recall = Hits Hits +Misses Selectivity = Hits + Strikes M * N Breakpoint = (threshold, Precision, Recall), s.t. Precision = Recall

16. Metrics Precision: signal - to - noise Recall: “coverage” Selectivity: improvement in # of comparisons vs. exhaustive search AvgH, AvgS, DiffR, Lag - separation between Hits and Strikes in candidate link lists Breakpoints - effects of filtering

17. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

18. RETRO: REquirements TRacing On-target

19. RETRO Architecture documents Build Representation IR toolbox Filter Feedback processor Analyst

20. RETRO + SuperTracePlus requirements documents SFEP RETRO Build Representation RETRO IR Toolbox STP Interactive Link Anlysis RETRO Feedback STP Report Generation Traceability Reports A STP RETRO Analyst Review

21. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

22. The Universe of Tests methodthesaurusthresholdfeedback TF-IDF LSI* Yes No Top 1 Top 2 Top 3 Top 4 [0.0…0.5] X XX * LSI: number of dimensions + – low-level documents – high+low-level documents – high-level, low-level documents separately

23. Datasets MODIS 20 high-level 49 low-level 41 true links CM-1 ~200 high-level ~300 low-level # true links - under construction

24. MODIS, TF-IDF, Thesaurus Top2 Feedback

25. MODIS, TF-IDF, Thesaurus Top2 Feedback Filtering at Iteration 0 Breakpoint

26. MODIS, TF-IDF, Thesaurus Top2 Feedback

27. Above 70%

28. MODIS, TF-IDF, No Thesaurus Top3, Feedback

29. MODIS, Comparing Feedback Traces

30. Above 70%

31. MODIS, Secondary Measures

33. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

34. NASA Research Information Technology Readiness Level -- 6.5 for RETRO Integrated with existing software system Engineering feasibility demonstrated Limited documentation available Most functionality available for demonstration and test Most software bugs removed Potential applications  Tracing bug reports to code  Identifying related/duplicate bug reports Ease of finding, or availability of, data or case studies  Data available  Issue is answerset Barriers to research or application  Answerset availability  IV&V analysts for human factors studies Publications  Paper accepted to RE 2004  1 journal paper submitted, one in progress

35. Outline Requirements Tracing and Information Retrieval Methods Metrics RETRO Experimental Results NASA Research information Technology Readiness Level Potential applications Ease of finding, or availability of, data or case studies Barriers to research or application Future work

36. Next Steps, Conclusions, Plans, Ideas IR methods work : need to implement more Productize RETRO (Check!) Data Integration with existing tools (Check!) Other IV&V problems may be alleviated Study “human factors”