1. Automated Requirements Traceability Study of the Analyst Presented by Jeff Holden Advisor Alex Dekhtyar

2. What is requirements traceability? “The ability to describe and follow the life of a requirement, in both a forwards and backwards direction”. [gotel]

3. Requirements process Output of tracing generates Requirements Traceability Matrix (RTM)  Specifies connections between low and high level elements

4. Why care about tracing? Verification & Validation (V&V/IV&V)  Required for mission & safety critical systems Test coverage analysis Change impact analysis Reverse engineering

5. Typical tracing process Manual tracing  Norm for industry  Laborious & error-prone Automated systems  Use information retrieval methods  Quick, can produce good results  Mission critical systems need verified

6. Semi-Automated tracing Tracing tool generates candidate RTM Analyst validates the RTM to produce a final RTM  Quicker, analyst validates rather than creates.

7. Typical view on tracing quality Precision  Percent of links found that are true links. Recall  Percent of true links found. F-# measure  Harmonic mean between precision & recall  Use F-2: weights recall heavier than precision Easier for analyst to resolve errors of commission than omission.

8. Does better candidate RTM lead to better final RTMs? Proposed in 2005 Initial study: 4 users  Not statistically significant  Showed an interesting finding, better may not be better.

9. Pilot study findings

10. Is high quality good? Initial experiment David Cuddeback  35 responses  Old RETRO  Showed “region” trends

11. My additions Expanded automated study to new RETRO  Simpler, more user-friendly UI  Enhanced logging capabilities MORE DATA!!! Conducted manual tracing study  Utilized the same data set

12. RTM locations

13. RTM submissions

14. Region trends – low recall, low precision Low precision, low recall  Improvement of precision & recall  Maintain ~same RTM size

15. Region trends – high recall, low precision Low precision, high recall  Focus on removing links  Improve precision, some time at cost of recall

16. Region trends – low recall, high precision High precision, low recall  Opposite trend, focus on adding links  Increase recall, normally at cost of precision

17. Region trends – high recall, high precision High precision, high recall  Almost all decrease quality of final RTM

18. Preliminary results!!! Good initial != Good final No consensus on “true RTM” Final RTM converge on “hotspot” Automated tools may assist in finding errors of omission better than manual! Its hard to get good precision + recall!

19. Contributions (so far) Improved experimental RETRO.NET Expanded upon experimental framework to work with other tools & other tracing methods MORE DATA!!! (52 more data points)  Up to ~90 data points total Currently writing up & submitting early findings

20. Planned next steps Work with existing IR methods, filters, and feedback mechanisms.  Determine if real methods can get “good” results  Validate findings on real IR methods in similar experimental setup Conduct usability study on RETRO.NET

21. Thesis goal Create a tracing tool that analysts can use to reliably generate quality final RTM in a efficient manner.

22. Questions?!?