1. Using Natural Language Program Analysis to Locate and understand Action-Oriented Concerns David Shepherd, Zachary P. Fry, Emily Hill, Lori Pollock, and K. Vijay-Shanker Presented By: Paul Heintzelman

2. Global Concepts Concept assignment problem Hybrid of structural and natural language information Concern Comprehension Action-oriented relations between identifiers –Represented by Action-oriented identifier graph model (AOIG)

3. Why Action-Oriented Concerns In OOP –Code is organized by objects Objects are nouns Objects and actions conflict –Code organized by objects causes actions to be scattered Therefore in OOP action-oriented concerns tend to be scattered and more difficult to locate

4. Paper Contributions AOIG –Interactive query expansion algorithm –A result graph construction algorithm –An Eclipse plug-in Evaluation –Comparison of search effectiveness of tools –Per task analysis –Comparison of user effort

5. AOIG

6. State of the Art Search-based approaches –Lexical searches Lead to over-generalized searches –Information retrieval Does not separate verbs and objects Uses word frequency Program navigation –Uses structural information e.g. call, inheritance graphs... –Accurate but difficult to seed Dynamic approaches –Requires test case to enact concept

7. Challenges Map high level concepts to queries –Aid user in mapping concepts Inability to search with high precision and recall –Search NLP representation of concern Understanding large result sets –Return results in an explorable graph

8. Overview of Approach User formulates a query –Query must include verb-direct object pairings User expands query –Recommendations based on query words and source code Searches the AOIG –Interact with result graph

9. Independent Variables Search Tools –Find-Concept –ELex built in Eclipse search –GES Google Eclipse search (modified) Search Tasks –Application concept pairing Human Subjects –13 professional programmers –5 grad students

10. Applications –4 large open source java projects 9 concepts taken from bug reports –1 training application 2 concepts Application Concept Pairing

11. Forming the Initial Query User generates abstract initial query –e.g. “automatically finish the word” User decomposes abstract query into verb-direct object pairs –e.g. “finish” and “word” Find concept maintains both verb query and direct object query Initial query expansion –User is presented with alternative forms of words in both queries

12. Query Expansion Iterative steps –Generate recommended list Similar semantics is weighted more heavily than similar use 10 ranked recommendations –User examines recommendations User selects words to add to queries User can view a list of methods fitting the current queries Stop when user is satisfied –Augment user query with get, set, execute, construct Use AOIG to map verb-direct object pairs to source code –Generate result graph

13. Word Recommendation Similar semantics –Stemming Recommends different forms of words in either list e.g. If “finish” is in verb-query, “finished” will be recommended –Synonyms Recommend a word if synonym exists in either list e.g. Recommend “complete” if “finish” is in list Similar use –Recommend words that occur near words in either query –e.g. Recommend “word” if “complete” is in the verb query and “complete word” is in the AOIG

14. Evolution of a Query

15. Result Graph

16. Find-Concept Process

17. Research Questions Which search tool is most effective at locating concerns by forming and executing a query? Which search tool requires the least amount of human effort to form an effective query?

18. Evaluation Effectiveness –Use the harmonic mean of precision and recall (f-measure) (2 * precision * recall)/(precision + recall) –Result set is compared to evaluation set Evaluation set is 90% generated by a member unfamiliar with the work of this paper Effort –Measured amount of time required to form each query

19. Experimental Setup Training –Subjects are guided through the use of each tool on the two training tasks Task setup –Users are presented concepts in a visual form –Users confirm that they understood each task

20. Experimental Procedure 9 tasks 18 programmers 6 groups 6 of every task tool combination

21. Results Find-Concept vs. ELex –Consistently outperformed ELex Find-Concept vs. GES –Outperformed GES on 4 tasks –Outperformed by GES on 2 tasks AOIG to blame? –Performed equally to GES on 3 tasks

22. Effectiveness

23. Effort Human Effort was very similar with all tools

24. Threats to Validity The selected tasks favored one tool –Concerns selected from bug reports Evaluation sets created for evaluation –90% generated by member unfamiliar with work Results may not generalize to all Java applications –Tested on reasonably-sized applications Results may not generalize to all types of concepts

25. Conclusion Interactive query expansion algorithm Graph construction algorithm Find-Concept performs well against state of the art tools All evaluated tools required similar human effort

26. Future Work Create a more effective AOIGBuilder Evaluate the effect of application’s quality and size on results Evaluate the effect of incorporating naming conventions Perform a study on how many tasks focus on actions Automate query expansion

27. Additional threats to validity Effort and Effectiveness are not really independent Relies heavily on unjustified heuristic –Augmenting query Search tools are often used in conjunction with structural tools