Using Natural Language Program Analysis to Locate and understand Action-Oriented Concerns David Shepherd, Zachary P. Fry, Emily Hill, Lori Pollock, and K. Vijay-Shanker Presented By: Paul Heintzelman

Global Concepts Concept assignment problem Hybrid of structural and natural language information Concern Comprehension Action-oriented relations between identifiers –Represented by Action-oriented identifier graph model (AOIG)

Why Action-Oriented Concerns In OOP –Code is organized by objects Objects are nouns Objects and actions conflict –Code organized by objects causes actions to be scattered Therefore in OOP action-oriented concerns tend to be scattered and more difficult to locate

Paper Contributions AOIG –Interactive query expansion algorithm –A result graph construction algorithm –An Eclipse plug-in Evaluation –Comparison of search effectiveness of tools –Per task analysis –Comparison of user effort

AOIG

State of the Art Search-based approaches –Lexical searches Lead to over-generalized searches –Information retrieval Does not separate verbs and objects Uses word frequency Program navigation –Uses structural information e.g. call, inheritance graphs... –Accurate but difficult to seed Dynamic approaches –Requires test case to enact concept

Challenges Map high level concepts to queries –Aid user in mapping concepts Inability to search with high precision and recall –Search NLP representation of concern Understanding large result sets –Return results in an explorable graph

Overview of Approach User formulates a query –Query must include verb-direct object pairings User expands query –Recommendations based on query words and source code Searches the AOIG –Interact with result graph

Independent Variables Search Tools –Find-Concept –ELex built in Eclipse search –GES Google Eclipse search (modified) Search Tasks –Application concept pairing Human Subjects –13 professional programmers –5 grad students

Applications –4 large open source java projects 9 concepts taken from bug reports –1 training application 2 concepts Application Concept Pairing

Forming the Initial Query User generates abstract initial query –e.g. “automatically finish the word” User decomposes abstract query into verb-direct object pairs –e.g. “finish” and “word” Find concept maintains both verb query and direct object query Initial query expansion –User is presented with alternative forms of words in both queries

Query Expansion Iterative steps –Generate recommended list Similar semantics is weighted more heavily than similar use 10 ranked recommendations –User examines recommendations User selects words to add to queries User can view a list of methods fitting the current queries Stop when user is satisfied –Augment user query with get, set, execute, construct Use AOIG to map verb-direct object pairs to source code –Generate result graph

Word Recommendation Similar semantics –Stemming Recommends different forms of words in either list e.g. If “finish” is in verb-query, “finished” will be recommended –Synonyms Recommend a word if synonym exists in either list e.g. Recommend “complete” if “finish” is in list Similar use –Recommend words that occur near words in either query –e.g. Recommend “word” if “complete” is in the verb query and “complete word” is in the AOIG

Evolution of a Query

Result Graph

Find-Concept Process

Research Questions Which search tool is most effective at locating concerns by forming and executing a query? Which search tool requires the least amount of human effort to form an effective query?

Evaluation Effectiveness –Use the harmonic mean of precision and recall (f-measure) (2 * precision * recall)/(precision + recall) –Result set is compared to evaluation set Evaluation set is 90% generated by a member unfamiliar with the work of this paper Effort –Measured amount of time required to form each query

Experimental Setup Training –Subjects are guided through the use of each tool on the two training tasks Task setup –Users are presented concepts in a visual form –Users confirm that they understood each task

Experimental Procedure 9 tasks 18 programmers 6 groups 6 of every task tool combination

Results Find-Concept vs. ELex –Consistently outperformed ELex Find-Concept vs. GES –Outperformed GES on 4 tasks –Outperformed by GES on 2 tasks AOIG to blame? –Performed equally to GES on 3 tasks

Effectiveness

Effort Human Effort was very similar with all tools

Threats to Validity The selected tasks favored one tool –Concerns selected from bug reports Evaluation sets created for evaluation –90% generated by member unfamiliar with work Results may not generalize to all Java applications –Tested on reasonably-sized applications Results may not generalize to all types of concepts

Conclusion Interactive query expansion algorithm Graph construction algorithm Find-Concept performs well against state of the art tools All evaluated tools required similar human effort

Future Work Create a more effective AOIGBuilder Evaluate the effect of application’s quality and size on results Evaluate the effect of incorporating naming conventions Perform a study on how many tasks focus on actions Automate query expansion

Additional threats to validity Effort and Effectiveness are not really independent Relies heavily on unjustified heuristic –Augmenting query Search tools are often used in conjunction with structural tools