1. kimmens@vub.ac.be1December 2000 Advanced OOSA State of the Art on Software Architecture Declarative Meta Programming Session 5: Lightweight Architectural Tools Nantes, EMOOSE 2000–2001 Dr. Kim Mens, PROG, VUB

2. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 2 Course overview ÊIntroduction ËSoftware Architecture ÌDeclarative Meta Programming ÍSoftware Classification ÎLightweight Architectural Tools ÏAutomated Architectural Conformance Checking ÐAn Architecture-driven Software Development Tool ÑAssignments

3. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 3 Session overview n Lightweight architectural tools n Lightweight source-model extraction n Software reflexion models n Conceptual module querying n Summary n Lightweight architectural tools n Lightweight source-model extraction n Software reflexion models n Conceptual module querying n Summary

4. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 4 Lightweight architectural tools n Lightweight source-model extraction u automatically reverse engineer high-level source models from implementation n Software reflexion models u automatically check conformance of extracted source models to desired high-level models n Conceptual module querying u support reengineering u allows software engineers to perform queries simultaneously about both existing and desired source structure u a conceptual module is a set of lines of source that are treated as a logical unit

5. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 5 References n Lecture “Connecting Task to Source” by Gail Murphy. n Murphy, Notkin & Sullivan. Software Reflexion Models: Bridging the Gap Between Source and High-Level Models. In Proceedings of the FSE’95 Symposium, pp. 18-28. ACM Press, 1995.Software Reflexion Models: Bridging the Gap Between Source and High-Level Models. n Murphy & Notkin. Reengineering with Reflexion Models: A Case Study. In IEEE Computer 30, 8, pp. 29-36, 1997.Reengineering with Reflexion Models: A Case Study. n Reflexion Model tool: http://www.cs.ubc.ca/~murphy/software/rmtool/index.html n Murphy & Notkin. Lightweight Lexical Source Model Extraction. ACM Transactions on Software Engineering and Methodology, vol. 5, no. 3, July 1996, pp. 262-292.Lightweight Lexical Source Model Extraction. n Lightweight Source Model Extraction tool: http://www.cs.ubc.ca/~murphy/software/lsme/index.html n Baniassad & Murphy. Conceptual Module Querying for Software Reengineering. In Proceedings of the 20th International Conference on Software Engineering, pp. 64-73. IEEE, 1998.Conceptual Module Querying for Software Reengineering. RECAP

6. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 6 n Lightweight conformance checking techniques n Lightweight source model extraction n Software reflexion models n Conceptual module querying n Summary Reference: n Murphy & Notkin 1996. Lightweight Lexical Source Model Extraction. ACM Transactions on Software Engineering and Methodology, vol. 5, no. 3, July 1996, pp. 262-292.Lightweight Lexical Source Model Extraction. Session overview

7. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 7 Lightweight source-model extraction n Source-model extraction u automatically extract high-level info from implementation call graphs, file dependencies, event interactions, cross-reference lists, program-dependence graphs, … u independent of implementation language C, C++, CLOS, Eiffel, … u information can be extracted from source code, macros, comments, … n Lightweight u small and easy to write specifications of desired source model u lexical specifications

8. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 8 Lexical approach n Scan only for constructs relevant to the source model u for example: extracting a call graph only look for calls in the source code skip all data declarations, control constructs, … n Similar to lexically-based tools like ‘grep’ n But additional support for u matching source constructs in context u and across multiple lines n Alternative approach: parsing u more precise u but places syntactic constraints on the source code u and building/modifying parsers is not trivial

9. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 9 Approximate information n Extracted information is approximate u not all intended constructs may be extracted u some unintended constructs may be extracted n Approach trades precision in extraction for increased u efficiency more efficient than parsing but less precise false positives and missing information u flexibility information can be extracted from code, macros, comments, … u tolerance information can be extracted from non-compilable code, code that is not linked, …

10. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 10 Architecture of the source-model extraction system Source-model specification Source Code Source Model Scanner Generate Analyser Generate Intermediate Representation

11. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 11 Source model specification n Pattern definitions: u the patterns of interest in the source code n Action definitions: u actions to execute when a pattern is matched in the code n are used to generate a n Post-processing analysis operations: u operations that combine local info extracted from individual source code files into a global source code model n are used to generate an Source-model specification defines Scanner Analyser

12. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 12 Source model specification: pattern definitions n Code patterns are described as regular expressions For example: extracting function names from C source [ ] \( [ { }+ ] \) [ { ; }+ ] \{ u optional type specification, followed by u name of the function and left parenthesis, followed by u optional list of formal arguments and right parenthesis, u optional list of declarations of the formal argument types, u and finally a curly brace to start the function body n This pattern does not capture all function definitions  e.g. it misses type declarations of the form int * x; u but can be refined to capture 99% of the cases

13. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 13 Source model specification: action definitions n Attach action code to patterns to be executed when the pattern is matched in the source code n For example: print names of extracted functions [ ] @ write ( functionName ) @ \( [ { }+ ] \) [ { ; }+ ] \{

14. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 14 The generated tools n More info on the generated scanner and analyser, see Murphy & Notkin 1996 Scanner Generate Analyser Generate

15. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 15 n Lightweight conformance checking techniques n Lightweight source model extraction n Software reflexion models n Conceptual module querying n Summary Reference: u Murphy, Notkin & Sullivan. Software Reflexion Models: Bridging the Gap Between Source and High-Level Models. In Proceedings of the FSE’95 Symposium, pp. 18-28. ACM Press, 1995.Software Reflexion Models: Bridging the Gap Between Source and High-Level Models. u Murphy & Notkin. Reengineering with Reflexion Models: A Case Study. In IEEE Computer 30, 8, pp. 29-36, 1997.Reengineering with Reflexion Models: A Case Study. Slides mainly based on lecture “Connecting Task to Source” by Gail Murphy. Session overview

16. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 16 Software reflexion models n A typical estimation scenario n Software reflexion model n Software reflexion model technique n Experiences n A typical estimation scenario n Software reflexion model n Software reflexion model technique n Experiences

17. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 17 n You are asked to provide, within 5 days, an estimate of the effort required to modify an implementation of a Unix operating system to page over a distributed network NetBSD Kernel Source Code Slide taken from lecture “Connecting Task to Source” by Gail Murphy A typical estimation scenario

18. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 18 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Software visualization

19. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 19 Practical problems n Difficult to understand software structure directly from the (very large) source code n Visualisation tools suffer from essentially the same problem n Software reflexion models u limit the scope by comparing a (small) desired diagram with the source code u do not work on the source code directly but on an extracted source model

20. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 20 Software reflexion models n A typical estimation scenario n Software reflexion model n Software reflexion model technique n Experiences

21. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 21 n High-level model of a Unix virtual memory subsystem drawn by a domain expert Slide taken from lecture “Connecting Task to Source” by Gail Murphy High-level model

22. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 22 Software reflexion model n After comparing the high-level model with an (extracted) source model, a software reflexion model is generated n This is an annotated version of the high-level model u convergences discovered interactions in the source that were expected by the high-level model u divergences discovered interactions that were not expected u absences interactions that were expected but were not found n Each of the lines is annotated with a number indicating the number of discovered interactions

23. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 23 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Software reflexion model

24. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 24 Software reflexion models n A typical estimation scenario n Software reflexion model n Software reflexion model technique n Experiences

25. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 25 Slide taken from lecture “Connecting Task to Source” by Gail Murphy 4 Reflexion Model Extraction Tool 2 Software System Artefacts Source Model Mapping 3 RM Tools Model 1 Software reflexion model technique

26. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 26 Slide taken from lecture “Connecting Task to Source” by Gail Murphy 1. State a high-level model n Syntactic n Multiple relations n “everyone has one or more”

27. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 27 Slide taken from lecture “Connecting Task to Source” by Gail Murphy 2. Extract a source model n Use existing tools (e.g., cflow, Field, etc.) n Lightweight lexical source model extractor (Murphy/Notkin) n May contain multiple relations

28. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 28 Source Model EntitiesHigh-Level Model Entities file=pager.cPager file=vm_map.*VirtualAddressMaint. dir=vm func=active VMPolicy Slide taken from lecture “Connecting Task to Source” by Gail Murphy 3. State a declarative mapping n Name source model entities using: u physical and logical software structure u regular expressions n Many-to-many mapping

29. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 29 Slide taken from lecture “Connecting Task to Source” by Gail Murphy 4. Investigate a reflexion model

30. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 30 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Iteration n Want to investigate the data relationships? u augment the source model u update the mapping: var=queue.*active VMPolicy u recompute...

31. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 31 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Refined reflexion model

32. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 32 Software reflexion models n A typical estimation scenario n Software reflexion model n Software reflexion model technique n Experiences

33. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 33 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Experience

34. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 34 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Excel: experimental reengineering n A Microsoft engineer computed reflexion models several times a day for four weeks u 120,000 calls and global variable references u map file with over 1000 entries u high-level model with 15 entities and 96 interactions u 4 minutes to compute on a 486 n Some lessons learned: u map files evolved to be larger than expected u scale places pressure on managing the information

35. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 35 n Lightweight conformance checking techniques n Lightweight source model extraction n Software reflexion models n Conceptual module querying n Summary Reference: Elisa L. A. Baniassad & Gail. C. Murphy. Conceptual Module Querying for Software Reengineering. In Proceedings of the 20th International Conference on Software Engineering, pp. 64-73. IEEE, 1998.Conceptual Module Querying for Software Reengineering. Slides mainly based on lecture “Connecting Task to Source” by Gail Murphy. Session overview

36. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 36 Conceptual module querying n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules

37. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 37 Goal n Support software reengineering u reengineering is more than source code analysis u reengineering requires reasoning about source code… identification and extraction of new software components from existing source code u …but also about desired source structure interactions of new components with remaining source code interactions among newly-formed components n Allow software engineer to simultaneously perform queries about the existing and desired source structure n A conceptual module is a set of lines of source that are treated as a logical unit

38. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 38 Conceptual modules are like software classifications Both conceptual modules and software classifications n group a set of related source code entities n represent some logical or conceptual unit n may cross-cut the source code u e.g., code lines included in a CM may be split across existing procedure boundaries

39. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 39 Conceptual module querying n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules

40. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 40 A typical reengineering scenario Slide taken from lecture “Connecting Task to Source” by Gail Murphy

41. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 41 Reengineering scenario... main procedure: Input Pipe sort procedure: Slide taken from lecture “Connecting Task to Source” by Gail Murphy

42. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 42 Conceptual module querying n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules

43. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 43 Different reengineering approaches n Five different program understanding tools: u xrefdb (part of Field) cross-reference database tool u Unravel slicing tool u Lackwit tool based on type-inferencing u Software reflexion model u Conceptual module tool n Task: u create an input pipe component from the GNU sort program.

44. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 44 Cross-reference database n Identify variables of interest n For each variable u where is the variable declared? u where is the variable referenced? n Collate results n Repeat n Result: Using Field’s xrefdb, 126 lines were returned of which only 30% were relevant Slide taken from lecture “Connecting Task to Source” by Gail Murphy

45. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 45 Slicer n Compute backward slices on variables in pre-identified lines of code n Slice computation took several hours; only partial were viewed n Result: Slices computed with Unravel were > 750 nodes in size and included many irrelevant procedures Slide taken from lecture “Connecting Task to Source” by Gail Murphy

46. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 46 Type inferencer n Compute which procedures affect the values of particular variables n The Lackwit tool produces graphs summarizing how values are transmitted through a program n Provides information on relevant procedures but not on relevant source code lines n Result: Lackwit graphs are often large and are reported in terms of the existing structure Slide taken from lecture “Connecting Task to Source” by Gail Murphy

47. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 47 Software reflexion model Difficult to ascertain interface of the module No support for querying the source model Syntactic comparison Slide taken from lecture “Connecting Task to Source” by Gail Murphy

48. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 48 Conceptual module querying n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules

49. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 49 Conceptual module approach Query Source code Source Model Extraction Tool Source Model Form a CM Conceptual Module Tool Interface Analysis Query Results Slide taken from lecture “Connecting Task to Source” by Gail Murphy

50. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 50 Source model n Instead of reasoning directly about source code, the approach reasons about an extracted source model n Source model is directed towards a procedural language and consists of three relations: u variable dependence relation uses and definitions of variables (described by line number) u control transfer relation procedure calls (line number of call site and called procedure) u procedure start relation source line number on which a procedure starts Source code Source Model Extraction Tool Source Model

51. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 51 Query language n Supports formation (definition) of conceptual modules n Supports investigation of interactions involving conceptual modules Query Form a CM Conceptual Module Tool

52. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 52 Forming a conceptual module n Map lines of source code to a logical module n Two ways to map the code: u by specifying line numbers (individual, ranges, etc.) u by specifying pieces of existing logical structure (i.e., variables or procedures) e.g., the fp variable n Each module has a name n Formation can be iterative For sort, we ended up including about 24 lines in the input pipe conceptual module. Slide taken from lecture “Connecting Task to Source” by Gail Murphy

53. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 53 Conceptual modules are like virtual software classifications Just like software classifications, conceptual modules can be specified either n extensionally, i.e. by explicit enumeration of the elements (line numbers) that belong to it n intentionally, by using a more semantic specification n This is similar to the distinction between ordinary and virtual software classifications

54. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 54 n First define conceptual module n Tool then scans the source model to determine the lines of code with occurrences of u input, local and output variables u calls from CM to procedures in existing source n These lines are included as part of definition of the CM Forming a conceptual module: interface analysis Conceptual Module Tool Source Model Interface Analysis

55. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 55 n Picture of interactions between CM’s

56. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 56 Interface analysis rules n Input variable u used on line inside module, but defined on line outside module n Output variable u defined on line inside module, but used on line outside module n Local variable u Two-phase analysis for local variables: 1. Use-def pairs: all uses & defs in module implies local variable. 2. Only uses or defs: consider identified input/output variables; promote to local if all uses and defs are inside module. n Control transfer u call to procedure outside module

57. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 57 Input Variables: sortalloc, main.ofp, main.minus, … Output Variables: main.mergeonly, sort.ofp, sortalloc, … Local Variables: main.files, main.nfiles, sort.files Control Transfers: xmalloc at sort.c 1796 fillbuf at sort.c 248 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Interface analysis: an example n Local (interface) analysis is used to summarize how the module interacts directly with the existing code

58. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 58 Conceptual module querying n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules

59. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 59 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Querying about conceptual modules n Once one or more conceptual modules are formed, the re-engineer typically needs to perform queries: u How do conceptual modules relate to each other? u How do conceptual modules relate to existing source? n Tool provides pre-coded queries and a programmable interface through which a user can code queries. Query Form a CM Conceptual Module Tool

60. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 60 Conceptual module relationships A B direct def use A B indirect def use AB overlap A B contains Slide taken from lecture “Connecting Task to Source” by Gail Murphy n 4 types of useful queries

61. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 61 Conceptual module relationships n Simply determining the presence of a relationships between two modules is generally not sufficient to perform a reengineering task n Therefore, a programmable interface is provided which allows the engineer to access u input, output and local variable names u defs and uses of conceptual module variables u lines of code spanned by a conceptual module u calls made to and by (code in) a conceptual module u relationship info between conceptual modules including common line numbers, variables and calls

62. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 62 Programmable Interface SET common = new SET(); // Get the use-def chains for all input and local variables of that module. Module first = (Module)Module.ModuleTable.elementAt(0); common=DefUse.GetFullUseDefChain(first); for(int i=1; i<Module.ModuleTable.size(); i++) { // Get the use-def chains for the next module Module current = (Module)Module.ModuleTable.elementAt(i); SET curr_chain = DefUse.GetFullDefUseChain(current); // Intersect the chains to determine common definition points common = DefUse.INTERSECTION(common, curr_chain); } common.print(); Slide taken from lecture “Connecting Task to Source” by Gail Murphy

63. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 63 Conceptual module querying n Goal n A typical reengineering scenario n Different reengineering approaches n Conceptual module approach n Querying about conceptual modules n Experiences with conceptual modules

64. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 64 Experiences with conceptual modules SUIF xrefdb SUIF = tools built on SUIF provides use-def pairs xrefdb = Field’s xrefdb provides uses & defs Slide taken from lecture “Connecting Task to Source” by Gail Murphy

65. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 65 Query Context and Form n Context is expressed in 2 parts: u A good re-engineering tool should allow a software engineer to express the context of a query being performed u identify region of source over which to query conceptual module identifies region u restrict the region for which results are reported interface analysis summarizes local results n Form includes both input and output: u some tasks require queries over grouped items u report results in terms of task results are reported in terms of target structure; can use conceptual module structure to query against source Slide taken from lecture “Connecting Task to Source” by Gail Murphy

66. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 66 Session overview n Lightweight conformance checking techniques n Lightweight source model extraction n Software reflexion models n Conceptual module querying n Summary

67. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 67 Partial and approximate techniques n Each of these characteristics can be an effective way to attack scale. n These characteristics can be combined to provide software engineers with a “smoother” means of managing source investigations. Bottom line for most developments is that time is money. approximate conservative Slide taken from lecture “Connecting Task to Source” by Gail Murphy

68. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 68 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Summary Software Reflexion Model “Definitely confirmed suspicions about the structure of Excel. Further, it allowed me to pinpoint the deviations.... very easy to ignore stuff that is not interesting and thereby focus on the part of Excel I want to know more about.”  Microsoft Engineer Conceptual Module “Not only did the tool verify the independent nature of the ZDD functionality and allow me to rip out all that code, but, the process of using your tool forced me to analyze and understand the code in a way that I had not been doing  and that ultimately it very quickly gave me the perspective I needed.”  Yvonne Coady

69. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 69 Slide taken from lecture “Connecting Task to Source” by Gail Murphy Summary n Demonstrated benefits of task-aware program understanding techniques u current techniques are structurally task-aware n Demonstrated role for approximate information u reflexion model technique makes engineer responsible u conceptual modules takes some of responsibility n Goal is to get to “what-if” tools that would allow engineers to leverage, cost-effectively, connections between design and source

70. State of the Art on Software Architecture — Declarative Meta ProgrammingSession 5, page 70 Session overview n Lightweight conformance checking techniques n Lightweight source model extraction n Software reflexion models n Conceptual module querying n Summary Time for a break