1. Refactoring Erlang Programs Huiqing Li Simon Thompson University of Kent

2. Overview What is refactoring? Examples The process of refactoring Tool building and infrastructure What is in Wrangler … demo Latest advances: data, processes, erlide.

3. Introducing refactoring

4. Soft-ware There’s no single correct design … … different options for different situations. Maintain flexibility as the system evolves.

5. Refactoring Refactoring means changing the design or structure of a program … without changing its behaviour. RefactorModify

6. Examples

7. Generalisation -module (test). -export([f/1]). add_one ([H|T]) -> [H+1 | add_one(T)]; add_one ([]) -> []. f(X) -> add_one(X). -module (test). -export([f/1]). add_one (N, [H|T]) -> [H+N | add_one(N,T)]; add_one (N,[]) -> []. f(X) -> add_one(1, X). -module (test). -export([f/1]). add_int (N, [H|T]) -> [H+N | add_int(N,T)]; add_int (N,[]) -> []. f(X) -> add_int(1, X). Generalisation and renaming

8. Generalisation -export([printList/1]). printList([H|T]) -> io:format("~p\n",[H]), printList(T); printList([]) -> true. printList([1,2,3]) -export([printList/2]). printList(F,[H|T]) -> F(H), printList(F, T); printList(F,[]) -> true. printList( fun(H) -> io:format("~p\n", [H]) end, [1,2,3]).

9. Generalisation -export([printList/1]). printList([H|T]) -> io:format("~p\n",[H]), printList(T); printList([]) -> true. -export([printList/1]). printList(F,[H|T]) -> F(H), printList(F, T); printList(F,[]) -> true. printList(L) -> printList( fun(H) -> io:format("~p\n", [H]) end, L).

10. Asynchronous to synchronous pid! {self(),msg} {Parent,msg} -> body pid! {self(),msg}, receive {pid, ok}-> ok {Parent,msg} -> Parent! {self(),ok}, body

11. Refactoring

12. Refactoring = Transformation + Condition Transformation Ensure change at all those points needed. Ensure change at only those points needed. Condition Is the refactoring applicable? Will it preserve the semantics of the module? the program?

13. Transformations fullstopone

14. Condition > Transformation Renaming an identifier "The existing binding structure should not be affected. No binding for the new name may intervene between the binding of the old name and any of its uses, since the renamed identifier would be captured by the renaming. Conversely, the binding to be renamed must not intervene between bindings and uses of the new name."

15. Which refactoring exactly? Generalise f by making 23 a parameter of f: f(X) -> Con = 23, g(X) + Con + 23. This one occurrence? All occurrences (in the body)? Some of the occurrences … to be selected.

16. Compensate or crash? -export([oldFun/1, newFun/1]). oldFun(L) -> newFun(L). newFun(L) -> … …. -export([newFun/1]). newFun(L) -> … …. or?

17. Refactoring tools

18. Tool support Bureaucratic and diffuse. Tedious and error prone. Semantics: scopes, types, modules, … Undo/redo Enhanced creativity

19. Semantic analysis Binding structure Dynamic atom creation, multiple binding occurrences, pattern semantics etc. Module structure and projects No explicit projects for Erlang; cf Erlide / Emacs. Type and effect information Need effect information for e.g. generalisation.

20. Erlang refactoring: challenges Multiple binding occurrences of variables. Indirect function call or function spawn: apply (lists, rev, [[a,b,c]]) Multiple arities … multiple functions: rev/1 Concurrency Refactoring within a design library: OTP. Side-effects.

21. Static vs dynamic Aim to check conditions statically. Static analysis tools possible … but some aspects intractable: e.g. dynamically manufactured atoms. Conservative vs liberal. Compensation?

22. Architecture of Wrangler

23. Wrangler in Emacs

25. Wrangler refactorings Rename variable/function/modu le Generalise function definition Move a function definition to another (new) module Function extraction Fold expression against function Expression search Detect duplicate code Tuple function parameters From tuple to record

26. Wrangler demo

28. Tool building

29. Wrangler and RefactorErl Lightweight. Better integration with interactive tools (e.g. emacs). Undo/redo external? Ease of implementing conditions. Higher entry cost. Better for a series of refactorings on a large project. Transaction support. Ease of implementing transformations.

30. Integration … with IDEs Back to the future? Programmers' preference for emacs and gvim … … though some IDE interest: Eclipse, NetBeans … Issue of integration with multiple IDEs: building common interfaces.

31. Integration … with tools Test data sets and test generation. Makefiles, etc. Working with macros e.g. QuickCheck uses Erlang macros … … in a particular idiom.

32. APIs … programmer / user API in Erlang to support user-programmed refactorings: declarative, straightforward and complete but relatively low-level. Higher-level combining forms? OK for transformations, but need a separate condition language.

33. Verification and validation Possible to write formal proofs of correctness: check conditions and transformations different levels of abstraction possibly-name binding substitution for renaming etc. more abstract formulation for e.g. data type changes. Use of Quivq QuickCheck to verify refactorings in Wrangler.

34. Clone detection

35. The Wrangler Clone Detector Uses syntactic and static semantic information. Syntactically well-formed code fragments … identical after consistent renaming of variables, … with variations in literals, layout and comments. Integrated within the refactoring environment.

36. The Wrangler Clone Detector Make use of token stream and annotated AST. Token–based approaches  Efficient.  Report non-syntactic clones. AST-based approaches.  Report syntactic clones.  Checking for consistent renaming is easier.

37. The Wrangler Clone Detector Source Files Tokenisation Token Stream Normalisation Normalised Token Stream Suffix Tree Construction Suffix tree

38. The Wrangler Clone Detector Source Files Tokenisation Token Stream Normalisation Normalised Token Stream Suffix Tree Construction Suffix tree Clone Collector Initial Clones Clone Filter Filtered Initial Clones Clone Decomposition Parsing + Static Analysis Annotated ASTs Syntactic Clones

39. The Wrangler Clone Detector Source Files Tokenisation Token Stream Normalisation Normalised Token Stream Suffix Tree Construction Suffix tree Clone Collector Initial Clones Clone Filter Filtered Initial Clones Clone Decomposition Parsing + Static Analysis Annotated ASTs Syntactic Clones Consistent Renaming Checking Clones to report

40. The Wrangler Clone Detector Source Files Tokenisation Token Stream Normalisation Normalised Token Stream Suffix Tree Construction Suffix tree Clone Collector Initial Clones Clone Filter Filtered Initial Clones Clone Decomposition Parsing + Static Analysis Annotated ASTs Syntactic Clones Consistent Renaming Checking Clones to report Formatting Reported Code Clones

41. Clone detection demo

45. Support for clone removal Refactorings to support clone removal.  Function extraction.  Generalise a function definition.  Fold against a function definition.

46. Case studies Applied the clone detector to Wrangler itself with threshold values of 30 and 2.  36 final clone classes were reported …12 are across modules, and 3 are duplicated function definitions.  Without syntactic checking and consistent variable renaming checking, 191 would have been reported. Applied to third party code base (32k loc, 89 modules),109 clone classes reported.

47. Data-oriented refactorings

48. -module(tup1). -export([gcd/1]). gcd({X,Y}) -> if X>Y -> gcd({X-Y,Y}); Y>X -> gcd({Y-X,X})‏; true -> X end. Tupling parameters -module(tup1). -export([gcd/2]). gcd(X,Y) -> if X>Y -> gcd(X-Y,Y); Y>X -> gcd(Y-X,X); true -> X ‏ end. 2

49. -module(rec1). -record(rec,{f1, f2}). g(#rec{f1=A, f2=B})-> A + B. h(X, Y)-> g(#rec{f1=X,f2=X}), g(#rec{ f1=element(1,Y), f2=element(2,Y)}). Introduce records … -module(rec1). g({A, B})-> A + B. h(X, Y)-> g({X, X}), g(Y). f1 f2

50. Introduce records in a project Need to replace other expressions … Replace tuples with record Record update expression Record access expression Chase dependencies across functions … … and across modules.

51. Refactoring and Concurrency

52. Wrangler and processes Refactorings which address processes Register a process. Rename a registered process. From function to process. Add tags to messages sent / received.

53. Challenges to implementation Data gathering is a challenge because Processes are syntactically implicit. Pid to process links are implicit. Communication structure is implicit. Side effects.

54. Underlying analysis Analyses include Annotation of the AST, using call graph. Forward program slicing. Backwards program slicing.

55. Wrangler and Erlide

56. Erlide is an Eclipse plugin for Erlang. Distribution simplified. Integration with the edit undo history. Notion of project. Refactoring API in the Eclipse LTK. Ongoing support for Erlide from Ericsson.

59. Issues on integration LTK has a fixed workflow for interactions. New file vs set of diffs as representation. Fold and generalise interaction pattern. Cannot support rename / create file. Other refactorings involve search … a different API.

60. Conclusions

61. Future work Concurrency: continue work. Refactoring within a design library: OTP. Working with Erlang Training and Consulting. Continue integration with Eclipse + other IDEs. Test and property refactoring in. Clone detection: fuller integration.

62. Ackonwledgements Wrangler development funded by EPSRC. The developers of syntax-tools, distel and Erlide. George Orosz and Melinda Toth. Zoltan Horvath and the RefactorErl group at Eotvos Lorand Univ., Budapest.

63. http://projects.cs.kent.ac.uk