1. Unit Testing for Domain-Specific Languages 1 Hui Wu, 1 Jeff Gray and 2 Marjan Mernik 1 University of Alabama at Birmingham, USA {wuh,gray}@cis.uab.edu 2 University of Maribor, Slovenia marjan.mernik@uni-mb.si DSL 2009 Oxford, UK July 15, 2009 Programming Methodologies Laboratory

2. Emperors New DSLs As we show off our new DSLs and approaches for creating them, are there some obvious limitations that we may be ignoring? 2 But where are the tools I can use?

3. OOPSLA 2008 DSL Panel: The Good, the Bad, and the Ugly Four of the five participants listed lack of tool support as a top challenge for DSLs –Utility of a new DSL is severely diminished if supporting tools needed by a software developer and/or end-user are not available –Poor interoperability and reuse of such tools 3

4. 44 What Kinds of Tools are Needed? Domain Experts program at DSL level DSL translated into General Purpose Language (GPL) Domain Experts deal with DSL Integrated Development Environment (IDE) Editor Compiler Visualizer Debugger Domain Experts deal with GPL Test Engine Profiler

5. 55 Mismatch Between Abstraction Levels …. commands : ( c:command cs:commands | ) ; command : ( RIGHT { fileio.print("//move right"); fileio.print("x=x+1;"); fileio.print("time=time+1;"); fileio.print(" "); } |LEFT { fileio.print("//move left"); fileio.print("x=x-1;"); fileio.print("time=time+1;"); fileio.print(" "); … …. public final void commands() throws RecognitionException, TokenStreamException { try { // for error handling { switch ( LA(1)) { case CALL: case INIT: case SET: case PRINT: { command(); commands(); break; } case END: { break; } default: { throw new NoViableAltException(LT(1), getFilename()); … public final void function_name() throws RecognitionException, TokenStreamException { try { // for error handling { switch ( LA(1)) { case RIGHT: { match(RIGHT); fileio.print("//move right"); fileio.print("move_right();"); fileio.print(" "); break; } case LEFT: { match(LEFT); sllinenumber=dsllinenumber+1; fileio.print("//move left"); fileio.print("move_left();"); fileio.print(" "); break; } … ANTLR Grammar Generated Java Parser Code

6. Why is basic DSL tool support lacking? Why are such tools scarce? –Building new DSL tools from scratch for each new DSL can be time consuming, error prone and costly –Tooling implementation effort generally not reusable for a different context (even in same domain) 6 Example –ANTLRStudio http://www.placidsystems.com/antlrstudio.aspx –Fixed tool support customized for specific DSL

7. Idea of this paper Introduce the idea of a framework for building DSL tools that reuse existing GPL tools Assumptions/Limitations: –Only applicable to source-to-source translation (DSL reuses GPL compiler and other tools of GPL context) –DSL tool implementation dependent on available GPL mappings; requires adaptation to existing grammars –Does not consider case when one DSL statement is mapped to non-contiguous GPL code (e.g., an aspect language) –Paper uses simple examples for illustration; scalability of approach not certain 7

8. 88 DSL to GPL Source Mapping … 3 knight: 4 position(+0,+1); 5 position(+0,+1); 6 position(+1,+0); 7 knight: 8 … 9 Init position(0,0); 10 left; 11 down; 12 knight; 13 Set position(5,6); 14 up; 15 right; 16 Print position; … 6 public static void move_knight(){ 7x=x+0; 8y=y+1; 9x=x+0; 10y=y+1; 11x=x+1; 12y=y+0;} 13 public static void main(String[] args) { 14x=0; 15y=0; … 18move_knight(); … 20x = 5; 21y = 6; … 26System.out.println("x coordinate="+x+""+ 27 "y coordinate= " + y);} … {13, "Robot.java", 20, 21, "main", "none"}

9. 99 Reusing GPL Tools in DSL Context

10. Background Context: DSL Debuggers Our initial work for DSL tool support focused on debuggers The DSL Debugging Framework (DDF) provides a context for understanding the source mapping necessary to reuse GPL tools in a DSL context 10

11. 11 Debugging Methods Mapping MappingDSLGPL Source Code n_i maps to m_i to m_j Line Number: n_1 n_2 n_... n_i n_i+1 n_... n_j n_j+1 n_... Line Number: m_1 m_2 m_... m_i m_i+1 m_... m_j m_j+1 m_... BreakpointSet breakpoint at n_iSet breakpoint at m_i Step OverStep over line at n_iStep Over algorithm Step IntoStep into line at n_iStep Into algorithm TerminateTerminate at line n_iTerminate at line m_i ResumeResume at line n_iResume at line m_i

12. 12 Debugging Results Mapping: Tangled Concerns within Grammar 1 | INIT var:VARIABLES LPAREN init_num1:NUMBER COMMA init_num2:NUMBER RPAREN 2 { 3dsllinenumber=dsllinenumber+1; 4fileio.print("x="+init_num1.getText()+";"); 5fileresult.print("x_coordinate=print x"); 6gplbeginline=fileio.getLinenumber(); 7fileio.print("y="+init_num2.getText()+";"); 8fileresult.print("y_coordinate=print y"); 9fileio.print("time=0"+";"); 10gplendline=fileio.getLinenumber(); 11 filemap.print("mapping.add(new Map("+dsllinenumber+",\"Robot.java\","+ 12 gplbeginline +","+gplendline+","+"\""+funcname+"\""+","+"\""+funcall+"\""+"));"); 13fileresult.print(variable=var.getText()+"(x_coordinate,y_coordinate)"); 14 } Black: Basic functionality Blue: Source code mapping Red: Debugging results mapping

13. 13 Imperative DSL Debugger User has access to traditional debugging operations Variables defined in terms of DSL, not GPL

14. 14 Declarative DSL Debugger (FDL) 1 Car : all (Carbody, Transmission, Engine, Horsepower, opt(pullsTrailer)) 2 Transmission : oneof (automatic, manual) 3 Engine : moreof (electric, gasoline) 4 Horsepower : oneof (lowPower, mediumPower, highPower) 5 include pullsTrailer 6 pullsTrailer requires highPower

15. 15 Declarative DSL Debugger (BNF)

16. 16 Hybrid Robot Debugger (Java within DSL)

17. 17 Hybrid SWUL Debugger (DSL within Java)

18. 18 Current Focus: DSL Testing Before locating software errors how do we know there are bugs inside a DSL program?

19. 19 Complement to the DSL Debug Framework (DDF) – the DUTF assists in identifying the presence of errors and the DDF assists in isolating the specific location of the error Architecture and process of construction is similar to the DDF architecture (Figure 1 in paper); source code mapping similar DSL Unit Testing Framework (DUTF)

20. 20 DSL Unit Testing Framework (DUTF)

21. 21 Robot DSL Unit Test Case 1 TestCase testknight { 2 Init position(0,0); 3 Expectedposition(1,2); 4 knight; 5 AssertEqual (Expectedposition, position); 6 } … GPL Unit Test Case (JUnit) 11 public void testknight() { 12robot.x = 0; 13robot.y =0; 14int x=1; 15int y=2; 16robot.move_knight(); 17assertEquals(x, robot.x); 18assertEquals(y, robot.y); 19 } … {1, “TestRobot.java”,11,“testknight”} A testing assertion at the DSL level may result in multiple assertions at the GPL level depending on variable mapping DSL Unit Testing Framework (DUTF): Test Cases Mapping

22. 22 Car FDL Unit Test Case 1 TestCase testFeatures { 2 Expectedfeature:(carbody, manual, highPower); 3 use Car.FDL(All); 4 Constraint C1: include pullsTrailer; 5 AssertTrue(contain(Expectedfeature, feature)); 6 AssertEqual(6, numberof feature); 7 } GPL Unit Test Case (JUnit) 11 public void testFeatures () { 12 testFeatures.add("carbody"); 13 testFeatures.add("manual"); 14 testFeatures.add("highPower"); … 27 assertTrue(compareFeatures(testFeatures,parse(fc,root,cons))); 28 assertEquals(6,getFeatureListNumber(parse(fc,root,cons))); … DSL Unit Testing Framework (DUTF): Test Cases Mapping

23. 23 DSL Unit Testing Framework (DUTF) Robot Language Unit Test Engine Correct knight method 1 begin knight: 2 position (+0,+1); 3 position (+0,+1); 4 position (+1,+0); 5 end knight: Incorrect knight method 1 begin knight: 2 position (+0,+1); 3 position (+1,+1); 4 position (+1,+0); 5 end knight: User sees results as typical JUnit test report, but with DSL test cases

24. 24 DSL Unit Testing Framework (DUTF) FDL Unit Test Engine

25. Video Demonstration DUTF Testing Demo 2.1 available at http://www.cis.uab.edu/softcom/DDF/ 25

26. 26 Adapting Existing DSL Grammars for DSL Tool Support Crosscutting concerns emerge in different software artifacts (e.g., model, grammar, and source code) AspectG: A domain-specific aspect language for language grammars written in ANTLR –Program transformation rules are generated from a domain-specific aspect language focused on language specification in ANTLR –Generated rules modify grammars, not source code

27. 27 dsllinenumber=dsllinenumber+1; gplbeginline=fileio.getLinenumber(); gplendline=fileio.getLinenumber(); filemap.print(" mapping.add(new Map("+dsllinenumber+", \"Robot.java\","+gplbeginline+","+gplendline+"));"); Addition of DSL Tool Support: A Crosscutting Grammar Concern dsllinenumber=dsllinenumber+1; gplbeginline=fileio.getLinenumber(); gplendline=fileio.getLinenumber(); filemap.print(" mapping.add(new Map("+dsllinenumber+", \"Robot.java\","+gplbeginline+","+gplendline+"));"); … command :( RIGHT { fileio.print("//move right"); fileio.print("x=x+1;"); fileio.print("time=time+1;"); fileio.print(" "); } |LEFT { fileio.print("//move left"); fileio.print("x=x-1;"); fileio.print("time=time+1;"); fileio.print(" "); } | … Base Grammar Duplicate Tool Aspect Code What if this line changes? Change here Change ……

28. 28 pointcut productions(): within(command.*); pointcut count_gpllinenumber(): within(command.*) && match (fileio.print("time=time+1;")); before(): productions() {dsllinenumber=dsllinenumber+1;} before(): count_gpllinenumber() {gplbeginline=fileio.getLinenumber();} after(): count_gpllinenumber() {gplendline=fileio.getLinenumber();} after(): productions() {filemap.print(" mapping.add(new Map("+ dsllinenumber+", \"Robot.java\","+gplbeginline+ ","+gplendline+"));"); } … command :( RIGHT { dsllinenumber=dsllinenumber+1; fileio.print("x=x+1;"); gplbeginline=fileio.getLinenumber(); fileio.print("time=time+1;"); gplendline=fileio.getLinenumber(); fileio.print(" "); filemap.print(" mapping.add(new Map("+ dsllinenumber+", \"Robot.java\","+gplbeginline+ ","+gplendline+"));"); } … Join Point Model for AspectG ANTLR Grammar AspectG

29. 29 Summary and Lessons Learned Tool development requires deep understanding of DSL, our framework, and the required mappings JUnit tests for equality in assertion checks may not match semantics of domain and need to be adapted Among 22 software components in DUTF, there are 3,001 lines of code that are generalized and can be reused to generate different DSL unit test engines DSL CategoryDSL NameNumber of Specific Functions or Classes Customized Lines of Code Imperative DSLRobot Language2239 Declarative DSLFDL4482

30. 30 Future Work An extension of the current framework that enables DSL profiling Adaptation of DUTF to address more complex DSLs Investigation into extending DUTF to other platforms –Application of different IDE platforms (Microsoft Visual Studio.Net) and GPLs (C# and C++) using Nunit and other testing tools

31. 31 Questions? Video demonstrations and papers available at: http://www.cis.uab.edu/softcom/DDF/ Acknowledgement: The work presented in this paper was supported in part by NSF CAREER grant (CCF-0643725) and the IBM Eclipse Innovation Grant (EIG). Programming Methodologies Laboratory