1 Unit Testing for Domain-Specific Languages 1 Hui Wu, 1 Jeff Gray and 2 Marjan Mernik 1 University of Alabama at Birmingham, USA 2 University of Maribor, Slovenia IFIP DSL WC’09 Oxford England July 2009

2 The Benefits of Using DSLs Case studies and empirical evaluations documented in the research literature have identified the following benefits of DSLs –DSLs can hide the lower level programming language details from end-user programmers –DSLs assist in software maintenance whereby end- users can directly use the DSLs to make required routine modifications –DSLs assist end-users in writing more concise, descriptive, and platform independent programs –The use of DSLs increases flexibility, productivity, reliability, and usability; shorten the application development time and reduce the development cost significantly

33 Motivation of Proposed Research Domain Experts program at DSL level DSL translated into General Purpose Language (GPL) Domain Experts deal with DSL Integrated Development Environment (IDE) Editor Compiler Visualizer Debugger Domain Experts deal with GPL Test Engine Profiler

44 Mismatch Between Abstraction Levels …. commands : ( c:command cs:commands | ) ; command : ( RIGHT { fileio.print("//move right"); fileio.print("x=x+1;"); fileio.print("time=time+1;"); fileio.print(" "); } |LEFT { fileio.print("//move left"); fileio.print("x=x-1;"); fileio.print("time=time+1;"); fileio.print(" "); … …. public final void commands() throws RecognitionException, TokenStreamException { try { // for error handling { switch ( LA(1)) { case CALL: case INIT: case SET: case PRINT: { command(); commands(); break; } case END: { break; } default: { throw new NoViableAltException(LT(1), getFilename()); … public final void function_name() throws RecognitionException, TokenStreamException { try { // for error handling { switch ( LA(1)) { case RIGHT: { match(RIGHT); fileio.print("//move right"); fileio.print("move_right();"); fileio.print(" "); break; } case LEFT: { match(LEFT); sllinenumber=dsllinenumber+1; fileio.print("//move left"); fileio.print("move_left();"); fileio.print(" "); break; } … ANTLR Grammar Generated Java Parser Code

55 Categories of Domain-Specific Languages Imperative DSL –Robot DSL Declarative DSL –Feature Description Language (FDL) –Backus–Naur Form (BNF) language syntax specification Hybrid DSL (for future work) –Swing User-interface Language (SWUL) –Hybrid Robot DSL

66 Categories of Domain-Specific Languages Imperative DSL: Centered around assignment expressions or control flow statements –Robot Language … 17 Down: 18 position(+0,-1) 19 Down: … 21 Init position(0,0) 22 Call left 23 Call down 24 Call knight 25 Set position(5,6) 26 Call up 27 Call right 28 Print position …

77 Declarative DSL: declares the relationship between inputs and outputs –Feature Description Language (FDL) 1 Car : all (Carbody, Transmission, Engine, Horsepower, opt(pullsTrailer)) 2 Transmission : oneof (automatic, manual) 3 Engine : moreof (electric, gasoline) 4 Horsepower : oneof (lowPower, mediumPower, highPower) 5 include pullsTrailer 6 pullsTrailer requires highPower Categories of Domain-Specific Languages – Backus–Naur Form (BNF) language syntax specification: a set of derivation rules START ::= begin COMMANDS end ; COMMANDS :: = COMMAND COMMANDS | epsilon ; COMMAND :: = left | right | up | down ;

88 Eclipse Plug-In Based Software Development JUnit Eclipse Plug-in

99 Plug-In Based Software Development Syntax-Directed Translation Aspect-Oriented Programming on DSL Grammars Overview of the Approach Design Patterns Generative Programming Eclipse Model-View-Controller Adapter Pattern AspectG ANTLR pointcut productions(): within(command.*); before(): productions() { dsllinenumber=dsllinenumber+1;} … public class Robot{ … x=0; y=0; time=0; //move left move_left();

10 DSL Application Software Errors! Before locating software errors how do we know there are bugs inside a DSL application?

11 DSL Unit Testing Framework (DUTF) Complement to the DSL Debug Framework (DDF) – the DUTF assists in identifying the presence of errors and the DDF assists in isolating the specific location of the error Architecture and process of construction is similar to the DDF architecture

12 DSL Unit Testing Framework (DUTF)

13 Source Code Mapping … 3 knight: 4 position(+0,+1); 5 position(+0,+1); 6 position(+1,+0); 7 knight: 8 … 9 Init position(0,0); 10 left; 11 down; 12 knight; 13 Set position(5,6); 14 up; 15 right; 16 Print position; … 6 public static void move_knight(){ 7x=x+0; 8y=y+1; 9x=x+0; 10y=y+1; 11x=x+1; 12y=y+0;} 13 public static void main(String[] args) { 14x=0; 15y=0; … 18move_knight(); … 20x = 5; 21y = 6; … 26System.out.println("x coordinate="+x+""+ 27 "y coordinate= " + y);} … {13, "Robot.java", 20, 21, "main", "none"}

14 DSL Unit Testing Framework (DUTF) Test Cases Mapping Robot DSL Unit Test Case 1 TestCase testknight { 2 Init position(0,0); 3 Expectedposition(1,2); 4 knight; 5 AssertEqual (Expectedposition, position); 6 } … GPL Unit Test Case (JUnit) 11 public void testkinght() { 12robot.x = 0; 13robot.y =0; 14int x=1; 15int y=2; 16robot.move_knight(); 17assertEquals(x, robot.x); 18assertEquals(y, robot.y); 19 } … {1, “TestRobot.java”,11,“testknight”}

15 DSL Unit Testing Framework (DUTF) Test Cases Mapping Car FDL Unit Test Case 1 TestCase testFeatures { 2 Expectedfeature:(carbody, manual, highPower); 3 use Car.FDL(All); 4 Constraint C1: include pullsTrailer; 5 AssertTrue(contain(Expectedfeature, feature)); 6 AssertEqual(6, numberof feature); 7 } GPL Unit Test Case (JUnit) 11 public void testFeatures () { 12 testFeatures.add("carbody"); 13 testFeatures.add("manual"); 14 testFeatures.add("highPower"); … 27 assertTrue(compareFeatures(testFeatures,parse(fc,root,cons))); 28 assertEquals(6,getFeatureListNumber(parse(fc,root,cons))); …

16 DSL Unit Testing Framework (DUTF) Robot Language Unit Test Engine Correct knight method 1 begin knight: 2 position (+0,+1); 3 position (+0,+1); 4 position (+1,+0); 5 end knight: Incorrect knight method 1 begin knight: 2 position (+0,+1); 3 position (+1,+1); 4 position (+1,+0); 5 end knight:

17 DSL Unit Testing Framework (DUTF) FDL Unit Test Engine

18 Lessons Learned and Limitations The concept of comparing raw values may not be relevant at the DSL level Among 22 software components in DUTF, there are 3,001 lines of code that are generalized and reused to generate the different DSL unit test engines. More complex and feature-rich DSLs will likely require additional customization DSL CategoryDSL NameNumber of Specific Functions or Classes Customized Lines of Code Imperative DSLRobot Language2239 Declarative DSLFDL4482

19 Future Work An extension of the current framework that enables DSL profiling Investigation into the scalable, reliable, and extensible DSL Unit Test framework –Application of different IDE platforms (Microsoft Visual Studio.Net) and GPLs (C# and C++) –Adaptation of DUTF to address more complex DSLs (Embedded DSLs)

20 Related Works in the Area of DSL Tools The End-Users Shaping Effective Software （ EUSES ） –Represents collaboration among several dozen researchers who aim to improve the software development capabilities provided to end-users. ASF+SDF –Generate program analysis and transformation tools, and produce software renovation tools –The unit testing tool support has not been reported JTS –Complicated mechanism (e.g., Language extension) –Does not focus on language tools generation LISA –Can generate editor, parser, and visualizer for new languages –Debugger, test engine, and profiler are not the target language tools, LISA can be used as the front-end of our framework SmartTools –Base on Java and XML technologies –Debugger, test engine, and profiler are not target language tools Other Related Testing Tools –Parameterized unit testing –Testing grammar-driven functionality –Generating unit tests using symbolic execution –Generating test inputs of AspectJ programs

21 Questions? Video demonstrations and papers available at: Acknowledgement: The work presented in this paper was supported in part by NSF CAREER grant (CCF ) and the IBM Eclipse Innovation Grant (EIG).

22 Backup Slides

23 Categories of DSL End-Users Admin Assistants Business Person Auto Factory Worker Scientist Spreadsheet Business Query Systems Modeling Language DSL for Physics

24 JUnit and NUnit Basic Unit Test Actions Comparison Unit Test ActionsJUnit (Java)NUnit (.Net languages) Equality Assertion assertEquals (expected, actual) Assert.AreEqual (expected, actual); Condition Test assertTrue(actual)Assert.IsTrue(actual) assertFalse(actual)Assert.IsFalse(actual) assertNull(actual)Assert.IsNull(actual) Utility Method fail()Assert.Fail() Identity Assert assertSame (expected, actual) Assert.AreSame (expected, actual)