Automation of Testing in the Distributed Common Ground System (Army) Michael Eng Mentor: Mark Pumphrey, Greg Cordero Northrop Grumman Mission Systems

Outline Purpose DCGS-A overview JUnit Automation of testing Problems encountered Results/conclusion

Purpose Explore methods of software testing and testing automation Help to test and debug the DCGS-A project Improve software quality

DCGS-A Distributed Common Ground System- Army Modular & decentralized Integrate multiple types/sources of intelligence Java, JavaBeans, Oracle Similar efforts taking place in other branches of the military

Why Test? System crashes in field System is hacked Loss of communication between commander and soldiers No one knows where the enemy is Intelligence database cannot be updated System is hacked Self-explanatory

JUnit Unit testing framework for Java Test classes correspond to main classes E.g., MyClass.java & MyClassTest.java Run  status bar- green or red Used with Eclipse to test DCGS-A Perfect for testing individual instances of objects, methods, etc. http://www.onjava.com/pub/a/onjava/2005/07/13/pisces.html?page=2

Methodology/Procedure Each method with logic is tested Every constructor is tested Unit testing- method input arguments, conditional statements tested Edge testing Dummy objects Hardcoded characteristics- force all possibilities “logic…that is, not a simple get or set method” THIS INCLUDES GARBAGE DATA AND RANDOMLY GENERATED DATA- FUZZ TESTING

Procedure continued JUnit4TestRunner, Logger JUnit formatting JUnit Javadoc notation- @Before, @After, @Test Assertions E.g., assertEquals(array[0], “Proper answer”); assertNull([something that returns null]); Expected void test (expected =NullPointerException.class) Exception testing Javadoc notation is placed before the method to signify to the TestRunner that the method is a test method Before is used to reset variables before each test After usually is used to free up memory and resources or reset variables after each test

Example Test Program class myClassTest class myClass{ void testParse() myClass.parse(null); void testParse2() myClass.parse(“”); void testParse3() myClass.parse(“ABC”); void testParse4() myClass.parse(“3##”); void testParse5() myClass.parse(“53”); void testParse6() myClass.parse( new Object().toString()); void testParse7() myClass.parse(new Integer(54)); void testParse8() myClass.parse(new String[2]); And so on… class myClass{ static char parse (String s){ …logic to parse out the first number character in a String (e.g. ‘3’ in “ABC3WW5”)… return that number as a char or null if not found } Say we have a class MyClass with method parse that gets the first number character in an input String and returns it or null if there’s no numbers in the string. On the right we have a JUnit test class. So, we test the method by sending null. Then we send an empty String and expect a null output, or maybe an Exception if the method doesn’t check for that sort of thing (which we would then fix in the main class). Then we send a String that’s all letters and expect it to return null. Then a string that actually has a number. Then an all-number string. Then an Object forced into a String. Then an Integer object. Then a String array instead of an object. We could even go so far as to send a string one character too large (32767 is the limit) to see how gracefully the program handles overflows. And so on.

Procedure continued One method can have many, many cases Vast majority of possibilities must be tested Errors, if-else ladders, arguments, etc. “Hardened” classes important System must be protected Reflection, buffer overflow attacks “One method can have many, many cases”…and most, if not all need to be covered in testing “System must be protected…tactics like reflection and data overflow attacks have to be handled- for example reflection can be stopped by making critical values private and final, and many overflow attacks can be stopped with bounds checking” http://www.dabbledoo.com/ee/images/uploads/gamertell/access_denied.jpg

Automation Tests are written, now what? JUnit AllTests- template for mass runs Project has over 700 test methods “Each subdirectory has an AllTests file that basically loads either the AllTests files below it, or loads each test class into it and runs the tests. It actually only takes a little under 10 seconds to run the whole system’s tests. So that saves a LOT of time.” http://java.sun.com/developer/technicalArticles/JavaLP/JavaToMac/images/WindowsJUnit.gif

Problems Encountered External frameworks Test dependency hell Undocumented or still being integrated Test dependency hell E.g., testing a class nested in another that depends on unavailable/unfinished properties file Mock objects/files Not really a problem, but constant updates to data classes still in development/pre-debugging Constant updates of old tests http://gallery.nofactzone.net/albums/userpics/normal_headdesk.jpg

Results/Conclusion Created over 20 test classes Modified over 10 existing test classes and main classes (bug fixes, etc.) Still working on getting 100% function, path, and condition coverage for classes that need testing Modifying existing tests to reflect updates in development