Automation of Software Testing for the Distributed Common Ground System – Army (DCGS-A) Michael Eng Mentors: Mark Pumphrey, Greg Cordero Northrop Grumman Mission Systems

Outline Purpose DCGS-A overview JUnit Automation of Testing Problems Encountered Results/Conclusion

Explore methods of software testing and testing automation Purpose Explore methods of software testing and testing automation Help to test and debug the DCGS-A project Improve software quality Meet functional and contractual requirements Ensure functionality in deployment

What is DCGS-A? Distributed Common Ground System- Army Signals Intelligence branch SIGINT Modular & decentralized Integrate multiple types/sources of intelligence Infrastructure of Java, JavaBeans, Oracle Similar efforts taking place in other branches of the military

CPP Shelter on Armored M1152 –Assured Mobility for BCT, Div, and Above Common Shelter in Battle Command

Why Test Software? Software developed for the US Military must be, to the absolute extent possible, free of error because lives and mission are at stake. System crashes in field Loss of communication between commander and soldiers No one knows where the enemy is Intelligence database cannot be updated System is hacked Self-explanatory

JUnit Unit testing framework for Java Part of xUnit family of frameworks Test classes correspond to main classes E.g., MyClass.java & MyClassTest.java Test and main directories parallel each other Run  status bar- green or red Used with Eclipse to test DCGS-A Perfect for testing individual instances of objects, methods, etc. http://www.onjava.com/pub/a/onjava/2005/07/13/pisces.html?page=2

Methodology/Procedure Each method with logic is tested Every constructor is tested Unit testing- method input arguments, conditional statements tested Edge testing Dummy objects Hardcoded characteristics- force all possibilities JUnit4TestRunner, Logger JUnit formatting JUnit Javadoc notation- @Before, @After, @Test Assertions E.g., assertEquals(array[0], “Proper answer”); assertNull([something that returns null]); Expected void test (expected =NullPointerException.class) Exception testing “logic…that is, not a simple get or set method” THIS INCLUDES GARBAGE DATA AND RANDOMLY GENERATED DATA- FUZZ TESTING

Procedure Continued One method can have many, many cases Vast majority of possibilities must be tested Errors, if-else ladders, arguments, etc. “Hardened” classes important System must be protected Reflection, buffer overflow attacks Make sure values are private/final Bounds checking for overflow attacks “One method can have many, many cases”…and most, if not all need to be covered in testing “System must be protected…tactics like reflection and data overflow attacks have to be handled- for example reflection can be stopped by making critical values private and final, and many overflow attacks can be stopped with bounds checking” http://www.dabbledoo.com/ee/images/uploads/gamertell/access_denied.jpg

Example Test Program class myClass{ class myClassTest static char parse (String s){ …logic to parse out the first number character in a String (e.g. ‘3’ in “ABC3WW5”)… return that number as a char or null if not found } class myClassTest void testParse() myClass.parse(null); void testParse2() myClass.parse(“”); void testParse3() myClass.parse(“ABC”); void testParse4() myClass.parse(“3##”); void testParse5() myClass.parse(“53”); void testParse6() myClass.parse( new Object().toString()); void testParse7() myClass.parse(new Integer(54)); void testParse8() myClass.parse(new String[2]); And so on… Say we have a class MyClass with method parse that gets the first number character in an input String and returns it or null if there’s no numbers in the string. On the right we have a JUnit test class. So, we test the method by sending null. Then we send an empty String and expect a null output, or maybe an Exception if the method doesn’t check for that sort of thing (which we would then fix in the main class). Then we send a String that’s all letters and expect it to return null. Then a string that actually has a number. Then an all-number string. Then an Object forced into a String. Then an Integer object. Then a String array instead of an object. We could even go so far as to send a string one character too large (32767 is the limit) to see how gracefully the program handles overflows. And so on.

Automation Tests are written, now what? JUnit AllTests- template for mass runs Project has over 700 test methods Before automation- run each test class manually Hours After automation- run one root test class 10 seconds “Each subdirectory has an AllTests file that basically loads either the AllTests files below it, or loads each test class into it and runs the tests. It actually only takes a little under 10 seconds to run the whole system’s tests. So that saves a LOT of time.” http://java.sun.com/developer/technicalArticles/JavaLP/JavaToMac/images/WindowsJUnit.gif

Problems Encountered External frameworks Test dependency hell Undocumented or still being integrated Test dependency hell E.g., testing a class nested in another that depends on unavailable/unfinished properties file Not really a problem, but constant updates to data classes still in development/pre-debugging Constant updates of old tests http://gallery.nofactzone.net/albums/userpics/normal_headdesk.jpg

Results/Conclusion Created over 20 test classes Modified over 10 existing test classes (bug/build fixes, etc.) Found and fixed 10 bugs using test classes Wrong classifications Incorrect date calculations Incorrect order of if-else ladders Still working on getting 100% function, path, and condition coverage for classes that need testing Modifying existing tests to reflect updates in development

Questions?