1. Unit Testing & Test Coverage Why,What,(Who)
Andrea Guarise
INFN Torino
Milano, June 7th, 2004

2. Introduction
Unit Testing: It is the procedure that furnishes the capability to test a software module in a reprudicible, efficient and automated way.
Test Suite: A collection of unit tests.
Test Coverage: The measurement of how many parts of the code are really tested, it is an expression of the goodeness of the Test Units.

3. Why The Four reasons for Unit Testing:
Before writing code, it forces you to detail your requirements in a useful fashion.
While writing code, it keeps you from over-coding. When all test cases pass, the function is complete.
When refactoring code, it assures you that the new version behaves the same way as the old version.
When maintaining code, it helps you cover your ass when someone comes screaming that your latest change broke their old code. (“But Sir, all the unit tests passed when I checked It in...”
Mark Pilgrim, from “Dive Into Python”

4. Why (2) Why Test Coverage:
Simply because it is useless to have a beautiful test suite that doesn’t test all your code.
A good test suite should always have instruments to tell you if you forgot to test something. So that at least you know where most likely the users of your code will find bugs.
This is a direct consequence of the Murphy’s law.

5. We use many different programming languages
What
Unit Test Framework:
It can be really useful to find a single test framework for all our software modules.
But there is a little problem ahead of this:
We use many different programming languages
Therefore it’s necessary to change our perspective:
We need to find a set of testing framework (one for each language) that behaves all in the same manner.

6. What (2) The xUnit Framework:
xUnit is not a real test framework. It is the name used to address all the test frameworks that adhere to a common set of “capabilities” and test patterns.
It isn’t yet a formalised standard but there are chanches that it will be.
One inportant thing about xUnit framework is that test code must always be separate from tested code.
There are xUnit compliant test frameworks for more or less every programming language. Our goal is to find one of these framework for each of the languages we use:
C++, C, Java, Perl, Python (did I forget any?)

7. What (3) Some General rules: The general rules we have to follow are:
Each module must use only one unit test framework.
Unit test code goes under the module/test/ directory.
The unit test output is in XML format and goes under the module/reports diroctory.
The build fails if the unit test fails.

8. What (4) cppUnit: http://cppunit.sourceforge.net C++:
There are many implementations available, we analyzed some of them, and the choice has been:
cppUnit:

9. What (4)
Java:
The choice for java is really easy the de facto standard is
jUnit:
The Idea of xUnit itself is borrowed from the jUnit framework. Frameworks for the other languages, a part from boost.test, are portings of jUnit.
It only adds dependencies at build level.
This is one of the requirements of the xUnit framework, the test units must not interfere with the code itself. There is no need to instrument the code.

10. What (5)
Perl:
We tried the: Test::Unit unit testing framework available on
It is intended to be a porting of jUnit for OO based perl programming, altough not all the classic xUnit functionalities are present yet.
The documentation is in “POD” format. There is the documentation of the APIs some simple example and the usage instruction for the tools shipped with the framework.
It only adds dependencies on the “build” node, no dependencies are needed on the machine where the tested code is installed.

11. What (6)
Python:
We did not try the framework, but the standard implementation is pyUnit:
It is fully documanted, with API and examples.
It doesn’t add dependencies to the executables, but only on the build node.
We did not know about any currently maintained test coverage analysis tool for Python.

12. Test Coverage
As for unit testing, we need tools to measure test coverage for all of the above mentioned languages.
Unfortunately tools for different languages report their results in different way, and all of them requires some effort in order to elaborate the results to get some usable statistics. So it may not be straightforward to really exploit these tools:

13. Test Coverage (2) C/C++: Java: Hansel: http://hansel.sourceforge.net
GNU GCC + GCOV
It works, reports the percentage of code covered, and how many times each useful line of code has been run.
GNU GCC + COVTOOL:
On the paper seems most powerful, but it doesn’t build on gcc 3.x.x
Java:
Hansel:
NoUnit:
Not tested (does any one want to spare some time?)
Python:
There is a module named coverage.py available at:
But it is rather old and seems to be not official (if someone wants to check....).
Perl:
Code coverage is available under CPAN, some tests have been performed and it seems to work fine.

14. (Who?) The short answer: We
The long answer: (This is what we proposed in last meeting, but it needs to be reviewed, see conclusions)
For each SW module we have two team of developers:
Team A (The developers): responsible of developing the module and writing the basic unit tests (at least 50% code coverage), excuting them each time they feel it is necesssary, write basic Documentation.
Team B (The Guardian Angels): study Team A code and tests, perform test coverage analysis on team A tests and add missing cases (90% code coverage), propose changes to both code and tests, help Team A in maintaining the unit tests and the test suites. Review and finalize Documentation.
And then a Test coordinator: Integrate the test suites with the help of each “team B”. Check the global test coverage analysis and that the unit tests adhere to the project standards.

15. Conclusions
In the meeting we learned how to use basic cppUnit functionalitites in our cpp tests, with a simple generic example, and a real life example with socketpp classes.
We also learned how to use code coverage analysis tools to discover missing test cases.
We realized that:
It is possible to test currently available client/server sw pieces only relying on real instances of the servers (or of the clients) instantiated from whithin the test suites. This is not really compliant with true Unit Testing practices, but has been agreed with Alberto Di Meglio.
This approach however has some disadvantages, like difficulties in tracking down the source of problems and most important it creates dependencies between client and server modules.
A reimplementation of the server interfaces allowing the creation of mock implementation of the servers, for testing purposes has been proposed, and we agree that this would be the right solution, but it is definitely a medium/long term solution.

16. Conclusions(2)
So for the short term we plan to follow the joint client/server test approach
But we need to agree with Alberto the details of this approach and moreover the fact that the coverage analysis be performed to the client and server modules jointly.
We defined the initial test cases for the LB service.
We also started analyizing test cases for the NS service.
Questions:
Deadlines? What about the fact that we will go to WS (it is a waste of time to write unitTests for code that has to be thrown away?)
Test cases: who starts: the new people (the GA), that aren’t trained with the code and will need more time? Or the developers that can quickly implement the skeleton for the tests and then let the remaining work to the GA?
How much effort are we expected to put in this activity? Who (The names)
C framework, we have to choose it. (I will discard the not compliant one)
Report schema: Alberto has to give us the XML schema for the reports, otherwise we aren’t able to go on. (I’ll send the request to Alberto.)