Thomas Sundberg

February 18, 2012

Test coverage in a multi module Maven project

Filed under: Automation, Java, Maven, Test Automation — Tags: , , , , , — Thomas Sundberg @ 09:35

Test driving a project should result in a test coverage of 100% of the code. But how can we know what test coverage we have? Is it even important?

Know what you measure

Numbers is a complicated subject. They are a little like alcohol. To much and dependant of them is a problem. None may be boring. The Swedish word ‘lagom’ is very applicable here. It means not to much and not to little, just enough.

To measure is to know

As an engineer I know that if you can’t measure a value, you don’t know anything about it. But measuring the wrong thing is just as bad. You must know what the values you get represents. If you don’t, then you should at least not talk about them.

Is it important

Is test coverage even important? What is important is that you are certain that the functionality you want to support actually works as expected and required. Numbers are easy to misunderstand and test coverage test coverage can be useless if the tests are bad. Suppose that all code is executed but nothing is asserted? Then we might think that we have a great test coverage, but if nothing is asserted then it is actually worthless. So test coverage may be a good value to track, but it may also be totally useless information. The only way to know is to have confidence in the test code that actually generated the execution. A bad test will result in an execution that may or may not be correct.

But enough with ranting about numbers. How do you measure the test coverage? As usual, use the right tools for the job. One tool is Cobertura. It works great, but it has some problems when it comes to Maven projects with multiple modules.

Instrumented classes

Cobertura works like this: It instruments classes and writes to a log file each time a line is executed. The log is then combined with the source code to calculate which rows that actually were executed.

Let’s start with the simple case that works out of the box.

The simple case – one Maven module

A simple example is when you have a simple project in one module. It is easy in this case to measure the test coverage using a Maven plugin.

The project I use in ths example looks like this:

one-module-example
|-- pom.xml
`-- src
    |-- main
    |   `-- java
    |       `-- se
    |           `-- sigma
    |               `-- calculator
    |                   `-- Calculator.java
    `-- test
        `-- java
            `-- se
                `-- sigma
                    `-- calculator
                        `-- CalculatorTest.java

We have a really simple calculator that calculates fibonacci numbers. The source looks like this:

src/main/java/se/sigma/calculator/Calculator.java

package se.sigma.calculator; public class Calculator { public int nextFibonacci(int a, int b) { return a + b; } }

The test code is also simple and looks like this:

src/test/java/se/sigma/calculator/CalculatorTest.java

package se.sigma.calculator; import org.junit.Test; import static org.hamcrest.core.Is.is; import static org.junit.Assert.assertThat; public class CalculatorTest { @Test public void shouldCalculateFibonacci() { Calculator calculator = new Calculator(); int expected = 13; int actual = calculator.nextFibonacci(5, 8); assertThat(actual, is(expected)); } }

The test coverage is 100% and it can be measured using Cobertura as shown in the Maven Pom below:

pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <groupId>se.thinkcode</groupId> <artifactId>one-module-example</artifactId> <version>1.0</version> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> </properties> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>cobertura-maven-plugin</artifactId> <version>2.5.1</version> <executions> <execution> <phase>process-classes</phase> <goals> <goal>cobertura</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> </dependency> </dependencies> </project>

The key part here is the cobertura-maven-plugin that is executed in the process-classes phase. It will instrument the compiled classes so Cobertura can record the execution of each line.

The report from a build looks like this:

It tells us that we have 100% coverage of the system and that two out of two lines of code has been executed. We have the complexity 1. There is only one path through the code. There are no conditions that will give us a different paths of execution.

Drilling down into the package se.sigma.calculator will give you the view below:

Further drilling into the class Calculator will give you the view below:

We note that each executed line is counted and the number of times it has been executed is reported. This doesn’t tell us anything about the quality of a test, it just tells us that the line has been executed. The quality of the tests has to be validated by examining the test cases themselves. I don’t know of any good tool that will help you here. You must use your experience as developer and your domain knowledge about the domain problem to know if this is a good test that tests the correct things or not.

As a side effect, we are publishing a nice way to browse the source code.

This case was easy since it is just one Maven module. A multi module Maven project is trickier if the production code and test code test is defined in different modules. The reason lays in the design of Maven. Maven will execute all build steps for the entire module before moving on to the next module. This is the only reasonable way to design a tool like Maven, but it clashes somewhat with a measuring tool like Cobertura.

A more complicated case – multiple Maven modules

If we have tests defined in one module and the production code in another module then we will have a problem with the behaviour of Maven. It performs all it’s build phases for each module before it continues with the next module. This is by design and normally a good thing.

Given a setup as below:

multi-module-failing-example
|-- pom.xml
|-- product
|   |-- pom.xml
|   `-- src
|       `-- main
|           `-- java
|               `-- se
|                   `-- sigma
|                       `-- calculator
|                           `-- Calculator.java
`-- test
    |-- pom.xml
    `-- src
        `-- test
            `-- java
                `-- se
                    `-- sigma
                        `-- calculator
                            `-- CalculatorTest.java

The production code and the test code is the same as before. A dependency has been added from the test module to the product so the Calculator can be found from the test. The only real difference is that I choose to separate production and test code in different Maven modules. The reasons for doing that may differ, in my cases I tend to have slow integration or acceptance test defined like this. It normally works very well, but test coverage is difficult to calculate using Maven.

I include the Maven poms for reference. They are the only thing that differs in this example compared to the first example.

The root pom is defined as:

pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <groupId>se.thinkcode</groupId> <artifactId>multi-module-failing-example</artifactId> <version>1.0</version> <packaging>pom</packaging> <modules> <module>product</module> <module>test</module> </modules> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> </properties> <build> <plugins> <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>cobertura-maven-plugin</artifactId> <version>2.5.1</version> <executions> <execution> <phase>post-integration-test</phase> <goals> <goal>cobertura</goal> </goals> </execution> </executions> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> </dependency> </dependencies> </project>

It has the same cobertura-maven-plugin defined and a report is generated. It also refers to the two modules that contains the product and the test in it’s modules section.

The product pom is defined as:

product/pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <parent> <groupId>se.thinkcode</groupId> <artifactId>multi-module-failing-example</artifactId> <version>1.0</version> </parent> <groupId>se.thinkcode</groupId> <artifactId>calculator</artifactId> <version>1.0</version> </project>

The test pom is defined as below:

test/pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <parent> <groupId>se.thinkcode</groupId> <artifactId>multi-module-failing-example</artifactId> <version>1.0</version> </parent> <groupId>se.thinkcode</groupId> <artifactId>calculator-test</artifactId> <version>1.0</version> <dependencies> <dependency> <groupId>se.thinkcode</groupId> <artifactId>calculator</artifactId> <version>1.0</version> </dependency> </dependencies> </project>

It has a dependency to the production code so the test can get hold of the Calculator and get something to execute.

The result is a below:

We see that we have 0% code coverage. Even though we know that we should have 100% in this case.

Drilling down into the package tells us the same thing, no execution of the code has been recorded.

Further drilling revels the Calculator class. We see that we had the possibility to run two lines. We also notice that Cobertura didn’t record any execution of the code.

This is by design. The classes the test executes has not been instrumented yet.

A solution is to use a combination of tools. Maven and Ant in combination can solve this problem. Maven is great for compiling the classes and executing the tests. All issues with class path and similar are handled nicely by Maven. Ant is extremely flexible and can be used to instrument the compiled classes and then merge the result after executing the tests to a coverage report.

A working multiple Maven module example

The file layout in this example is the same as to the example above, with the exception that I have added an Ant build.xml into the mix.

multi-module-example
|-- build.xml
|-- pom.xml
|-- product
|   |-- pom.xml
|   `-- src
|       `-- main
|           `-- java
|               `-- se
|                   `-- sigma
|                       `-- calculator
|                           `-- Calculator.java
`-- test
    |-- pom.xml
    `-- src
        `-- test
            `-- java
                `-- se
                    `-- sigma
                        `-- calculator
                            `-- CalculatorTest.java

The execution has to be done in four steps.

  1. Compile all code
  2. Instrument the code
  3. Execute all tests
  4. Consolidate and build the report

It means that you have to execute the commands below in this order:

mvn clean compile
ant instrument
mvn test
ant report

The root pom is defined as below:

pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <groupId>se.thinkcode</groupId> <artifactId>multi-module-example</artifactId> <version>1.1</version> <packaging>pom</packaging> <modules> <module>product</module> <module>test</module> </modules> <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> </properties> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-clean-plugin</artifactId> <version>2.4.1</version> <configuration> <filesets> <fileset> <directory>.</directory> <includes> <include>**/*.ser</include> </includes> </fileset> </filesets> </configuration> </plugin> </plugins> </build> <dependencies> <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.10</version> </dependency> </dependencies> </project>

It doesn’t use any Cobertura plugin. The cobertura instrumentation and reporting will be done by the Ant script.

It uses the Maven Clean plugin to remove files Cobertura uses to store which lines has been executed. They are called cobertura.ser as default. I add sum.ser in the Ant script and therefore I want to remove all *.ser files from the project during the clean phase. This really doesn’t matter for this example, it is just house keeping.

The Ant build script is defined as:

build.xml

<project> <target name="instrument"> <!-- Add all modules that should be included below --> <!-- <antcall target="instrumentAModule"> <param name="module" value="MODULE_NAME_TO_REPLACE"/> </antcall> --> <antcall target="instrumentAModule"> <param name="module" value="product"/> </antcall> </target> <target name="report" depends="merge"> <property name="src.dir" value="src/main/java/"/> <cobertura-report datafile="sum.ser" format="html" destdir="./target/report"> <!-- Add all modules that should be included below --> <!-- fileset dir="./MODULE_NAME_TO_REPLACE/${src.dir}"/ --> <fileset dir="./product/${src.dir}"/> </cobertura-report> </target> <target name="merge"> <cobertura-merge datafile="sum.ser"> <fileset dir="."> <include name="**/cobertura.ser"/> </fileset> </cobertura-merge> </target> <target name="instrumentAModule"> <property name="classes.dir" value="target/classes"/> <cobertura-instrument todir="./${module}/${classes.dir}"> <fileset dir="./${module}/target/classes"> <include name="**/*.class"/> </fileset> </cobertura-instrument> </target> <property environment="env"/> <property name="COBERTURA_HOME" value="/Users/tsu/java/cobertura-1.9.4.1"/> <property name="cobertura.dir" value="${COBERTURA_HOME}"/> <path id="cobertura.classpath"> <fileset dir="${cobertura.dir}"> <include name="cobertura.jar"/> <include name="lib/**/*.jar"/> </fileset> </path> <taskdef classpathref="cobertura.classpath" resource="tasks.properties"/> </project>

The two main targets to understand are

  • instrument
  • report

The instrument target will prepare the compiled classes with Cobertura instrumentation so they are able to record their usage.

The report target is used to combine the result from the execution and build to coverage report.

To get this example up and running, you need to download Cobertura. I used the link http://cobertura.sourceforge.net/download.html and downloaded the binary version of Cobertura 1.9.4.1. I unzipped the downloaded file into the directory /Users/tsu/java/cobertura-1.9.4.1 on my computer. You will ned to update the reference in the build.xml to match your specific location.

The production pom is identical to the production pom in the previous example, it doesn’t know anything about code coverage or Cobertura.

product/pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <parent> <groupId>se.thinkcode</groupId> <artifactId>multi-module-example</artifactId> <version>1.1</version> </parent> <groupId>se.thinkcode</groupId> <artifactId>calculator</artifactId> <version>1.1</version> </project>

The test project is almost identical to the test module pom above.

test/pom.xml

<?xml version="1.0" encoding="UTF-8"?> <project> <modelVersion>4.0.0</modelVersion> <parent> <groupId>se.thinkcode</groupId> <artifactId>multi-module-example</artifactId> <version>1.1</version> </parent> <groupId>se.thinkcode</groupId> <artifactId>calculator-test</artifactId> <version>1.1</version> <dependencies> <dependency> <groupId>se.thinkcode</groupId> <artifactId>calculator</artifactId> <version>1.1</version> </dependency> <dependency> <groupId>net.sourceforge.cobertura</groupId> <artifactId>cobertura</artifactId> <version>1.9.4.1</version> </dependency> </dependencies> </project>

The test project need a dependency to the Cobertura plugin to be able to record the execution during the test phase. The production code has been instrumented to use Cobertura so it needs to be available in the test classpath.

Executing the commands

mvn clean compile
ant instrument
mvn test
ant report

should now create a report like this:

It tells us that we have been able to count the times all lines has been executed.

Drilling down into the package se.sigma.calculator will give you the view below:

Further drilling into the class Calculator will show each executed line of code.

That’s it folks!
If the classes are properly instrumented before the test execution, then Cobertura will be able to record the execution of each line and a coverage report is possible to compile afterwards.

Acknowledgements

This post has been reviewed by Malin Ekholm.

Thank you very much for your feedback!

Resources

About these ads

28 Comments »

  1. Great post. I’ve used it and it works excellently. I adapted it using the maven antrun plugin so you can have everything in the parent pom file, optionally with external ant script files. Using the plugin you don’t need the cobertura installation and only need to run mvn twice and you have your report. If you’re interested in more details, let me know.

    Comment by Wouter — August 8, 2012 @ 14:19

    • hi Wouter,
      can you please provide your pom.xml for multi module project.

      Comment by gourab — November 5, 2012 @ 22:14

      • Hi Wouter,
        Yes, that would be great if you could post your pom file here. Thanks a lot.

        Comment by moudug — November 15, 2012 @ 13:29

    • Hi Wouter,
      I’m having the problem of multi-module and test coverage. I read this post, but it works with cobertura plugin and in my project we don’t use this plugin. Could you please send me your pom without cobertura? is that has to do with the command run? Is it possible to run ant command with a project maven?

      Regards,

      Sarang (igiagante@gmail.com)

      Comment by Ignacio — November 12, 2012 @ 16:49

      • We use cobertura :P.

        Comment by Ignacio — November 13, 2012 @ 13:37

    • can uu post u r pom please ..i am facing issues/////

      Comment by anandkatti108 — February 7, 2013 @ 13:35

  2. Hi
    we are getting stuck in maven pom.xml to get coverage report if the test case are executing fine but data tables are not available now becasue they are old for some modules and we wont use further but it is affecting the overall coverage report

    can you please help and my email id is shankerdeee1@yahoo.co.in

    Comment by shanker — August 20, 2012 @ 08:51

  3. Hi Thomas,

    I received the error: [taskdef] Could not load definitions from resource tasks.properties. It could not be found. could you help me please? Thanks

    Regards,

    Sarang

    Comment by Ignacio — November 14, 2012 @ 14:51

    • Hi!

      Does the cobertura.classpath contain the proper jars? If it doesn’t then I would expect this error to occur.
      Double check your paths, there might be an error there.

      HTH
      Thomas

      Comment by Thomas Sundberg — November 14, 2012 @ 20:29

      • I could resolve all problems. The path of my cobertura.dir was wrong. Now, I have another problem. When I run report and want to see the result of the coverage, it appear all in red, 0% coverage over all class tested. What could be?

        Comment by Ignacio — November 14, 2012 @ 23:36

      • One reason may be that you have somehow failed to annotate the classes before executing the tests. Another may be that you haven’t combined the .ser files properly. I would double check the order the steps are executed in.

        HTH
        Thomas

        Comment by Thomas Sundberg — November 15, 2012 @ 06:14

      • I tried to post the ant.xml with the structure of my project and I did not have success. Please, send me your e-mail, and I will send you if you do not matter.

        Comment by Ignacio — November 15, 2012 @ 14:38

      • The files cobertura.ser and sum.ser are located in a folder call ant, which contains the ant.xml file. Is correct
        to have these files in the root project?

        Structure project:

        -Project root
        — ant
        — ant.xml
        — cobertura.ser
        — sum.ser
        — target
        — reports

        * The commands are ran in the next steps:

        — mvn clean compile
        — ant instrument
        — mvn test
        — ant report


        Comment by Ignacio — November 15, 2012 @ 14:43

  4. Thomas!! I got it. One path was wrong. Now, I have to configure jenkins to run these commands in order to get the correct coverage. Any suggestion for that?

    Thanks!

    Comment by Ignacio — November 16, 2012 @ 15:01

    • Hi!

      Glad you found the problem.

      My suggestion if you want to run this using Jenkins is to create freestyle project. Check out the source and add four consecutive steps. First the Maven compile step, then the Ant instrument step, then the Maven test step and finally the Ant report step. The last thing you would like to do is to have Jenkins to display the result. You can add support for Cobertura reports at the end of the job configuration. I don’t remember the details and I don’t have a Jenkins that I could try with to get the commands correct. But Jenkins has a Cobertura plugin that you can use. I think it s a plugin at least, it could be built in as well.

      HTH
      Thomas

      Comment by Thomas Sundberg — November 17, 2012 @ 08:55

      • [cobertura-merge] Cobertura 1.9.4.1 – GNU GPL License (NO WARRANTY) – See COPYRIGHT file
        [cobertura-merge] Exception in thread “main” java.lang.NoClassDefFoundError: org.apache.log4j.Logger

        Comment by anandkatti108 — February 7, 2013 @ 13:34

  5. Hi Thomas,

    Thanks for this interesting post. I’m wondering how surefire plugin knows about the generated .ser file in your example.
    I tried to pass it via system.properties.variable or -D option but it seems it doesn’t work.
    Tests are executed against instrumented libraries but .ser file is not updated.
    If I run unit tests with ant and by providing the .ser it works fine.

    Regards

    Comment by jnc — April 30, 2013 @ 15:22

    • Sorry for the wrong information, but writing it down finally helped me to find the trick.

      Surefire was running tests against non instrumented libraries (the one declared in project dependency section), as I instrument previously generated artifacts.

      By excluding these project dependencies in surefire configuration and by adding additional classpath element pointing to the instrumented dependencies it works like a charm.

      Thanks again for your post.

      Comment by jnc — April 30, 2013 @ 15:41

      • Hi!

        I am glad that you found my post valuable and that you found the problem. I obviously acted as a rubber duck without even knowing it :)

        /Thomas

        Comment by Thomas Sundberg — May 1, 2013 @ 10:01

  6. Thomas,

    THANK YOU for posting this solution. I spent a day looking for a solution, many of which are either out of date or simple don’t work. My frustration has ended with your post. It was easy to read and easy to understand and I don’t even know how to use Ant, but picked up on it quickly.

    Thanks again!! =)

    Comment by Billy — April 30, 2013 @ 23:00

  7. Thanks A lot Thomas, it helped me a lot .. I implement your solution with maven-antrun-plugin http://mebada.blogspot.com/2013/05/how-to-use-maven-with-cobertura-code.html it may help

    Comment by worldpython — May 27, 2013 @ 16:49

  8. I do have an antrun example, but not one that describes how to solve the problem with multi module test coverage.

    http://thomassundberg.wordpress.com/2011/02/11/maven-and-antrun/

    /Thomas

    Comment by Thomas Sundberg — October 8, 2013 @ 22:07


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Silver is the New Black Theme. Create a free website or blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 60 other followers

%d bloggers like this: