JavaPulse

a finger on the pulse of the freelance Java™ market in the Netherlands

Coverage: EMMA, Cobertura, Maven

Posted on | 2 September 2008 |
Tags: | | |

I finished adding test coverage reports to the nightly build a couple of weeks ago, but only got around to write about it now. This post describes the details of my research into both EMMA and Cobertura. In the end, I went with EMMA mainly because of the server support.

Summary

EMMA Cobertura
Dump and reset coverage data without shutting down application server Yes No
Source files needed for report No Yes
Wildcard to specify multiple source directories in Ant reporting Yes No
External dependencies (Cobertura’s ASM version is a potential conflict) No Yes
Released Maven plugin No Yes
Active development No Yes

Applications running on a server require the following phases for coverage: instrumentation, running, data collection, and report generation.

Note: In Maven, I’ve put all EMMA related cofiguration in an ‘emma’ profile and all Cobertura-related configuration in an ‘cobertura’ profile, each activated by the coverage property (-Dcoverage=emma or -Dcoverage=cobertura) so only one can be run at a time.

Instrumentation

There are not much difference between EMMA and Cobertura in the instrumentation phase. Both Maven plugins are similar to use and configure, although Cobertura seems to be more clear in specifying paths.

Field Reflection

Note that because classes are that are instrumented have added fields to hold coverage data, it is not possible to reflect through all fields of an instrumented class because EMMA/Cobertura fields are not accessible at runtime. Therefore, it is better to reflect using names of fields that you know are there instead of through all fields. If code change is not possible, exclude the classes to be reflected from the instrumentation.

Our project is a multi-module Maven project. To instrument all modules, only the master pom must be altered.

EMMA

<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>emma-maven-plugin</artifactId>
<inherited>true</inherited>
<executions>
<execution>
<id>instr-emma</id>
<phase>process-classes</phase>
<configuration>
<filters>
<filter>+com.mycompany.*</filter>
<filter>-com.mycompany.*.SomeClass</filter>
<filter>-com.mycompany.*.*Test</filter>
</filters>
</configuration>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

To access the emma-maven-plugin, the CodeHaus snapshot repository must be added to your pom:

<pluginRepositories>
<pluginRepository>
<id>snapshot.codehaus.org</id>
<name>CodeHaus Plugin Snapshots</name>
<url>http://snapshots.repository.codehaus.org</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>

Cobertura

Cobertura is very similar:

<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<inherited>true</inherited>
<executions>
<execution>
<id>instr-cobertura</id>
<phase>process-classes</phase>
<configuration>
<instrumentation>
<includes>
<include>com/mycompany/**/*.class</include>
</includes>
<excludes>
<exclude>com/mycompany/**/SomeClass.class</exclude>
<exclude>com/mycompany/**/*Test.class</exclude>
</excludes>
</instrumentation>
</configuration>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Running

Running is the same with or without coverage.
Unit tests are run with the surefire plugin in Maven. While integration tests that require applications to be running on a server must be packaged and deployed before they are run from the client side.

EMMA

Because an application running with EMMA connects to the socket ‘47653′ by default, a running application server and the test client will conflict for the socket. Therefore, if the test client is running on the same machine as the application server, configure the the socket for the test client by setting the system property -Demma.rt.control.port=12345 for the test client.

System Properties

Although the surefire plugin is supposed to allow passing in system properties (see http://maven.apache.org/plugins/maven-surefire-plugin/examples/system-properties.html), it did not work when I tried it. Therefore, I moved back to Ant to run the tests.

Data Collection

EMMA

The main reason I chose EMMA is because of the ability to dump and reset coverage data without shutting down the application server. EMMA (from version 2.1.5320) provides a tool (ctl) for getting a dump from a server and resetting the data. The ctl tool is accessible from the command line or from Ant. This is useful for running several types of tests where you want to collect coverage data for each separately. Here I use Ant so that it can be platform independent (instead of a .bat or .sh file).

After declaring the emma task (see Appendix), the following is the Ant target to be run from the POM’s of the test projects:

<target name=”emma-server-dump”>
<delete file=”${project.name}/coverage-${server}.ec” failonerror=”false”/>
<emma>
<ctl connect=”${server}:47653″>
<command name=”coverage.get” args=”${project.name}/coverage-${server}.ec” />
<command>coverage.reset</command>
</ctl>
</emma>
</target>

The properties are as follows:

  • ${project.name}: the project where to put the coverage data file
  • ${server}: the server from which to collect coverage data

    • Cobertura

    Although Cobertura provides an API (See http://cobertura.sourceforge.net/faq.html) for dumping coverage data from an application running on a server, it must be coded, deployed, and exposed to be triggered yourself. Furthermore, there is no API to reset the data. Therefore, to achieve the same functionality as EMMA, the server must be restarted between different test runs.

    Report Generation

    For both EMMA and Cobertura, it is not possible to specify more than one source directory in the report configuration in Maven. Therefore, generating the report in Maven is only useful for single module Maven projects or for running unit tests where only one source directory is to be considered for the report. Because of this limitation, I reverted to Ant to generate the coverage report for both EMMA and Cobertura.

    EMMA

    The following is a portion of the Ant build file after the emma task has been defined (see Appendix).

    <target name=”emma-report”>
    <delete failonerror=”false” dir=”${report.dir}” />
    <mkdir dir=”${report.dir}” />
    <emma>
    <report>
    <sourcepath>
    <dirset dir=”${top.dir}”>
    <include name=”**/src” />
    <include name=”**/ejbModule” />
    <include name=”**/Javasource” />
    </dirset>
    </sourcepath>
    <fileset dir=”${top.dir}”>
    <include name=”${project.dirs}/target/coverage.em” />
    <include name=”${project.name}/coverage*.ec” />
    <exclude name=”*Test*/target/coverage.em” />
    </fileset>
    <html outfile=”${report.dir}/index.html” />
    </report>
    </emma>
    </target>

    The properties are as follows:

  • ${report.dir}: the directory where the coverage report will be generated
  • ${top.dir}: top project directory when the master pom resides
  • ${project.dirs}: project(s) from where to get instrumentation meta-data (.em). can be a specific name or include wildcard (*)
  • ${project.name}: project(s) from where to get the coverage data (.ec). can be a specific name or include wildcard (*)

    • Cobertura

    Because it is not possible to specify more than one source directory in a wildcard, configuring cobertura for reporting is much more annoying. Furthermore, meta-data files and coverage data files (.ser) must be first merged because only one such file can be given for the report.

    <target name=”merge”>
    <cobertura-merge datafile=”${report.dir}/${test.name}.ser”>
    <fileset dir=”${top.dir}”>
    <include name=”${project1}/target/cobertura/cobertura.ser” />
    <exclude name=”${project2}/target/cobertura/cobertura.ser” />
    </fileset>
    </cobertura-merge>
    </target>
    <target name=”cobertura-report” depends=”merge”>
    <cobertura-report format=”html” destdir=”${report.dir}” datafile=”${report.dir}/${test.name}.ser”>
    <fileset dir=”${src.dir.1}”><include name=”**/*.java” /></fileset>
    <fileset dir=”${src.dir.2}”><include name=”**/*.java” /></fileset>
    <fileset dir=”${src.dir.3}”><include name=”**/*.java” /></fileset>
    …
    </cobertura-report>
    </target>

    The “cobertura-report” target depends on the “merge” target.
    The properties are as follows:

  • ${report.dir}: the directory where the coverage report will be generated
  • ${test.name}: the name of the ser file where the (merged) coverage data resides. Must be only one file and merged previously.
  • ${project1}…${project2}: the project(s) to include/exclude from merge. Can contain wildcards because using Ant fileset.
  • ${src.dir.1}..${src.dir.3}: the source directories specified one at a time.

    • Appendix: Declaring EMMA / Cobertura in Ant

    EMMA

    <property name=”emma.home” value=”/usr/local/emma”/>
    <path id=”emma.lib”>
    <pathelement location=”${emma.home}/emma.jar” />
    <pathelement location=”${emma.home}/emma_ant.jar” />
    </path>
    <taskdef resource=”emma_ant.properties” classpathref=”emma.lib”/>

    Cobertura

    <property name=”cobertura.dir” value=”/usr/local/cobertura” />
    <path id=”cobertura.classpath”>
    <fileset dir=”${cobertura.dir}”>
    <include name=”cobertura.jar” />
    <include name=”lib/**/*.jar” />
    </fileset>
    </path>
    <taskdef classpathref=”cobertura.classpath” resource=”tasks.properties” />

    Comments

    Leave a Reply