Coverage: EMMA, Cobertura, Maven
Posted on | 2 September 2008 | 3 Comments
Tags: cobertura | coverage | emma | maven
I finished adding test coverage reports to the nightly build a couple of weeks ago, but only got around to write about it now. This post describes the details of my research into both EMMA and Cobertura. In the end, I went with EMMA mainly because of the server support.
Summary
| EMMA | Cobertura | |
|---|---|---|
| Dump and reset coverage data without shutting down application server | Yes | No |
| Source files needed for report | No | Yes |
| Wildcard to specify multiple source directories in Ant reporting | Yes | No |
| External dependencies (Cobertura’s ASM version is a potential conflict) | No | Yes |
| Released Maven plugin | No | Yes |
| Active development | No | Yes |
Applications running on a server require the following phases for coverage: instrumentation, running, data collection, and report generation.
Note: In Maven, I’ve put all EMMA related cofiguration in an ‘emma’ profile and all Cobertura-related configuration in an ‘cobertura’ profile, each activated by the coverage property (-Dcoverage=emma or -Dcoverage=cobertura) so only one can be run at a time.
Instrumentation
There are not much difference between EMMA and Cobertura in the instrumentation phase. Both Maven plugins are similar to use and configure, although Cobertura seems to be more clear in specifying paths.
Field Reflection
Note that because classes are that are instrumented have added fields to hold coverage data, it is not possible to reflect through all fields of an instrumented class because EMMA/Cobertura fields are not accessible at runtime. Therefore, it is better to reflect using names of fields that you know are there instead of through all fields. If code change is not possible, exclude the classes to be reflected from the instrumentation.
Our project is a multi-module Maven project. To instrument all modules, only the master pom must be altered.
EMMA
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>emma-maven-plugin</artifactId>
<inherited>true</inherited>
<executions>
<execution>
<id>instr-emma</id>
<phase>process-classes</phase>
<configuration>
<filters>
<filter>+com.mycompany.*</filter>
<filter>-com.mycompany.*.SomeClass</filter>
<filter>-com.mycompany.*.*Test</filter>
</filters>
</configuration>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
To access the emma-maven-plugin, the CodeHaus snapshot repository must be added to your pom:
<pluginRepositories>
<pluginRepository>
<id>snapshot.codehaus.org</id>
<name>CodeHaus Plugin Snapshots</name>
<url>http://snapshots.repository.codehaus.org</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>
Cobertura
Cobertura is very similar:
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<inherited>true</inherited>
<executions>
<execution>
<id>instr-cobertura</id>
<phase>process-classes</phase>
<configuration>
<instrumentation>
<includes>
<include>com/mycompany/**/*.class</include>
</includes>
<excludes>
<exclude>com/mycompany/**/SomeClass.class</exclude>
<exclude>com/mycompany/**/*Test.class</exclude>
</excludes>
</instrumentation>
</configuration>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
Running
Running is the same with or without coverage.
Unit tests are run with the surefire plugin in Maven. While integration tests that require applications to be running on a server must be packaged and deployed before they are run from the client side.
EMMA
Because an application running with EMMA connects to the socket ‘47653′ by default, a running application server and the test client will conflict for the socket. Therefore, if the test client is running on the same machine as the application server, configure the the socket for the test client by setting the system property -Demma.rt.control.port=12345 for the test client.
System Properties
Although the surefire plugin is supposed to allow passing in system properties (see http://maven.apache.org/plugins/maven-surefire-plugin/examples/system-properties.html), it did not work when I tried it. Therefore, I moved back to Ant to run the tests.
Data Collection
EMMA
The main reason I chose EMMA is because of the ability to dump and reset coverage data without shutting down the application server. EMMA (from version 2.1.5320) provides a tool (ctl) for getting a dump from a server and resetting the data. The ctl tool is accessible from the command line or from Ant. This is useful for running several types of tests where you want to collect coverage data for each separately. Here I use Ant so that it can be platform independent (instead of a .bat or .sh file).
After declaring the emma task (see Appendix), the following is the Ant target to be run from the POM’s of the test projects:
<target name=”emma-server-dump”>
<delete file=”${project.name}/coverage-${server}.ec” failonerror=”false”/>
<emma>
<ctl connect=”${server}:47653″>
<command name=”coverage.get” args=”${project.name}/coverage-${server}.ec” />
<command>coverage.reset</command>
</ctl>
</emma>
</target>
The properties are as follows:
Cobertura
Although Cobertura provides an API (See http://cobertura.sourceforge.net/faq.html) for dumping coverage data from an application running on a server, it must be coded, deployed, and exposed to be triggered yourself. Furthermore, there is no API to reset the data. Therefore, to achieve the same functionality as EMMA, the server must be restarted between different test runs.
Report Generation
For both EMMA and Cobertura, it is not possible to specify more than one source directory in the report configuration in Maven. Therefore, generating the report in Maven is only useful for single module Maven projects or for running unit tests where only one source directory is to be considered for the report. Because of this limitation, I reverted to Ant to generate the coverage report for both EMMA and Cobertura.
EMMA
The following is a portion of the Ant build file after the emma task has been defined (see Appendix).
<target name=”emma-report”>
<delete failonerror=”false” dir=”${report.dir}” />
<mkdir dir=”${report.dir}” />
<emma>
<report>
<sourcepath>
<dirset dir=”${top.dir}”>
<include name=”**/src” />
<include name=”**/ejbModule” />
<include name=”**/Javasource” />
</dirset>
</sourcepath>
<fileset dir=”${top.dir}”>
<include name=”${project.dirs}/target/coverage.em” />
<include name=”${project.name}/coverage*.ec” />
<exclude name=”*Test*/target/coverage.em” />
</fileset>
<html outfile=”${report.dir}/index.html” />
</report>
</emma>
</target>
The properties are as follows:
Cobertura
Because it is not possible to specify more than one source directory in a wildcard, configuring cobertura for reporting is much more annoying. Furthermore, meta-data files and coverage data files (.ser) must be first merged because only one such file can be given for the report.
<target name=”merge”>
<cobertura-merge datafile=”${report.dir}/${test.name}.ser”>
<fileset dir=”${top.dir}”>
<include name=”${project1}/target/cobertura/cobertura.ser” />
<exclude name=”${project2}/target/cobertura/cobertura.ser” />
</fileset>
</cobertura-merge>
</target>
<target name=”cobertura-report” depends=”merge”>
<cobertura-report format=”html” destdir=”${report.dir}” datafile=”${report.dir}/${test.name}.ser”>
<fileset dir=”${src.dir.1}”><include name=”**/*.java” /></fileset>
<fileset dir=”${src.dir.2}”><include name=”**/*.java” /></fileset>
<fileset dir=”${src.dir.3}”><include name=”**/*.java” /></fileset>
…
</cobertura-report>
</target>
The “cobertura-report” target depends on the “merge” target.
The properties are as follows:
Appendix: Declaring EMMA / Cobertura in Ant
EMMA
<property name=”emma.home” value=”/usr/local/emma”/>
<path id=”emma.lib”>
<pathelement location=”${emma.home}/emma.jar” />
<pathelement location=”${emma.home}/emma_ant.jar” />
</path>
<taskdef resource=”emma_ant.properties” classpathref=”emma.lib”/>
Cobertura
<property name=”cobertura.dir” value=”/usr/local/cobertura” />
<path id=”cobertura.classpath”>
<fileset dir=”${cobertura.dir}”>
<include name=”cobertura.jar” />
<include name=”lib/**/*.jar” />
</fileset>
</path>
<taskdef classpathref=”cobertura.classpath” resource=”tasks.properties” />
Comments
3 Responses to “Coverage: EMMA, Cobertura, Maven”
Leave a Reply
January 11th, 2009 @ 11:34
[...] Open Wednesday talk 22C – Matthew 17:24-18:14 Lessons 1-3 Saved by Harini1008 on Tue 30-12-2008 Coverage: EMMA, Cobertura, Maven Saved by BB1A on Mon 29-12-2008 Boom boom boom boom part 2 Saved by joviaguilar on Thu 18-12-2008 [...]
March 9th, 2009 @ 12:35
[...] Remove restriction on script filenames First saved by jocktheripper1888 | 8 days ago Coverage: EMMA, Cobertura, Maven First saved by Grunzinger017 | 10 days ago Maven plugin for Eclipse First saved by wizzadofozz [...]
December 5th, 2009 @ 23:30
Your points about server support are valid, but you should be aware that EMMA and its successor JaCoCo base their coverage reporting on “basic blocks”, which do not match 100% with a programmer’s view of the source code, leading to false gaps in reported coverage. Cobertura does not have this flaw, and if you are shooting for 100% coverage that can be important.