Summary

Applications running on a server require the following phases for coverage: instrumentation, running, data collection, and report generation.

Note: In Maven, I’ve put all EMMA related cofiguration in an ‘emma’ profile and all Cobertura-related configuration in an ‘cobertura’ profile, each activated by the coverage property (-Dcoverage=emma or -Dcoverage=cobertura) so only one can be run at a time.

Instrumentation

There are not much difference between EMMA and Cobertura in the instrumentation phase. Both Maven plugins are similar to use and configure, although Cobertura seems to be more clear in specifying paths.

Field Reflection

Note that because classes are that are instrumented have added fields to hold coverage data, it is not possible to reflect through all fields of an instrumented class because EMMA/Cobertura fields are not accessible at runtime. Therefore, it is better to reflect using names of fields that you know are there instead of through all fields. If code change is not possible, exclude the classes to be reflected from the instrumentation.

Our project is a multi-module Maven project. To instrument all modules, only the master pom must be altered.

EMMA

<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>emma-maven-plugin</artifactId>
<inherited>true</inherited>
<executions>
<execution>
<id>instr-emma</id>
<phase>process-classes</phase>
<configuration>
<filters>
<filter>+com.mycompany.*</filter>
<filter>-com.mycompany.*.SomeClass</filter>
<filter>-com.mycompany.*.*Test</filter>
</filters>
</configuration>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

To access the emma-maven-plugin, the CodeHaus snapshot repository must be added to your pom:

<pluginRepositories>
<pluginRepository>
<id>snapshot.codehaus.org</id>
<name>CodeHaus Plugin Snapshots</name>
<url>http://snapshots.repository.codehaus.org</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</pluginRepository>
</pluginRepositories>

Cobertura

Cobertura is very similar:

<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<inherited>true</inherited>
<executions>
<execution>
<id>instr-cobertura</id>
<phase>process-classes</phase>
<configuration>
<instrumentation>
<includes>
<include>com/mycompany/**/*.class</include>
</includes>
<excludes>
<exclude>com/mycompany/**/SomeClass.class</exclude>
<exclude>com/mycompany/**/*Test.class</exclude>
</excludes>
</instrumentation>
</configuration>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>

Running

Running is the same with or without coverage.
Unit tests are run with the surefire plugin in Maven. While integration tests that require applications to be running on a server must be packaged and deployed before they are run from the client side.

EMMA

Because an application running with EMMA connects to the socket ‘47653′ by default, a running application server and the test client will conflict for the socket. Therefore, if the test client is running on the same machine as the application server, configure the the socket for the test client by setting the system property -Demma.rt.control.port=12345 for the test client.

System Properties

Although the surefire plugin is supposed to allow passing in system properties (see http://maven.apache.org/plugins/maven-surefire-plugin/examples/system-properties.html), it did not work when I tried it. Therefore, I moved back to Ant to run the tests.

Data Collection

EMMA

The main reason I chose EMMA is because of the ability to dump and reset coverage data without shutting down the application server. EMMA (from version 2.1.5320) provides a tool (ctl) for getting a dump from a server and resetting the data. The ctl tool is accessible from the command line or from Ant. This is useful for running several types of tests where you want to collect coverage data for each separately. Here I use Ant so that it can be platform independent (instead of a .bat or .sh file).

After declaring the emma task (see Appendix), the following is the Ant target to be run from the POM’s of the test projects:

<target name=”emma-server-dump”>
<delete file=”${project.name}/coverage-${server}.ec” failonerror=”false”/>
<emma>
<ctl connect=”${server}:47653″>
<command name=”coverage.get” args=”${project.name}/coverage-${server}.ec” />
<command>coverage.reset</command>
</ctl>
</emma>
</target>

The properties are as follows:

Cobertura

Although Cobertura provides an API (See http://cobertura.sourceforge.net/faq.html) for dumping coverage data from an application running on a server, it must be coded, deployed, and exposed to be triggered yourself. Furthermore, there is no API to reset the data. Therefore, to achieve the same functionality as EMMA, the server must be restarted between different test runs.

Report Generation

For both EMMA and Cobertura, it is not possible to specify more than one source directory in the report configuration in Maven. Therefore, generating the report in Maven is only useful for single module Maven projects or for running unit tests where only one source directory is to be considered for the report. Because of this limitation, I reverted to Ant to generate the coverage report for both EMMA and Cobertura.

EMMA

The following is a portion of the Ant build file after the emma task has been defined (see Appendix).

<target name=”emma-report”>
<delete failonerror=”false” dir=”${report.dir}” />
<mkdir dir=”${report.dir}” />
<emma>
<report>
<sourcepath>
<dirset dir=”${top.dir}”>
<include name=”**/src” />
<include name=”**/ejbModule” />
<include name=”**/Javasource” />
</dirset>
</sourcepath>
<fileset dir=”${top.dir}”>
<include name=”${project.dirs}/target/coverage.em” />
<include name=”${project.name}/coverage*.ec” />
<exclude name=”*Test*/target/coverage.em” />
</fileset>
<html outfile=”${report.dir}/index.html” />
</report>
</emma>
</target>

The properties are as follows:

Cobertura

Because it is not possible to specify more than one source directory in a wildcard, configuring cobertura for reporting is much more annoying. Furthermore, meta-data files and coverage data files (.ser) must be first merged because only one such file can be given for the report.

<target name=”merge”>
<cobertura-merge datafile=”${report.dir}/${test.name}.ser”>
<fileset dir=”${top.dir}”>
<include name=”${project1}/target/cobertura/cobertura.ser” />
<exclude name=”${project2}/target/cobertura/cobertura.ser” />
</fileset>
</cobertura-merge>
</target>
<target name=”cobertura-report” depends=”merge”>
<cobertura-report format=”html” destdir=”${report.dir}” datafile=”${report.dir}/${test.name}.ser”>
<fileset dir=”${src.dir.1}”><include name=”**/*.java” /></fileset>
<fileset dir=”${src.dir.2}”><include name=”**/*.java” /></fileset>
<fileset dir=”${src.dir.3}”><include name=”**/*.java” /></fileset>
…
</cobertura-report>
</target>

The “cobertura-report” target depends on the “merge” target.
The properties are as follows:

Appendix: Declaring EMMA / Cobertura in Ant

EMMA

<property name=”emma.home” value=”/usr/local/emma”/>
<path id=”emma.lib”>
<pathelement location=”${emma.home}/emma.jar” />
<pathelement location=”${emma.home}/emma_ant.jar” />
</path>
<taskdef resource=”emma_ant.properties” classpathref=”emma.lib”/>

Cobertura

<property name=”cobertura.dir” value=”/usr/local/cobertura” />
<path id=”cobertura.classpath”>
<fileset dir=”${cobertura.dir}”>
<include name=”cobertura.jar” />
<include name=”lib/**/*.jar” />
</fileset>
</path>
<taskdef classpathref=”cobertura.classpath” resource=”tasks.properties” />

Finally, I noticed that the Maven build (maven-dependency-plugin:copy-dependencies) added more external jars in the classpath than was previously in the Ant build. It contained gnujaxp which was added transitively by jfreechart, also used by the application. By excluding gnujaxp, the Axis call finally worked as expected. I guess the moral of the story is: when getting completely unexpected behavior, there is a good chance that it is dependency conflicts. Looking carefully at the Maven copied dependencies could reveal any unnecessary dependencies. It would be useful to know which part of the code uses which dependencies. In any case, I wonder how many implementation of jaxp are out there.

This is a not very well-documented or very obvious part of Maven because the <id> tag is not required in the pom as a child of <execution>. Usually ids are used so that they can be referenced but this is not the case here.

Specifying an id will allow separate configuration for each execution, but it does not reload dependencies. So if you need to use the same plugin with different dependencies, you still need to do a workaround such as the one I posted here under the section ‘Different dependencies for different executions of the same plugin’.

The latest version (1.0) of castor-maven-plugin for Maven2 in the central repo uses Castor version 0.9.7. However, the newest version of Castor (1.1.2.1) is not backward compatible! The castor plugin for Maven1 has been upgraded, but not for Maven2. Has Castor gone out of fashion since Maven2 was introduced?

Anyway, my project needed Castor 1.1.2.1 for some new functionality, so I downloaded to sources for castor-maven-plugin and the pom from the central repository upgraded the dependencies to Castor 1.1.2.1.

Instead of:
<dependency>
 <groupId>castor</groupId>
 <artifactId>castor</artifactId>
 <version>0.9.7</version>
</dependency>

I now have:
<dependency>
 <groupId>org.codehaus.castor</groupId>
 <artifactId>castor</artifactId>
 <version>1.1.2.1</version>
</dependency>
<dependency>
 <groupId>org.codehaus.castor</groupId>
 <artifactId>castor-codegen</artifactId>
 <version>1.1.2.1</version>
</dependency>
<dependency>
 <groupId>org.codehaus.castor</groupId>
 <artifactId>castor-jdo</artifactId>
 <version>1.1.2.1</version>
</dependency>

Some classes for the Castor builder has been moved, so the import statements for CastorSourceGenerator had to be changed:

From:
import org.exolab.castor.builder.CollectionInfo;
import org.exolab.castor.builder.FieldInfoFactory;

To:
import org.exolab.castor.builder.info.CollectionInfo;
import org.exolab.castor.builder.factory.FieldInfoFactory;

After rebuilding everything, a new version 1.0.1-SNAPSHOT was produced for castor-maven-plugin that uses the latest version (1.1.2.1) of Castor.

I haven’t yet had time to investigate how to get it into the central repository so you can download the RAR compressed directory of castor-maven-plugin version 1.0.1-SNAPSHOT here. It is to be placed at ${local.repo}/org/codehaus/mojo/castor-maven-plugin/ together with maven-metadata-local.xml.

Rebinding to a different phase and compiling generated sources after normal classes are already compiled
Normally, wsdl2java is bound to the generate-sources phase and both java2wsdl and admin are bound to the process-classes phase. But I don’t have the wsdl to process in the wsdl2java until after the process-classes phase. So I bound the wsdl2java to the generate-test-sources phase and trick Maven into compiling the generated sources as test sources by setting the testSourceDirectory to the wsdl2java generated sources and testOutputDirectory to the same as the normal sources.

<testSourceDirectory>${basedir}/target/generated-sources/axistools/wsdl2java</testSourceDirectory>
<testOutputDirectory>${basedir}/target/classes</testOutputDirectory>

Configuring a plugin for multiple execution

Since this post was published, I found that there is another way to get the multiple executions of the same pom to be reinstantiated. So the following section is no longer valid. See Addendum: Configuring a Maven2 plugin for multiple execution. However, the next section ‘Different dependencies for different executions of the same plugin’ is still valid until the bug is fixed.

Then I ran into the Maven bug #MNG-1323 – plugins are instantiated only once and cached according to the groupId/artifactId during any given Maven run. This means that configurations are not reapplied and dependencies are not reloaded in subsequent executions of the same plugin. To workaround the first problem, I had to put all the configuration properties in the first invocation (java2wsdl):

<configuration>
  <!– java2wsdl –>
  <filename>${project.name}.wsdl</filename>
  <classOfPortType><path to interface or class of webservice></classOfPortType>
  <location><url for access to webservice></location>
  <namespace><namespace used in SOAP></namespace>
  <style>rpc</style>
  <typeMappingVersion>1.1</typeMappingVersion>
  <use>encoded</use>
  <!– wsdl2java –>
  <all>true</all>
  <serverSide>true</serverSide>
  <sourceDirectory>${basedir}/target/generated-sources/axistools/java2wsdl</sourceDirectory>
  <wsdlFiles><wsdlFile>${project.name}.wsdl</wsdlFile></wsdlFiles>
  <noWrapped>false</noWrapped>
  <mappings>
    <mapping>
      <namespace><namespace used for webservice, specified above></namespace>
      <targetPackage><package for generated sources for wsdl2java></targetPackage>
    </mapping>
  </mappings>
  <implementationClassName><path to implementation class of webservice></implementationClassName> <!– This will be generated as a template to be filled in, but because I started with a webservice implementation, the generated one should be delete from the target classes before packaging –>
  <servicePortName>${project.name}</servicePortName>
  <wrapArrays>false</wrapArrays> <!– This one must be specified to prevent mapping of arrays into lists. Notice that this was not configurable for axis ant tasks where it is false by default. –>
  <!– admin –>
  <inputFiles>
    <inputFile>${basedir}/target/generated-sources/axistools/wsdl2java/<remainder path to generated deploy descriptor>/deploy.wsdd</inputFile>
  </inputFiles>
  <isServerConfig>true</isServerConfig>
  <configOutputDirectory>${basedir}/target/${project.artifactId}-${version}/WEB-INF</configOutputDirectory>
</configuration>

Different dependencies for different executions of the same plugin
To workaround the second problem of applying different dependencies to different executions, I had to reinstall the plugin multiple times to different artifactId’s because the plugin are cached according to groupId/artifactId, so changing the version wouldn’t work. Then I found out that install:install-file doesn’t work for a plugin because the plugin descriptor would not be found. So I had to get the sources and invoke ‘mvn install’ for each set of dependencies after specifying the artifactId:

<artifactId>axistools-maven-plugin-dependencies1</artifactId>

The install from the sources will eventually call plugin:updateRegistry which makes the plugin descriptor available.

In Maven1, it was possible to specify the order of plugin execution within a single phase using the preGoal tag in a plugin configuration. However, this mechanism was removed in Maven2, causing the bug – MNG-2258: Wrong execution order of plugins in same phase. Since it is not expected to be fixed before Maven 2.1, the workaround that I use is to bind the various plugins to different phases, sometimes misusing phases to preserve order. Knowing the standard bindings, plugins can be bound to various phases that are part of the build lifecycle of the given packaging standard. Plugins bound to a phase that already has a default binding will always be executed after the default plugin. This workaround is preferable to resorting to Ant, because it uses Maven plugins, and so prevents having to configure classpaths for the Ant tasks. It is also preferable because when this bug is fixed in Maven, there would be minimal changes to the build process in the pom.

http://cvs.peopleware.be/training/maven/maven2/buildLifecyclePhases.html

As part of a huge refactoring effort, I’m working with the TestNG, Maven, Netbeans. Not surprisingly, everything has not been a breeze. So I put together these steps to set up test development using TestNG and Maven, and debugging with any IDE. In fact, TestNG can be easily replaced by JUnit here. It’s more about getting around the Maven build life cycle during test development.

1) Skipping the Maven test phase until test development is complete

The first problem I encountered was that the jars get created at the install phase near the end of the Maven build life cycle, which means that if the tests fail, the jar never gets created. Apparently, this is not meant for test development. The solution is to turn off the test phase by setting skip to true in the configuration of the surefire plugin, until all tests pass and can be used for automated testing.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<skip>true</skip>
<suiteXmlFiles>
<suiteXmlFile>src/test/config/testng.xml</suiteXmlFile>
</suiteXmlFiles>
</configuration>
</plugin>

2) Copying all dependencies to target/lib

This is usually done for any jar that is an executable (contains a main method), so that you can reference the dependencies easily from within the jar manifest.

<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>dependency-maven-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>${project.build.directory}/lib</outputDirectory>
<overWriteReleases>false</overWriteReleases>
<overWriteSnapshots>true</overWriteSnapshots>
</configuration>
</execution>
</executions>
</plugin>

3) Referencing dependencies in the jar manifest

The following creates a manifest in the jar with references to all the dependencies previously copied to lib. Note that the classPrefix is lib.

<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<jarName>myapp</jarName>
<archive>
<manifest>
<mainClass>com.mycompany.MyApp</mainClass>
<addClasspath>true</addClasspath>
<classpathPrefix>lib</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>

4) Packaging the tests

Next, I add a Maven goal to produce a jar of the test sources.

mvn clean install jar:test-jar

This produces a jar (myapp-tests.jar) in your target directory next to your normal jar (myapp.jar)

5) Running the tests with TestNG

Then, I run the tests on the command line using TestNG instead of via Maven.

java -cp ./target/myapp.jar;./target/myapp-tests.jar; org.testng.TestNG ./src/test/config/testng.xml

I put the testng.xml in the config directory instead of resources because it is a runtime configuration and does not need to be part of the jar. Everything in the resources directory gets added to the jar.

6) Debugging the tests
Debugging the tests is simple.

java -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,server=y,address=8888 -cp ./target/myapp.jar;./target/myapp-tests.jar; org.testng.TestNG ./src/test/config/testng.xml

From your IDE, attach your debugger to port 8888 using a SocketAttach to attach to an external program.