Maven and TestNG

Posted by Roger Keays, 8 November 2006, 9:58 AM

I'm fairly new to maven, but so far it has really impressed me with its simplicity, productivity gains and IDE independence. So I was really looking forward to porting some unit tests to TestNG, running mvn test and watching in awe as maven found my tests, ran them and generated a pretty html report of the test results. Unfortunately, it didn't turn out quite like that.

It would seem there are some unfortunate bugs in surefire (maven's test runner) and these extended my five minute job into six hours of bewilderment - with occasional profanities. Anyway, to cut a long story short, here is a list of problems I encountered using TestNG with maven, and the ultimate solution I settled for.

The Problems

Each of the Test* classes were found by surefire, but none of the tests in them were run. It turns out that this is a known bug [1], and the workaround is use an xml test suite instead of relying on class name matching.
A TestNG exception caused by a missing parameter for an @BeforeSuite method was being hidden by surefire, and the build failing mysteriously. I only discovered the problem when I changed @BeforeSuite to @Test.
The surefire-report reruns the test cases. Again, this is a known bug [2] and isn't critical, but it is quite annoying. To only execute the tests once, you can run mvn surefire-report:report instead of mvn test.
The test report repeats the results from all the previous tests in each section, so you end up with longer and longer lists of repeated data (another known bug [3]).
The straw which broke the camels back: Nested exceptions are not shown in the test output. Many of my test failures were caused by "An expection has occurred, please check the nested exception for details".

A Solution

Fortunately, maven can run ant scripts and the testng jars include some ant tasks for executing testng 'natively'. We can configure surefire not to run the test cases and use the <testng> ant task instead. Also, the <junitreport> task can be used to generate a similar html report to that generated by the surefire-report plugin.

<build>
  <plugins>
    <plugin>
      <artifactId>maven-antrun-plugin</artifactId>
      <executions>

        <!-- this ant script runs testng natively -->
        <execution>
          <id>testng</id>
          <phase>test</phase>
          <configuration>
            <tasks>
              <taskdef resource="testngtasks" classpath="testng.jar"
                       classpathref="maven.test.classpath"/>
              <testng classpathref="maven.test.classpath"
                      outputdir="target/test-reports">
                <xmlfileset dir="src/test/suites" includes="*.xml"/>
              </testng>
              <junitreport todir="target/test-reports">
                <fileset dir="target/test-reports">
                  <include name="**/*.xml" />
                </fileset>
                <report format="noframes" todir="target/test-reports" />
              </junitreport>
            </tasks>
          </configuration>
          <goals><goal>run</goal></goals>
        </execution>
      </executions>
      <dependencies>
        <dependency>
          <groupId>ant</groupId>
          <artifactId>ant-junit</artifactId>
          <version>1.6.2</version>
        </dependency>
      </dependencies>
    </plugin>

    <!-- disable surefire plugin (too many problems!) -->
    <plugin>
      <groupId>org.apache.maven.plugins</groupId>
      <artifactId>maven-surefire-plugin</artifactId>
      <configuration>
        <skip>true</skip>
      </configuration>
    </plugin>
  </plugins>
</build>

Of course you'll need to make your project dependent on TestNG also:

<dependency>
  <groupId>org.testng</groupId>
  <artifactId>testng</artifactId>
  <version>5.1</version>
  <scope>test</scope>
  <classifier>jdk15</classifier>
</dependency>

References

[1] http://jira.codehaus.org/browse/MSUREFIRE-134
[2] http://jira.codehaus.org/browse/MSUREFIREREP-6
[3] http://jira.codehaus.org/browse/MSUREFIRE-172

<< Experiences With DynaFaces

Back to Blog

Glassfish vs Tomcat >>

Comment posted by: Binil Thomas on 8 November 2006, 10:37 PM

Hi Roger,

Thanks for the informative post.
Do you know a way to configure a TestNG listener when running the tests from Surefire?

Thanks,
Binil

Comment posted by: Roger Keays on 9 November 2006, 5:06 PM

Hey Binil,

I don't know if that can be done with Surefire, but it's another one that can be done with the ant task by using the listeners attribute.

Comment posted by: Tomek on 5 March 2007, 10:14 PM

maven-surefire-plugin ver. 2.3 fixes bug nr 1
the other two should be fixed in 2.4

thx for this post
Tomek
http://kaczanowscy.pl/tomek

Comment posted by: Daren on 14 May 2008, 3:51 PM

Thanks for the great post! I can't believe we're halfway through 2008 and the surefire/TestNG combination is still so unstable! I've tried every version combination of surefire and TestNG and have lost all hope of getting it to work. This Ant workaround is just what the Doctor ordered.

Thanks! - Daren