TestNG

Table of Contents

Introduction

Writing a test is typically a three-step process:

• Write the business logic of your test and insert TestNG annotations in your code.
• Add the information about your test (e.g. the class name, the groups you wish to run, etc...) in a testng.xml file or in build.xml.
• Run TestNG.

The concepts used in this documentation are as follows:

• A suite is represented by one XML file. It can contain one or more tests and is defined by the <suite> tag.
• A test is represented by <test> and can contain one or more TestNG classes.
• A TestNG class is a Java class that contains at least one TestNG annotation. It is represented by the <class> tag and can contain one or more test methods.
• A test method is a Java method annotated by @Test in your source.

The rest of this manual will explain the following:

• A list of all the annotations with a brief explanation. This will give you an idea of the various functionalities offered by TestNG but you will probably want to consult the section dedicated to each of these annotations to learn the details.
• A description of the testng.xml file, its syntax and what you can specify in it.
• A detailed list of the various features and how to use them with a combination of annotations and testng.xml.

Annotations

@BeforeSuite @AfterSuite @BeforeTest @AfterTest @BeforeGroups @AfterGroups @BeforeClass @AfterClass @BeforeMethod @AfterMethod		Configuration information for a TestNG class: @BeforeSuite: The annotated method will be run before all tests in this suite have run. @AfterSuite: The annotated method will be run after all tests in this suite have run. @BeforeTest: The annotated method will be run before any test method belonging to the classes inside the <test> tag is run. @AfterTest: The annotated method will be run after all the test methods belonging to the classes inside the <test> tag have run. @BeforeGroups: The list of groups that this configuration method will run before. This method is guaranteed to run shortly before the first test method that belongs to any of these groups is invoked. @AfterGroups: The list of groups that this configuration method will run after. This method is guaranteed to run shortly after the last test method that belongs to any of these groups is invoked. @BeforeClass: The annotated method will be run before the first test method in the current class is invoked. @AfterClass: The annotated method will be run after all the test methods in the current class have been run. @BeforeMethod: The annotated method will be run before each test method. @AfterMethod: The annotated method will be run after each test method.
	alwaysRun	For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to. For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.
	dependsOnGroups	The list of groups this method depends on.
	dependsOnMethods	The list of methods this method depends on.
	enabled	Whether methods on this class/method are enabled.
	groups	The list of groups this class/method belongs to.
	inheritGroups	If true, this method will belong to groups specified in the @Test annotation at the class level.
@DataProvider		Marks a method as supplying data for a test method. The annotated method must return an Object[][] where each Object[] can be assigned the parameter list of the test method. The @Test method that wants to receive data from this DataProvider needs to use a dataProvider name equals to the name of this annotation.
	name	The name of this data provider. If it's not supplied, the name of this data provider will automatically be set to the name of the method.
	parallel	If set to true, tests generated using this data provider are run in parallel. Default value is false.
@Factory		Marks a method as a factory that returns objects that will be used by TestNG as Test classes. The method must return Object[].
@Listeners		Defines listeners on a test class.
	value	An array of classes that extend org.testng.ITestNGListener.
@Parameters		Describes how to pass parameters to a @Test method.
	value	The list of variables used to fill the parameters of this method.
@Test		Marks a class or a method as part of the test.
	alwaysRun	If set to true, this test method will always be run even if it depends on a method that failed.
	dataProvider	The name of the data provider for this test method.
	dataProviderClass	The class where to look for the data provider. If not specified, the data provider will be looked on the class of the current test method or one of its base classes. If this attribute is specified, the data provider method needs to be static on the specified class.
	dependsOnGroups	The list of groups this method depends on.
	dependsOnMethods	The list of methods this method depends on.
	description	The description for this method.
	enabled	Whether methods on this class/method are enabled.
	expectedExceptions	The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure.
	groups	The list of groups this class/method belongs to.
	invocationCount	The number of times this method should be invoked.
	invocationTimeOut	The maximum number of milliseconds this test should take for the cumulated time of all the invocationcounts. This attribute will be ignored if invocationCount is not specified.
	priority	The priority for this test method. Lower priorities will be scheduled first.
	successPercentage	The percentage of success expected from this method
	singleThreaded	If set to true, all the methods on this test class are guaranteed to run in the same thread, even if the tests are currently being run with parallel="methods". This attribute can only be used at the class level and it will be ignored if used at the method level. Note: this attribute used to be called sequential (now deprecated).
	timeOut	The maximum number of milliseconds this test should take.
	threadPoolSize	The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount. Note: this attribute is ignored if invocationCount is not specified

testng.xml

You can invoke TestNG in several different ways:

• With a testng.xml file
• With ant
• From the command line

This section describes the format of testng.xml (you will find documentation on ant and the command line below).

The current DTD for testng.xml can be found on the main Web site:  http://testng.org/testng-1.0.dtd (for your convenience, you might prefer to browse the HTML version).

testng.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
 
<suite name="Suite1" verbose="1" >
  <test name="Nopackage" >
    <classes>
       <class name="NoPackageTest" />
    </classes>
  </test>

  <test name="Regression1">
    <classes>
      <class name="test.sample.ParameterSample"/>
      <class name="test.sample.ParameterTest"/>
    </classes>
  </test>
</suite>

testng.xml

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >

<suite name="Suite1" verbose="1" >
  <test name="Regression1"   >
    <packages>
      <package name="test.sample" />
   </packages>
 </test>
</suite>

In this example, TestNG will look at all the classes in the package test.sample and will retain only classes that have TestNG annotations.

testng.xml

<test name="Regression1">
  <groups>
    <run>
      <exclude name="brokenTests"  />
      <include name="checkinTests"  />
    </run>
  </groups>
 
  <classes>
    <class name="test.IndividualMethodsTest">
      <methods>
        <include name="testMethod" />
      </methods>
    </class>
  </classes>
</test>

You can also define new groups inside testng.xml and specify additional details in attributes, such as whether to run the tests in parallel, how many threads to use, whether you are running JUnit tests, etc... 

By default, TestNG will run your tests in the order they are found in the XML file. If you want the classes and methods listed in this file to be run in an unpredictible order, set the preserve-order attribute to false

testng.xml

<test name="Regression1" preserve-order="false">
  <classes>

    <class name="test.Test1">
      <methods>
        <include name="m1" />
        <include name="m2" />
      </methods>
    </class>

    <class name="test.Test2" />

  </classes>
</test>

Please see the DTD for a complete list of the features, or read on.

Running TestNG

• Command line
• ant
• Eclipse
• IntelliJ's IDEA

Assuming that you have TestNG in your class path, the simplest way to invoke TestNG is as follows:

java org.testng.TestNG testng1.xml [testng2.xml testng3.xml ...]

This documentation can be obtained by invoking TestNG without any arguments.

You can also put the command line switches in a text file, say c:\command.txt, and tell TestNG to use that file to retrieve its parameters:

  C:> more c:\command.txt
  -d test-output testng.xml
  C:> java org.testng.TestNG @c:\command.txt

Additionally, TestNG can be passed properties on the command line of the Java Virtual Machine, for example

java -Dtestng.test.classpath="c:/build;c:/java/classes;" org.testng.TestNG testng.xml

java org.testng.TestNG -groups windows,linux -testclass org.test.MyTest

Important: The command line flags that specify what tests should be run will be ignored if you also specify a testng.xml file, with the exception of -includedgroups and -excludedgroups, which will override all the group inclusions/exclusions found in testng.xml.

Test methods, Test classes and Test groups

Test methods

<suite allow-return-values="true">

or

<test allow-return-values="true">

Test groups

TestNG allows you to perform sophisticated groupings of test methods. Not only can you declare that methods belong to groups, but you can also specify groups that contain other groups. Then TestNG can be invoked and asked to include a certain set of groups (or regular expressions) while excluding another set.  This gives you maximum flexibility in how you partition your tests and doesn't require you to recompile anything if you want to run two different sets of tests back to back.

Groups are specified in your testng.xml file and can be found either under the <test> or <suite> tag. Groups specified in the <suite> tag apply to all the <test> tags underneath. Note that groups are accumulative in these tags: if you specify group "a" in <suite> and "b" in <test>, then both "a" and "b" will be included.

For example, it is quite common to have at least two categories of tests

• Check-in tests.  These tests should be run before you submit new code.  They should typically be fast and just make sure no basic functionality was broken.
   
• Functional tests.  These tests should cover all the functionalities of your software and be run at least once a day, although ideally you would want to run them continuously.

Test1.java

public class Test1 {
  @Test(groups = { "functest", "checkintest" })
  public void testMethod1() {
  }

  @Test(groups = {"functest", "checkintest"} )
  public void testMethod2() {
  }

  @Test(groups = { "functest" })
  public void testMethod3() {
  }
}

testng.xml

<test name="Test1">
  <groups>
    <run>
      <include name="functest"/>
    </run>
  </groups>
  <classes>
    <class name="example1.Test1"/>
  </classes>
</test>

will run all the test methods in that classes, while invoking it with checkintest will only run testMethod1() and testMethod2().

Test1.java

@Test
public class Test1 {
  @Test(groups = { "windows.checkintest" }) 
  public void testWindowsOnly() {
  }

  @Test(groups = {"linux.checkintest"} )
  public void testLinuxOnly() {
  }

  @Test(groups = { "windows.functest" )
  public void testWindowsToo() {
  }
}

testng.xml

<test name="Test1">
  <groups>
    <run>
      <include name="windows.*"/>
    </run>
  </groups>

  <classes>
    <class name="example1.Test1"/>
  </classes>
</test>

Note: TestNG uses regular expressions, and not wildmats. Be aware of the difference (for example, "anything" is matched by ".*" -- dot star -- and not "*").

Method groups

testng.xml

<test name="Test1">
  <classes>
    <class name="example1.Test1">
      <methods>
        <include name=".*enabledTestMethod.*"/>
        <exclude name=".*brokenTestMethod.*"/>
      </methods>
     </class>
  </classes>
</test>

Groups of groups

testng.xml

<test name="Regression1">
  <groups>
    <define name="functest">
      <include name="windows"/>
      <include name="linux"/>
    </define>
 
    <define name="all">
      <include name="functest"/>
      <include name="checkintest"/>
    </define>
 
    <run>
      <include name="all"/>
    </run>
  </groups>
 
  <classes>
    <class name="test.sample.Test1"/>
  </classes>
</test>

Exclusion groups

TestNG allows you to include groups as well as exclude them.

A simple way to solve this problem is to create a group called "broken" and make these test methods belong to it.  For example, in the above example, I know that testMethod2() is now broken so I want to disable it:

Java

@Test(groups = {"checkintest", "broken"} )
public void testMethod2() {
}

testng.xml

<test name="Simple example">
  <groups>
    <run>
      <include name="checkintest"/>
      <exclude name="broken"/>
    </run>
  </groups>
 
  <classes>
    <class name="example1.Test1"/>
  </classes>
</test>

This way, I will get a clean test run while keeping track of what tests are broken and need to be fixed later.

Note:  you can also disable tests on an individual basis by using the "enabled" property available on both @Test and @Before/After annotations.

Partial groups

All.java

@Test(groups = { "checkin-test" })
public class All {

  @Test(groups = { "func-test" )
  public void method1() { ... }

  public void method2() { ... }
}

Parameters

Test methods don't have to be parameterless.  You can use an arbitrary number of parameters on each of your test method, and you instruct TestNG to pass you the correct parameters with the @Parameters annotation.

There are two ways to set these parameters:  with testng.xml or programmatically.

Parameters from testng.xml

Java

@Parameters({ "first-name" })
@Test
public void testSingleString(String firstName) { 
  System.out.println("Invoked testString " + firstName);
  assert "Cedric".equals(firstName);
}

testng.xml

<suite name="My suite">
  <parameter name="first-name"  value="Cedric"/>
  <test name="Simple example">
  <-- ... -->

The same technique can be used for @Before/After and @Factory annotations:

@Parameters({ "datasource", "jdbcDriver" })
@BeforeMethod
public void beforeTest(String ds, String driver) {
  m_dataSource = ...;                              // look up the value of datasource
  m_jdbcDriver = driver;
}

Parameters can be declared optional with the Optional annotation:

@Parameters("db")
@Test
public void testNonExistentParameter(@Optional("mysql") String db) { ... }

The @Parameters annotation can be placed at the following locations:

• On any method that already has a @Test, @Before/After or @Factory annotation.
• On at most one constructor of your test class.  In this case, TestNG will invoke this particular constructor with the parameters initialized to the values specified in testng.xml whenever it needs to instantiate your test class.  This feature can be used to initialize fields inside your classes to values that will then be used by your test methods.

Notes:

• The XML parameters are mapped to the Java parameters in the same order as they are found in the annotation, and TestNG will issue an error if the numbers don't match.
• Parameters are scoped. In testng.xml, you can declare them either under a <suite> tag or under <test>. If two parameters have the same name, it's the one defined in <test> that has precedence. This is convenient if you need to specify a parameter applicable to all your tests and override its value only for certain tests.

Parameters with DataProviders

Specifying parameters in testng.xml might not be sufficient if you need to pass complex parameters, or parameters that need to be created from Java (complex objects, objects read from a property file or a database, etc...). In this case, you can use a Data Provider to supply the values you need to test.  A Data Provider is a method on your class that returns an array of array of objects.  This method is annotated with @DataProvider:

Java

//This method will provide data to any test method that declares that its Data Provider
//is named "test1"
@DataProvider(name = "test1")
public Object[][] createData1() {
 return new Object[][] {
   { "Cedric", new Integer(36) },
   { "Anne", new Integer(37)}, 
 };
}

//This test method declares that its data should be supplied by the Data Provider
//named "test1"
@Test(dataProvider = "test1")
public void verifyData1(String n1, Integer n2) {
 System.out.println(n1 + " " + n2);
} 

Cedric 36
Anne 37

By default, the data provider will be looked for in the current test class or one of its base classes. If you want to put your data provider in a different class, it needs to be a static method or a class with a non-arg constructor, and you specify the class where it can be found in the dataProviderClass attribute:

StaticProvider.java

public class StaticProvider {
  @DataProvider(name = "create")
  public static Object[][] createData() {
    return new Object[][] {
      new Object[] { new Integer(42) }
    };
  }
}

public class MyTest {
  @Test(dataProvider = "create", dataProviderClass = StaticProvider.class)
  public void test(Integer n) {
    // ...
  }
}

• An array of array of objects (Object[][]) where the first dimension's size is the number of times the test method will be invoked and the second dimension size contains an array of objects that must be compatible with the parameter types of the test method. This is the cast illustrated by the example above.
• An Iterator<Object[]>. The only difference with Object[][] is that an Iterator lets you create your test data lazily. TestNG will invoke the iterator and then the test method with the parameters returned by this iterator one by one. This is particularly useful if you have a lot of parameter sets to pass to the method and you don't want to create all of them upfront.

@DataProvider(name = "test1")
public Iterator<Object[]> createData() {
  return new MyIterator(DATA);
} 

For example, the following code prints the name of the test method inside its @DataProvider:

@DataProvider(name = "dp")
public Object[][] createData(Method m) {
  System.out.println(m.getName());  // print test method name
  return new Object[][] { new Object[] { "Cedric" }};
}

@Test(dataProvider = "dp")
public void test1(String s) {
}

@Test(dataProvider = "dp")
public void test2(String s) {
}

test1
test2

@DataProvider(parallel = true)
// ...

<suite name="Suite1" data-provider-thread-count="20" >
... 

Parameters in reports

Parameters used to invoke your test methods are shown in the HTML reports generated by TestNG. Here is an example:

Dependencies

Sometimes, you need your test methods to be invoked in a certain order.  Here are a few examples:

• To make sure a certain number of test methods have completed and succeeded before running more test methods.
• To initialize your tests while wanting this initialization methods to be test methods as well (methods tagged with @Before/After will not be part of the final report).

Dependencies with annotations

You can use the attributes dependsOnMethods or dependsOnGroups, found on the @Test annotation.

• Hard dependencies. All the methods you depend on must have run and succeeded for you to run. If at least one failure occurred in your dependencies, you will not be invoked and marked as a SKIP in the report.
• Soft dependencies. You will always be run after the methods you depend on, even if some of them have failed. This is useful when you just want to make sure that your test methods are run in a certain order but their success doesn't really depend on the success of others. A soft dependency is obtained by adding "alwaysRun=true" in your @Test annotation.

@Test
public void serverStartedOk() {}

@Test(dependsOnMethods = { "serverStartedOk" })
public void method1() {}

In this example, method1() is declared as depending on method serverStartedOk(), which guarantees that serverStartedOk() will always be invoked first.

You can also have methods that depend on entire groups:

@Test(groups = { "init" })
public void serverStartedOk() {}

@Test(groups = { "init" })
public void initEnvironment() {}

@Test(dependsOnGroups = { "init.*" })
public void method1() {}

In this example, method1() is declared as depending on any group matching the regular expression "init.*", which guarantees that the methods serverStartedOk() and initEnvironment() will always be invoked before method1(). 

Note:  as stated before, the order of invocation for methods that belong in the same group is not guaranteed to be the same across test runs.

If a method depended upon fails and you have a hard dependency on it (alwaysRun=false, which is the default), the methods that depend on it are not marked as FAIL but as SKIP.  Skipped methods will be reported as such in the final report (in a color that is neither red nor green in HTML), which is important since skipped methods are not necessarily failures.

Both dependsOnGroups and dependsOnMethods accept regular expressions as parameters.  For dependsOnMethods, if you are depending on a method which happens to have several overloaded versions, all the overloaded methods will be invoked.  If you only want to invoke one of the overloaded methods, you should use dependsOnGroups.

For a more advanced example of dependent methods, please refer to this article, which uses inheritance to provide an elegant solution to the problem of multiple dependencies.

a(1)
a(2)
b(2)
b(2)

This behavior might not be desirable in certain scenarios, such as for example testing a sign in and sign out of a web browser for various countries. In such a case, you would like the following ordering:

signIn("us")
signOut("us")
signIn("uk")
signOut("uk")

  <suite name="Factory" group-by-instances="true">
or
  <test name="Factory" group-by-instances="true">

Dependencies in XML

  <test name="My suite">
    <groups>
      <dependencies>
        <group name="c" depends-on="a  b" />
        <group name="z" depends-on="c" />
      </dependencies>
    </groups>
  </test>

Factories

TestWebServer.java

public class TestWebServer {
  @Test(parameters = { "number-of-times" })
  public void accessPage(int numberOfTimes) {
    while (numberOfTimes-- > 0) {
     // access the web page
    }
  }
}

testng.xml

<test name="T1">
  <parameter name="number-of-times" value="10"/>
  <class name= "TestWebServer" />
</test>

<test name="T2">
  <parameter name="number-of-times" value="20"/>
  <class name= "TestWebServer"/>
</test>

<test name="T3">
  <parameter name="number-of-times" value="30"/>
  <class name= "TestWebServer"/>
</test>

WebTestFactory.java

public class WebTestFactory {
  @Factory
  public Object[] createInstances() {
   Object[] result = new Object[10];  
   for (int i = 0; i < 10; i++) {
      result[i] = new WebTest(i * 10);
    }
    return result;
  }
}

WebTest.java

public class WebTest {
  private int m_numberOfTimes;
  public WebTest(int numberOfTimes) {
    m_numberOfTimes = numberOfTimes;
  }

  @Test
  public void testServer() {
   for (int i = 0; i < m_numberOfTimes; i++) {
     // access the web page
    }
  }
}

Your testng.xml only needs to reference the class that contains the factory method, since the test instances themselves will be created at runtime:

<class name="WebTestFactory" />

The factory method can receive parameters just like @Test and @Before/After and it must return Object[].  The objects returned can be of any class (not necessarily the same class as the factory class) and they don't even need to contain TestNG annotations (in which case they will be ignored by TestNG).

Factories can also be used with data providers, and you can leverage this functionality by putting the @Factory annotation either on a regular method or on a constructor. Here is an example of a constructor factory:

  @Factory(dataProvider = "dp")
  public FactoryDataProviderSampleTest(int n) {
    super(n);
  }

  @DataProvider
  static public Object[][] dp() {
    return new Object[][] {
      new Object[] { 41 },
      new Object[] { 42 },
    };
  }

Class level annotations

Test1.java

@Test
public class Test1 {
  public void test1() {
  }

  public void test2() {
  }
}

For example:

Test1.java

@Test
public class Test1 {
  public void test1() {
  }

  @Test(groups = "g1")
  public void test2() {
  }
}

Parallelism and time-outs

Parallel suites

java org.testng.TestNG -suitethreadpoolsize 3 testng1.xml testng2.xml testng3.xml

Parallel tests, classes and methods

<suite name="My suite" parallel="methods" thread-count="5">

<suite name="My suite" parallel="tests" thread-count="5">

<suite name="My suite" parallel="classes" thread-count="5">

<suite name="My suite" parallel="instances" thread-count="5">

• parallel="methods": TestNG will run all your test methods in separate threads. Dependent methods will also run in separate threads but they will respect the order that you specified.


• parallel="tests": TestNG will run all the methods in the same <test> tag in the same thread, but each <test> tag will be in a separate thread. This allows you to group all your classes that are not thread safe in the same <test> and guarantee they will all run in the same thread while taking advantage of TestNG using as many threads as possible to run your tests.


• parallel="classes": TestNG will run all the methods in the same class in the same thread, but each class will be run in a separate thread.


• parallel="instances": TestNG will run all the methods in the same instance in the same thread, but two methods on two different instances will be running in different threads.

Additionally, the attribute thread-count allows you to specify how many threads should be allocated for this execution.

Note: the @Test attribute timeOut works in both parallel and non-parallel mode.

@Test(threadPoolSize = 3, invocationCount = 10,  timeOut = 10000)
public void testServer() {

Rerunning failed tests

java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -d test-outputs testng.xml
java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -d test-outputs test-outputs\testng-failed.xml

Note that testng-failed.xml will contain all the necessary dependent methods so that you are guaranteed to run the methods that failed without any SKIP failures.

JUnit tests

testng.xml

<test name="Test1" junit="true">
  <classes>
    <!-- ... -->

The behavior of TestNG in this case is similar to JUnit depending on the JUnit version found on the class path:

• JUnit 3:
  • All methods starting with test* in your classes will be run
  • If there is a method setUp() on your test class, it will be invoked before every test method
  • If there is a method tearDown() on your test class, it will be invoked before after every test method
  • If your test class contains a method suite(), all the tests returned by this method will be invoked
• JUnit 4:
  • TestNG will use the org.junit.runner.JUnitCore runner to run your tests

Running TestNG programmatically

Java

TestListenerAdapter tla = new TestListenerAdapter();
TestNG testng = new TestNG();
testng.setTestClasses(new Class[] { Run2.class });
testng.addListener(tla);
testng.run(); 

Similary, you can invoke TestNG on a testng.xml file or you can create a virtual testng.xml file yourself. In order to do this, you can use the classes found the package org.testng.xml: XmlClass, XmlTest, etc... Each of these classes correspond to their XML tag counterpart.

For example, suppose you want to create the following virtual file:

<suite name="TmpSuite" >
  <test name="TmpTest" >
    <classes>
      <class name="test.failures.Child"  />
    <classes>
    </test>
</suite>

XmlSuite suite = new XmlSuite();
suite.setName("TmpSuite");

XmlTest test = new XmlTest(suite);
test.setName("TmpTest");
List<XmlClass> classes = new ArrayList<XmlClass>();
classes.add(new XmlClass("test.failures.Child"));
test.setXmlClasses(classes) ;

List<XmlSuite> suites = new ArrayList<XmlSuite>();
suites.add(suite);
TestNG tng = new TestNG();
tng.setXmlSuites(suites);
tng.run(); 

Please see the JavaDocs for the entire API.

BeanShell and advanced group selection

If the <include> and <exclude> tags in testng.xml are not enough for your needs, you can use a BeanShell expression to decide whether a certain test method should be included in a test run or not. You specify this expression just under the <test> tag:

testng.xml

<test name="BeanShell test">
   <method-selectors>
     <method-selector>
       <script language="beanshell"><![CDATA[
         groups.containsKey("test1")
       ]]></script>
     </method-selector>
   </method-selectors>
  <!-- ... -->

Here are additional information on the BeanShell script:

• It must return a boolean value.  Except for this constraint, any valid BeanShell code is allowed (for example, you might want to return true during week days and false during weekends, which would allow you to run tests differently depending on the date).
   
• TestNG defines the following variables for your convenience:
    java.lang.reflect.Method method:  the current test method.
    org.testng.ITestNGMethod testngMethod:  the description of the current test method.
    java.util.Map<String, String> groups:  a map of the groups the current test method belongs to.
   
• You might want to surround your expression with a CDATA declaration (as shown above) to avoid tedious quoting of reserved XML characters).
   

Annotation Transformers

In order to achieve this, you need to use an Annotation Transformer.

An Annotation Transformer is a class that implements the following interface:

public interface IAnnotationTransformer {

  /**
   * This method will be invoked by TestNG to give you a chance
   * to modify a TestNG annotation read from your test classes.
   * You can change the values you need by calling any of the
   * setters on the ITest interface.
   * 
   * Note that only one of the three parameters testClass,
   * testConstructor and testMethod will be non-null.
   * 
   * @param annotation The annotation that was read from your
   * test class.
   * @param testClass If the annotation was found on a class, this
   * parameter represents this class (null otherwise).
   * @param testConstructor If the annotation was found on a constructor,
   * this parameter represents this constructor (null otherwise).
   * @param testMethod If the annotation was found on a method,
   * this parameter represents this method (null otherwise).
   */
  public void transform(ITest annotation, Class testClass,
      Constructor testConstructor, Method testMethod);
}

  java org.testng.TestNG -listener MyTransformer testng.xml

  TestNG tng = new TestNG();
  tng.setAnnotationTransformer(new MyTransformer());
  // ...

For example, here is how you would override the attribute invocationCount but only on the test method invoke() of one of your test classes:

  public class MyTransformer implements IAnnotationTransformer {
    public void transform(ITest annotation, Class testClass,
        Constructor testConstructor, Method testMethod)
    {
      if ("invoke".equals(testMethod.getName())) {
        annotation.setInvocationCount(5);
      }
    }
  }

Method Interceptors

• Methods run sequentially. These are all the test methods that have dependencies or dependents. These methods will be run in a specific order.
• Methods run in no particular order. These are all the methods that don't belong in the first category. The order in which these test methods are run is random and can vary from one run to the next (although by default, TestNG will try to group test methods by class).

public interface IMethodInterceptor {
  
  List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context);

}

• The same list you received in parameter but in a different order.
• A smaller list of IMethodInstance objects.
• A bigger list of IMethodInstance objects.

Shell

java -classpath "testng-jdk15.jar:test/build" org.testng.TestNG -listener test.methodinterceptors.NullMethodInterceptor
   -testclass test.methodinterceptors.FooTest

For example, here is a Method Interceptor that will reorder the methods so that test methods that belong to the group "fast" are always run first:

public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
  List<IMethodInstance> result = new ArrayList<IMethodInstance>();
  for (IMethodInstance m : methods) {
    Test test = m.getMethod().getConstructorOrMethod().getAnnotation(Test.class);
    Set<String> groups = new HashSet<String>();
    for (String group : test.groups()) {
      groups.add(group);
    }
    if (groups.contains("fast")) {
      result.add(0, m);
    }
    else {
      result.add(m);
    }
  }
  return result;
}

TestNG Listeners

• IAnnotationTransformer (doc, javadoc)
• IAnnotationTransformer2 (doc, javadoc)
• IHookable (doc, javadoc)
• IInvokedMethodListener (doc, javadoc)
• IMethodInterceptor (doc, javadoc)
• IReporter (doc, javadoc)
• ISuiteListener (doc, javadoc)
• ITestListener (doc, javadoc)

• Using -listener on the command line.
• Using <listeners> with ant.
• Using <listeners> in your testng.xml file.
• Using the @Listeners annotation on any of your test classes.
• Using ServiceLoader.

Specifying listeners with testng.xml or in Java

<suite>

  <listeners>
    <listener class-name="com.example.MyListener" />
    <listener class-name="com.example.MyMethodInterceptor" />
  </listeners>

...

@Listeners({ com.example.MyListener.class, com.example.MyMethodInterceptor.class })
public class MyTest {
  // ...
}

Note that the @Listeners annotation will apply to your entire suite file, just as if you had specified it in a testng.xml file. If you want to restrict its scope (for example, only running on the current class), the code in your listener could first check the test method that's about to run and decide what to do then.

Specifying listeners with ServiceLoader

With ServiceLoader, all you need to do is create a jar file that contains your listener(s) and a few configuration files, put that jar file on the classpath when you run TestNG and TestNG will automatically find them.

Here is a concrete example of how it works.

Let's start by creating a listener (any TestNG listener should work):

package test.tmp;

public class TmpSuiteListener implements ISuiteListener {
  @Override
  public void onFinish(ISuite suite) {
    System.out.println("Finishing");
  }

  @Override
  public void onStart(ISuite suite) {
    System.out.println("Starting");
  }
}

You should end up with the following directory structure, with only two files:

$ tree
|____META-INF
| |____services
| | |____org.testng.ITestNGListener
|____test
| |____tmp
| | |____TmpSuiteListener.class

$ cat META-INF/services/org.testng.ITestNGListener
test.tmp.TmpSuiteListener

$ jar cvf ../sl.jar .
added manifest
ignoring entry META-INF/
adding: META-INF/services/(in = 0) (out= 0)(stored 0%)
adding: META-INF/services/org.testng.ITestNGListener(in = 26) (out= 28)(deflated -7%)
adding: test/(in = 0) (out= 0)(stored 0%)
adding: test/tmp/(in = 0) (out= 0)(stored 0%)
adding: test/tmp/TmpSuiteListener.class(in = 849) (out= 470)(deflated 44%)

$ java -classpath sl.jar:testng.jar org.testng.TestNG testng-single.yaml
Starting
f2 11 2
PASSED: f2("2")
Finishing

Dependency injection

Native dependency injection

• Any @Before method or @Test method can declare a parameter of type ITestContext.
• Any @AfterMethod method can declare a parameter of type ITestResult, which will reflect the result of the test method that was just run.
• Any @Before and @After methods can declare a parameter of type XmlTest, which contain the current <test> tag.
• Any @BeforeMethod (and @AfterMethod) can declare a parameter of type java.lang.reflect.Method. This parameter will receive the test method that will be called once this @BeforeMethod finishes (or after the method as run for @AfterMethod).
• Any @BeforeMethod can declare a parameter of type Object[]. This parameter will receive the list of parameters that are about to be fed to the upcoming test method, which could be either injected by TestNG, such as java.lang.reflect.Method or come from a @DataProvider.
• Any @DataProvider can declare a parameter of type ITestContext or java.lang.reflect.Method. The latter parameter will receive the test method that is about to be invoked.

public class NoInjectionTest {

  @DataProvider(name = "provider")
  public Object[][] provide() throws Exception {
      return new Object[][] { { CC.class.getMethod("f") } };
  }

  @Test(dataProvider = "provider")
  public void withoutInjection(@NoInjection Method m) {
      Assert.assertEquals(m.getName(), "f");
  }

  @Test(dataProvider = "provider")
  public void withInjection(Method m) {
      Assert.assertEquals(m.getName(), "withInjection");
  }
}

Guice dependency injection

@Guice(modules = GuiceExampleModule.class)
public class GuiceTest extends SimpleBaseTest {

  @Inject
  ISingleton m_singleton;

  @Test
  public void singletonShouldWork() {
    m_singleton.doSomething();
  }

}

public class GuiceExampleModule implements Module {

  @Override
  public void configure(Binder binder) {
    binder.bind(ISingleton.class).to(ExampleSingleton.class).in(Singleton.class);
  }

}

@Guice(moduleFactory = ModuleFactory.class)
public class GuiceModuleFactoryTest {

  @Inject
  ISingleton m_singleton;

  @Test
  public void singletonShouldWork() {
    m_singleton.doSomething();
  }
}

public interface IModuleFactory {
 /**
   * @param context The current test context
   * @param testClass The test class
   *
   * @return The Guice module that should be used to get an instance of this
   * test class.
   */
  Module createModule(ITestContext context, Class<?> testClass);
}

package com.example;

public class ParentModule extends AbstractModule {
  @Override
  protected void conigure() {
    bind(MyService.class).toProvider(MyServiceProvider.class);
    bind(MyContext.class).to(MyContextImpl.class).in(Singleton.class);
  }
}

package com.example;

public class TestModule extends AbstractModule {
  private final MyContext myContext;

  @Inject
  TestModule(MyContext myContext) {
    this.myContext = myContext
  }
  
  @Override
  protected void configure() {
    bind(MySession.class).toInstance(myContext.getSession());
  }
}

package com.example;

@Test
@Guice(modules = TestModule.class)
public class TestClass {
  @Inject
  MyService myService;
  @Inject
  MySession mySession;
  
  public void testServiceWithSession() {
    myService.serve(mySession);
  }
}

Listening to method invocations

public interface IInvokedMethodListener extends ITestNGListener {
  void beforeInvocation(IInvokedMethod method, ITestResult testResult);
  void afterInvocation(IInvokedMethod method, ITestResult testResult);
}

Overriding test methods

Here is an example with JAAS:

public class MyHook implements IHookable {
  public void run(final IHookCallBack icb, ITestResult testResult) {
    // Preferably initialized in a @Configuration method
    mySubject = authenticateWithJAAs();
   
    Subject.doAs(mySubject, new PrivilegedExceptionAction() {
      public Object run() {
        icb.callback(testResult);
      }
    };
  }
}

Altering suites (or) tests

A classic example for this would be to try and leverage your existing suite file and try using it for simulating a load test on your "Application under test". At the minimum you would end up duplicating the contents of your <test> tag multiple times and create a new suite xml file and work with. But this doesn't seem to scale a lot.

TestNG allows you to alter a suite (or) a test tag in your suite xml file at runtime via listeners. You achieve this by providing a listener that implements IAlterSuiteListener. Please refer to Listeners section to learn about listeners.

Here is an example that shows how the suite name is getting altered in runtime:

public class AlterSuiteNameListener implements IAlterSuiteListener {

    @Override
    public void alter(List<XmlSuite> suites) {
        XmlSuite suite = suites.get(0);
        suite.setName(getClass().getSimpleName());
    }
}

• Through the <listeners> tag in the suite xml file.
• Through a Service Loader

Test results

Success, failure and assert

A test is considered successful if it completed without throwing any exception or if  it threw an exception that was expected (see the documentation for the expectedExceptions attribute found on the @Test annotation).

Your test methods will typically be made of calls that can throw an exception, or of various assertions (using the Java "assert" keyword).  An "assert" failing will trigger an AssertionErrorException, which in turn will mark the method as failed (remember to use -ea on the JVM if you are not seeing the assertion errors).

Here is an example test method:

@Test
public void verifyLastName() {
  assert "Beust".equals(m_lastName) : "Expected name Beust, for" + m_lastName;
}

import static org.testng.AssertJUnit.*;
//...
@Test
public void verify() {
  assertEquals("Beust", m_lastName);
}

Note that the above code use a static import in order to be able to use the assertEquals method without having to prefix it by its class.

Logging and results

It's very easy to generate your own reports with TestNG with Listeners and Reporters:

• Listeners implement the interface org.testng.ITestListener and are notified in real time of when a test starts, passes, fails, etc...
• Reporters implement the interface org.testng.IReporter and are notified when all the suites have been run by TestNG. The IReporter instance receives a list of objects that describe the entire test run.

Logging Listeners

public class DotTestListener extends TestListenerAdapter {
  private int m_count = 0;

  @Override
  public void onTestFailure(ITestResult tr) {
    log("F");
  }

  @Override
  public void onTestSkipped(ITestResult tr) {
    log("S");
  }

  @Override
  public void onTestSuccess(ITestResult tr) {
    log(".");
  }

  private void log(String string) {
    System.out.print(string);
    if (++m_count % 40 == 0) {
      System.out.println("");
    }
  }
} 

Here is how I invoke TestNG to use this new listener:

Shell

java -classpath testng.jar;%CLASSPATH% org.testng.TestNG -listener org.testng.reporters.DotTestListener test\testng.xml

Shell

........................................
........................................
........................................
........................................
........................................
.........................
===============================================
TestNG JDK 1.5
Total tests run: 226, Failures: 0, Skips: 0
===============================================

Logging Reporters

public void generateReport(List<ISuite> suites, String outputDirectory)

JUnitReports

TestNG contains a listener that takes the TestNG results and outputs an XML file that can then be fed to JUnitReport. Here is an example, and the ant task to create this report:

build.xml

<target name="reports">
  <junitreport todir="test-report">
    <fileset dir="test-output">
      <include name="*/*.xml"/>
    </fileset>
 
    <report format="noframes"  todir="test-report"/>
  </junitreport>
</target>

Note:  a current incompatibility between the JDK 1.5 and JUnitReports prevents the frame version from working, so you need to specify "noframes" to get this to work for now.

Reporter API

If you need to log messages that should appear in the generated HTML reports, you can use the class org.testng.Reporter:

    Reporter.log("M3 WAS CALLED");

XML Reports

TestNG offers an XML reporter capturing TestNG specific information that is not available in JUnit reports. This is particulary useful when the user's test environment needs to consume XML results with TestNG-specific data that the JUnit format can't provide. Below is a sample of the output of such a reporter:

<testng-results>
  <suite name="Suite1">
    <groups>
      <group name="group1">
        <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
        <method signature="com.test.TestOne.test1()" name="test1" class="com.test.TestOne"/>
      </group>
      <group name="group2">
        <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
      </group>
    </groups>
    <test name="test1">
      <class name="com.test.TestOne">
        <test-method status="FAIL" signature="test1()" name="test1" duration-ms="0"
              started-at="2007-05-28T12:14:37Z" description="someDescription2"
              finished-at="2007-05-28T12:14:37Z">
          <exception class="java.lang.AssertionError">
            <short-stacktrace>
              <![CDATA[
                java.lang.AssertionError
                ... Removed 22 stack frames
              ]]>
            </short-stacktrace>
          </exception>
        </test-method>
        <test-method status="PASS" signature="test2()" name="test2" duration-ms="0"
              started-at="2007-05-28T12:14:37Z" description="someDescription1"
              finished-at="2007-05-28T12:14:37Z">
        </test-method>
        <test-method status="PASS" signature="setUp()" name="setUp" is-config="true" duration-ms="15"
              started-at="2007-05-28T12:14:37Z" finished-at="2007-05-28T12:14:37Z">
        </test-method>
      </class>
    </test>
  </suite>
</testng-results>

This reporter is injected along with the other default listeners so you can get this type of output by default. The listener provides some properties that can tweak the reporter to fit your needs. The following table contains a list of these properties with a short explanation:

   1 - will generate all the results in one file.
   2 - each suite is generated in a separate XML file that is linked to the main file.
   3 - same as 2 plus separate files for test-cases that are referenced from the suite files.

   0 - no stacktrace (just Exception class and message).
   1 - a short version of the stack trace keeping just a few lines from the top
   2 - the complete stacktrace with all the inner exceptions
   3 - both short and long stacktrace

In order to configure this reporter you can use the -reporter option in the command line or the Ant task with the nested <reporter> element. For each of these you must specify the class org.testng.reporters.XMLReporter. Please note that you cannot configure the built-in reporter because this one will only use default settings. If you need just the XML report with custom settings you will have to add it manually with one of the two methods and disable the default listeners.

YAML

  <parameter name="n" value="42" />

  
    
      
        <exclude name="broken" />
      
    

    
      <class name="test.listeners.ResultEndMillisTest" />
    
  

and here is its YAML version:

name: SingleSuite
threadCount: 4
parameters: { n: 42 }

tests:
  - name: Regression2
    parameters: { count: 10 }
    excludedGroups: [ broken ]
    classes:
      - test.listeners.ResultEndMillisTest

You might find the YAML file format easier to read and to maintain. YAML files are also recognized by the TestNG Eclipse plug-in. You can find more information about YAML and TestNG in this blog post.  

Back to my home page.

Or check out some of my other projects:

• EJBGen:  an EJB tag generator.
• TestNG:  A testing framework using annotations, test groups and method parameters.
• Doclipse:  a JavaDoc tag Eclipse plug-in.
• J15:  an Eclipse plug-in to help you migrate your code to the new JDK 1.5 constructs.
• SGen:  a replacement for XDoclet with an easy plug-in architecture.
• Canvas:  a template generator based on the Groovy language.