Show @Ignored junit 4 tests in XML output on teamcity

Not long ago, I managed to migrate tests to junit 4. But when I ran the tests remotely on Teamcity build server (my project is using Ant as runner on Teamcity), the report didn't show any ignored tests that were marked @Ignore.

I was shocked to discover that this issue was first reported a few years ago here , and it is still uncertain when Ant will fix it. Fortunately, is the one that saves my purposes of migrating all the tests into junit 4.

Here is an example of how we can create a suite runner to run multiple tests or test suites. The purpose of this suite runner is to generate xml-based test reports that will be stored onto specified folder such that Teamcity can then import these report data from "Ant JUnit" format from the specified folder.

public class SomeSuiteRunner {
    
    public static void main() throws Exception {
        boolean success = new SomeSuiteRunner().run();
        System.exit(success ? 0 : 1);
    }

    public boolean run() throws Exception {    
        File reportDirectory = new File("/reports");
        org.apache.commons.io.FileUtils.deleteDirectory(reportDirectory); 
        reportDirectory.mkdirs();
        return runTests();
    }

    private boolean runTests() throws Exception {
        boolean isSuccessful = true;
        final Class testClass =  this.getClass().getClassLoader().loadClass("package.ATest");
        return runTestClass(testClass);
    }

    private boolean runTestClass(Class testClass) {
        JUnitCore runner = new JUnitCore();
        final XmlWritingListener listener = new XmlWritingListener("/reports");
        runner.addListener(listener);
        listener.startFile(testClass);
        try{
           return runner.run(testClass);
        } finally {
           listener.closeFile();
        }
    }
    
}

To be able to show the number of @Ignore tests on Teamcity, we need to config our project on Teamcity properly (in "Runner" settings):

1. choose an appropriate "Build runner"
2. choose "Ant JUnit" as the "Import data from XML"
3. tpye, for example, "/reports/*.xml" in the "Report paths"

Migrating tests from junit 3 to junit 4

For some legacy reason, the project that I am working on has been using junit 3 for all the unit and integration tests since day 1. Last week, I decided to do something different that could cheer up my mind and a step forward to more flexibilities in testing.

It could be an relative easy task, but it turned out to cost me a whole day to complete the migration of 700+ test classes.

1) Based on the suggestions on http://stackoverflow.com/questions/264680/best-way-to-automagically-migrate-tests-from-junit-3-to-junit-4 , I performed the initial easy-task by the following regular expressions:

// Add @Test
Replace:
^[ \t] +((publicpublic abstract) +void +test)
With:
@Test\n $1
Regular Expression: on
Case sensitive: on
File name filter: *Test.java, *TestCase.java

// Remove double @Test's on already @Test annotated files
Replace:
^[ \t]+@Test\n[ \t]+@Test
With:
@Test
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java


// Remove all empty setUp's
Replace:
^[ \*]+((publicprotected) +)?void +setUp\(\)[^\{]*\{\s*(super\.setUp\(\);)?\s*\}\n([ \t]*\n)?
With: nothing
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java

// Add @Before to all setUp's
Replace:
^([ \t]+@Override\n)?[ \t]+((publicprotected) +)?(void +setUp\(\))
With:
@Before\n public void setUp()
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java

// Remove double @Before's on already @Before annotated files
Replace:
^[ \t]+@Before\n[ \t]+@Before
With:
@Before
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java


// Remove all empty tearDown's
Replace:
^[ \*]+((publicprotected) +)?void +tearDown\(\)[^\{]*\{\s*(super\.tearDown\(\);)?\s*\}\n([ \t]*\n)?
With: nothing
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java

// Add @After to all tearDown's
Replace:
^([ \t]+@Override\n)?[ \t]+((publicprotected) +)?(void +tearDown\(\))
With:
@After\n public void tearDown()
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java

// Remove double @After's on already @After annotated files
Replace:
^[ \t]+@After\n[ \t]+@After
With:
@After
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java


// Remove old imports, add new imports
Replace:
^([ \t]*import[ \t]+junit\.framework\.Assert;\n)?[ \t]*import[ \t]+junit\.framework\.TestCase;
With:
import org.junit.After;\nimport org.junit.Before;\nimport org.junit.Test;\nimport static org.junit.Assert.*;
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java


// Remove all extends TestCase
Replace:
[ \t]+extends[ \t]+(TestCase)[ \t]+\{
With:
{
Regular Expression: on
Case sensitive: on
File name filter:
*Test.java, *TestCase.java



// Look for import junit.framework;
Find:
import junit\.framework
Manually fix
Regular Expression: on
Case sensitive: on


// Look for ignored tests (FIXME, disabled, ...)
Find:
public[ \t]+void[ \t]+\w+test
Manually fix
Regular Expression: on
Case sensitive: on


// Look for dummy/empty tests
Find:
public[ \t]+void[ \t]+test[\w\d]*\(\s*\)\s*\{\s*(//[^\n]*)?\s*\}
Manually fix
Regular Expression: on
Case sensitive: on

2) The project might not compile without an error after step 1, mostly because we still need to do the static import of various assertion methods (e.g. assertEquals, assertTrue, etc.)

3) Since we use MockObjectTestCase from JMock with jUnit 3, we need to have a class level annotation of "@RunWith(JMock.class)", remove the "extends MockObjectTestCase", and manually define

private final Mockery context = new Mockery();
...
xyz = context.mock(XYZ.class);
context.checking(new Expectations(){{}});

4) We then have to replace the test suite with junit 4 declaration styles:

@RunWith(SomeDirectoryScanningSuite.class)
public class SomeTestSuite {
}

The "SomeDirectoryScanningSuite" has to extend "org.junit.runners.Suite", for example

public class SomeDirectoryScanningSuite extends Suite {

public SomeDirectoryScanningSuite(Class testSuite) throws InitializationError, ClassNotFoundException {
super(testSuite, new Class[]{SomeTest.class, AnotherTest.class});
}
}

There are many different ways of specifying what test classes should be included in the above example.

The example below is a test suite of a bunch of test suites/classes:

@RunWith(Suite.class)
@Suite.SuiteClasses({
SomeTest.class, // junit 4 annotated test class
SomeTestSuite.class // junit 4 test suite
})
public class TestsSuite {}