The TestFixture supports combinatorial tests through the use of the CombinatorialTestAttribute.

A simple example

Suppose that we have a ICountX interface that defines a method that counts the number of 'x' in a string:

public interface ICountX
{
   int Count(string s);
}

You have two classes that implement that interface:

class SickCountX : ICountX
{
   public int Count(string s)
   {
       return 2;
    }
}

class CountX : ICountX
{
   public int Count(string s)
   {
        int count = 0;
        foreach(char c in s)
            if (c=='x')
               count++;
       return count;
   }
}

Now you would like to test that those two interface implementations work correctly. Let's start by writing a test method inside a TestFixture. The CountIsExact method receives a string, the number of x in the string and checks that a provided ICountX implementation computes the expected x count:

[TestFixture]
public class ICountXTest
{
   public class StringX
   {
       public StringX(string value, int xcount){Value = value; XCount = xcount;}
       public string Value;
       public int XCount;
       public override string ToString() { return String.Format("{0},{1}",this.Value,this.XCount);}
   }

   [CombinatorialTest]
   public void CountIsExact(
       ICountX counter,
       StringX s
       )
   {
       Assert.AreEqual(s.XCount, counter.Count(s.Value));
    }
}

Next, we create a few strings and xcount that can serve for creating test case. By simplicity we use the new C# 2.0 iterators to implement that:

[Factory(typeof(StringX))]
public IEnumerable Strings()
{
    yield return new StringX("",0);
    yield return new StringX("x",1);
    yield return new StringX("xa",1);
    yield return new StringX("xax",2);
    yield return new StringX("aaa",0);
}

Note that we have tagged the method with FactoryAttribute so that MbUnit knows this method creates StringX instances. We also create another method that will create instance of ICountX to test:

[Factory(typeof(ICountX))]
public IEnumerable Counters()
{
   yield return new SickCountX();
   yield return new CountX();
}

Now, we would like to cross product the output of Counters and StringX and feed the combination in the CountIsExact test method. To do so, we use the UsingFactories attribute to tag the parameters of the method:

[CombinatorialTest]
public void CountIsExact(
   [UsingFactories("Counters")] ICountX counter,
   [UsingFactories("Strings")] StringX s)
{...}

The fixture is ready, so we can give it a run. The output of the test execution is as follows:

[mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(,0))
 [mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(x,1))
 [mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(xa,1))
 [mbunit][Success] CountIsExact(Counters(SandBox.SickCountX),Strings(xax,2))
 [mbunit][Failure] CountIsExact(Counters(SandBox.SickCountX),Strings(aaa,0))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(,0))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(x,1))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xa,1))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(xax,2))
 [mbunit][Success] CountIsExact(Counters(SandBox.CountX),Strings(aaa,0))6 succeeded, 4 failed, 0 skipped, took 0,00 seconds.

It worked! As expected, we have 5 x 2 = 10 test case generated. Each test case name is generated out of the data source and different data values. Of course, you can "bind" data to the parameters using various other ways and the tests support more that 2 parameters as we will see in the following. We could also have added multiple Using Attributes.

Prerequisites

This new features uses the TestFuOperationsframework in the background.

Rationale

The test has the following workflow:

for each parameter of the test method,
    get all the domains D of the parameter from the [UsingAttributes] (there are a bunch of those).

This creates a collection of domain for each parameter which is also a domain itself, which we'll call parameter domain: PD = collection of D,

foreach tuple in the [CartesianProduct] of the parameter domains
    foreach ptuple in the [PairWizeProduct] of the domain in the tuple (each entry is a domain for the corresponding method parameters)
        Create a test case that will invoke the test method with the values in ptuple.

Note that in the example above,

PD = {
  { Counters }
  ,
  { Strings }
}

hence the Cartesian product has only one element.

{Counters}
 x
{Strings}
 = (Counters,Strings)

Built in Using Attributes

MbUnit comes with a number of built-in using attribute that should get you started almost all situations.

• UsingEnumAttribute
• UsingLinearAttribute
• UsingLiteralsAttribute
• UsingFactoriesAttribute
• UsingImplementationsAttribute
• ExtendingUsingAttributes, how to write your own UsingAttribute

What about tuple filtering ?

In some situation you may want to filter out tuple. For example, we want to check that a methods throws ArgumentNullException on null arguments:

[Factory]
public string[] Strings()
{
    return new string[] { null, "hello" };
}

[CombinatorialTest]
[ExpectedArgumentNullException]
public void TestWithValidator(
    [UsingFactories("Strings")] string s1,
    [UsingFactories("Strings")] string s2
)
{
    if (s1 == null)
        throw new ArgumentNullException();
    if (s2 == null)
        throw new ArgumentNullException();
}

note The ExpectedExceptionAttribute and all other decorators work on the combinatorial tests, however if you mark a test with the IgnoreAttribute or ExplicitAttribute it's factories will not be run.

If we run the test now, we will have unwanted tuples such as (null, null) and ("hello", "hello") included in our tests: the first does not give much information and the second will not throw and hence, is erroneous. In this kind of situation, we can specify a "filter" method that takes the same arguments as the test method and return a bool:

public bool IsValid(string s1, string s2)
{
   if (s1 == null && s2 == null)
      return false;
   if (s1 != null && s2 != null)
      return false;
   return true;
}

[CombinatorialTest(TupleValidatorMethod = "IsValid")]
[ExpectedArgumentNullException]
public void TestWithValidator(...

--- output
 [mbunit][Success] TestWithValidator(Strings(),Strings(hello))
 [mbunit][Success] TestWithValidator(Strings(hello),Strings())

As required, the unwanted tuples have been dropped from the test.