I have created a project on CodePlex hosting the solution which supports VS2005 and VS2008 (link at the end). The CodePlex release contains MSIs plus the full source code.
The MSTest extensions Attributes are as follows:
- TestTransaction – this provides equivalent functionality to MbUnit’s much lauded “RollBack” attribute.
- ExpectedExceptionMessage – this is an alternative to MSTest’s built-in “ExpectedException” Attribute. The built in “ExpectedException” Attribute is broken, it allows you to provide a message which you would expect is the exception message that is thrown by the expected type. But it isn’t. It’s actually a message that is displayed to the user in the test results log which is very confusing. This alternative allows you to supply the exception message. If the exception message thrown by the Test Method does not match the message in the ExpectedExceptionMessage Attribute, then the test fails.
- TestTimer – this allows us to put a timer on the unit tests so if a unit test takes longer than the time specified in the Attribute then the test fails.
An example usage of the custom Attributes for MSTest are as follows:
[TestClass]
public class TestTransactionAttributeTest : MSTestExtensionsTestFixture
{
[TestMethod]
[TestTransaction]
public void SomeTest1()
{
// Do database stuff, will be
// rolled back after test ends.
}
[TestMethod]
[TestTimer(10)]
public void SomeTest2()
{
// Do stuff, if it takes longer
// than 10 milliseconds the test
// will fail.
}
[TestMethod]
[ExpectedExceptionMessage(typeof(MSTestExtensionsException), "Message.")]
public void SomeTest3()
{
throw new MSTestExtensionsException("Message.");
// This test passes
// (see above comments).
}
}
Please note there are no references to any xUnit frameworks! All that is required is to inherit from “MSTestExtensionsTestFixture” and to decorate the Test Methods with the relevant Attribute.
Here’s how we get there...
A comprehension of Aspect Oriented Programming (AOP) is needed to understand how this all fits together. I won’t go into AOP in detail as there are many on-line resources that do a much better job than I would. This is as good a place as any to start: http://en.wikipedia.org/wiki/Aspect-oriented_programming.
MSTest, like the xUnit testing frameworks, uses Attributes on Classes and Methods to specify tests and their behaviour. AOP uses the same style of Attributes on Types to extend their behaviour. If you were building your own application, you might use a framework such as Spring.NET which would provide you with a transparent AOP mechanism. Unfortunately, as MSTest is effectively “closed” (there are no extension points) we don’t have any such options. We have to wire up the AOP features ourselves.
The only way to implement AOP is to use the following objects from the .NET Framework:
- System.ContextBoundObject – the object that you wish to decorate with Attributes as a means to change the behaviour must inherit from this class.
- System.Runtime.Remoting.Contexts.ContextAttribute – Attributes inheriting from this Attribute will be called at run-time when the type that they are decorating is called.
The combination of these two objects will allow us to intercept method calls. If we can intercept calls to Methods, we can intercept the calls to Test Methods and modify their behaviour. Every one of our Test Classes (or Fixtures if you prefer) will need to inherit from “ContextBoundObject” and be decorated with our own specialised “ContextAttribute”. Instead of having to do this every time we write a Test Class, we’ll create a Class from which we can inherit when we want to write a Test Class. We’ll call this base class “MSTestExtensionsTestFixture” (as per the previous examples).This is as follows:
[MSTestExtensionsTest]
public class MSTestExtensionsTestFixture : ContextBoundObject
{
}
The “MSTestExtensionsTest” Attribute that this base Test Class is decorated with needs explaining. The “MSTestExtensionsTest” Attribute inherits from the “ContextAttribute” mentioned above. The “ContextAttribute” class has a virtual method called “GetPropertiesForNewContext” which we override. This method allows us to add “Aspects” to the context (see above link for a detailed description of Aspects and other AOP concepts). These “Aspects” will intercept Method calls i.e. when the Test Methods are executed, allowing us to change their behaviour. The important “MSTestExtensionsTest” Attribute code is as follows:
[AttributeUsage(AttributeTargets.Class)]
public sealed class MSTestExtensionsTestAttribute : ContextAttribute
{
...
public override void GetPropertiesForNewContext(IConstructionCallMessage msg)
{
if (msg == null)
throw new ArgumentNullException("msg");
msg.ContextProperties.Add(new TestProperty<TestTimerAspect>());
msg.ContextProperties.Add(new TestProperty<TestTransactionAspect>());
msg.ContextProperties.Add(new TestProperty<ExpectedExceptionMessageAspect>());
}
...
}
The "TestProperty" is a generic Class to reduce the amount of boiler plate code in each Aspect Class. I will skip this for the sakes on brevity, please look at the attached source code for details.
Fast forwarding, this is what an Aspect looks like:
public class MyAspect : TestAspect<TestTimerAttribute>, IMessageSink, ITestAspect { ... [SecurityPermission(SecurityAction.LinkDemand, Flags = SecurityPermissionFlag.Infrastructure)] public IMessage SyncProcessMessage(IMessage msg) { if (msg == null) throw new ArgumentNullException("msg"); // The following line is the call // to the actual method we have // intercepted. We can do stuff // before we call this or after. // This lets us change behaviour. IMessage returnMessage = _nextSink.SyncProcessMessage(msg); return returnMessage; } ... }
Please review the in-line comments above. If this code was used as-is, it would simply call the Method that had been intercepted and we would notice no change in behaviour. However, we can add code before and/or after the call to “SyncProcessMessage” to do whatever we wish. This is the point at which we add our custom code to change the behaviour of the calls to the Test Methods.
From the above code, the “TestAspect” Class is another Class containing boiler plate code which I will skip over and the “ITestAspect” Interface is an Interface that our Aspects must satisfy.
An Aspect must be accompanied by an Attribute, an Attribute is simple and is essentially a placeholder for properties and ensures an Aspect is called. An example Attribute is as follows:
public sealed class MyAttribute : Attribute { private string _someProperty; public MyAttribute(string someProperty) { _testLength = someProperty; } public string SomeProperty { get { return _someProperty; } } }
And bringing it all together, if we were to use our (rather useless) Attribute it would look like so:
[TestClass]
public class MyTest : MSTestExtensionsTestFixture
{
[TestMethod]
[MyAttribute("some value")]
public void SomeTest()
{
// Unit test code.
}
}
Summary
I have skipped over the details of AOP as I would only be repeating other resources you can find on-line. However, I hope I’ve given you a taste of how the solution was implemented. Full source code is available from the CodePlex project
I think this shows that MSTest can be extended and via this mechanism can match a lot of the features of the xUnit frameworks. If you want to make further extensions, all you need to do is implement you own "TestAttribute" and an accompanying "TestAspect". Inherit from the "MSTestExtensionsFixture" and apply your Attribute and you are good to go.
Note the CodePlex solution supports VS2005 and VS2008. With the extensions that are possible and the performance improvements in VS2008 (MSTest is noticeably faster), then MSTest is not such a poor cousin to the xUnit frameworks anymore.
MSTest...it’s not all that bad, honest!
The CodePlex project hosting the solution can be found here: http://www.codeplex.com/MSTestExtensions
Callum
Posts
4 comments:
This is great! I'm still struggling with one thing though... accessing the test object's public members (like TestContext, if it's implemented). MSTest obviously hooks into it somehow, so I assume it's hopefully not impossible, but I don't have a very good comprehension of remoting internals... do you have any idea how I can hook into the TestContext object for an individual test execution?
This is great! I'm still struggling with one thing though... accessing the test object's TestContext. Do you have any idea how I can hook into the TestContext object for an individual test execution?
That's not something I've looked at and I think it would be tricky. I am currently travelling in South America but if I have any bright ideas when I get back I will post an update.
After banging my head against it for a few more hours, I decided just to include the MSExtensions source in my assembly, instead of just referencing the assembly, and used the hook in GetObjectSink() in Testproperty to save a copy of the proxy to the test class instance (in this case, a MarshalByRefObject), like this:
[SecurityPermission(SecurityAction.LinkDemand, Flags = SecurityPermissionFlag.Infrastructure)]
public IMessageSink GetObjectSink(MarshalByRefObject obj, IMessageSink nextSink)
{
T testAspect = new T();
testAspect.Object = obj;
testAspect.AddMessageSink(nextSink);
return testAspect;
}
Then, I can dynamically invoke the TestContext property from within TestAspect, and access it that way. In this manner, I can determine whether or not to run a test based on a matrix of data driven information from the text context (using the DataSource attribute), and the new custom attributes I created. If this is a poor way of doing this, let me know, but it seems to work fairly well. Thanks for providing this framework!
Post a Comment