GenTestAsm

Run Your Unmanaged Tests in nUnit

Programming Tools & Info

Download GenTestAsm (198K zip file)
How to use GenTestAsm

Unit Testing C++ Code

Automated testing in general and nUnit tool in particular has become an invaluable asset for me and my team. Unit tests have greatly improved my confidence in the quality of code and my ability to make changes. They saved my bacon numerous times by discovering subtle and not-so-subtle bugs after "risky" updates.

However, nUnit has one serious limitation: it works only with managed code. From time to time I do write code in C++, and probably will continue to do so in the foreseeable future.

When it comes to unit testing C++ code, there are essencially have three choices:

• Do not do unit testing at all.
• Use one of the unit testing packages designed specifically for C++, e.g. TUT C++ unit test framework.
• Find a way to run C++ tests in nUnit.

Not doing unit testing at all is a very risky approach. The code becomes brittle and the risk of making changes is too high. TUT is a nice tool, but it does not provide a GUI test runner like nUnit. Also, having to switch between two different tools looks like a nuisance.

Therefore, I concentrated on the last approach - finding a way to run C++ tests in nUnit.

The Battle Plan

General battle plan was as follows:

1. Make unmanaged tests callable from the outside world via DLL exports.
2. Write a tool that takes an unmanaged DLL, enumerates its exports, and generates a managed assembly loadable by nUnit. I called this tool GenTestAsm.
3. For each exported unmanaged function, automatically create a managed method marked with [Test] attribute.
4. Managed method calls unmanaged function via P/Invoke.

Enumerating DLL Exports

Sadly, Win32 does not provide out-of-the box API for enumerating DLL exports. Fortunately, format of DLL files is publicly available from Microsoft. I extract the list of exports by opening the executable file and analyzing the bytes. It is a little tedious, but not very complex task. The biggest annoyance is that PE file format uses relative virtual memory addresses (RVAs) instead of file offsets. This is great when the file is loaded in memory, but requires constant recalculations when working with the file on disk.

Generating Test Assembly

To generate test assembly, I first create C# source code and then compile it using CSharpCodeProvider class. This proved to be simpler and more straightforward than building the code through CodeDOM. This is also easier to test. If something goes wrong with the generated assembly, one can always look at the generated source code and scan it for abnormalities. I added an option to GenTestAsm that outputs generated source code instead of compiled binary.

Test Exports vs. Other Exports

It is definitely possible that a DLL with unmanaged tests exports a function that is not a test. When GenTestAsm creates the managed wrapper, it needs to know which exports are tests and which are not. nUnit separates tests from non-tests using attributes, but there are no attributes in the unmanaged world. I decided to use a simple naming convention instead. GenTestAsm generates managed test wrappers only for the exports whose names begin with certain prefix (by default "UnitTest"). Other exports are ignored.

Test Results

The next problem is how to handle test failures. In nUnit world a test is usually considered successful if it runs to completion, and failed if it throws an exception. Since my tests are written in unmanaged C++, their exceptions would be unmanaged C++ exceptions. I cannot let these exceptions escape into the managed wrapper. Therefore, I need some other mechanism to report test failures. I decided to use test's return value. Unmanaged tests must have signature

BSTR Test();

Return value of NULL means success, anything else means failure, and returned string is the error message. I chose BSTR over regular char*, because BSTR has well-defined memory management rules, and .NET runtime knows how to free it.

Writing a Trivial Test

Returning BSTR from the C++ test is nice, but it makes writing a test a little difficult. The author of the test must make sure that unhandled C++ exceptions don't escape the test. He also must format the error message and convert it to BSTR. If this were done by hand in each in every test, the code would become too verbose to be practical. Let's take a trivial test in C#:

// C#
public void CalcTest()
{
    Assert.AreEqual( 4, Calculator.Multiply(2,2) );
}

and see how equivalent test in C++ would look like:

// C++
__declspec(dllexport)
BSTR CalcTest()
{
   try
   {
       int const expected = 4;
       int actual = Calculator::Multiply(2,2);
       
       if (expected != actual)
       {
           std::wostringstream msg;
           msg << "Error in " << __FILE__ << " (" << __LINE__ << "): "
               << "expected " << expected << ", but got " << actual;
           return SysAllocString( msg.str().c_str() );
       }
   }
   catch (...)
   {
       return SysAllocString("Unknown exception");
   }

   return NULL;
}

This is too much boiler plate code. We need a support library here.

Support Library

With the help of a tiny #include file we can squeeze our C+ test back to 3 lines of code:

// C++
#include "TestFramework.h"

TEST(CalcTest)
{
    ASSERT_EQUAL( 4, Calculator::Multiply(2,2) );
}

TestFramework.h defines TEST macro that encapsulates the details of exception handling and BSTR conversion. It also defines a couple of ASSERT macros such as ASSERT_EQUAL.

The Big Lockdown

However, there is one catch. As you remember, I use P/Invoke to call my unmanaged tests. Internally, P/Invoke loads the unmanaged DLL and keeps it loaded until the managed process exits. In other words, if I used P/Invoke blindly, once you executed the tests, your managed DLL would become locked. You would not be able to recompile it until you closed nUnit GUI. This is an unpleasant speed bump.

One Way Out

Instead of invoking unmanaged DLL directly, GenTestAsm could, of course, call LoadLibrary(), and then GetProcAddress(). It could then do Marshal.GetDelegateForFunctionPointer() and invoke the resulting delegate. The problem is, this API is available only in .NET 2.0. I wanted GenTestAsm to be compatible with .NET 1.1, so I had to find a different solution.

Another Way Out

If something must be loaded forever, let it be not the test DLL, but some other, helper DLL that never changes. Current version of GenTestAsm P/Invokes into unmanaged helper (thunk), which then calls LoadLibrary(), GetProcAddress() and FreeLibrary(). This way, it is the thunk that gets locked, while the real test DLL remains free.

// C++
typedef BSTR (*TestFunc)();

extern "C"
__declspec(dllexport)
BSTR __cdecl RunTest( LPCSTR dll, LPCSTR name )
{
    HMODULE hLib = LoadLibrary(dll);
    if (hLib == NULL) return SysAllocString(L"Failed to load test DLL");
    
    TestFunc func = (TestFunc)GetProcAddress(hLib, name);
    
    if (func == NULL) return SysAllocString(L"Entry point not found");
    
    BSTR result = func();
    FreeLibrary(hLib);
    return result;
}

I put the thunk DLL as a resource into GenTestAsm.exe, and it is always written alongside generated managed assembly. Having two additional DLL files hanging around is a little annoying, but it is better than being unable to recompile your code.

Conclusion

To summarize. GenTestAsm is a tool that allows to run unmanaged (typically, C++) tests in popular nUnit environment. A tiny support library provides authors of unmanaged tests with basic assertion facilities, similar to those of nUnit. With GenTestAsm a team can use more uniform approach to unit testing of managed and unmanaged code. The same tool is used to run the tests, and test syntax is similar.