Run Your Unmanaged Tests in nUnit
Unit Testing C++ Code
Automated testing in general and nUnit tool in particular has become an invaluable asset for me and my team. Unit tests have greatly improved my confidence in the quality of code and my ability to make changes. They saved my bacon numerous times by discovering subtle and not-so-subtle bugs after "risky" updates.
However, nUnit has one serious limitation: it works only with managed code. From time to time I do write code in C++, and probably will continue to do so in the foreseeable future.
When it comes to unit testing C++ code, there are essencially have three choices:
- Do not do unit testing at all.
- Use one of the unit testing packages designed specifically for C++, e.g. TUT C++ unit test framework.
- Find a way to run C++ tests in nUnit.
Not doing unit testing at all is a very risky approach. The code becomes brittle and the risk of making changes is too high. TUT is a nice tool, but it does not provide a GUI test runner like nUnit. Also, having to switch between two different tools looks like a nuisance.
Therefore, I concentrated on the last approach - finding a way to run C++ tests in nUnit.
The Battle Plan
General battle plan was as follows:
- Make unmanaged tests callable from the outside world via DLL exports.
- Write a tool that takes an unmanaged DLL, enumerates its exports, and generates a managed assembly loadable by nUnit. I called this tool GenTestAsm.
- For each exported unmanaged function, automatically create a managed method marked with
- Managed method calls unmanaged function via P/Invoke.
Enumerating DLL Exports
Sadly, Win32 does not provide out-of-the box API for enumerating DLL exports. Fortunately, format of DLL files is publicly available from Microsoft. I extract the list of exports by opening the executable file and analyzing the bytes. It is a little tedious, but not very complex task. The biggest annoyance is that PE file format uses relative virtual memory addresses (RVAs) instead of file offsets. This is great when the file is loaded in memory, but requires constant recalculations when working with the file on disk.
Generating Test Assembly
To generate test assembly, I first create C# source code and then compile it using
CSharpCodeProvider class. This proved to be simpler and more straightforward than building the
code through CodeDOM. This is also easier to test. If something goes wrong with the generated
assembly, one can always look at the generated source code and scan it for abnormalities. I added an
option to GenTestAsm that outputs generated source code instead of compiled binary.
Test Exports vs. Other Exports
It is definitely possible that a DLL with unmanaged tests exports a function
that is not a test. When GenTestAsm creates the managed wrapper, it needs to know which
exports are tests and which are not. nUnit separates tests from non-tests using attributes,
but there are no attributes in the unmanaged world. I decided to use a simple naming
convention instead. GenTestAsm generates managed test wrappers only for the exports
whose names begin with certain prefix (by default "
UnitTest"). Other exports are ignored.
The next problem is how to handle test failures. In nUnit world a test is usually considered successful if it runs to completion, and failed if it throws an exception. Since my tests are written in unmanaged C++, their exceptions would be unmanaged C++ exceptions. I cannot let these exceptions escape into the managed wrapper. Therefore, I need some other mechanism to report test failures. I decided to use test's return value. Unmanaged tests must have signature
Return value of
NULL means success, anything else means failure, and returned string is the error
message. I chose
BSTR over regular
well-defined memory management rules, and .NET runtime knows how to free it.
Writing a Trivial Test
BSTR from the C++ test is nice, but it makes writing a test a little difficult.
The author of the test must make sure that unhandled C++ exceptions don't escape
the test. He also must format the error message and convert it to
BSTR. If this were done
by hand in each in every test, the code would become too verbose to be practical. Let's take
a trivial test in C#:
and see how equivalent test in C++ would look like:
This is too much boiler plate code. We need a support library here.
With the help of a tiny
#include file we can squeeze our C+
test back to 3 lines of code:
TEST macro that
encapsulates the details of exception handling and
It also defines a couple of
ASSERT macros such as
The Big Lockdown
However, there is one catch. As you remember, I use P/Invoke to call my unmanaged tests. Internally, P/Invoke loads the unmanaged DLL and keeps it loaded until the managed process exits. In other words, if I used P/Invoke blindly, once you executed the tests, your managed DLL would become locked. You would not be able to recompile it until you closed nUnit GUI. This is an unpleasant speed bump.
One Way Out
Instead of invoking unmanaged DLL directly, GenTestAsm could, of course, call
LoadLibrary(), and then
GetProcAddress(). It could then do
Marshal.GetDelegateForFunctionPointer() and invoke the resulting delegate. The problem is,
this API is available only in .NET 2.0. I wanted GenTestAsm to be compatible with .NET
1.1, so I had to find a different solution.
Another Way Out
If something must be loaded forever, let it be not the test DLL,
but some other, helper DLL that never changes. Current version of
GenTestAsm P/Invokes into unmanaged helper (thunk), which then calls
This way, it is the thunk that gets locked, while the real test DLL remains free.
I put the thunk DLL as a resource into GenTestAsm.exe, and it is always written alongside generated managed assembly. Having two additional DLL files hanging around is a little annoying, but it is better than being unable to recompile your code.
To summarize. GenTestAsm is a tool that allows to run unmanaged (typically, C++) tests in popular nUnit environment. A tiny support library provides authors of unmanaged tests with basic assertion facilities, similar to those of nUnit. With GenTestAsm a team can use more uniform approach to unit testing of managed and unmanaged code. The same tool is used to run the tests, and test syntax is similar.
|Programming Tools & Info
|Copyright (c) Ivan Krivyakov. Last updated: October 15, 2006