As part of one of my projects I needed to use Microsoft DIA SDK. In particular, I wanted to retrieve source files checksums stored in PDB files.
DIA SDK is a COM component written in C++. The recommended way to call a COM component from .NET is to use tlbimp.exe to convert the component’s type library file (.tlb) into a .NET assembly, and then call that assembly. If there is no type library, one can be created from the component’s interface definition file (.idl) by the MIDL compiler using the /tlb option.
The trouble is, the method for getting checksums, IDiaSourceFile::get_checksum(), is not type-library friendly, because it returns a raw byte array (byte*). TLB files cannot properly handle “raw byte array” value type, because it is not OLE Automation compatible, see below. get_checksum() method generated via the tlbimp process returns a single byte, which is not very useful.
Thus, the “easy” way to access the COM component does not work, but we have a number of alternatives:
- Program the whole thing in C++.
- Use Managed C++ for the part that deals with DIA SDK, and C# for the rest.
- Decompile the assembly generated by tlbimp.exe, fix the problem and recompile.
- Manually create a wrapper assembly in C#.
Program the whole thing in C++
This is something I did not attempt. Even though C++ had been my main programming language for about 10 years, my C++ development speed is 3-4 times slower than my C# development speed, and the bugs in C++ code are harder to catch. Of course, there might be some people out there who crank up C++ much faster than C#, but I am not one of them, and I am yet to meet one.
Use Managed C++
I gave it a try and it worked. The code is on GitHub. Note, that this is not the latest version of the repository.
I created a Managed C++ project, included DIA SDK header files directly, did all COM interop using C++, and wrapped it in a managed class. This was slightly painful, but bearable.
This approach caused an unexpected problem with 32-bit vs 64-bit compatibility. A C# project compiles into a purely managed binary that can be executed as 32-bit or 64-bit. A Managed C++ project compiles into a binary that can be either 32-bit or 64-bit, but not both; its bitness is fixed at compile time.
All parts of the program must have the same bitness at runtime: a 32-bit process can only load 32-bit DLLs, and a 64-bit process can only load 64-bit DLLs. Note, however, that 64-bit Windows can execute 32-bit programs, via some low level tricks, but 32-bit Windows cannot execute 64-bit programs.
The path of least resistance is to compile the MC++ code to 32-bit, and mark the main C# library as 32-bit only (x86). This means that the entire program will always run at 32-bit, even on a 64-bit operating system. DIA SDK does provide a 64-bit version of its COM component, but it would never be used. Practically speaking, this is not a huge deal, but I felt that this is not the best approach.
Another option would be to produce two MC++ binaries and load them dynamically from C# depending on the current bitness, but that is too much of a burden.
Forcing the entire program to 64-bit is not a good idea, because such program will not run on 32-bit Windows.
Decompile, fix and recompile the auto-generated wrapper
The wrapper generated by tlbimp.exe is in fact quite close to what we want, we just need to fix a couple of small issues. Technically, it can be decompiled to IL (Intermediate Language), fixed, and recompiled.
This approach is described on StackOverflow, and apparently did work for some people, but I did not try it, since I am not an IL guru.
Manually create a wrapper assembly in C#
This is the method I ended up using. The code is here: https://github.com/ikriv/IsItMySource.
Of course, I did not create the entire wrapper manually: I started with the automatically generated wrapper recompiled by Resharper.
It almost worked: I found out that Resharper sometimes rearranges the methods in the interfaces, which ruins binary compatibility, and wrong methods end up being called. I contacted Resharper support and they logged as issue RSRP-464903. I tried to use Telerik JustDecompile instead, but it was even worse: it sorted the methods alphabetically.
Thankfully, DIA SDK interface is not that large and Resharper follows a known pattern when rearranging methods. I ended up manually checking all the interfaces, and I hope I did manage to undo Resharper’s reordering, but there is still a chance I missed something.
It’s all fine, but what is “OLE Automation Compatible” and how can byte be banned?
COM was first introduced over 20 years ago, in the largely C/C++ ruled world. It was supposed to be a language-agnostic standard, so component interfaces (today we would call it “metadata”) were to be published not in C, but in a special Microsoft Interface Definition Language (MIDL). A MIDL compiler would then be used to create bindings to the component in various programming languages.
In practice, the only language MIDL can be compiled to is C.
The main problem with MIDL is that there is no API that would load a MIDL file and report what interfaces a component provides and how to call them. Compiling MIDL to C does not help: C also lacks reflection API of any kind. Theoretically, one could parse the MIDL file to get the required information, but MIDL syntax is quite close to C syntax, so parsing a MIDL file is complex, slow and inefficient task.
MIDL was fine for calling pre-defined components from C or C++, but it was inadequate for scripting arbitrary components from interpreted language like Visual Basic.
Consequently, Microsoft introduced so called OLE Automation. It supported a limited set of data types, but provided metadata querying API, and allowed to invoke methods by name. In OLE automation the metadata is stored in binary form in so called Type Libraries (TLB files). Programming environments like Visual Basic do not use the MIDL compiler: they rely on Automation to convert strings like wShell.OpenFile "myfile.txt" into method calls at run time.
OLE Automation does not support raw C arrays like unsigned char[] , because C arrays are problematic: they do not contain size information and are indistinguishable from a pointer to a single object. OLE Automation replaces raw C arrays with so called “Safe arrays”. COM methods that take or return raw C arrays are allowed, but they are not OLE Automation compatible, and raw array type cannot be encoded in a TLB file.
The MIDL compiler will convert a MIDL file to a TLB file when invoked with the /tlb option. However, this conversion is potentially lossy: only Automation compatible types can be represented in the TLB, and everything else is quietly ignored. The tlbimp.exe does not have access to the original MIDL file, all it sees is the TLB file, where C array is already replace with a single object, so it generates a method returning single object instead of an array.
The irony is that .NET COM Interop is perfectly capable of supporting all COM interfaces, not only Automation compatible ones. It should be theoretically possible to compile MIDL directly into a managed assembly, but no one undertook this task, and probably no one will: parsing MIDL code is complicated, the number of corner cases is large, and the general coolness of all this is relatively low.
Conclusion
There is no fully automated solution for calling COM component from .NET code, if the component is not 100% OLE Automation compatible.
One work around is to use Managed C++, but it requires unique programming skills (Managed C++ is a relatively exotic animal), and, practically speaking, forces the program to 32 bit.
Another work around is to build the wrapper manually, based on the (not working) wrapper generated by tlbimp.exe. This may be practical for small to medium sized interfaces, but not for large components. Popular .NET decompiling tools like Resharper/dotPeek and Telerik reorder methods in decompiled classes, which destroys compatibility with the original COM interface and causes mysterious runtime errors. Such reordering must be manually spotted and fixed. This further reduces the upper limit of the size of the component that can be handled using this method.