Scenario: You have a shell application that loads plugins. For isolation, each plugin is loaded in its own AppDomain.
Question: If plugin causes an exception, how do you prevent it from bringing down your process and gracefully unload it?
Short Answer: It is very difficult.
Long Answer: What happens to an unhandled exception depends on a number of factors:
- Whether the exception occurs on a UI thread or on a worker thread.
- Whether the exception can be marshaled to the main
AppDomain.
Exception can be marshaled to the main app domain if its type is [Serializable], and if its assembly is can be loaded into the main app domain. Most built-in exceptions such as [ApplicationException] fall into this category. However, plugin code not under your control may throw exceptions of any type, and chances are user-defined exception will not be [Serializable].
Worker Threads
In .NET 2.0 and higher, unhandled exception in a worker thread will kill the process, regardless of the app domain in which it happened. You can subscribe to the AppDomain.UnhandledException event, but you cannot prevent the process from terminating afterwards.
Prior to .NET 2.0 unhandled exceptions in worker threads did not lead to unconditional process termination. It is possible to revert to the old by adding the following lines to your app.config file:
<runtime>
<legacyUnhandledExceptionPolicy enabled="1"/>
</runtime>
What happens next depends on whether the exception can be marshaled to the default app domain.
Serializable Worker Thread Exception
First CLR raises UnhandledException in the app domain that caused the exception. In the event handler you may use an ad hoc communication channel to signal main application to unload the failing app domain. Of course, unloading must be deferred until after exception handling is finished, otherwise you will get AppDomainUnloaded exception. If event handler throws another exception, that exception will be ignored.
Then CLR will marshal the exception to the default app domain and raise UnhandledException there. I am not aware of a way to stop this. The sender parameter of the exception handler will be set to the failing AppDomain, which is nice. Marshaling the exception may have an unwanted side effect of loading the assembly where exception type is defined into the default app domain. This assembly will then remain loaded forever. If it is a plugin specific assembly, this behavior is hardly desirable.
Non serializable Worker Thread Exception
The processing will begin as above, but when CLR tries to marshal the exception to the default AppDomain, it will cause an unhandled SerializationException in the default app domain. The UnhandledException event will be raised for this new exception. There is no way to retrieve the original exception or find out what app domain caused it.
There is no easy way to distinguish such marshaling exceptions from other serialization exceptions that naturally occur in the default app domain, e.g. from saving a file or making remoting calls. If you choose to continue after an unhandled SerializationExceptions, you may miss genuine exceptions in the default app domain that may leave your main program in an unknown state. If you choose to terminate the process on unhandled SerializationException, any non-serializable exception in a plugin will terminate the process. Both solutions are bad.
UI Threads
UI threads run a loop that processes window messages. UI frameworks like WPF or Windows Forms typically wrap message processing call in a try/catch block. Any exceptions thrown while processing a message may be caught and neutralized before they become truly unhandled. For WPF it is done via the Dispatcher.UnhandledException event, for Windows Forms it is Application.ThreadException event. You will have to setup this handler in each app domain.
However, here we may also run into a marshaling problem. Depending on the situation, WPF may handle the exception in the app domain that caused it, or it may try to marshal the exception to the default app domain. I am not sure what the exact rule is, but I observed both variants. If WPF decides to marshal the exception, it may not succeed, and thus you will get a SerializationException just like you would with a non serializable worker thread exception. Again, there is no way to distinguish this marshaling exception from a genuine SerializationException that occurred in the default app domain.
Bottom Line
If you don’t control plugin’s code and don’t know what kind of exceptions it will throw and when, setting up reliable exception handling strategy is very difficult or maybe even impossible. You will need to use processes for true isolation.