Amusing fact #1: .NET framework does not have built-in class to load RSA private key from PKCS#1 (PEM) representation.
I borrowed this: http://www.codeproject.com/Articles/162194/Certificates-to-DB-and-Back

Amusing fact #2: creating an RSA key requires file system access. To the user profile. If you are running under an ASP.NET app pool user that has no profile, you get this:

System.Security.Cryptography.CryptographicException: The system cannot find the file specified.

   at System.Security.Cryptography.CryptographicException.ThrowCryptographicException(Int32 hr)
   at System.Security.Cryptography.Utils._CreateCSP(CspParameters param, Boolean randomKeyContainer, SafeProvHandle& hProv)
   at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
   at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle& safeProvHandle, SafeKeyHandle& safeKeyHandle)
   at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
   at {my method}

To fix, one must either create the user profile, or use CspProviderFlags.UseMachineKeyStore when creating RSACryptoServiceProvider. If you choose the latter, make sure the user has write access to C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys. I believe %AllUsersProfile%\Microsoft\Crypto\RSA\MachineKeys should be used to accommodate customized or non-English systems.

If you use machine storage, but the user does not have write access to the above folder, you’ll get
System.Security.Cryptography.CryptographicException: Access denied.

I am not certain which exact calls raises the exception: first Microsoft function on the call stack is GetKeyPair(), but I am not calling it: the actual culprit must be inlined. I suspect it is either new RSACryptoServiceProvider(CspParameters) or RSACryptoServiceProvider.ImportParameters(RsaParameterTraits), I did not have time to investigate which one.