A few days ago I had a support call where an ASP.net application deployed to a Windows 2008 R2 server and running on IIS 7.5 was crashing from time to time. When a page from the application crashed, a 500 error code was returned with an error message similar to the following:
Compiler Error Message: The compiler failed with error code -1073741502.
The error is then followed by the command line with which the .Net compiler was launched. This can be a very daunting error as it can appear and disappear randomly, and recycling the app pool or restarting IIS will not help. If you just need the error solved, go directly to the solution part of the article, if not, keep reading for the detailed explanation.
So what actually goes on:
ASP.net applications can be deployed in several modes to an IIS server (I discuss this in more detail in the following video: http://linqto.me/IISArchP3 ), but basically each source code file in an ASP.net application (be it a .cs – C# file or a .vb – VB.Net file) will have to be compiled before it can be executed on the server. Generally this occurs when a request for that particular par or resource comes in from an end user.
Suppose we have an application that has a page called Default.aspx. The page code behind is written in C# and is present in a file called Default.aspx.cs. When the first request for this page comes in from the very first user to hit the site, the .Net Runtime will notice that the page is not yet compiled and will request that the page be compiled – the C# code be transformed into a .dll that can be executed - .
When the page has to be compiled, what will happen is that the w3pw.exe (worker process) that hosts the ASP.net application will request the activation of the .Net compiler (csc.exe – C# compiler for C# code or vbc.exe for VB.net code). You can see a detailed architecture diagram of all IIS components and how they interact in this video: http://linqto.me/IISArchP1 and a detailed explanation of the inner workings of the worker process (w3wp.exe) in this video: http://linqto.me/IISArchP2 .
The .Net Runtime will then pause the execution of the request pending the execution of the compiler process (csc.exe or vbc.exe). You can actually see the compiler activating if you look at the Task Manager of your server when the first request comes in. The compiler will run with the same credentials (account) as those used for the application pool. By default this is Application Pool Identity is used as of IIS 7.5, but you can configure your own account in the 'Advanced Properties' of the application pool.
Should the compiler process encounter an error (be it a compile error that you introduced in your code or something else) it will then transfer the error message back to the worker process, which will just issue a 500 (internal server error) status code for the particular request back to the connecting client. So we now know that the error is coming from the csc.exe process that runs with the same credentials and in the same session with the w3wp.exe. Now to the error message that is raised in the particular case.
Looking at the return code value, the first thing to do is to convert from base 10 to base 16 – or HEX, and we can do this using the debugger or even the windows calculator:
.formats -0n1073741502
Evaluate expression:
Hex: c0000142
Decimal: -1073741502
Octal: 30000000502
Binary: 11000000 00000000 00000001 01000010
Chars: ...B
Time: ***** Invalid
Float: low -2.00008 high -1.#QNAN
Double: -1.#QNAN
Then using a tool to lookup the hexadecimal error code we can see the following:
!error c0000142
Error code: (NTSTATUS) 0xc0000142 (3221225794) - {DLL Initialization Failed} Initialization of the dynamic link library %hs failed. The process is terminating abnormally.
Hence the csc.exe process terminated because of a Dll initialization failure. This will most likely happen when the system is low on what is called Desktop Heap memory. This is a special kind of memory that is allocated at boot time for each of the sessions opened by an account on the machine. Hence, all processes that are running using the credentials of that particular account will share resources from the Desktop Heap that is allocated for that account's session.
Should there be too many processes running under the same account's credentials, the desktop heap can be close to exhaustion, so resources might not be available to dlls that need to load inside a particular process. Here is an article on how to resize the Desktop Heap segments: Link – but what I recommend is a different solution.
Solution:
Now that we know that this error is generated by Desktop Heap exhaustion, a simple solution is to create a dedicated account for the application pool for which we are seeing the error. Hence, the worker process and the csc.exe process will have the entire Desktop Heap segment available account to themselves, hence bypassing the resource consumption problem.
As a final note, I would like to underline that the default configuration for IIS application pools, in which each pool dynamically is allocated its own identity (account) by IIS when application pool starts, bypasses the problem as well, since each application pool, as well as the compiler instances it needs to launch, do not need to compete for the scarce resources of a shared desktop heap.