Introduction to debug Fatal Execution Engine Error in .net code

  We have one .net application running in production server. It runs 2 hours or so and quits every day. It runs fine until I made some change intended for performance improvement. However, it turns out that modified code had bug. When application is put to run for long time for test, it crashes usually after some time which varies from 2 hours to 5 hours. When it crashes, it just disappears without any trace except the following errors in the application event log:

.NET Runtime version 2.0.50727.3615 - Fatal Execution Engine Error (000006427F44AE16) (80131506)

Faulting application myapp.exe, version 1.0.0.0, stamp 4d7753b2, faulting module mscorwks.dll, version 2.0.50727.3615, stamp 4be90280, debug? 0, fault address 0x000000000026e55a.

It is very hard to debug such crash since the exception can’t be caught with regular try-catch block. After 3 weeks investigation, I eventually kinda of solve this problem with two tools provided by Microsoft. 

cscript

windbg

 

With the help from cscript, one can catch the exception and create memory dump when it happens. The uniqueness of this tool is that it can catch the exception from .NET Runtime.  In my case I use the following command to see what exception happens.

cscript adplus.vbs -crash -FullOnFirst -p <PID>

Here <PID> is the process id. And the culprit turns out to be access violation exception in mscorwks.dll. Then I use the below command with one configuration file to obtain full memory dump instead of mini memory dump of the crashed process.

cscript.exe adplus.vbs -c c:/ADPlus.config

Configuration file:

<ADPlus><Settings><RunMode>crash</RunMode><Option> Quiet </Option><Option>NoFreeSpaceChecking</Option><OutputDir>C:/Program Files/Debugging Tools for Windows</OutputDir><ProcessName>BOServer.exe</ProcessName></Settings><PreCommands></PreCommands><Exceptions><!-- First we redefine all exceptions --><Config><Code>AllExceptions</Code><Actions1>Log</Actions1><Actions2>Log;EventLog</Actions2></Config><!-- Next we define a special behavior for the AccessViolation exception --><Config><Code>AV</Code><Actions1>FullDump;Log;EventLog</Actions1><Actions2>FullDump;Log;EventLog</Actions2></Config></Exceptions></ADPlus>  

With full memory dump, I can use tool windbg to execute the following command to make deep analysis.

windbg -y SymbolPath -i ImagePath -z DumpFileName

 I found my .NET application crashes during the execution of.NET function SqlBulkCopy::WriteToServer, one DataTable object, one parameter of WriteToServer has heap corruption problem. Though now I still can’t understand why heap corruption occurs in the data memeory of that DataTable object, I managed to use one override form of WriteToServer function to work around this problem. The override function takes DataRow[] array as parameter, and doesn’t need DataTable. (Since I don’t understand why memory corruption happens, this problem is kinda of solved, I think. If anyone feels interested, or has any problem, please contact me. Thanks. ^_^)

During my experience with windbg, I use the following command to get result. More commands can be searched via internet. Moreover, these is one really great blog for using windbg to debug .net application, I strongly recommend visiting it. Here is the address: http://blogs.msdn.com/b/tess/

The command I used:

CommandDescription.loadby sos mscorwksload sos in frame 2.0!clrstack -pview clr function argument!clrstack -aview clr stack variables.!clrstack -lview clr stack local variable~*e!clrstackList all managed code stack!do addrdump object at address!dd addrDisplay address!da addrPrint the content of array!dumpmt addrDisplay type of object at that address!gcroot addrDisplay which object is holding object at address!verifyheapVerify whether heap has problem