Introduction to postmortem debugging

A program crash is a very nasty thing. Unfortunately, we are all not perfect, and even using the most secure development methods (for example, TDD), we are not immune from the fact that the program will fall. Especially bad if it falls already at the customer. But on the way to the ideal, we always have the tools that can help to investigate the fall of programs, identify errors, and most importantly correct them.
One thing is sad that many, even very experienced developers are not familiar with this toolkit and many companies do not include this practice in their work. I'm talking about postmortem debugging.
In this article I want to show the basics of working with this beast, and perhaps I will push the developers to expand their knowledge in the field of ice cream. So, developers are invited to read C ++ Windows, team leads, and, well, it will be a good idea for heads of development departments to familiarize themselves with it.

We break the program

Let's start with the fact that we will write a small program that will fall down when you try to run it,
for this we use VS2008 and create a console application in C ++ with the following code:

#include "stdafx.h"

int _tmain( int argc, _TCHAR* argv[])
{
int * p = NULL;
p[0] = 10;
return 0;
}

* This source code was highlighted with Source Code Highlighter .

as you can see, we created a null pointer and try to stuff a value there. Compile the program in the Release configuration and run it from under the explorer. We will see something like this:

If we press the “Debug program” button, the debugger will most likely come up and show you the place where the error occurred. This project was with default settings, that's why my studio went up and showed the place of the error in the source code.
If for some reason you don’t know how the studio managed to do this, then look into the project settings:

By default, the highlighted values ​​were set. This means that, together with the program, a pdb file is created containing debugging information about our program. Due to the fact that we launched the program on the same computer where it was written, when debugging, Visual Studio picked up this pdb file and, through the paths stored in it to the sources, could open them. Further, Visual Studio unfolded the drop stack and showed us a line of code which caused an exception. And if the crash happened on someone else's computer, where there is neither a pdb file, nor Visual Studio, nor source? In this case, we ourselves must catch the exceptional situation and collect all the possible data about how it arose.

Set unhandled exception handler

Rewrite the program as follows:

#include "stdafx.h"
#include <windows.h>
#include <io.h>

LONG WINAPI CustomUnhandledExceptionFilter( PEXCEPTION_POINTERS pExInfo )
{
_tprintf( TEXT( "Exception!" ) );
return EXCEPTION_EXECUTE_HANDLER;
}

int _tmain( int argc, _TCHAR* argv[])
{
LPTOP_LEVEL_EXCEPTION_FILTER hOldFilter = SetUnhandledExceptionFilter( CustomUnhandledExceptionFilter );

int * p = NULL;
p[0] = 10;

SetUnhandledExceptionFilter( hOldFilter );

return 0;
}

* This source code was highlighted with Source Code Highlighter .

As you can see, we added the CustomUnhandledExceptionFilter function here, which only prints the word “Exception!” To the console. And in the main function we set the filter of unhandled exceptions at the beginning and return the old one at the end.
Compile the program, run ... everything! no more error messages. Now the word “Exception!” Is only displayed in the console.
We proceed to the next step, the preservation of information about the fall.
')

Let there be light DbgHelp.lib

Now the fun begins. Go to www.microsoft.com/whdc/DevTools/Debugging/default.mspx
and download the Debugging Tools For Windows from there for your system. This is the installer for the WinDBG debugger and debug SDK.
We install all this on a computer and make sure that the studios have registered the paths to the SDK that went to Debugging Tools For Windows.

Further, depending on the project, we prescribe the DbgHelp.lib library and modify the CustomUnhandledExceptionFilter function as follows:

#include <dbghelp.h>

LONG WINAPI CustomUnhandledExceptionFilter( PEXCEPTION_POINTERS pExInfo )
{
HANDLE hFile;

hFile = CreateFile( TEXT( "c:\\minidump.dmp" ), GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL );

if ( NULL == hFile || INVALID_HANDLE_VALUE == hFile )
return EXCEPTION_EXECUTE_HANDLER;

MINIDUMP_EXCEPTION_INFORMATION eInfo;
eInfo.ThreadId = GetCurrentThreadId();
eInfo.ExceptionPointers = pExInfo;
eInfo.ClientPointers = FALSE;

MiniDumpWriteDump( GetCurrentProcess(), GetCurrentProcessId(), hFile,
MiniDumpNormal, &eInfo, NULL, NULL);
CloseHandle( hFile );

return EXCEPTION_EXECUTE_HANDLER;
}

* This source code was highlighted with Source Code Highlighter .

Here we connected the header file dbghelp.h and made the call to the MinidumpWriteDump function to save the application crash.
It remains only to copy the dbghelp.dll file from the Debugging Tools For Windows folder to the Release folder of our application and run it. The minidump.dmp file will appear on the “c: \” drive - this is the corpse of our application, we will dissect it.

Morbid anatomy app

Run the WinDbg debugger from the Debugging Tools For Windows package. Go to the menu File-> Open Crash Dump ... open the file minidump.dmp and negatively answer the question about saving the workspace. Next we go to the menu File-> Symbol File Path ... there we enter the path to the folder Release of our project, enter the same path in the window File-> Image File Path ... In the window File-> Source File Path ... we prescribe the folder with the sources of our project. Here we are ready for preparation.
And we write the cherished command in the command line of the debugger:! !analyze -v
and in a couple of seconds we will get a listing containing something like this:

FAULTING_IP:
CrashHandler! Wmain + 11 [d: \ work \ crashhandler \ crashhandler.cpp @ 35]
00401091 c7010a000000 mov dword ptr [ecx], 0Ah

EXCEPTION_RECORD: ffffffff - (.exr 0xffffffffffffffff)
ExceptionAddress: 00401091 (CrashHandler! Wmain + 0x00000011)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 00000000
NumberParameters: 2
Parameter [0]: 00000001
Parameter [1]: 00000000
Attempt to write to address 00000000

PROCESS_NAME: CrashHandler.exe

ADDITIONAL_DEBUG_TEXT:
Use the '! Findthebuild' command to search for the target build information.
If the build information is available, run '! Find thebuild -s; .reload 'set symbol path and load symbols.

FAULTING_MODULE: 75bf0000 kernel32

DEBUG_FLR_IMAGE_TIMESTAMP: 4bb07e29

MODULE_NAME: CrashHandler

ERROR_CODE: (NTSTATUS) 0xc0000005 - EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - EXCEPTION_PARAMETER1: 00000001

EXCEPTION_PARAMETER2: 00000000

WRITE_ADDRESS: 00000000

FOLLOWUP_IP:
CrashHandler! Wmain + 11 [d: \ work \ crashhandler \ crashhandler.cpp @ 35]
00401091 c7010a000000 mov dword ptr [ecx], 0Ah

FAULTING_THREAD: 0000109c

BUGCHECK_STR: APPLICATION_FAULT_NULL_POINTER_WRITE_WRONG_SYMBOLS

PRIMARY_PROBLEM_CLASS: NULL_POINTER_WRITE

DEFAULT_BUCKET_ID: NULL_POINTER_WRITE

LAST_CONTROL_TRANSFER: from 0040120d to 00401091

STACK_TEXT:
0018ff44 0040120d 00000001 003c2dc0 003c3c90 CrashHandler! Wmain + 0x11 [d: \ work \ crashhandler \ crashhandler.cpp @ 35]
0018ff88 75c03677 7efde000 0018ffd4 773b9d72 CrashHandler! __ tmainCRTStartup + 0x10f [f: \ dd \ vctools \ crt_bld \ self_x86 \ crt \ src \ crtexe.c @ 583]
WARNING: Stack unwind information not available. Following frames may be wrong.
0018ff94 773b9d72 7efde000 6819a792 00000000 kernel32! BaseThreadInitThunk + 0x12
0018ffd4 773b9d45 00401355 7efde000 00000000 ntdll! RtlInitializeExceptionChain + 0x63
0018ffec 00000000 00401355 7efde000 00000000 ntdll! RtlInitializeExceptionChain + 0x36

STACK_COMMAND: ~ 0s; .ecxr; kb

FAULTING_SOURCE_CODE:
31: int * p = NULL;
32: p [0] = 10;
33:
34: SetUnhandledExceptionFilter (hOldFilter);
> 35:
36: return 0;
37:}
38:

SYMBOL_STACK_INDEX: 0

SYMBOL_NAME: CrashHandler! Wmain + 11

FOLLOWUP_NAME: MachineOwner

IMAGE_NAME: CrashHandler.exe

BUCKET_ID: WRONG_SYMBOLS

FAILURE_BUCKET_ID: NULL_POINTER_WRITE_c0000005_CrashHandler.exe! Wmain


In this listing there is everything to understand what happened with our program.
in the EXCEPTION_RECORD section there is information about the exception, in the STACK_TEXT section you will receive a drop stack with addresses, function names, source codes and code lines.

Now you can take the exe file of your program and the dbghelp.dll file, transfer it to another computer, and run it there. Take the minidump.dmp file from there and again parse this file on your work computer. And the result will be the same. This is postmortem debugging.
In the next section I will give some tips.

Further development

In the example above, we made a primitive program that, when dropped, collects its own dump and was able to parse this dump in order to get a stack of crashes. And now a few sensible thoughts on the organization of work with this kind of things.

1. Common sense dictates to us that directly the handler for unhandled exceptions should be placed in a separate dll for reuse in your programs. To register a handler, it would be good to make a class, an instance of which is created on the stack at the entry point of your application (then we will be able to set our handler in the constructor and remove in the destructor
2. In addition to the dump, you can also create a file with information about the exception by manually parsing the data structures in the callback of the DumpWriteDump function (read the help to this function for clarity). And in general, you can even fasten an archiver to the dump collector to reap everything, but do not forget to insert a reentrant protection into the handler (in case an exception occurs during the processing of an exception)
3. To obtain reliable data on the drop stack, we need to have debugging symbols from all modules that have been loaded by the application. To do this, you should deploy the debug information server, about how to do this, I wrote in my article: habrahabr.ru/blogs/development/89094/#habracut
4. The dbghelp.dbg library has long been bundled with Windows, with Visual Studio, etc. therefore, in order not to fence dll-hell, distribute it along with your application and load exactly the copy you need.

Where to dig further

If you liked this article and you want to use Postmortem Debugging, then you should study the following:

• Help from Debugging Tools For Windows, from the SDK and from the WinDbg debugger
• Oleg Starodumov’s blog , debuginfo.com , has useful articles and utilities.
• Look at the old, but very useful implementation of the dump collection library www.codeproject.com/KB/debug/XCrashReportPt1.aspx (there are 4 parts)
• Read John Robbins debug books, as well as his blog (his nickname is Bugslayer).

Well, fantasize. For example, one of my former colleague, a very competent specialist, made a library that collected ladies, sent them to a special mailbox of the company. A robot hung on this mailbox, which parsed the dumps, looked for familiar components on the stack, and sent letters with dumps to the programmers responsible for these modules. If I am not mistaken by this method, he even wrote an article in the RSDN Journal.

Good luck and a reckless code to you, if you have any questions - ask, discuss.