The solution is described for Solaris and Sun Studio, using the attached example project.  On Windows or Linux ,the process is similar, but the tools and parameters differ. Please regard the platform specific information at the end of this article.

In summary the steps are

1)      Find the crashing function and instruction offset by the call stack

2)      Disassemble the crashing address and the instructions around it

3)      Create an annotated assembly build of the compile unit which contains the function

4)      Match the compiled assembler code with the crashed instruction

Prepare the example

Compile the attached example by calling

> make Makefile_sol_cpp

Start an osagent, then the Server, then the Client

> osagent &

> Server &

> Client &

When the Client is started, the Server will crash with Segmentation Fault and a core dump is written.

Step 1

Check the call stack with

> mdb core

> ::stack

The topmost entry on the call stack will be something like

__1cLAccountImplFcrash6ML_L_ 0x78(135c70, 7, 138f78, 0, 3, b)

The call stack reveal the crash is located in AccountImpl::crash at the instruction with offset 0x78.

Step 2

Disassemble the crashing instruction by calling

> __1cLAccountImplFcrash6ML_L_ 0x78::dis

The output will be similar to

__1cLAccountImplFcrash6ML_L_ 0x78:      ldsb      [%i3 1], %o0

Check the content of the registers

> ::regs

It states %i3 = 0x00000000. The function can be lengthy. It may not be possible to find the matching line in the C source code.

Step 3

The compile unit which contains the crashing function is BankImpl.C. Use the output of the build process to find the compilation parameters for the file.

> make –f Makefile_sol_cpp

Find in the build output the command to build the compile unit.

CC -mt -DTHREAD -DPTHREADS -DSOLARIS -I. -I/opt/Borland/VB8/include -I/opt/Borland/VB8/include/stubs  -c -o BankImpl.o BankImpl.C

“-c -o BankImpl.o” means to compile without linking and specifies the name of the object file. Remove it, and replace it by “-S”, which means to compile only and generate assembly code.

> CC -mt -DTHREAD -DPTHREADS -DSOLARIS -I. -I/opt/Borland/VB8/include -I/opt/Borland/VB8/include/stubs  -S BankImpl.C

This will create BankImpl.s, which contains assembly code mixed with the original source code as annotations.

Step 4

Open BankImpl.s generated in Step 3, search for AccountImpl::crash and compare the following assembly instructions with the mdb disassembly of Step 2.  The output is almost similar to the one seen in Step 2.

!   69       test[1] = test2[1];
               ldsb        [%i3 1],%o0

               stb          %o0,[%fp-10]

! 69 specifies the line number of the crash. It is also easier to interpret the assembly now. ldsb loads a signed byte from an address, in this case the content of register %i3 plus 1. %i3 is test2, which is NULL and caused the SIGSEGV.

Compile Environment

Be aware that the generated assembly code strongly depends on the OS, Chipset and Compiler Version.  For best result, use the exact same environment for the binary and the assembly build.  If the environment is not exactly the same, it may still be possible to match the assembly build with the output of mdb. Look for the sequences of same instructions with same parameter format.  While the constants will be the same, the offset from begin of the function and the registers used may differ.

Windows, Visual Studio .Net Compiler

Use the /FA option to create a .asm file. It contains the assembly code, not mixed with the original C code, but the line numbers of the source code.  Use WinDbg instead of mdb to create and analyze a core dump. It is part of the Windows SDK.

Linux, GNU g Compiler

The –S option works like on Solaris, but the resulting file contains no annotations. Make a compile first and use objdump –S , it will output intermixed assembler and C source code.  Use dbg or gdb instead of mdb to analyze the core dump.