Windows Debugging Tips

Windows Debugging Tips

How do I get user memory dump

For Crash
  1. cdb -p $pid
  2. enter ‘g’ to continue ( go)
  3. when crashed -> .dump /ma /u c:\temp\foo.dmp
  4. ‘qd’ to quit from the debugger with detaching
  5. c:\temp\foo_YYYY-MM-DD_HH-MM-SS-XXX_YYYY.dmp will be created
For Hang
  1. cdb -pv -p $pid -c “.dump /ma /u c:\temp\foo.dmp; qd”
  2. wait for 30-60 seconds
  3. take another memory dump
  4. wait for 30-60 seconds
  5. take another memory dump
  6. repeat until you get 3 dump files

How do I get kernel memory dump

  • If the OS is not hanging, you can take kernel dump by lived, or by killing cssrs.exe + Registry to get full dump as below.
  • Sometimes the entire server hangs in kernel mode (by Anti Virus etc).
  • In that case, set Kernel Full Dump and get the kernel dump by crashing OS.
  • If it’s HP machine, disable boot timeout in BIOS (that will prevent taking full dump).
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl -> 1
  • See for more info
  • Kill csrss.exe (which will crash the box), or hold down right ctrl + press scroll lock key twice after configuring it in Registry
  • You’ll get the dump in c:\windows\memory.dmp

How do I debug with memory dump on a different platform

  • When you debug using a dump file taken on another box, sos debugging may fail to map the clr structures using mscordackws if the target box’s .Net framework patch level is different.
  • Setting sympath by ‘.symfix’ and ‘.reload’ will download necessary files from MSFT.
  • But if it doesn’t work, you may need to do this.
    • copy sos.dll and mscordacwks.dll from the original machine which generated the dump into your local (i.e. c:\temp)
    • open the dump, then “.load c:\temp\sos.dll”
  • If it doesn’t work, follow the next steps
0:010> !clrstack
* Symbols can not be loaded because symbol path is not initialized. *
*                                                                   *
* The Symbol Path can be set by:                                    *
*   using the _NT_SYMBOL_PATH environment variable.                 *
*   using the -y <symbol_path> argument when starting the debugger. *
*   using .sympath and .sympath+                                    *
PDB symbol for mscorwks.dll not loaded
Failed to load data access DLL, 0x80004005
Verify that 1) you have a recent build of the debugger (6.2.14 or newer)
2) the file mscordacwks.dll that matches your version of mscorwks.dll is
in the version directory
3) or, if you are debugging a dump file, verify that the file
mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path.
4) you are debugging on the same architecture as the dump file.
For example, an IA64 dump file must be debugged on an IA64
You can also run the debugger command .cordll to control the debugger's
load of mscordacwks.dll.  .cordll -ve -u -l will do a verbose reload.
If that succeeds, the SOS command should work on retry.
  • get the mscordackws from the box where the dump was taken
  • rename it to mscordacwks_<arch>_<arch>_<version>.dll where arch is either AMD64 or x86 and version is the dll’s version
    • Sample from a 32bit box: mscordacwks_x86_x86_2.0.50727.3634.dll
    • Sample from a 32bit process on a 64bit box: mscordacwks_x86_AMD64_2.0.50727.5681.dll
    • Sample from a 64bit process on 64bit box: mscordacwks_AMD64_AMD64_2.0.50727.5681.dll
  • copy the dll to your box somewhere ( like to c:\dbgsymbols )
  • add it to the path with .exepath+ c:\dbgsymbols
  • .reload
  • .loadby sos mscorwks (.NET2) or .loadby sos clr (.NET4)

How do I resolve symbols when setting breakpoints

  • If you can’t set breakpoint in windbg or visual studio debugger, most likely symbol for the function is not loaded. Add the symbol path as below.
  • If you still can’t resolve the symbol in source code debugging in  Visual Studio, your code may be newer than the build binaries/pdb. Rebuild / regenerate binaries with pdb and try it again.
  • common: open IE and go to any external site to make sure you have external http access. if not, set http proxy.
  • windbg:
.sympath+ path-to-your-pdb
  • Visual Studio

1. Debug -> Options and Settings -> Debugging -> Symbols

Add your pdb path

How do I debug Windows service or w3wp process which will crash as soon as it starts

  1. Open Regedit
  2. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\$exename.exe
  3. Create (REG_SZ) Debugger = full-path-to-cdb -server tcp:port=8090
  4. Run the target web application to debug
  5. windbg -remote tcp:server=localhost,port=8090
    1. cdb will be attached to the remote debugger on the same machine
  6. ‘g’ to go and take memory dump by “.dump /ma /u c:\temp\foo.dmp”
  7. ‘q’ to quit the process or kill the target w3wp process and remote cdb
  8. Delete ‘Debugger’ value under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\$exename.exe

How do I run debugger instead of Dr. Watson when any application crashes

  • use aedebug key in Registy. ‘ntsd -iae’ will install ntsd instead of Dr.Watson.

How do I make Dr Watson do full dump instead of mini dump

  • When you set AdDebug->Auto=1 & Debuggers = “c:\foo\cdb.exe” -p %ld -c “.dump /ma /u c:\crash.dmp;.kill;qd”, it should make a full dump in c:. It usually works, but it didn’t work well on some environments. In that case, Dr Watson’s full dump feature may be better (not sure).
drwtsn32 –i (this reverted AeDebug -> Debuggers to Dr Watson).
Drwtsn32 -> GUI appeared -> change the followings and save.
Log file path: d:\logfiles
Crash dump: d:\logfiles\drwatson.dmp
Choose ‘full’ for crash dump type
Left the others default.
That changed configs under

How do I find loaded debug dlls

load psscor2 or 4
!FindDebugModules -full

How do I get a list of object for a specific type

  •  Use “dumpheap -stat” to get a list of available types
  •  Then run “!dumpheap -type $typname” to get a list of the target objects
0:000> !dumpheap -stat
00007fff7a1c41e8       10          240 Hoge.Bar
00007fff7a1c4100       10          240 Hoge.Foo
00007fffd8942d00       26         1456 System.RuntimeType
00007fffd8940bd8      226         9174 System.String
00007fffd8941250       28        36520 System.Object[]

0:003> !dumpheap -type Hoge.Bar
         Address               MT     Size
000000a404725190 00007fff7a1c41e8       24
000000a404725280 00007fff7a1c41e8       24
000000a404725370 00007fff7a1c41e8       24
000000a404725460 00007fff7a1c41e8       24
000000a404725550 00007fff7a1c41e8       24
000000a404725640 00007fff7a1c41e8       24
000000a404725730 00007fff7a1c41e8       24
000000a404725820 00007fff7a1c41e8       24
000000a404725910 00007fff7a1c41e8       24
000000a404725a00 00007fff7a1c41e8       24
              MT    Count    TotalSize Class Name
00007fff7a1c41e8       10          240 Hoge.Bar
Total 10 objects
0:003> !do 000000a404725190
Name:        Hoge.Bar
MethodTable: 00007fff7a1c41e8
EEClass:     00007fff7a2d2450
Size:        24(0x18) bytes
File:        C:\Users\sokoide\Projects\Spike\Hoge\bin\Debug\Hoge.exe
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fffd8940bd8  4000002        8        System.String  0 instance 000000a4047251e0 <Name>k__BackingField
0:003> !do 000000a4047251e0
Name:        System.String
MethodTable: 00007fffd8940bd8
EEClass:     00007fffd82d1aa8
Size:        34(0x22) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      bar0
              MT    Field   Offset                 Type VT     Attr            Value Name
00007fffd89437c8  40000ab        8         System.Int32  1 instance                4 m_stringLength
00007fffd8941f38  40000ac        c          System.Char  1 instance               62 m_firstChar
00007fffd8940bd8  40000ad       18        System.String  0   shared           static Empty
                                 >> Domain:Value  000000a402b72860:NotInit  <<

How do I get opened handles

  • Attach a debugger (i.e. windbg -p pid)
  • “!handle” to get handle info. To see more in detail, “!handle address f”
  • sample output
Handle 634
Type         Key
Attributes   0
GrantedAccess 0x20019:
HandleCount  2
PointerCount 3
Name         \REGISTRY\MACHINE\SOFTWARE\Microsoft\Fusion\PublisherPolicy\Default
Object Specific Information
Key last write time:  10:52:50. 7/31/2008
Key name Default
  • Since handles are Kernel object, you can’t get more info from user mode memory dump. Use Sysinternal’s handle.exe to get more info about the handle, or use Kernel debugger / dump.

How do I find handle leak

  • Attach a debugger (i.e. windbg -p pid)
  • “!htrace -enable”
  • “!htrace -snapshot”
  • “g” to resume the process
  • (run your app for a while)
  • ctrl-break to break the process
  • “!htrace -diff” to show the diff of handles
  • sample output
0:003> !htrace -diff
Handle tracing information snapshot successfully taken.
0x9 new stack traces since the previous snapshot.
Ignoring handles that were already closed...
Outstanding handles opened since the previous snapshot:
Handle = 0x00000634 - OPEN
Thread ID = 0x00000b90, Process ID = 0x000015f4
  • See individual handle info by “!handle address f” as written in the previous section

How do I find native memory leak

set _NT_SYMBOL_PATH=symsrv*symsrv.dll*c:\dbgsymbols*;C:${path-to-your-pdb}
gflags -i w3wp.exe +ust (or gflags -r +ust and reboot for system-wide to create User mode Stack Trace database)
umdh -p:${process-id} -d -f:snap1
run test
umdh -p:${process-id} -d -f:snap2
run test
umdh -p:${process-id} -d -f:snap3
umdh -v -d snap1 snap2 > diff1_2
umdh -v -d snap2 snap3 > diff2_3
check the diff (or by using Excel)
gflags -i w3wp.exe ffffffff

How do I find managed memory leak

  • Get 2 memory dumps
  • Run these and compare them
    • !dumpheap -stat
    • !dumpheap -type $type
    • !do $obj

How do I find native heap corruption

  • Full pageheap (more memory for guard pages)
gflags -i yourprocess.exe +hpa            (+hpa means turning on Page Heap flag to enable pageheap corruption checking)
run test
the process breaks and debugger launches (depends on AeDebug) when the guard page is accessed
gflags -i yourprocess.exe ffffffff
(or delete HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\yourprocess.exe)  (disable pageheap corruption checking)
  • Standard pageheap (little memory overhead)
gflags -p /enable yourprocess.exe
gflags -p /disable yourprocess.exe

How do I get run a command / get detail when a specific managed exception is thrown

Set this if you want to break at 1st chance InvalidOperationException and ‘g’ to continue.’1′ is a pseudo register which will be set when it breaks

!soe -create System.InvalidOperationException 1

Set this if you want to break at 2nd chance InvalidOperationException and =g= to continue.

!soe -create2 System.InvalidOperationException 1

Set this if you want to break at any 1st chance exception derived from Hoge.PageException.

!soe -derive Hoge.PageException 1

Set this if you want to do something when a specific exception is thrown. The example logs and take dumps at SEHException.

sxe -c @".loadby sos clr;gn" ld:C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
sxe -c @"!soe System.Runtime.InteropServices.SEHException 1;.if(@$t1==0) {} .else {.echo 1st chance clr exception with SEHException at: ;.time; .dump /ma /u C:\\temp\\hoge.dmp;}gn;" clr

Note: In WinDbg scripts we can use 20 different user-defined variables called pseudo-registers. Their names are $t0 – $t19. If you want to obtain the pseudo-register value then use @ symbol, for example, @$t0. We can use %p type field character in .printf to interpret the value as a pointer. You can use like this.

r $t1 = 1
r $t2 = 2
r $t0 = @$t1 + @$t2
.printf "%p+%p = %p\n", @$t1, @$t2, @$t0

$t1 will be 1 when it stops on exception by set by !soe -create2 System.InvalidOperationException 1

$t5 will be 1 when it stops on exception by set by !soe -create2 System.InvalidOperationException 5

How do I debug wow64 processes

  • To debug wow64(32bit mode) with 64bit debugger, you can switch the mode as below. Then you’ll be able to do similar debugging to what’s with a 32bit debugger.
.effmach x86
.load wow64exts

How do I force load PDB files

  • Basically, you should store pdb files when the binaries are built. When you rebuild it later and generate the pdb from the same source code, the pdb will have a newer timestamp and they fails to match although the symbols are correct.
  • If you believe that the pdb is correct (just timestamp diff), you can force load by the option SYMOPT_LOAD_ANYTHING. See SYMOPT_LOAD_ANYTHING = 0x40 in debugger help.
0:000> .symopt+0x40
Symbol options are 0x30277:
0x00000002 - SYMOPT_UNDNAME
0x00000010 - SYMOPT_LOAD_LINES
0:000> .sympath+ path-to-the-pdb-dir
0:000> .reload
  • You can check the age/signature as below.
!chksym Foo.Bar.dll C:\temp\foo\bar\hoge.pdb

How do I automatically dump memory from my .Net app

  • Link/deploy your app with the appropriate cpu bitness (32/64) of dbghelp.dll / dbgeng.dll
  • Do something like this.
class DumpDemo
    private string logDir = @"c:\temp";
    private void Hoge()
            // run your code here
        // catch and dump only when you see unexpected exceptions.
        catch (Exception)
            Util.MiniDumpInDirectory(logDir, PInvoke.MINIDUMP_TYPE.MiniDumpWithFullMemory);
public class Util
    public static void MiniDumpInDirectory(String directory, PInvoke.MINIDUMP_TYPE minidumpType)
        FileStream fsToDump = null;
        Process thisProcess = Process.GetCurrentProcess();
        DateTime now = DateTime.Now;
        string dumpFilenameHeader = Path.Combine(directory, thisProcess.ProcessName);
        string fileToDump = string.Format("{0}_{1:X4}-{2:D4}-{3:D2}-{4:D2}_{5:D2}-{6:D2}-{7:D2}-{8:D3}.dmp",
        dumpFilenameHeader, thisProcess.Id,
        now.Year, now.Month, now.Day,
        now.Hour, now.Minute, now.Second, now.Millisecond);
        using (fsToDump = File.Create(fileToDump))
            PInvoke.MiniDumpWriteDump(thisProcess.Handle, thisProcess.Id,
            IntPtr.Zero, IntPtr.Zero, IntPtr.Zero);
public class PInvoke
    public enum MINIDUMP_TYPE
        MiniDumpNormal = 0x00000000,
        MiniDumpWithDataSegs = 0x00000001,
        MiniDumpWithFullMemory = 0x00000002,
        MiniDumpWithHandleData = 0x00000004,
        MiniDumpFilterMemory = 0x00000008,
        MiniDumpScanMemory = 0x00000010,
        MiniDumpWithUnloadedModules = 0x00000020,
        MiniDumpWithIndirectlyReferencedMemory = 0x00000040,
        MiniDumpFilterModulePaths = 0x00000080,
        MiniDumpWithProcessThreadData = 0x00000100,
        MiniDumpWithPrivateReadWriteMemory = 0x00000200,
        MiniDumpWithoutOptionalData = 0x00000400,
        MiniDumpWithFullMemoryInfo = 0x00000800,
        MiniDumpWithThreadInfo = 0x00001000,
        MiniDumpWithCodeSegs = 0x00002000
    public static extern bool MiniDumpWriteDump(
    IntPtr hProcess,
    Int32 ProcessId,
    IntPtr hFile,
    IntPtr ExceptionParam,
    IntPtr UserStreamParam,
    IntPtr CallackParam);
    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool Wow64DisableWow64FsRedirection(ref IntPtr ptr);
    [DllImport("kernel32.dll", SetLastError = true)]
    public static extern bool Wow64RevertWow64FsRedirection(IntPtr ptr);
    public static extern IntPtr GetCurrentProcess();
    [DllImport("kernel32.dll", CharSet = CharSet.Auto)]
    public static extern IntPtr GetModuleHandle(string moduleName);
    [DllImport("kernel32", CharSet = CharSet.Auto, SetLastError = true)]
    public static extern IntPtr GetProcAddress(IntPtr hModule,
    [MarshalAs(UnmanagedType.LPStr)]string procName);
    [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
    [return: MarshalAs(UnmanagedType.Bool)]
    public static extern bool IsWow64Process(IntPtr hProcess, out bool wow64Process);

How do I get method arguments when it’s <no data> (.Net)

How do I get function arguments (Others)

  • In 32bit native apps, kb shows first 3 arguments, and you can find others in the stack.
  • If “!clrstack -p” shows the arguments, it’s the easiest way. However, it doesn’t in many cases because 32/64bit .Net doesn’t keep it in stack. Same for 64bit native apps. Golang uses stack for all function calls.
  • However, sometimes copy of args are stored in stack. Check if it has a copy first. If it doesn’t, you’ll need to get it as soon as the function is called.
  • Here is how to set a breakpoint as soon as the function is called to get the arguments.
  1. attach a debugger, load sos.dll
  2. find the target function address by “!name2ee *!Foo.Bar.Hoge.TargetFunctionName”
  3. set a break point at the address (or use !bpmd -md TargetMd)
  4. continue the process
  5. check registers by ‘r’
  • Windows Calling Conventions:
    • x64 C/C++/.Net: x64 convention (retval=rax, 1st arg=rcx, 2nd=rdx, 3rd=r8, 4th=r9, 5th and later = stack for int, use FP registers for FP)
    • x86 .Net: fastcall convention (similar to x64 convention)
    • x86 C/C++: stdcall convention (retval=eax, use stack for all args)
    • x64 Go: uses stack for both retval and args, but not doing “PUSH”
    • x86 Go: (not tested yet)
  • Here is when Hoge.Program.Func1(1,2,3,4,5,6,7). See rcx=’this’, rdx=1, r8=2, r9=3, and 4/5/6/7 are in the stack (rsp+0x60, 0x68, 0x70, 0x78).
0:000> !name2ee *!Hoge.Program.Func1
Module:      00007fffd82d1000
Assembly:    mscorlib.dll
Module:      00007fff7a1c2fc8
Assembly:    Hoge.exe
Token:       0000000006000009
MethodDesc:  00007fff7a1c4020
Name:        Hoge.Program.Func1(Int32, Int32, Int32, Int32, Int32, Int32, Int32)
JITTED Code Address: 00007fff7a2e0540
0:000> bp 00007fff7a2e0540
0:000> g
Breakpoint 0 hit
00007fff`7a2e0540 44894c2420      mov     dword ptr [rsp+20h],r9d ss:0000002e`fe5ee9b8=03000000
0:000> r
rax=00007fff7a2e0540 rbx=0000002efe5eec88 rcx=0000002e81405a78
rdx=0000000000000001 rsi=0000002efe5eec18 rdi=0000002efe5eedb0
rip=00007fff7a2e0540 rsp=0000002efe5ee998 rbp=0000002efe5eeb00
 r8=0000000000000002  r9=0000000000000003 r10=00007fff7a1c4020
r11=0000002efe5ee8a0 r12=0000002efe5eed28 r13=0000002efe5eede0
r14=0000002efe5eec88 r15=0000002efe5eebb0
iopl=0         nv up ei pl nz na po nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206
00007fff`7a2e0540 44894c2420      mov     dword ptr [rsp+20h],r9d ss:0000002e`fe5ee9b8=03000000

0:000> !U .
Normal JIT generated code
Hoge.Program.Func1(Int32, Int32, Int32, Int32, Int32, Int32, Int32)
Begin 00007fff7a2e0540, size 65
c:\Users\sokoide\Projects\Spike\Hoge\Program.cs @ 41:
00007fff`7a2e0540 44894c2420      mov     dword ptr [rsp+20h],r9d
00007fff`7a2e0545 4489442418      mov     dword ptr [rsp+18h],r8d
00007fff`7a2e054a 89542410        mov     dword ptr [rsp+10h],edx
00007fff`7a2e054e 48894c2408      mov     qword ptr [rsp+8],rcx
00007fff`7a2e0553 4883ec38        sub     rsp,38h
>>> 00007fff`7a2e0557 c744242000000000 mov     dword ptr [rsp+20h],0
00007fff`7a2e055f 48b838341c7aff7f0000 mov rax,7FFF7A1C3438h
00007fff`7a2e0569 8b00            mov     eax,dword ptr [rax]
00007fff`7a2e056b 85c0            test    eax,eax
00007fff`7a2e056d 7405            je      00007fff`7a2e0574
00007fff`7a2e056f e87c34b25f      call    clr!JIT_DbgIsJustMyCode (00007fff`d9e039f0)
00007fff`7a2e0574 90              nop
c:\Users\sokoide\Projects\Spike\Hoge\Program.cs @ 42:
00007fff`7a2e0575 8b4c2450        mov     ecx,dword ptr [rsp+50h]
00007fff`7a2e0579 8b442448        mov     eax,dword ptr [rsp+48h]
00007fff`7a2e057d 03c1            add     eax,ecx
00007fff`7a2e057f 03442458        add     eax,dword ptr [rsp+58h]
00007fff`7a2e0583 03442460        add     eax,dword ptr [rsp+60h]
00007fff`7a2e0587 03442468        add     eax,dword ptr [rsp+68h]
00007fff`7a2e058b 03442470        add     eax,dword ptr [rsp+70h]
00007fff`7a2e058f 03442478        add     eax,dword ptr [rsp+78h]
00007fff`7a2e0593 89442420        mov     dword ptr [rsp+20h],eax
00007fff`7a2e0597 eb00            jmp     00007fff`7a2e0599
c:\Users\sokoide\Projects\Spike\Hoge\Program.cs @ 43:
00007fff`7a2e0599 8b442420        mov     eax,dword ptr [rsp+20h]
00007fff`7a2e059d eb00            jmp     00007fff`7a2e059f
00007fff`7a2e059f 90              nop
00007fff`7a2e05a0 4883c438        add     rsp,38h
00007fff`7a2e05a4 c3              ret
0:000> dd rsp
0000002e`fe5ee960  fe5eec88 0000002e fe5eeb00 0000002e
0000002e`fe5ee970  fe5eed28 0000002e fe5eede0 0000002e
0000002e`fe5ee980  fe5eec88 0000002e fe5eebb0 0000002e
0000002e`fe5ee990  7a1c4020 00007fff 7a2e0512 00007fff
0000002e`fe5ee9a0  81405a78 0000002e 00000001 00000000
0000002e`fe5ee9b0  00000002 00000000 00000003 00000000
0000002e`fe5ee9c0  00000004 0000002e 00000005 0000002e
0000002e`fe5ee9d0  00000006 0000002e 00000007 0000002e


Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.