Windows Debugging Tips
Windows Debugging Tips
How do I get user memory dump
For Crash
- cdb -p $pid
- enter ‘g’ to continue ( go)
- when crashed -> .dump /ma /u c:\temp\foo.dmp
- ‘qd’ to quit from the debugger with detaching
- c:\temp\foo_YYYY-MM-DD_HH-MM-SS-XXX_YYYY.dmp will be created
For Hang
- cdb -pv -p $pid -c “.dump /ma /u c:\temp\foo.dmp; qd”
- wait for 30-60 seconds
- take another memory dump
- wait for 30-60 seconds
- take another memory dump
- repeat until you get 3 dump files
How do I get kernel memory dump
- If the OS is not hanging, you can take kernel dump by lived, or by killing cssrs.exe + Registry to get full dump as below.
- Sometimes the entire server hangs in kernel mode (by Anti Virus etc).
- In that case, set Kernel Full Dump and get the kernel dump by crashing OS.
- If it’s HP machine, disable boot timeout in BIOS (that will prevent taking full dump).
- HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl -> 1
- See http://support.microsoft.com/kb/972110/en-us for more info
- Kill csrss.exe (which will crash the box), or hold down right ctrl + press scroll lock key twice after configuring it in Registry
- http://msdn.microsoft.com/en-us/library/ff545499.aspx
- http://support.microsoft.com/kb/244139
- You’ll get the dump in c:\windows\memory.dmp
How do I debug with memory dump on a different platform
- When you debug using a dump file taken on another box, sos debugging may fail to map the clr structures using mscordackws if the target box’s .Net framework patch level is different.
- Setting sympath by ‘.symfix’ and ‘.reload’ will download necessary files from MSFT.
- But if it doesn’t work, you may need to do this.
- copy sos.dll and mscordacwks.dll from the original machine which generated the dump into your local (i.e. c:\temp)
- open the dump, then “.load c:\temp\sos.dll”
- If it doesn’t work, follow the next steps
0:010> !clrstack ********************************************************************* * Symbols can not be loaded because symbol path is not initialized. * * * * The Symbol Path can be set by: * * using the _NT_SYMBOL_PATH environment variable. * * using the -y <symbol_path> argument when starting the debugger. * * using .sympath and .sympath+ * ********************************************************************* PDB symbol for mscorwks.dll not loaded Failed to load data access DLL, 0x80004005 Verify that 1) you have a recent build of the debugger (6.2.14 or newer) 2) the file mscordacwks.dll that matches your version of mscorwks.dll is in the version directory 3) or, if you are debugging a dump file, verify that the file mscordacwks_<arch>_<arch>_<version>.dll is on your symbol path. 4) you are debugging on the same architecture as the dump file. For example, an IA64 dump file must be debugged on an IA64 machine. You can also run the debugger command .cordll to control the debugger's load of mscordacwks.dll. .cordll -ve -u -l will do a verbose reload. If that succeeds, the SOS command should work on retry.
- get the mscordackws from the box where the dump was taken
- rename it to mscordacwks_<arch>_<arch>_<version>.dll where arch is either AMD64 or x86 and version is the dll’s version
- Sample from a 32bit box: mscordacwks_x86_x86_2.0.50727.3634.dll
- Sample from a 32bit process on a 64bit box: mscordacwks_x86_AMD64_2.0.50727.5681.dll
- Sample from a 64bit process on 64bit box: mscordacwks_AMD64_AMD64_2.0.50727.5681.dll
- copy the dll to your box somewhere ( like to c:\dbgsymbols )
- add it to the path with .exepath+ c:\dbgsymbols
- .reload
- .loadby sos mscorwks (.NET2) or .loadby sos clr (.NET4)
How do I resolve symbols when setting breakpoints
- If you can’t set breakpoint in windbg or visual studio debugger, most likely symbol for the function is not loaded. Add the symbol path as below.
- If you still can’t resolve the symbol in source code debugging in Visual Studio, your code may be newer than the build binaries/pdb. Rebuild / regenerate binaries with pdb and try it again.
- common: open IE and go to any external site to make sure you have external http access. if not, set http proxy.
- windbg:
.symfix .sympath+ path-to-your-pdb .reload
- Visual Studio
1. Debug -> Options and Settings -> Debugging -> Symbols
Add your pdb path
How do I debug Windows service or w3wp process which will crash as soon as it starts
- Open Regedit
- Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\$exename.exe
- Create (REG_SZ) Debugger = full-path-to-cdb -server tcp:port=8090
- Run the target web application to debug
- windbg -remote tcp:server=localhost,port=8090
- cdb will be attached to the remote debugger on the same machine
- ‘g’ to go and take memory dump by “.dump /ma /u c:\temp\foo.dmp”
- ‘q’ to quit the process or kill the target w3wp process and remote cdb
- Delete ‘Debugger’ value under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\$exename.exe
How do I run debugger instead of Dr. Watson when any application crashes
- use aedebug key in Registy. ‘ntsd -iae’ will install ntsd instead of Dr.Watson.
How do I make Dr Watson do full dump instead of mini dump
- When you set AdDebug->Auto=1 & Debuggers = “c:\foo\cdb.exe” -p %ld -c “.dump /ma /u c:\crash.dmp;.kill;qd”, it should make a full dump in c:. It usually works, but it didn’t work well on some environments. In that case, Dr Watson’s full dump feature may be better (not sure).
drwtsn32 –i (this reverted AeDebug -> Debuggers to Dr Watson). Drwtsn32 -> GUI appeared -> change the followings and save. Log file path: d:\logfiles Crash dump: d:\logfiles\drwatson.dmp Choose ‘full’ for crash dump type Left the others default. That changed configs under HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\DrWatson
How do I find loaded debug dlls
load psscor2 or 4 !FindDebugModules -full
How do I get a list of object for a specific type
- Use “dumpheap -stat” to get a list of available types
- Then run “!dumpheap -type $typname” to get a list of the target objects
0:000> !dumpheap -stat Statistics: ... 00007fff7a1c41e8 10 240 Hoge.Bar 00007fff7a1c4100 10 240 Hoge.Foo ... 00007fffd8942d00 26 1456 System.RuntimeType 00007fffd8940bd8 226 9174 System.String 00007fffd8941250 28 36520 System.Object[] 0:003> !dumpheap -type Hoge.Bar Address MT Size 000000a404725190 00007fff7a1c41e8 24 000000a404725280 00007fff7a1c41e8 24 000000a404725370 00007fff7a1c41e8 24 000000a404725460 00007fff7a1c41e8 24 000000a404725550 00007fff7a1c41e8 24 000000a404725640 00007fff7a1c41e8 24 000000a404725730 00007fff7a1c41e8 24 000000a404725820 00007fff7a1c41e8 24 000000a404725910 00007fff7a1c41e8 24 000000a404725a00 00007fff7a1c41e8 24 Statistics: MT Count TotalSize Class Name 00007fff7a1c41e8 10 240 Hoge.Bar Total 10 objects 0:003> !do 000000a404725190 Name: Hoge.Bar MethodTable: 00007fff7a1c41e8 EEClass: 00007fff7a2d2450 Size: 24(0x18) bytes File: C:\Users\sokoide\Projects\Spike\Hoge\bin\Debug\Hoge.exe Fields: MT Field Offset Type VT Attr Value Name 00007fffd8940bd8 4000002 8 System.String 0 instance 000000a4047251e0 <Name>k__BackingField 0:003> !do 000000a4047251e0 Name: System.String MethodTable: 00007fffd8940bd8 EEClass: 00007fffd82d1aa8 Size: 34(0x22) bytes File: C:\WINDOWS\Microsoft.Net\assembly\GAC_64\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll String: bar0 Fields: MT Field Offset Type VT Attr Value Name 00007fffd89437c8 40000ab 8 System.Int32 1 instance 4 m_stringLength 00007fffd8941f38 40000ac c System.Char 1 instance 62 m_firstChar 00007fffd8940bd8 40000ad 18 System.String 0 shared static Empty >> Domain:Value 000000a402b72860:NotInit <<
How do I get opened handles
- Attach a debugger (i.e. windbg -p pid)
- “!handle” to get handle info. To see more in detail, “!handle address f”
- sample output
Handle 634 Type Key Attributes 0 GrantedAccess 0x20019: ReadControl QueryValue,EnumSubKey,Notify HandleCount 2 PointerCount 3 Name \REGISTRY\MACHINE\SOFTWARE\Microsoft\Fusion\PublisherPolicy\Default Object Specific Information Key last write time: 10:52:50. 7/31/2008 Key name Default
- Since handles are Kernel object, you can’t get more info from user mode memory dump. Use Sysinternal’s handle.exe to get more info about the handle, or use Kernel debugger / dump.
How do I find handle leak
- Attach a debugger (i.e. windbg -p pid)
- “!htrace -enable”
- “!htrace -snapshot”
- “g” to resume the process
- (run your app for a while)
- ctrl-break to break the process
- “!htrace -diff” to show the diff of handles
- sample output
0:003> !htrace -diff Handle tracing information snapshot successfully taken. 0x9 new stack traces since the previous snapshot. Ignoring handles that were already closed... Outstanding handles opened since the previous snapshot: -------------------------------------- Handle = 0x00000634 - OPEN Thread ID = 0x00000b90, Process ID = 0x000015f4
- See individual handle info by “!handle address f” as written in the previous section
How do I find native memory leak
set _NT_SYMBOL_PATH=symsrv*symsrv.dll*c:\dbgsymbols*http://msdl.microsoft.com/download/symbols;C:${path-to-your-pdb}
gflags -i w3wp.exe +ust (or gflags -r +ust and reboot for system-wide to create User mode Stack Trace database)
umdh -p:${process-id} -d -f:snap1
run test
umdh -p:${process-id} -d -f:snap2
run test
umdh -p:${process-id} -d -f:snap3
umdh -v -d snap1 snap2 > diff1_2
umdh -v -d snap2 snap3 > diff2_3
check the diff (or by using Excel)
gflags -i w3wp.exe ffffffff
How do I find managed memory leak
- Get 2 memory dumps
- Run these and compare them
- !dumpheap -stat
- !dumpheap -type $type
- !do $obj
How do I find native heap corruption
- Full pageheap (more memory for guard pages)
gflags -i yourprocess.exe +hpa (+hpa means turning on Page Heap flag to enable pageheap corruption checking) run test the process breaks and debugger launches (depends on AeDebug) when the guard page is accessed gflags -i yourprocess.exe ffffffff (or delete HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\yourprocess.exe) (disable pageheap corruption checking)
- Standard pageheap (little memory overhead)
gflags -p /enable yourprocess.exe gflags -p /disable yourprocess.exe
How do I get run a command / get detail when a specific managed exception is thrown
Set this if you want to break at 1st chance InvalidOperationException and ‘g’ to continue.’1′ is a pseudo register which will be set when it breaks
!soe -create System.InvalidOperationException 1
Set this if you want to break at 2nd chance InvalidOperationException and =g= to continue.
!soe -create2 System.InvalidOperationException 1
Set this if you want to break at any 1st chance exception derived from Hoge.PageException.
!soe -derive Hoge.PageException 1
Set this if you want to do something when a specific exception is thrown. The example logs and take dumps at SEHException.
sxe -c @".loadby sos clr;gn" ld:C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
sxe -c @"!soe System.Runtime.InteropServices.SEHException 1;.if(@$t1==0) {} .else {.echo 1st chance clr exception with SEHException at: ;.time; .dump /ma /u C:\\temp\\hoge.dmp;}gn;" clr
Note: In WinDbg scripts we can use 20 different user-defined variables called pseudo-registers. Their names are $t0 – $t19. If you want to obtain the pseudo-register value then use @ symbol, for example, @$t0. We can use %p type field character in .printf to interpret the value as a pointer. You can use like this.
r $t1 = 1 r $t2 = 2 r $t0 = @$t1 + @$t2 .printf "%p+%p = %p\n", @$t1, @$t2, @$t0
$t1 will be 1 when it stops on exception by set by !soe -create2 System.InvalidOperationException 1
$t5 will be 1 when it stops on exception by set by !soe -create2 System.InvalidOperationException 5
How do I debug wow64 processes
- To debug wow64(32bit mode) with 64bit debugger, you can switch the mode as below. Then you’ll be able to do similar debugging to what’s with a 32bit debugger.
.effmach x86 .load wow64exts
How do I force load PDB files
- Basically, you should store pdb files when the binaries are built. When you rebuild it later and generate the pdb from the same source code, the pdb will have a newer timestamp and they fails to match although the symbols are correct.
- If you believe that the pdb is correct (just timestamp diff), you can force load by the option SYMOPT_LOAD_ANYTHING. See SYMOPT_LOAD_ANYTHING = 0x40 in debugger help.
0:000> .symopt+0x40 Symbol options are 0x30277: 0x00000001 - SYMOPT_CASE_INSENSITIVE 0x00000002 - SYMOPT_UNDNAME 0x00000004 - SYMOPT_DEFERRED_LOADS 0x00000010 - SYMOPT_LOAD_LINES 0x00000020 - SYMOPT_OMAP_FIND_NEAREST 0x00000040 - SYMOPT_LOAD_ANYTHING 0x00000200 - SYMOPT_FAIL_CRITICAL_ERRORS 0x00010000 - SYMOPT_AUTO_PUBLICS 0x00020000 - SYMOPT_NO_IMAGE_SEARCH 0:000> .sympath+ path-to-the-pdb-dir 0:000> .reload
- You can check the age/signature as below.
!chksym Foo.Bar.dll C:\temp\foo\bar\hoge.pdb
How do I automatically dump memory from my .Net app
- Link/deploy your app with the appropriate cpu bitness (32/64) of dbghelp.dll / dbgeng.dll
- Do something like this.
class DumpDemo
{
private string logDir = @"c:\temp";
private void Hoge()
{
try
{
// run your code here
}
// catch and dump only when you see unexpected exceptions.
catch (Exception)
{
Util.MiniDumpInDirectory(logDir, PInvoke.MINIDUMP_TYPE.MiniDumpWithFullMemory);
throw;
}
}
}
public class Util
{
public static void MiniDumpInDirectory(String directory, PInvoke.MINIDUMP_TYPE minidumpType)
{
FileStream fsToDump = null;
Process thisProcess = Process.GetCurrentProcess();
DateTime now = DateTime.Now;
string dumpFilenameHeader = Path.Combine(directory, thisProcess.ProcessName);
string fileToDump = string.Format("{0}_{1:X4}-{2:D4}-{3:D2}-{4:D2}_{5:D2}-{6:D2}-{7:D2}-{8:D3}.dmp",
dumpFilenameHeader, thisProcess.Id,
now.Year, now.Month, now.Day,
now.Hour, now.Minute, now.Second, now.Millisecond);
using (fsToDump = File.Create(fileToDump))
{
PInvoke.MiniDumpWriteDump(thisProcess.Handle, thisProcess.Id,
fsToDump.SafeFileHandle.DangerousGetHandle(),
PInvoke.MINIDUMP_TYPE.MiniDumpWithFullMemory,
IntPtr.Zero, IntPtr.Zero, IntPtr.Zero);
fsToDump.Close();
}
}
}
public class PInvoke
{
public enum MINIDUMP_TYPE
{
MiniDumpNormal = 0x00000000,
MiniDumpWithDataSegs = 0x00000001,
MiniDumpWithFullMemory = 0x00000002,
MiniDumpWithHandleData = 0x00000004,
MiniDumpFilterMemory = 0x00000008,
MiniDumpScanMemory = 0x00000010,
MiniDumpWithUnloadedModules = 0x00000020,
MiniDumpWithIndirectlyReferencedMemory = 0x00000040,
MiniDumpFilterModulePaths = 0x00000080,
MiniDumpWithProcessThreadData = 0x00000100,
MiniDumpWithPrivateReadWriteMemory = 0x00000200,
MiniDumpWithoutOptionalData = 0x00000400,
MiniDumpWithFullMemoryInfo = 0x00000800,
MiniDumpWithThreadInfo = 0x00001000,
MiniDumpWithCodeSegs = 0x00002000
}
[DllImport("dbghelp.dll")]
public static extern bool MiniDumpWriteDump(
IntPtr hProcess,
Int32 ProcessId,
IntPtr hFile,
MINIDUMP_TYPE DumpType,
IntPtr ExceptionParam,
IntPtr UserStreamParam,
IntPtr CallackParam);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool Wow64DisableWow64FsRedirection(ref IntPtr ptr);
[DllImport("kernel32.dll", SetLastError = true)]
public static extern bool Wow64RevertWow64FsRedirection(IntPtr ptr);
[DllImport("kernel32.dll")]
public static extern IntPtr GetCurrentProcess();
[DllImport("kernel32.dll", CharSet = CharSet.Auto)]
public static extern IntPtr GetModuleHandle(string moduleName);
[DllImport("kernel32", CharSet = CharSet.Auto, SetLastError = true)]
public static extern IntPtr GetProcAddress(IntPtr hModule,
[MarshalAs(UnmanagedType.LPStr)]string procName);
[DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool IsWow64Process(IntPtr hProcess, out bool wow64Process);
}
How do I get method arguments when it’s <no data> (.Net)
How do I get function arguments (Others)
- In 32bit native apps, kb shows first 3 arguments, and you can find others in the stack.
- If “!clrstack -p” shows the arguments, it’s the easiest way. However, it doesn’t in many cases because 32/64bit .Net doesn’t keep it in stack. Same for 64bit native apps. Golang uses stack for all function calls.
- However, sometimes copy of args are stored in stack. Check if it has a copy first. If it doesn’t, you’ll need to get it as soon as the function is called.
- Here is how to set a breakpoint as soon as the function is called to get the arguments.
- attach a debugger, load sos.dll
- find the target function address by “!name2ee *!Foo.Bar.Hoge.TargetFunctionName”
- set a break point at the address (or use !bpmd -md TargetMd)
- continue the process
- check registers by ‘r’
- Windows Calling Conventions:
- x64 C/C++/.Net: x64 convention (retval=rax, 1st arg=rcx, 2nd=rdx, 3rd=r8, 4th=r9, 5th and later = stack for int, use FP registers for FP)
- x86 .Net: fastcall convention (similar to x64 convention)
- x86 C/C++: stdcall convention (retval=eax, use stack for all args)
- x64 Go: uses stack for both retval and args, but not doing “PUSH”
- x86 Go: (not tested yet)
- Here is when Hoge.Program.Func1(1,2,3,4,5,6,7). See rcx=’this’, rdx=1, r8=2, r9=3, and 4/5/6/7 are in the stack (rsp+0x60, 0x68, 0x70, 0x78).
0:000> !name2ee *!Hoge.Program.Func1 Module: 00007fffd82d1000 Assembly: mscorlib.dll -------------------------------------- Module: 00007fff7a1c2fc8 Assembly: Hoge.exe Token: 0000000006000009 MethodDesc: 00007fff7a1c4020 Name: Hoge.Program.Func1(Int32, Int32, Int32, Int32, Int32, Int32, Int32) JITTED Code Address: 00007fff7a2e0540 0:000> bp 00007fff7a2e0540 0:000> g Breakpoint 0 hit 00007fff`7a2e0540 44894c2420 mov dword ptr [rsp+20h],r9d ss:0000002e`fe5ee9b8=03000000 0:000> r rax=00007fff7a2e0540 rbx=0000002efe5eec88 rcx=0000002e81405a78 rdx=0000000000000001 rsi=0000002efe5eec18 rdi=0000002efe5eedb0 rip=00007fff7a2e0540 rsp=0000002efe5ee998 rbp=0000002efe5eeb00 r8=0000000000000002 r9=0000000000000003 r10=00007fff7a1c4020 r11=0000002efe5ee8a0 r12=0000002efe5eed28 r13=0000002efe5eede0 r14=0000002efe5eec88 r15=0000002efe5eebb0 iopl=0 nv up ei pl nz na po nc cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206 00007fff`7a2e0540 44894c2420 mov dword ptr [rsp+20h],r9d ss:0000002e`fe5ee9b8=03000000 0:000> !U . Normal JIT generated code Hoge.Program.Func1(Int32, Int32, Int32, Int32, Int32, Int32, Int32) Begin 00007fff7a2e0540, size 65 c:\Users\sokoide\Projects\Spike\Hoge\Program.cs @ 41: 00007fff`7a2e0540 44894c2420 mov dword ptr [rsp+20h],r9d 00007fff`7a2e0545 4489442418 mov dword ptr [rsp+18h],r8d 00007fff`7a2e054a 89542410 mov dword ptr [rsp+10h],edx 00007fff`7a2e054e 48894c2408 mov qword ptr [rsp+8],rcx 00007fff`7a2e0553 4883ec38 sub rsp,38h >>> 00007fff`7a2e0557 c744242000000000 mov dword ptr [rsp+20h],0 00007fff`7a2e055f 48b838341c7aff7f0000 mov rax,7FFF7A1C3438h 00007fff`7a2e0569 8b00 mov eax,dword ptr [rax] 00007fff`7a2e056b 85c0 test eax,eax 00007fff`7a2e056d 7405 je 00007fff`7a2e0574 00007fff`7a2e056f e87c34b25f call clr!JIT_DbgIsJustMyCode (00007fff`d9e039f0) 00007fff`7a2e0574 90 nop c:\Users\sokoide\Projects\Spike\Hoge\Program.cs @ 42: 00007fff`7a2e0575 8b4c2450 mov ecx,dword ptr [rsp+50h] 00007fff`7a2e0579 8b442448 mov eax,dword ptr [rsp+48h] 00007fff`7a2e057d 03c1 add eax,ecx 00007fff`7a2e057f 03442458 add eax,dword ptr [rsp+58h] 00007fff`7a2e0583 03442460 add eax,dword ptr [rsp+60h] 00007fff`7a2e0587 03442468 add eax,dword ptr [rsp+68h] 00007fff`7a2e058b 03442470 add eax,dword ptr [rsp+70h] 00007fff`7a2e058f 03442478 add eax,dword ptr [rsp+78h] 00007fff`7a2e0593 89442420 mov dword ptr [rsp+20h],eax 00007fff`7a2e0597 eb00 jmp 00007fff`7a2e0599 c:\Users\sokoide\Projects\Spike\Hoge\Program.cs @ 43: 00007fff`7a2e0599 8b442420 mov eax,dword ptr [rsp+20h] 00007fff`7a2e059d eb00 jmp 00007fff`7a2e059f 00007fff`7a2e059f 90 nop 00007fff`7a2e05a0 4883c438 add rsp,38h 00007fff`7a2e05a4 c3 ret 0:000> dd rsp 0000002e`fe5ee960 fe5eec88 0000002e fe5eeb00 0000002e 0000002e`fe5ee970 fe5eed28 0000002e fe5eede0 0000002e 0000002e`fe5ee980 fe5eec88 0000002e fe5eebb0 0000002e 0000002e`fe5ee990 7a1c4020 00007fff 7a2e0512 00007fff 0000002e`fe5ee9a0 81405a78 0000002e 00000001 00000000 0000002e`fe5ee9b0 00000002 00000000 00000003 00000000 0000002e`fe5ee9c0 00000004 0000002e 00000005 0000002e 0000002e`fe5ee9d0 00000006 0000002e 00000007 0000002e
[EOD]