[RFC] Enhance backtrace for microsoft system DLL calls

["Pierre Muller" <muller at ics dot u-strasbg dot fr>]

  I had troubles when trying to backtrace
when the debugge was stopped 
inside the windows DLL's.

  After some investigation, I 
found out that many exported routines
from the Microsoft operating system
start with a no-op assembler instruction
'mov %edi,%edi'.
  The presence of this instruction
leads to a complete failure of the 
function prologue analysis,
leading in turn to a bad backtrace.

  I am a small script that would look for all
the exported functions that start with 'mov %edi, %edi'
and this is what I got:
ntdll.dll: 532 functions out of 1311 start with that pattern
kernel32.dll: 721 functions out of 947.
gdi32.dll: 439 functions out of 609. 
shell32.dll: 268 functions out of 309.
user32.dll: 531 functions out of 732.
(on a Windows XP operating system).

   The patch simply discard a leading 'mov %edi,%edi'
instruction in the i386_analyze_frame_setup function.

  The main question is whether this patch is acceptable
for gdb as it is in a i386 common file, while 
it most probably only applies to MS operating system.

  The problem is that I found no other location where 
this could be done, but maybe someone in the list
has a better overview and a good idea where to put that.

  Tested on cygwin with no regressions.
If you put a breakpoint on ZwSuspendThread (in ntdll.dll)
you will get this with current CVS head:

Breakpoint 4, 0x7c90e84f in ntdll!ZwSuspendThread ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
(top-gdb) bt
#0  0x7c90e84f in ntdll!ZwSuspendThread ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#1  0x7c839744 in SuspendThread ()
   from /cygdrive/c/WINDOWS/system32/kernel32.dll
#2  0x00000680 in ?? ()
#3  0x0022c550 in ?? ()
#4  0x0022c568 in ?? ()
During symbol reading, struct/union type gets multiply defined: struct
language_
defn.
#5  0x0046b34a in thread_rec (id=1664, get_context=4374409)
    at ../../purecvs/gdb/win32-nat.c:265
Backtrace stopped: frame did not save the PC

With the attached patch, you get that:
Breakpoint 4, 0x7c90e84f in ntdll!ZwSuspendThread ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
(top-gdb) bt
#0  0x7c90e84f in ntdll!ZwSuspendThread ()
   from /cygdrive/c/WINDOWS/system32/ntdll.dll
#1  0x7c839744 in SuspendThread ()
   from /cygdrive/c/WINDOWS/system32/kernel32.dll
During symbol reading, struct/union type gets multiply defined: struct
language_
defn.
#2  0x0046b34a in thread_rec (id=3820, get_context=1)
    at ../../purecvs/gdb/win32-nat.c:265
#3  0x0046b815 in win32_fetch_inferior_registers (regcache=0x1063d9a8, r=8)
    at ../../purecvs/gdb/win32-nat.c:422



Pierre Muller

ChangeLog entry:

2007-12-10  Pierre Muller  <muller@ics.u-strasbg.fr>

	* i386-tdep.c (i386_analyze_frame_setup): Ignore `mov %edi,%edi'
instruction
	used at entry of some operating system calls.

	


Index: gdb/i386-tdep.c
===================================================================
RCS file: /cvs/src/src/gdb/i386-tdep.c,v
retrieving revision 1.246
diff -u -p -r1.246 i386-tdep.c
--- gdb/i386-tdep.c     6 Dec 2007 16:32:59 -0000       1.246
+++ gdb/i386-tdep.c     10 Dec 2007 16:22:21 -0000
@@ -650,6 +650,17 @@ i386_analyze_frame_setup (CORE_ADDR pc,

   read_memory_nobpt (pc, &op, 1);

+  if (op == 0x8b) /* Ignore no-op instruction `mov %edi, %edi' */
+    {
+      read_memory_nobpt (pc + 1, &op, 1);
+      if (op == 0xff)
+       {
+         pc += 2;
+         read_memory_nobpt (pc, &op, 1);
+       }
+      else
+       op = 0x8b;
+    }
   if (op == 0x55)              /* pushl %ebp */
     {
       /* Take into account that we've executed the `pushl %ebp' that