[rfa] i386-tdep.c: add some insn's to prologue analyzer

[Michael Elizabeth Chastain <mec at shout dot net>]

This patch improves the i386 prologue analyzer so that it can
handle a few more instructions in the prologue.  This fixes
pr gdb/1253 and pr gdb/1255.

When gdb backtraces through a function without debug info,
it needs to analyzes the function prologue to figure out what is
happening with the frame pointer.  If the analysis fails, then gdb
assumes the function is frameless.  This is not always the case.

Specifically, it's common for some threads in a multi-threaded program
to be inside 'sleep'.  The test suite depends on backtracing through
'sleep' and I think that real users with real programs are likely to
have programs like this too.

My machine is native i686-pc-linux-gnu, red hat 8.0,
glibc 2.2.93-5-rh.  The prologue for 'sleep' on my machine is:

  push   %ebp
  xor    %ecx, %ecx
  mov    %esp, %ebp
  push   %edi
  xor    %edx, %edx
  lea    0xfffffe50(%ebp),%edi
  push   %esi
  push   %ebx
  call   __i686.get_pc_thunk.bx
  add    $0x7bfab, %ebx
  sub    $0x1cc, %esp

Fortunately, gdb only needs to read as far as the 'mov %esp, %ebp'
to understand that the function is frame-ful.

This patch allows the following instructions to occur between
'push %ebp' and 'mov %esp, %ebp':

  xor %ebx, %ebx
  xor %ecx, %ecx
  xor %edx, %edx
  sub %ebx, %ebx
  sub %ecx, %ecx
  sub %edx, %edx

On the hand, this approach is incomplete.  Check out the cool
machinery in s390-tdep.c.  This would be really great to have for
every tdep.

On the other hand, this approach is safe.  It can't make any situation
worse and it improves a very common situation.  And the code is
completely confined to i386_analyze_frame_setup.

Testing: I tested this with the executables from pr gdb/1253 and pr
gdb/1255.  It works with both executables (dwarf-2 and stabs+).
I am currently running my test bed with all the gcc's that I have
with both dwarf-2 and stabs+ (only one binutils, though -- binutils 2.14).

I would like to commit this to HEAD.  Then let it mellow for a few days
and then commit to gdb_6_0-branch as well.

If this passes my test bed, may I have approval to commit this to
gdb HEAD?

Michael C

2003-08-04  Michael Chastain  <mec@shout.net>

	* i386-tdep.c (i386_analyze_frame_setup): Recognize more
	instructions that gcc likes to mingle into the prologue.

Index: i386-tdep.c
===================================================================
RCS file: /cvs/src/src/gdb/i386-tdep.c,v
retrieving revision 1.162
diff -c -3 -p -r1.162 i386-tdep.c
*** i386-tdep.c	18 Jul 2003 21:31:50 -0000	1.162
--- i386-tdep.c	5 Aug 2003 01:15:05 -0000
*************** i386_analyze_frame_setup (CORE_ADDR pc, 
*** 439,444 ****
--- 439,446 ----
  			  struct i386_frame_cache *cache)
  {
    unsigned char op;
+   unsigned char op2;
+   int skip_reg_inst;
  
    if (current_pc <= pc)
      return current_pc;
*************** i386_analyze_frame_setup (CORE_ADDR pc, 
*** 455,460 ****
--- 457,498 ----
        /* If that's all, return now.  */
        if (current_pc <= pc + 1)
  	return current_pc;
+ 
+       /* Check for register instructions that have migrated into
+          the prologue.  This happens with glibc 'sleep'.
+ 	 See pr gdb/1255.  -- chastain 2003-08-04 */
+       do 
+       {
+ 	skip_reg_inst = 0;
+ 	op = read_memory_unsigned_integer (pc + 1, 1);
+ 	switch (op)
+ 	  {
+ 	    case 0x31:
+ 	      op2 = read_memory_unsigned_integer (pc + 2, 1);
+ 	      switch (op2)
+ 		{
+ 		  case 0xdb:  /* xor %ebx, %ebx */
+ 		  case 0xc9:  /* xor %ecx, %ecx */
+ 		  case 0xd2:  /* xor %edx, %edx */
+ 		    pc += 2;
+ 		    if (current_pc <= pc + 1)
+ 		      return current_pc;
+ 		    skip_reg_inst = 1;
+ 		}
+ 	    case 0x29:
+ 	      op2 = read_memory_unsigned_integer (pc + 2, 1);
+ 	      switch (op2)
+ 		{
+ 		  case 0xdb:  /* sub %ebx, %ebx */
+ 		  case 0xc9:  /* sub %ecx, %ecx */
+ 		  case 0xd2:  /* sub %edx, %edx */
+ 		    pc += 2;
+ 		    if (current_pc <= pc + 1)
+ 		      return current_pc;
+ 		    skip_reg_inst = 1;
+ 		}
+ 	  }
+       } while (skip_reg_inst);
  
        /* Check for `movl %esp, %ebp' -- can be written in two ways.  */
        op = read_memory_unsigned_integer (pc + 1, 1);