The gdb x86 function prologue parser

[Jason Molenda <jmolenda at apple dot com>]

Hello, I'm Jason Molenda and I work on gdb for Apple.  :-)

As was announced yesterday, we'll be transitioning from PPC to x86  
over the next couple of years.  Over the last few months I had a  
delightful little introduction to the x86 architecture (read: read  
the IA32 PDFs until I couldn't see straight any more) and worked to  
get our x86 gdb support up to snuff for this software release.

One of the biggest problems we found was the x86 function prologue  
parser is remarkably weak.  We have a very mature and featureful  
prologue parser on our ppc side and an amazing number of bugs were  
directed my way as we had people pounding on the x86 side.  We aren't  
using DWARF yet, so CFI can't save our bacon -- the prologue parser  
has to work or our gdb fails.

There are a couple classes of changes I made, and I spent today  
trying to massage them into some kind of presentable form.  This is  
not perfect -- well, to be honest, this is just a first sketch -- but  
this is a HUGE improvement over the existing facilities.  I wrote the  
code while under immense deadline pressure so I'm not particularly  
interested in how I implemented any of it.  But changes akin to these  
are necessary if you want to debug programs with optimized code on  
the stack and without CFI.  I'll guarantee it.

Roughly speaking, here are the class of changes included in this patch.

1. i386_match_insn() bug fixes.  It wouldn't work for an instruction  
pattern of 1 byte, and it would never check anything beyond the 2nd  
byte (notice where the final "return insn" is located).  I've added  
patterns that can match prologue instructions, so exceptions had to  
be added for the big two (push %ebp, mov %esp, %ebp) and the  
equivalent ones used in the first frame (_start()).

2. i386_frame_setup_skip_insns table expansion.  Because you can't  
skip over an unknown instruction on x86 without knowing its length,  
this was of paramount importance.  Initially I waited for users to  
tell me of prologues that gdb was failing on, but this was taking too  
long and there were too many instructions scheduled into prologues  
for me to hear of them in time.  So I wrote a little maintenance  
command (not included in the patch to keep things simple) which would  
tell you if gdb could parse through the prologue of a given  
function.  Then with a couple of shell scripts, I could have gdb try  
to analyze the prologues of every function in every library on my  
MacOS X system and show me the ones it failed on.  I'd add them to  
this list.  I also made a little testsuite generator where the input  
looks like

# SOURCE: RedHat FedoraCore2 /lib/ld-2.3.3.so _dl_reloc_bad_type
# PROLOGUE
  push %ebp
  shl $0x5, %ecx # [ 0xc1 0xe1 0x05 ]
  mov %esp, %ebp
  sub $0x8, %esp
# EPILOGUE
  add $0x8, %esp
  pop %ebp
  ret

and a script that transforms the patterns into a test program and a  
Dejagnu expect script.  So you can ensure that you don't regress the  
prologue parser.  This was the lesson we learned in writing our PPC  
parser -- we have this wonderfully ornate parser with lots of  
exceptions and known tricks, but no testsuite for it.  So whenever we  
change it we're cringing because the gdb testsuite has nothing useful  
in it.  (you need optimized, no debug info test cases to be sure it's  
still working right).  The testsuite stuff isn't included in this  
patch, but I'll put that together soon and send it along if anyone's  
interested.

3. relatively minor changes to i386_analyze_frame_setup().  It had to  
have the push %ebp as the very first instruction or it would give  
up.  That's really bad -- the compiler can (and does) schedule all  
sorts of stuff before that instruction.

4. new function, i386_find_esp_adjustments().  This is used in a  
frameless leaf function where the compiler may create space on the  
stack for local variables and stuff, but doesn't call anything so it  
doesn't save the caller's frame pointer.  And it allows -fomit-leaf- 
frame-pointer codegen to be debugged.  -fomit-frame-pointer is a  
whole lot more complicated, but this wasn't so bad.  (we didn't end  
up enabling -fomit-leaf-frame-pointer in this release because of the  
schedule time constraints, but that's why I wrote it)

5. Huge i386_frame_cache() changes.  There's no way around it, this  
function is just not right.  It doesn't handle frameless functions  
correctly at all.  It's written without a clear understanding of the  
different classes of functions it needs to handle and works primarily  
by luck.  And for goodness sakes, if we can't figure out anything  
about a function that's not at the top of the stack, don't you think  
it'd be reasonable to assume that the function has set up a stack  
frame and saved the caller's EBP?  Sure seems like a reasonable  
assumption to me.  Why can't this function do something even that  
basic?  This function really cheesed my mellow.

Mark, I want to say that I'm not directing any of these criticisms  
towards you -- I've been looking over the changes you've made and  
they're definite improvements over the existing code.  The existing  
code bites, though.  I can't even begin to imagine how annoyed  
developers using the FSF gdb on x86 must be.  The changes I'm sending  
here are not a panacea/beautiful/perfect, but *functionally* they're  
a huge improvement.  Now that our release is out there I'll be more  
than happy to revisit the decisions/implementation that I came up  
with on little sleep. :-)


Oh, and I ran my "find all prologues gdb can't parse" on a FedoraCore  
2 system I have handy here at the office today and added the patterns  
of the biggest offenders.  There are still a few patterns I need to  
add to get 100% parsing success but I want to go home :-) so that'll  
be for another day.

We're in the middle of an all-week Apple developer's conference, so  
my replies may not be very speedy (I'm off-line 10-12 hours a day; I  
slipped away to get this patch together today) until the weekend.   
But I'll try to stay on top of any questions or comments and address  
them promptly.

I'm looking forward to making x86 gdb excellent.  There's definitely  
more work to do and I intend to be a part of that.

Jason Molenda
Apple Computer

Attachment: patch.txt
Description: Text document