Re: [RFA]: Add misc_obstack to object files.

[Daniel Berlin <dan at cgsoftware dot com>]

Elena Zannoni <ezannoni@cygnus.com> writes:

> Daniel Berlin writes:
> > Elena Zannoni <ezannoni@cygnus.com> writes:
> > 
> > > Daniel Berlin writes:
> > > > Elena Zannoni <ezannoni@cygnus.com> writes:
> > > > 
> > > > > Daniel Berlin writes:
> > > > > > 
> > > > > > This way, we stop putting things in the psymbol_obstack that don't
> > > > > > belong.  
> > > > > > 
> > > > > 
> > > > > Dan, can you elaborate a bit on why we need this?
> > > > Because these things don't belong in the psymbol obstack.
> > > > They are section offsets, not partial symbols.
> > > > 
> > > 
> > > Dan, I am really sorry, but still I cannot see a real reason for this
> > > change emerging from your mail. 
> > 
> > Um.
> > Okay.
> > According to objfiles.h:
> > 
> >     struct obstack psymbol_obstack;     /* Partial symbols */
> >     struct obstack symbol_obstack;      /* Full symbols */
> >     struct obstack type_obstack;        /* Types */
> > 
> > 
> > Notice it doesn't say that section offsets, or other random junk,
> > should be on the psymbol_obstack.
> > It says "Partial symbols".
> > Now, if you want to remove the partial symbols, for that object file,
> > theoretically, you just remove the psymbol_obstack.  However, because
> > we allocate other random things, like section offset tables, etc,
> > which have zip to do with partial symbols, on that obstack, we won't
> > remove just the partial symbols by removing that obstack.  We'll blow
> > away the section offset table, the psymtab structures, etc.
> > This should not be occurring.
> > It makes as much sense as allocating object file section offsets on the
> > type_obstack, which is for types only. That is to say, none.
> > 
> 
> Yes, and this is the bug I was alluding to. You mentioned you observed
> that things like the section tables were freed too early. When does
> this occur?

When you attempt to free the psymbol obstack after you are done  with
all the psymbols for an objfile.  However, gdb currently doesn't do
this. This is part of the reason you can't.

>  Did you try to move around the call to
> free_obstack(psymbol_obstack)?

There are only calls to do this type of thing when we remove object
files or reread symbols.

It would, of course, to make sense to free the partial symbols for an
objfile once we were sure all the symtabs for that objfile had been
read in.
This does not currently occur.  The bug occurs when you attempt to do
this.
The actual reason it never occurs in gdb is because the stabs reader
incorrectly expects partial symbols 
to live past the readin of full symbols (why would you want partial
symbols when you have full ones?), in order to do some duplicate
elimination the cheap, easy, and stabs specific, way.
It does this at the expense of every other symbol reader, since if you
attempt to free the psymbol obstack for stabs, even after you are done
with all the partial symbols for a given objfile, you'll start seeing
weird stabs reader errors.
However, this I consider a seperate problem, since gdb currently
doesn't ever free the psymbol obstack except when it frees objfiles.

>  What motivated you to try and do that?
The fact that we waste effectively double the memory to read symbols
for everything except stabs, because none of the other symbol readers
assume the psymtabs will live past complete symtab conversion, and
thus, assume they will be freed.   The STABS reader happens to
eliminate enough psymbols through it's cheap easy method that you
don't notice the fact that it's keeping all the psymbols around, never
freeing them.

Besides attacking this part of problem by fixing the stabs reader, 
you must also make it so you can actually free the psymbol obstack when you
are done with all the psymtabs for an objfile, without affecting
anything else about that objfile.  I.E. make it so the psymbol_obstack
only contains psymbols.
Otherwise, fixing the stabs reader does you no good, since you still
will blow away unrelated things when trying to free the psymbols.

So in order to fix the problem of keeping psymbols around until
their objfile goes away, rather than until the symtabs for that
objfile are all read in (Or better yet, until a given psymab is
converted into a symtab), I need to start with the small things like
moving the structures out of the psymbol_obstack.
> That's really what I am asking.
> 
> > It's the equivalent of having a trash bin marked "white paper only",
> > and having people knowing throw all kinds of colored paper in there,
> > for lack of a better place to throw colored paper.
> > They don't belong.
> > Not only that, but if you later wanted to remove the white paper, you'd have to sort out
> > the colored paper, and do something with it, first.
> > 
> > I'm just adding a bin for colored paper, so people will put that stuff
> > in there instead in the first place, and it will be plainly clear that they are supposed
> > to be doing so.
> > 
> > 
> > > You are doing this in response to some bug you saw, I assume, but what that
> > > bug is has not been explained.
> > No, i'm doing it because partial symbols belong on the psymbol
> > obstack, but section offsets, and other random things don't.
> > In practice, we've not followed this, because nobody bothered to
> > create somewhere *else* to put them. I'm attempting to bring us
> > to doing what we should have been doing in the first place, by
> > creating this place to put them.
> > 
> 
> Yes, true.
> 
> > Had I been trying to fix a bug, i would explain what the bug was, and
> > where it showed up.
> > 
> > This is not a bug, in that nothing is specifically *broken*. It's
> > perfectly okay to allocate these structures on that obstack, given
> > that gdb never frees the obstack, it's just not in line with what the
> > comments say, and the purpose of the psymbol obstack.
> > 
> 
> Isn't the psymbol_obstack freed when you re-read symbols in? 
Yes.
And when you free objfiles.
However, this is not really it's minimal lifetime.
The usefulness of a psymbol_obstack for an objfile ends the second
you've "converted" (i put it in quotes since conversion usually
involves transforming one thing into a nother, where here we are
really effectively copying, and transforming the copy, since we never
remove the original) all the psymtabs into symtabs.

> 
> > > 
> > > 
> > > > > Why are you removing
> > > > > the partial symbol table from the psymbol_objstack?
> > > > I'm not. At all in fact.
> > > > I'm specifically trying to make it so *all* that is in the
> > > > psymbol_obstack is the partial symbol table.
> > > > we've been stuffing random stuff like section offsets into the
> > > > psymbol_obstack, because it was handy.
> > > > This is wrong.
> > > > If we need a misc place to put info like section offsets, that isn't
> > > > large enough to deserve it's own obstack on it's own, then all the
> > > > misc stuff together should go in a misc_obstack.
> > > 
> > > Then your patch includes a diff that is unrelated.
> > Um, how so?
> > Please point out the portion of the diff unrelated.
> > The only thing even related to psymtabs there is the fact that they
> > psymtab structure itself is allocated on the misc obstack, since it
> > isn't a partial symbol either. It's a partial symbol table structure,
> > containing pointers to partial symbols. 
> 
> Yes, that's the one.
> 
> I would think that the psymtab itself is better coupled with
> psymbol_objstack, than misc_obstack.  

Okay, that's fine.


> I would think it safer to free both of them or none, than to risk
> freeing the section_offsets and also the psymbol table by mistake.

Sure.
> 
> > The actual partial symbols that fill that table are still allocated on
> > the psymbol_obstack.
> > As they should be.
> 
> yes, but looking at the way the symbol table is allocated, it follows
> the same logic, symbol_obstack includes the symtab itself as well.

Real symbols are different. Their lifetime really ends with the
objfile. Partial symbols are just there until we have a need to read
all the real symbols. So their lifetime is different.

> 
> I mean, the real problem that you are trying to address is that you
> need a different lifetime for psymbols vs psymtab, and for psymbols vs
> section_offsets. So, why is this necessary? 

I think i've explained this above, but in different parts of the
message so, for clarity's sake, this particular thing is necessary because:
The lifetime of psymbols for an objfile is until all the symtabs for
that objfile are read in (Theoretically, if you were careful, you
could make the lifetime until those psymbols are converted to symbols,
but it's tricky, because of how obstacks work. The whole freeing
everything that came after the object you want to free stuff). Once
this is done, the psymbols serve 0 purpose. Provably, no lookup that
fails on the symbols could work on the psymbols, unless your reader is
broken (since they are coming from the same info). 
> Are you addressing a
> limitation that you have found while trying to do something else?
Yes.
This is the first step of addressing the limitation that we keep
psymbols around longer than necessary.  This becomes a factor for
STABS when you have very large programs (IE duplicate symbol
elimination doesn't save you, because you really have 500k distinct
symbols and types), and for other debug formats which don't have
linker support for duplicate elimination, much quicker.

Even with a gdb compiled with dwarf2, where gdb isn't that large of a
program in terms of psymbols (because c has few duplicates compare to
C++, and because C++ programs have often have more of the things that
we generate psymbols for) shows the following, after all symtabs are read in (IE when
psymbols are, bye, definition, useless)

(top-gdb) maintenance print statistics
Statistics for '/buildspace2/dwarf2gdbtree/src/gdb/./gdbdwarf2':
  Number of "minimal" symbols read: 7149
  Number of "partial" symbols read: 289144
  Number of "full" symbols read: 445652
  Number of "types" defined: 162155
  Total memory used for psymbol obstack: 12389013
  Total memory used for psymbol cache: 745176
  Total memory used for symbol obstack: 35083179
  Total memory used for type obstack: 28099722

Out of 75 meg of memory, we are wasting 1/6th on psymbols.
This is probably average for C programs.

For C++ programs, you tend to see a psymbol obstack equal in size to
the symbol obstack.
In other words, we waste double the memory on symbols.

> Why
> is it a problem that section_offsets, psymbols and psymtab have the
> same lifetime?
section_offsets logically have the lifetime of the objfile.
psymbols and the associated psymtabs have a lifetime that lasts
exactly until the full symbols for those psymbols are read in (IE the
psymtab is converted to a symtab).
Right now, the we treat them like they last until the objfile goes
away.
I'm working on treating them like they last until all the symbols for
a given objfile are read in (which happens fairly quickly unless you
have a huge, monolithic program).
This is because treating them like they have the lifetime they really
do would require being able to free all the psymbols for a given
psymtab without touching any other psymbols.  Because of how obstacks
work, doing this would require not using obstacks for psymbols, or
rather, making the obstack part of the psymtab (so you can free all
the memory associated with the psymtab without touching anything else)
Eventually, that is the goal, but that's way off.
Since it is a complex change to do that anyway, and it's more of an 
incremental step from being able to free psymbols at *some particular* point
before the objfile goes away, this actually gets us closer to being
able to do that.

> [I am not arguing that section offsets are logically related to psymbols]
> 
> > > 
> > > > 
> > > > >  Was this intended?
> > > > > There are several other things that get allocated on the
> > > > > psymbol_objstack, for instance look in xcoffread.c, somread.c,
> > > > > etc. Why just remove the section offset stuff?
> > > > Because I haven't gotten to the others yet.
> > > > And there is no harm in taking time to make sure it gets done right.
> > > > 
> > > 
> > > If you have a roadmap in mind, you should explain it when you submit
> > > the patch.
> > 
> > I have no roadmap for when i'd be getting to that.
> > Eventually.
> > I could do them all now if it helps, but it'll make the patch bigger.
> > 
> 
> I would suggest that you put the section offsets stuff on its own
> separate obstack for now, but do that across the board, to avoid
> inconsistencies. That would be a self contained patch.
> Follow up patches would move other pieces.
Okeydokey.

> 
> Elena


-- 
"When I was a kid, I went to the store and asked the guy, "Do you
have any toy train schedules?"
"-Steven Wright