Re: Towards multiprocess GDB

[Michael Snyder <msnyder at specifix dot com>]

On Fri, 2008-07-18 at 13:40 -0700, Stan Shebs wrote:
> CodeSourcery has a project to add "multiprocess" capability to GDB,
> and with this message I'd like to kick off some discussion of what
> that means and how to make it happen.
> 
> To put it simply, the goal of the project is to make this command work
> in some useful way:
> 
>   gdb prog1 prog2 pid2 prog3 prog4
> 
> As the command suggests, we're talking about multiple programs or
> executables being controlled by a single GDB, in contrast to a single
> program with multiple processes or forks, a la Michael's machinery for
> Linux forks.

Daniel's, really, I just extended it.

This is great, it's been a long time coming.  I've had lots of
inquiries about it, and even people asking me to implement it.

Mostly I've referred them to Eclipse, which as you may know
can martial together several gdb sessions under one eclipse
session.

>  So although we often use the term "multiprocess", it's
> perhaps more precise to call it "multiprogram" or "multiexec" GDB.
> 
> The first thing is to figure out is how this should all work for the GDB
> user. The command above seems like a pretty obvious extension;
> programmers debugging client/server pairs have long wanted to be able
> to do just that, and we've always had to tell them they have to start
> up two GDBs and juggle. Since both core files and process ids are
> distinguishable from executable names, we can allow intermixing of all
> these on the command line, so that multiple core file and pid
> arguments apply to the preceding executables.

Yarg.  I can imagine wanting to debug several core files at once
(failure of a multi-process application, maybe), but mixed core
files and live processes?  I think maybe we should disallow that...
What would happen when you said "continue"?

> Once the debugger is started, but before any of the programs have run,
> it seems obvious to have a notion of "current program", so that commands
> like "list main" and "break main" will work as usual. The user should
> have a way to list the programs ("info programs") and a way to set the
> current one. It might also make sense to have a menu option a la C++,
> so that "list client_only_fn" works irrespective of the current program,
> while "list main" might ask the user which main() is wanted. Another
> possibility is to introduce a "program apply <names>|all <cmd>" on the
> analogy of the existing thread apply command.

Yep, I think we'd do well to consider the thread debugging paradigm
as a model, with this maybe as a second layer on top of that
(although users might want to be able to give commands that cross
both layers, eg. "give me the registers for process A, thread B").

In the case of threads, we have already taught GDB to maintain a
separate set of program state (and debugger state) info for each
thread.  We could do the same for processes, with the state info
set extended to include (eventually) a symbol file, environment, 
argv list etc.

> Notice that we don't really want to use the term "process" for any of
> this so far, because nothing is running yet and there are no
> processes/inferiors; this part is all about the symbol side.
> 
> Commands like "file" should get an alternate form or behavior that
> adds rather than replaces.

While preserving the present behavior, of course...   ;-)

>  Conversely, the user will also want a way
> to take programs out of the debugging session. This is not quite the
> same as detaching, because the user may want, say, the server to
> continue doing its serving thing, but also to have the list etc
> commands only work on the client code, no longer be ambiguous.

You would want to be able to kill, detach etc. on a per-process
basis of course (here I assume the process is running so I say
process explicitly).

But I echo somebody else's question -- how do you see what you
describe as being different from detach (other than by applying 
to only one of several child processes)?



> When it's time to run, the user will want the ability to run anywhere
> from one to all of the programs, each with its own argument list. It
> should be possible to do this with a single command, so that the user
> isn't scrambling to put in all the run commands quickly enough.

Yep -- while it might be a challenge to come up with a command
syntax that would allow the user to specify the argv args for all
the child processes at once, we could for instance have "set args"
commands for each of several processes we wanted to start at once,
and then a single "run all" sort of thing.

Of course, this sort of implies giving gdb some knowledge about
each of several child processes BEFORE they are actually created.

Then we could also apply some attribute to each of the (running
or potential) processes that specified something like a "run set",
if you will (and if you remember that concept from the HPDF) --
ie. I want these to run and those to not-run.


> Once programs are running, execution control should work in a fashion
> generally analogous to what we have for threads now. When something is
> stopped, it needs to report program, process, and thread; step and
> continue will need a way to specify the program or process being
> stepped or continued. User-friendliness suggests that program name
> should be accepted as a synonym for process id if there is only one
> process for the executable.

And again -- appologies that I have not closely followed the current
discussions about threads and running/not-running, but you would
probably
want to be able to specify some subset of processes-and-threads that
were considered runnable, and some that would be held in the stopped
state.


> Data display will need to have some way to identify the process from
> which the data is being taken. It may be useful to have a "process
> apply" for print commands, so that the output includes the value of
> the expression in each process, especially useful for values that are
> expected to be the same in each.

Absolutely -- although in the absence of a gui, this could rapidly
become cumbersome.  I think most of what we currently can do with
threads can extend more-or-less naturally to processes.  

I see the first big challenge as being the handling of multiple
symbol tables.


> Implementationwise, we will need to replace the single exec target
> with a list of execs, and modify symbol machinery to support a
> many-many relationship between programs and symbol tables. Although
> my inclination is to create a new symbol table for each process'
> image of each shared library, that may be excessively expensive.
> 
> In addition to thoughts on desired user interface, I would welcome
> suggestions on how to add this feature incrementally; the abovementioned
> bits are a lot to add all at once!

I am a big lover of the incremental approach.

How about this as a first increment?  Could we extend
what we now have with forks, so that GDB could individually
control the separate process ids (including threads, if they
have them), while forestalling dealing with separate symbol
files because forks can all share the same one?