RE: Multi-threaded debugging within GDB & eCOS

["Alex Schuilenburg" <alexs at cygnus dot co dot uk>]

> GDB will "lock up".  More precisely, GDB will sit quietly waiting for
> the target to do something, not realizing that the user has caused a
> deadlock by manually suspending thread A.  This is as expected, since
> GDB has no way to know that the program is in this state.  In
> practice, the user gets impatient, hits ^C, sees the current state,
> and goes "oh yeah, my fault" and fixes.

I figured it would.


> GDB's default behavior is to give all threads a chance to run when
> resuming, which generally prevents this kind of situation.  We
> introduced the scheduler-locking flag for those cases where the user
> really really needs to lock out other threads, even if it might mean
> the thread being stepped will block.

Hugo and I had a discussion a while ago about the usefulness of this flag
and come to the conclusion that it was not, or rather the semantics of it
should be changed.  Locking the scheduler is dangerous for this very reason.
You may have interrupts occuring which need to be serviced on the target
hardware and this method will lock out the service thread, and in certain
cases even crash the hardware.  So you really need to have a less intrusive
method of debugging, particularly for deeply embedded real-time debugging.

The solution we came up with for eCos was something similar to what I had
done before.  See below.

> (I'd be interested to know if you have workarounds for this
> fundamental problem though.)

Sort of.  I wrote a debugger for Helios which was targetted at
multi-threaded non-intrusive debugging.  For this I termed the phrase
"loosly coupled debugging".  Essentially the target hardware (e.g. threads
not being debugged) had to keep on running, even when a bp was hit by a
thread being debugged, else the h/w would die.  I had a debug agent running
on the h/w which would talk back to the debugger and let it know when a bp
was hit, when an exception occurred etc, but more importantly it would run
in conjunction with the application which was being debugged.  That is, the
debugger on the host could query the agent on the target at any time to
query the status of the h/w and the status of actively running threads.  You
could do a backtrace which was a "sample" where a thread on the remote app
was at when the bt command was executed, and set breakpoints within the app
while the app was still running.

More importantly I introduced the freezing and thawing of threads.  This
required some support from the RTOS.  If you froze a thread, the RTOS would
no longer schedule it for execution.  Similarly threads were thawed to allow
them to resume.  So the user would freeze only the threads which they did
not want to interact with the thread they were debugging.  Naturally you
could inspect the stack etc of a frozen thread as the full register context
of the frozen thread was preserved and available to the agent, and hence the
debugger.  There was no concept of ^C as you always had a command line.  If
you wanted to stop the target entirely you would simply freeze all threads.
Naturally the debug agent would be exempt.

The cool thing was that you could leave the debug agent in your final app.
So you could walk up to the running h/w in the field, hook a portable to the
serial port, and attach the debugger.  Hence you could query the target to
find out what it was doing, what state threads were in, and if you had the
matching source and symbol tables available, even start debugging it.  Like
attaching gdb to an active process on UNIX (which is where I snarfed the
idea from).

Things were a bit more complex (aren't they always) since you could debug
the agent itself and still had to provide the ability to debug the RTOS
which was controlling everything, including the agent.  So threads could
only be frozen if they were not consuming a system resource.  That is,
something which the debug agent or RTOS would need to fulfill its
obligations.  Of cource you could over-ride this to debug the kernel and
system calls, but in these instances the agent could fall back into a
gdb-stub mode, and the user suffered the consequences of freezing the
hardware.  At least they were warned...

Cheers
-- Alex