Re: [RFC] GDB performance testing infrastructure

[Yao Qi <yao at codesourcery dot com>]

On 08/27/2013 09:49 PM, Agovic, Sanimir wrote:
> > * Remote debugging.  It is slower to read from the remote target, and
> > >     worse, GDB reads the same memory regions in multiple times, or reads
> > >     the consecutive memory by multiple packets.
> > >
> Once gdb and gdbserver share most of the target code, the overhead will be
> caused by the serial protocol roundtrips. But this will take a while...

Sanimir, thanks for your comments!

One of the motivations of the performance testing is to measure the
overhead of RSP in some scenarios, and look for the opportunities to
improve it, or add a completely new protocol, which is an extreme case.

Once the infrastructure is ready, we can write some tests to see how 
efficient or inefficient RSP is.

> > >   * Tracepoint.  Tracepoint is designed to be efficient on collecting
> > >     data in the inferior, so we need performance tests to guarantee that
> > >     tracepoint is still efficient enough.  Note that we a test
> > >     `gdb.trace/tspeed.exp', but there are still some rooms to improve.
> > >
> Afaik the tracepoint functionality is quite separated from gdb may be tested
> in isolation. Having a generic benchmark framework covering the most parts of
> gdb is probably_the_  way to go but I see some room for specialized benchmarks
> e.g. for tracepoints with a custom driver. But my knowledge is too vague on
> the topic.

Well, it is sort of design trade-off.  We need a framework generic 
enough to handle most of the testing requirements for different GDB 
modules, (such as solib, symbols, backtrace, disassemble, etc), on the 
other hand, we want each test is specialized for the corresponding GDB 
module, so that we can find more details.

I am inclined to handle testing to _all_ modules under this generic
framework.

> > >   2. Detect performance regressions.  We collected the performance data
> > >      of each micro-benchmark, and we need to detect or identify the
> > >      performance regression by comparing with the previous run.  It is
> > >      more powerful to associate it with continuous testing.
> > >
> Something really simple, so simple that one could run it silently with every
> make invokation. For a newcomer, it took me some time to get used to make
> check e.g. setup, run, and interpret the tests with various settings. Something
> simpler would help to run it more often.

Yes, I agree, everything should be simple.  I assume that people
running performance testing should be familiar with GDB regular
regression test, like 'make check'.  We'll provide 'make check-perf' to 
run performance testing ,and it doesn't add extra difficulties on top of 
'make check', from user's point of view, IMO.

> I like to add the Machine Interface (MI) to the list, but it is quite rudimentary:
>
> $ gdb -interpreter mi -q debugee
> [...]
> -enable-timings
> ^done
> (gdb)
> -break-insert -f main
> ^done,bkpt={...},time={wallclock="0.00656",user="0.00000",system="0.00000"}
> [...]
> (gdb)
> -exec-step
> ^running
> *running,thread-id="1"
> (gdb)
> *stopped,[...],time={wallclock="0.19425",user="0.09700",system="0.04200"}
> (gdb)
>
> With -enable-timings[1] enabled, every result record has a time triple
> appended, even for async[2] ones. If we come up with a full mi parser
> one could run tests w/o timings. A mi result is quite json-ish.

Thanks for the input.

> (To be honest I do not know how timings are composed of =D)
>
> In addition there are some tools for plotting benchmark results[3].
>
> [1]http://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI-Miscellaneous-Commands.html
> [2]https://sourceware.org/gdb/onlinedocs/gdb/GDB_002fMI-Async-Records.html
> [3]http://speed.pypy.org/

I am using speed to track and show the performance data I got from the 
GDB performance tests.  It is able to associate the performance data to 
the commit, so easy to find which commit causes regressions.  However, 
my impression is that speed or its dependent packages are not 
well-maintained nowadays.

After some search online, I like the chromium performance test and its 
plot, personally.  It is integrated with buildbot (a customized version).

  http://build.chromium.org/f/chromium/perf/dashboard/overview.html

However, as I said in this proposal, let us focus on goal #1 first, get
the framework ready and collect performance data.

--
Yao (éå)