Re: New ELF linker code added to GNU binutils

["Joseph S. Myers" <joseph at codesourcery dot com>]

On Tue, 25 Mar 2008, Ian Lance Taylor wrote:

> I used DejaGNU for years, and I really don't like it.  It is difficult
> to work with and I think it encourages bad test writing.  I've written
> the gold testsuite to focus on what I think matters, which is that you
> can use the linker with the compiler to produce programs that do the
> right thing when they are run.

(The way in which I consider DejaGnu encourages bad test writing is that 
it's easy to write a testsuite without a well-defined enumeration of the 
test assertions (the names after PASS or FAIL in the output: in bad 
testsuites the set of names depends on the results of the tests, or some 
names are duplicated); I prefer the QMTest approach where it must be 
possible to statically enumerate the test assertions separately from 
running the tests.)

There's more than programs doing the right thing when run to testing 
linker correctness.  The following examples apply *even if you assume you 
can always run execution tests on the target*.

* The linker is implementing external ABI specifications, which enable 
interoperation with third-party compilers and assemblers and debuggers and 
other tools and operating systems that execute the resulting binaries, and 
assertions "this relocation in the input file causes a GOT entry to be 
generated in the output file against this symbol with this relocation" are 
relevant as well.

(The present gold testsuite looks like it only has target-independent C++ 
tests; target-dependent tests of each input relocation are in my view 
desirable, whether they are written as execution tests or not.  The linker 
is not used just with GCC-generated or binutils-generated objects; 
certainly not just with those generated by a particular GCC version.)

* Also relevant are tests that the linker implements a particular 
optimization, where more than one output is correct but some outputs may 
not be optimized.

* Tests that the linker implements a CPU erratum fix, if done with dump 
files, don't rely on having a CPU with the erratum in order to test the 
fix.  (See <http://sourceware.org/ml/binutils/2007-01/msg00233.html> for 
example with rationale for such a fix present in binutils.  The dump tests 
therein did subsequently show up a bug elsewhere in the linker with a bad 
qsort comparison function when tested on Windows host.  If you don't like 
erratum fixes in the linker, see the more recent --fix-v4bx and 
--fix-v4bx-interworking options, where no CPU bug is involved but 
execution tests would still only be effective on certain CPUs.)

* And the output should of course run on all implementations conforming to 
a given specification, not just one; this can never be tested perfectly, 
but various aspects of the output can be examined.

* The ABI may specify aspects of the output that cannot be determined 
simply by running the program; for example, the ARM EABI rules on how to 
merge object attributes in input files to produce attributes on the 
output, and on what cases should be diagnosed as incompatible.  (Yes, this 
is not very well tested in the ld testsuite at present; the test coverage 
for the attributes I added for Power and MIPS ABI tagging is better.)

> Of course, a consequence is that the testsuite only works for native
> development (apart from the unittests, of which there are currently
> only two).  I recognize that this is a real deficiency for cross
> development.  I would prefer to address that by using some sort of
> script to run the program in the remote environment.  It would be fine
> with me if that script uses DejaGNU.

You need this for running programs on the host (the linker, compiler etc. 
themselves), which may not be the build system, as well as for running 
programs on the target.

> > * Testing an installed toolchain through creating a site.exp file and 
> > running runtest with appropriate options, rather than testing in the build 
> > directory, so you can test the (relocatable) tools in exactly the 
> > arrangements in which they will be distributed and used, or compare two 
> > different tool versions on the same testsuite version.
> 
> It would be reasonably easy to modify the existing gold testsuite to
> work this way.  We could simply change the uses of gcctestdir/ld and
> -Bgcctestdir/ to use make variables instead.  Then you could pass the
> appropriate make variables to run the testsuite.

Another variable you need is the compiler's multilib options.

> Basically I really think DejaGNU has only one thing going for it,
> which I admit is rather important: it has a lot of information about
> running programs on different target boards.  I'm happy to use that
> part of it, but I would prefer to stay away from it for the testsuite
> in general.

- And about running programs on hosts, not just targets.

- And about how to link programs for different boards.  (Remember all the 
"generic" ELF targets where by design $target-gcc will not link by 
default, without a linker script selecting particular board support code.  
DejaGnu board files contain the information about how to link.)

Then there's the DejaGnu .sum and .log output format and scripts set up to 
process it.  I think it's a good idea even for non-DejaGnu testsuites to 
produce output files in this format with the results of each test 
assertion.  (It's possible of course to process the output of "make" - I 
do so even for the glibc testsuite, which is much worse in this regard - 
but testsuites can be helpful and create output in the usual form as 
well.)

Yes, arbitrary testsuites can be wrapped so they use DejaGnu support for 
running programs on hosts and targets, and the output postprocessed into 
DejaGnu format.  But every testsuite being randomly and arbitrarily 
different from every other is a pain to deal with, which is why I prefer 
testsuites to use the same standard infrastructure as each other; and in 
practical terms DejaGnu is that standard infrastructure for toolchain 
testing, even if QMTest (or your-preferred-test-harness-here) is in some 
ways more theoretically elegant.

> That is, I think a linker testsuite should test functionality.  It
> shouldn't test that the output is unchanged from previous versions.  I

It should however test output where that output is part of the 
functionality (e.g. correspondence of input relocations to output 
specified in the ABI).  And it's an important feature (although not tested 
as such in the present ld testsuite, and as with other features there will 
be bugs) that the output for a given version and target does not depend on 
the host system.

Tests of unchanged output can be a pain (as I've observed when changing 
anything related to MIPS GOT output), but they are also good at showing up 
output changing when it shouldn't, or different on different hosts when it 
shouldn't be (the qsort comparison function mentioned above).

> > The --version output is a bit strange:
> >
> > GNU gold (GNU binutils (GNU Binutils) 2.18.50.20080325) version 1.4
> >
> > (note the "GNU binutils (GNU Binutils)").  BFD_VERSION_STRING includes 
> > both the package version ("(GNU Binutils)" by default, possibly overridden 
> > by --with-pkgversion) and a version number - it's not expected for 
> > binutils components to have their own independent version numbers such as 
> > "1.4".  The word "version" isn't part of the standard --version output.
> 
> Yikes.  This was better when I wrote it--something must have changed
> since the binutils snapshot I was using for development.  I'll clean
> it up.

Yes.  The output for all binutils programs was made to follow the GNU 
Coding Standards more closely, with --with-pkgversion and --with-bugurl 
support added to support distributors putting their own "package name and 
package version" in place of "GNU Binutils" and their own bug-reporting 
URLs in place of the FSF ones.

-- 
Joseph S. Myers
joseph@codesourcery.com