This is the mail archive of the
gdb-patches@sources.redhat.com
mailing list for the GDB project.
Re: [testsuite] add gdb.cp/gdb1355.exp
- From: Michael Elizabeth Chastain <mec at shout dot net>
- To: carlton at kealia dot com, gdb-patches at sources dot redhat dot com
- Date: Wed, 17 Sep 2003 22:01:43 -0400
- Subject: Re: [testsuite] add gdb.cp/gdb1355.exp
Okay, I'm willing to make it an XFAIL rather than a KFAIL.
I remember those threads. The XFAIL feels odd but it does stand
for "external" and this bug is indeed a textbook external bug.
As far as making it a FAIL goes, I think David's argument comes down to:
people pay attention to FAIL, but not to XFAIL or KFAIL. I really don't
like that situation. I see the question as: whether to fight the
situation and use more accurate XFAIL/KFAIL rather than generic FAIL, or
to acquiesce to situation and give people what they expect to see.
I favor the former.
> Scenario 1: The bug is fixed. A year later, the bug in question
> regresses. But, if you just run 'make check', this isn't so obvious:
> no message gets printed in the output, and while the numbers at the
> end of the output will change, those numbers fluctuate for lots of
> reasons. So it will only be noticed by somebody who actually looks at
> gdb.sum (or does the moral equivalent); ideally that will happen, but
> I'd rather not count on it (especially if the bug is on an obscure
> platform).
I'm willing to count on it. Because if they just grep for '^FAIL'
then they will miss any tests that barf out completely with an
ERROR at the beginning and abort. I have little sympathy for
people who analyze gdb.sum that way.
dc> Scenario 2: The fix works on platform A, but not on platform B. And
dc> nobody using platform B has been paying close enough attention to the
dc> discussion of the bug to realize that the test is supposed to start
dc> passing. Then, nobody might ever realize that something's wrong:
dc> platform A users think that everything is fine (which is the case for
dc> them), and platform B have no indication from the testsuite that
dc> something is wrong and that the patch didn't work as indicated.
Ah, this is the old scenario "change the KFAIL/XFAIL to a FAIL"
makes people on platform B aware that something has regressed
(I always investigate transitions from KFAIL -> FAIL).
I have some sympathy for this.
dc> Which is why, in the situation at question, I like it that you have the
dc> explicit failure branches, containing an informative comment as to when
dc> the bug occurred - I just want the letter 'k' removed in a few places.
dc> :-)
So we have two proposals here: I'd like to make this an XFAIL,
and David would like to make it a FAIL.
Daniel, or anyone else, do you have a preference?
Michael C