This is the mail archive of the
binutils@sources.redhat.com
mailing list for the binutils project.
Re: to XFAIL or not to XFAIL
Date: Mon, 9 Oct 2000 15:43:21 -0400
From: DJ Delorie <dj@delorie.com>
> However, we should not lose track of whether some binutils behaviour,
> though expected, actually indicates a bug. Hence, I believe XFAIL
> should be used for your case (a) (and your patch should not be
> applied).
As much as I agree that keeping track of persistently failing cases is
annoying, I disagree. A test should fail if it indicates a bug in
binutils, regardless of how annoying it is to see the failure every
single time you run the testsuite.
> If anything, I think a test which does not indicate a bug, your case
> (b), should result in a PASS. Perhaps DejaGNU should provide some
> easy way to say ``reverse the sense of this test for this target.''
It's always been my understanding that XFAIL was to be used for
specific targets under which a test would fail through no fault of the
program being tested. In other words, "we expect it to fail on this
target, but there's nothing we can do about it." In other words,
there should be *no* cases which XFAIL for *all* hosts/targets.
Using XFAIL to make it easier to ignore persistent problems is evil.
There are two possible views of the testsuite:
1) It is a regression testsuite which tells you whether the code has
changed significantly due to some patch you just made.
2) It is a bug testsuite which tells you whether the code is working.
I believe that both uses are valuable.
If we use FAIL both for new failures and for tests which are known to
fail, then it is difficult to use the testsuite as a regression
testsuite.
If we use XFAIL for tests which are known to fail, then we distinguish
new failures from old failures. This makes it possible to use the
testsuite as both a regression testsuite and a real testsuite. To
check the real bugs, look at both FAIL and XFAIL results.
Ian