This is the mail archive of the binutils@sources.redhat.com mailing list for the binutils project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

Re: to XFAIL or not to XFAIL


   Date: Mon, 9 Oct 2000 15:43:21 -0400
   From: DJ Delorie <dj@delorie.com>

   > However, we should not lose track of whether some binutils behaviour,
   > though expected, actually indicates a bug.  Hence, I believe XFAIL
   > should be used for your case (a) (and your patch should not be
   > applied).

   As much as I agree that keeping track of persistently failing cases is
   annoying, I disagree.  A test should fail if it indicates a bug in
   binutils, regardless of how annoying it is to see the failure every
   single time you run the testsuite.

   > If anything, I think a test which does not indicate a bug, your case
   > (b), should result in a PASS.  Perhaps DejaGNU should provide some
   > easy way to say ``reverse the sense of this test for this target.''

   It's always been my understanding that XFAIL was to be used for
   specific targets under which a test would fail through no fault of the
   program being tested.  In other words, "we expect it to fail on this
   target, but there's nothing we can do about it."  In other words,
   there should be *no* cases which XFAIL for *all* hosts/targets.

   Using XFAIL to make it easier to ignore persistent problems is evil.

There are two possible views of the testsuite:

1) It is a regression testsuite which tells you whether the code has
   changed significantly due to some patch you just made.

2) It is a bug testsuite which tells you whether the code is working.

I believe that both uses are valuable.

If we use FAIL both for new failures and for tests which are known to
fail, then it is difficult to use the testsuite as a regression
testsuite.

If we use XFAIL for tests which are known to fail, then we distinguish
new failures from old failures.  This makes it possible to use the
testsuite as both a regression testsuite and a real testsuite.  To
check the real bugs, look at both FAIL and XFAIL results.

Ian

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]