Re: adding KFAILs to C++ testsuite

[Daniel Jacobowitz <drow at mvista dot com>]

On Sun, Dec 22, 2002 at 11:17:28AM -0600, Michael Elizabeth Chastain wrote:
> Hi Fernando,
> 
> > We hope to have 0 FAILed tests as a baseline, independent on the 
> > platform, target etc.  Tjis way, if you make some changes and you see 
> > something != 0 you can start looking for what you have broken ;-)
> 
> Sort of, in my opinion.  I regard "0 FAILs" as an unattainable goal
> (although I remember that Daniel J had one configuration with 0 FAILs
> with some patches he had that got lost in patch-approval limbo).
> 
> New FAILs are going to come in all the time as things change.  I don't
> want to drop everything and nag people at high priority every time a
> FAIL happens.

I think it'll be possible to keep it down, actually... for instance, to
have releases at 0 FAILs on a substantial number of platforms.

> My goals are:
> 
>   At any moment, every covered configuration has 10 or fewer FAILs.
>   Each new FAIL spends a maximum of 4 weeks before becoming something else
>     (it gets fixed, or it becomes a KFAIL or an XFAIL).

Sounds pretty good to me.

> I think it's too much work + test suite thrashing to edit the test
> suite after every run where a new FAIL happens.  I'd like to give
> people a little time to just fix the bugs and save a bunch of
> test suite patch submission activity.

Oh, certainly.  We should understand bugs before KFAILing them.

-- 
Daniel Jacobowitz
MontaVista Software                         Debian GNU/Linux Developer