RE: [patch, testsuite] test newlib.string/strcmp-1.c takes too long

["Greta Yorsh" <Greta dot Yorsh at arm dot com>]

On 02 May 2012 14:36, Corinna Vinschen wrote:
> On May  2 11:22, Richard Earnshaw wrote:
> > On 02/05/12 09:19, Corinna Vinschen wrote:
> > > On May  1 17:49, Greta Yorsh wrote:
> > >> The test of strcmp in newlib.string testsuite can take too long to
> run on
> > >> simulators, resulting in timeout failures.
> > >>
> > >> This patch creates a short version of the test, by reducing the
> length and
> > >> the number of strings to compare. This patch also adds a new macro
> LONG_TEST
> > >> to control whether a short or a more comprehensive test should be
> used.
> > >> LONG_TEST is not set by default.
> > >
> > > I'm wondering if that's the right way to solve the problem.  In
> theory,
> > > whatever you set these values to, you could find a simulator which
> is
> > > hitting your timeout values.  Wouldn't it make more sense to raise
> the
> > > timeout values in the testsuite?
> > >
> > >
> > > Corinna
> > >
> >
> >
> > But to what level?  The test is enormous (over 25 minutes of
> simulation
> > on one run we did), and probably overkill unless you're actively
> testing
> > a rewrite of strcmp.
> 
> strcmp is one of the first candidates for a target-specific rewrite in
> assembler.  So there's a good chance that you see a rewrite as soon as
> a new target gets supported.  Shouldn't we rather opt on the safe side
> by default?
> 
> > I think defaulting to a cut-down version is a good idea.  It might
> still
> > be necessary to bump the timeout, but the overall cost during normal
> > testing doesn't need to be this high.  Don't forget that with
> multilib
> > testing you can end up running that test multiple times.
> 
> How long does the cut-down test run in the same scenario as the 25 mins
> observed above?

It will probably take around one and a half minutes instead of 25 minutes. 

The number of calls to strcmp in the long test is 16 times that in the short test (because the number of calls 
is proportional to the product of MAX_BLOCK_SIZE, MAX_DIFF, MAX_LEN, MAX_ZERO, and each of these values in the long test is twice that in the short test).

On my machine using qemu for arm target, newlib.string testsuite takes 34 seconds with the short strcmp test instead of 9 minutes with the long test.

Thanks,
Greta