Re: [PATCH] randomize benchtests

[OndÅej BÃlka <neleai at seznam dot cz>]

On Fri, May 17, 2013 at 12:24:30PM +0200, Torvald Riegel wrote:
> On Mon, 2013-04-22 at 14:56 +0200, OndÅej BÃlka wrote:
> > On Mon, Apr 22, 2013 at 05:44:14PM +0530, Siddhesh Poyarekar wrote:
> > > On 22 April 2013 17:30, OndÅej BÃlka <neleai@seznam.cz> wrote:
> > > > +      clock_gettime (CLOCK_PROCESS_CPUTIME_ID, &start);
> > > > +      for (k = 0; k < iters; k++)
> > > > +        {
> > > > +         i = rand_r (&seed)%NUM_SAMPLES;
> > > > +         BENCH_FUNC(i);
> > > > +        }
> > > > +      clock_gettime (CLOCK_PROCESS_CPUTIME_ID, &end);
> > > 
> > > This is wrong.  The interval also has the time taken to call rand_r.
> > > 
> > This is not wrong. You are interested only on differences between
> > implementations and adding same time from rand_r calls does not change
> > that. 
> 
> But if we should be changing the rand_r implementation in the future
> (e.g., if we might be getting HW support for it on a certain
> architecture), then this will lead to a difference in all our

>From stdlib/rand_r.c:

/* This algorithm is mentioned in the ISO C standard, here extended
   for 32 bits.  */

Given that we can break applications that depend on rand_r
reproducibility it will not change. 
If you want fully specified generator use drand48.

> performance numbers between the prior code revisions and the newer ones.
> Remember that we eventually also want to find performance regressions.
> 
> I think that we should make the inner loops as reproducible as possible,
> so this should either be using a custom pseudo RNG, or calibrate it
> against a loop with just a rand_r call, or don't get the random numbers
> in the loop.  The latter might not really be an option too, because we'd
> then need to read those from a precomputed random set in memory, which
> might be even more of a distortion.