This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [PATCH] Run benchmarks for constant time instead of constant iterations
- From: OndÅej BÃlka <neleai at seznam dot cz>
- To: Siddhesh Poyarekar <siddhesh at redhat dot com>
- Cc: libc-alpha at sourceware dot org
- Date: Wed, 24 Apr 2013 17:44:27 +0200
- Subject: Re: [PATCH] Run benchmarks for constant time instead of constant iterations
- References: <20130424130839 dot GG14810 at spoyarek dot pnq dot redhat dot com> <20130424143241 dot GA5192 at domone dot kolej dot mff dot cuni dot cz> <20130424150027 dot GI14810 at spoyarek dot pnq dot redhat dot com>
On Wed, Apr 24, 2013 at 08:30:27PM +0530, Siddhesh Poyarekar wrote:
> On Wed, Apr 24, 2013 at 04:32:41PM +0200, OndÅej BÃlka wrote:
> > I work at even better idea, benchmark should run until certain level of
> > precision is reached. I decided to have with probability 90% error of 1%.
> >
> > This depends on preheat cpu patch, collecting data at 800MHz and then
> > switching to 2.5GHz will cause running time of minute until slow data
> > are ruled out as error.
> >
> > Your patch is nice, with it I could make less modifications.
> >
> > I want do rougthly following. Perhaps you could merge it to your patch?
>
> That's an interesting idea. By 'precision' and 'error' I assume you
> mean the stability of the measurements and the deviation of the
> current measurement from the mean. Is that a correct interpretation
> of what you're saying or do you mean something else?
>
Moreless. it means that http://en.wikipedia.org/wiki/Confidence_interval
is within given fraction of a mean.
> Howeber, it could be possible that the measurements could stabilize
> early on and give us a worse result than it actually is, no? To avoid
> that we may need to set a high enough initial number of iterations,
> which again could lead to vastly different runtimes for different
> architectures and may even have to be decided and hard-coded on an
> individual function basis. This essentially brings us back to the
> problem I'm trying to solve with this, which is to avoid the great
> variance in benchmark runtimes across architectures.
>
This does not happen in practice. Most of time variance is estimated
with sufficient precision within 100 trials. Then estimate to number of
iterations needed is pretty accurate.
> Siddhesh