This is the mail archive of the libc-alpha@sourceware.org mailing list for the glibc project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[PATCH] Framework for performance benchmarking of functions


Hi,

Currently there is no consistent way to compare improvements in
performance of functions within glibc because of which we generally
end up (re)writing small programs to test improvements.  In a lot of
cases, performance regressions go through unnoticed and are only
caught when someone sees the impact on their applications.

Attached patch is a very simple framework to get the run time of
functions over a number of iterations.  The idea is to run functions
over a known set of inputs (if any) for a fixed number of iterations
to get the average execution time.  The method of adding a function to
this is a simple matter of adding an entry for it in the makefile and
defining a set of inputs in a file.  The specifics are documented in
the patch itself.  I have added input files for exp and pow functions,
that I intend to expand on to cover all its paths (including the slow
paths).

The way to run the benchmark is to simply `make bench`.  The output is
in $objdir/benchtests/bench.out.  Repeatedly doing a `make bench`
results in the previous run being backed up, so that at any point in
time, you have the results of the current and previous run.  The
output looks like this:

exp: TOTAL: 9513212ns, MAX: 10005ns, MIN: 89ns, AVG: 95.132120ns
pow: TOTAL: 20898164ns, MAX: 20465ns, MIN: 161ns, AVG: 208.981640ns

I have also set up a branch 'siddhesh/bench' where one could try this
out.

Further plans:
-------------

This is a very simple first step, so there are a few more things we
can do to make this more powerful.  Apart from the framework itself,
the major work here would be to identify functions we care about and
inputs that cover all paths of that function, especially the slow
paths.  I will be doing this for the log, exp and pow functions to
begin with.

I believe some folks do automatic builds of latest master at regular
intervals.  These build systems could also run a `make bench` and post
results to track performance of functions over time.  The output can
also be easily tokenized and fed into a database which a web app can
pull from and display neat looking graphs.


Siddhesh

	* Makefile.in (bench): New target.
	* Rules (bench): Likewise.
	(binaries-bench): Generate binaries for functions to
	benchmark.
	* benchtests/Makefile: New makefile for benchmark tests.
	* benchtests/bench-skeleton.c: New skeleton file for benchmark
	programs.
	* benchtests/exp-inputs: New input file for EXP function.
	* benchtests/pow-inputs: New input file for POW function.
	* scripts/bench.pl: New script to generate source files for
	benchmark programs.


diff --git a/Makefile.in b/Makefile.in
index d73a78f..df75b8f 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -3,7 +3,7 @@ srcdir = @srcdir@
 # Uncomment the line below if you want to do parallel build.
 # PARALLELMFLAGS = -j 4
 
-.PHONY: all install
+.PHONY: all install bench
 
 all .DEFAULT:
 	$(MAKE) -r PARALLELMFLAGS="$(PARALLELMFLAGS)" -C $(srcdir) objdir=`pwd` $@
@@ -11,3 +11,6 @@ all .DEFAULT:
 install:
 	LANGUAGE=C LC_ALL=C; export LANGUAGE LC_ALL; \
 	$(MAKE) -r PARALLELMFLAGS="$(PARALLELMFLAGS)" -C $(srcdir) objdir=`pwd` $@
+
+bench:
+	$(MAKE) -C $(srcdir)/benchtests $(PARALLELMFLAGS) objdir=`pwd` $@
diff --git a/Rules b/Rules
index 5c5aa60..cbfe3e5 100644
--- a/Rules
+++ b/Rules
@@ -83,7 +83,7 @@ common-generated += dummy.o dummy.c
 
 # This makes all the auxiliary and test programs.
 
-.PHONY: others tests
+.PHONY: others tests bench
 ifeq ($(multi-arch),no)
 tests := $(filter-out $(tests-ifunc), $(tests))
 xtests := $(filter-out $(xtests-ifunc), $(xtests))
@@ -191,6 +191,27 @@ $(objpfx)%.out: /dev/null $(objpfx)%	# Make it 2nd arg for canned sequence.
 	$(make-test-out) > $@
 
 endif	# tests
+
+# Build and run benchmark programs.
+binaries-bench := $(addprefix $(objpfx)bench-,$(bench))
+
+bench: $(binaries-bench)
+	if [ -f $(objpfx)bench.out ]; then \
+	  mv -f $(objpfx)bench.out $(objpfx)bench.out.old; \
+	fi
+	for run in $^; do \
+	  $${run} >> $(objpfx)bench.out; \
+	done
+
+$(binaries-bench): %: %.o \
+  $(sort $(filter $(common-objpfx)lib%,$(link-libc))) \
+  $(addprefix $(csu-objpfx),start.o) $(+preinit) $(+postinit)
+	$(+link)
+
+$(objpfx)bench-%.c: %-inputs bench-skeleton.c
+	$(..)scripts/bench.pl $(patsubst %-inputs,%,$<) \
+	  $($*-ITER) $($*-ARGLIST) $($*-RET) > $@
+
 
 .PHONY: distclean realclean subdir_distclean subdir_realclean \
 	subdir_clean subdir_mostlyclean subdir_testclean
diff --git a/benchtests/Makefile b/benchtests/Makefile
new file mode 100644
index 0000000..7e514f0
--- /dev/null
+++ b/benchtests/Makefile
@@ -0,0 +1,57 @@
+# Copyright (C) 2013 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+
+# Makefile for benchmark tests.  The only useful target here is `bench`.
+
+# Adding a new function `foo`:
+# ---------------------------
+
+# - Append the function name to the bench variable
+
+# - Define foo-ITER with the number of iterations you want to run
+
+# - Define foo-ARGLIST as a colon separated list of types of the input
+#   arguments.  Use `void` if function does not take any inputs.  Put in quotes
+#   if the input argument is a pointer, e.g.:
+
+#      malloc-ARGLIST: "void *"
+
+# - Define foo-RET as the type the function returns.  Skip if the function
+#   returns void.  One could even skip foo-ARGLIST if the function does not
+#   take any inputs AND the function returns void.
+
+
+# - Make a file called `foo-inputs` with one input value per line, an input
+#   being a comma separated list of arguments to be passed into the function.
+#   See pow-inputs for an example.
+
+subdir := benchtests
+bench := exp pow
+
+exp-ITER = 100000
+exp-ARGLIST = double
+exp-RET = double
+LDFLAGS-bench-exp = -lm
+
+pow-ITER = 100000
+pow-ARGLIST = double:double
+pow-RET = double
+LDFLAGS-bench-pow = -lm
+
+include ../Makeconfig
+include ../Rules
diff --git a/benchtests/bench-skeleton.c b/benchtests/bench-skeleton.c
new file mode 100644
index 0000000..1dd1de7
--- /dev/null
+++ b/benchtests/bench-skeleton.c
@@ -0,0 +1,57 @@
+/* Skeleton for benchmark programs.
+   Copyright (C) 2013 Free Software Foundation, Inc.
+   This file is part of the GNU C Library.
+
+   The GNU C Library is free software; you can redistribute it and/or
+   modify it under the terms of the GNU Lesser General Public
+   License as published by the Free Software Foundation; either
+   version 2.1 of the License, or (at your option) any later version.
+
+   The GNU C Library is distributed in the hope that it will be useful,
+   but WITHOUT ANY WARRANTY; without even the implied warranty of
+   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+   Lesser General Public License for more details.
+
+   You should have received a copy of the GNU Lesser General Public
+   License along with the GNU C Library; if not, see
+   <http://www.gnu.org/licenses/>.  */
+
+#include <string.h>
+#include <stdint.h>
+#include <stdio.h>
+#include <time.h>
+
+int
+main (int argc, char **argv)
+{
+  int i, j;
+  int64_t total = 0, max = 0, min = 0x7fffffffffffffff;
+  struct timespec start, end;
+
+  memset (&start, 0, sizeof (start));
+  memset (&end, 0, sizeof (end));
+
+  for (i = 0; i < ITER; i++)
+    {
+      for (j = 0; j < NUM_SAMPLES; j++)
+	{
+	  clock_gettime (CLOCK_MONOTONIC_RAW, &start);
+	  BENCH_FUNC(j);
+	  clock_gettime (CLOCK_MONOTONIC_RAW, &end);
+
+	  int64_t cur = (end.tv_nsec - start.tv_nsec
+			 + (end.tv_sec - start.tv_sec) * 1000000000);
+
+	  if (cur > max)
+	    max = cur;
+
+	  if (cur < min)
+	    min = cur;
+
+	  total += cur;
+	}
+    }
+
+  printf (FUNCNAME ": TOTAL: %ldns, MAX: %ldns, MIN: %ldns, AVG: %lfns\n",
+	  total, max, min, (double) total / (ITER * NUM_SAMPLES));
+}
diff --git a/benchtests/exp-inputs b/benchtests/exp-inputs
new file mode 100644
index 0000000..713c222
--- /dev/null
+++ b/benchtests/exp-inputs
@@ -0,0 +1 @@
+42.0
diff --git a/benchtests/pow-inputs b/benchtests/pow-inputs
new file mode 100644
index 0000000..2f7cc03
--- /dev/null
+++ b/benchtests/pow-inputs
@@ -0,0 +1 @@
+42.0, 42.0
diff --git a/scripts/bench.pl b/scripts/bench.pl
new file mode 100755
index 0000000..3c01db9
--- /dev/null
+++ b/scripts/bench.pl
@@ -0,0 +1,82 @@
+#! /usr/bin/perl -w
+# Copyright (C) 2013 Free Software Foundation, Inc.
+# This file is part of the GNU C Library.
+
+# The GNU C Library is free software; you can redistribute it and/or
+# modify it under the terms of the GNU Lesser General Public
+# License as published by the Free Software Foundation; either
+# version 2.1 of the License, or (at your option) any later version.
+
+# The GNU C Library is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# Lesser General Public License for more details.
+
+# You should have received a copy of the GNU Lesser General Public
+# License along with the GNU C Library; if not, see
+# <http://www.gnu.org/licenses/>.
+
+
+use strict;
+use warnings;
+# Generate a benchmark source file for a given input.
+
+if ($#ARGV lt 1) {
+  die "Usage: bench.pl <function> <iterations> [parameter types] [return type]"
+}
+
+my $arg;
+my $func = $ARGV[0];
+my $iters = $ARGV[1];
+my @args;
+my $ret = "void";
+
+if ($#ARGV ge 2) {
+  @args = split(':', $ARGV[2]);
+}
+
+if ($#ARGV eq 3) {
+  $ret = $ARGV[3];
+}
+
+my $decl = "extern $ret $func (";
+
+if ($#args lt 0 || $args[0] eq "void") {
+  print "$decl void);\n";
+  print "#define BENCH_FUNC(j) $func();\n";
+  print "#define NUM_SAMPLES (1)\n";
+}
+else {
+  my $num = 0;
+  my $bench_func = "#define BENCH_FUNC(j) $func (";
+  my $struct = "struct args {";
+
+  foreach $arg (@args) {
+    if ($num gt 0) {
+      $bench_func = "$bench_func,";
+      $decl = "$decl,";
+    }
+
+    $struct = "$struct $arg arg$num;";
+    $bench_func = "$bench_func in[j].arg$num";
+    $decl = "$decl $arg";
+    $num = $num + 1;
+  }
+
+  print "$decl);\n";
+  print "$bench_func);\n";
+  print "$struct } in[] = {";
+
+  open INPUTS, "<$func-inputs" or die $!;
+
+  while (<INPUTS>) {
+    chomp;
+    print "{$_},\n";
+  }
+  print "};\n";
+  print "#define NUM_SAMPLES (sizeof (in) / sizeof (struct args))\n"
+}
+
+print "#define ITER $iters\n";
+print "#define FUNCNAME \"$func\"\n";
+print "#include \"bench-skeleton.c\"\n";


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]