Improving libm-test.inc structure and maintenance

["Joseph S. Myers" <joseph at codesourcery dot com>]

libm-test.inc is a 14000-line file, containing testcases for most libm
functions where the inputs were generally manually chosen but the
expected outputs were computed with MPFR or other tools.  When adding
libm tests, it is necessary to generate expected outputs and put them
in the file; there is no standard automation for generating them.  It
is also necessary to generate the expectations for exceptions and
errno setting, and similarly put them in the file, and likewise the
conditionals for which tests are run for which floating-point formats.

The tests are described by lines containing calls to function-like
entities such as TEST_f_f.  Those calls are interpreted by
gen-libm-test.pl, which converts them to actual C code (libm-test.c)
in the build directory.

Most tests are run only in the default (round-to-nearest) rounding
mode, although some are run in other rounding modes.  If a given test
is to be run in each rounding mode, it must be repeated four times in
the file in separate functions, with the appropriate results for each
rounding mode manually placed in each call to TEST_*.

I propose two main changes to how these tests are handled:

* Instead of embedding TEST_* calls that are converted to code inside
  functions, such calls should go in arrays of testcases defined
  outside the functions; the functions should then loop over the
  arrays.  This would probably speed up compilation of these tests, as
  they would contain large constant data instead of large functions
  that need optimization / code generation.  With appropriate
  definition of the arrays (for example, containing test names in
  forms that don't name the function being called), the same tests
  could be used for multiple function names that are supposed to alias
  each other - or for testing a function in each rounding mode with
  the same tests, if the arrays contain expectations for each rounding
  mode.

* The expected results, for cases of finite inputs where the
  mathematical expected results are finite, should be generated by
  MPFR, which should also generate expectations for exceptions and
  errno in such cases, and appropriate conditionals for which
  floating-point formats each test is run in.

  I envisage two new checked-in files to support this, say
  auto-libm-test-in and auto-libm-test-out.  The first file would
  contain a list of testcases, listing the function and the input
  arguments for each test.  The second, generated by a new checked-in
  program (using MPFR and MPC), would also contain the expected
  results (including errno and exception settings) for each such test,
  for each floating-point format.  The new program could either
  completely regenerate the file, or (the default) only generate
  results for newly added tests (full regeneration would take several
  minutes on a fast system - MPC is *slow* for some of the inverse
  trig / hyperbolic functions).

  The test inputs might be listed as decimal or hex floats, or maybe
  as special values such as pi.  In any case, each input would be
  converted to a set of floating-point values, by rounding up and down
  in each floating-point format.  This would produce a set of tests
  (from which duplicates would be automatically removed).  For each
  format, there would then be a subset of those tests for which the
  values are representable in that format, and those would be the
  tests run for that format.  Other information that would be included
  with the inputs would be whether to disable some tests for
  TEST_INLINE; whether to disable for particular formats because of
  bugs; whether to allow spurious exceptions or permit expected
  exceptions to be missing because of bugs; similarly, whether to
  allow errno settings to be missing because of bugs.

  (There are six supported formats - flt-32, dbl-64, ldbl-96 (Intel),
  ldbl-96 (m68k), ldbl-128, ldbl-128ibm - though the differences
  between the two ldbl-96 variants only affect subnormals.
  ldbl-128ibm would be treated for the purposes of the generators as a
  format with a fixed 106-bit precision except for subnormals, except
  that LDBL_MAX is slightly smaller than it would be for an ordinary
  106-bit format.)

  gen-libm-test.pl would then process auto-libm-test-out to generate C
  code inside the tables of tests for given functions, where they say
  (for example) AUTO_TESTS (cos).  So the tables would combine manual
  tests (for NaNs, infinities, etc.) and automatic ones (for finite
  inputs and outputs - though the finite outputs may overflow /
  underflow for particular formats).  Results would automatically be
  generated for all rounding modes; the C code in libm-test.inc would
  at least initially be responsible for running tests in each mode
  where desired (appropriate macros / functions might be added to make
  it easy to do so).

I suggest that the "names" of the tests that go in libm-test-ulps just
name the function, rounding mode (as applicable) and inputs (as hex
floats, removing the existing support for prettier names of certain
test inputs), but not the expected output.  This would of course
invalidate existing ulps and require them to be regenerated from
scratch, but apart from x86/x86_64 we try to do such regeneration
before each release anyway, and it's probably a good idea for
x86/x86_64 despite the machine-dependence.

Apart from the above, I'm doubtful that the checks for each function
being unimplemented and setting errno to ENOSYS make sense any more -
and there may be cases where they wrongly cause tests to be skipped,
if e.g. a function calls fesetround which is unimplemented for a given
processor (but still produces reasonable results anyway).  We don't
have unimplemented libm functions for any system at present; I think
the last such case was fixed by my addition of mips64 sqrtl on
2007-05-23.

I'd envisage changes such as the above being implemented incrementally
- first setting up support for tables of tests and moving tests into
such tables bit by bit, before then setting up support for automatic
test generation and moving existing tests into it.  This means that
initial versions of support for tables, or for automatic generation,
don't need to support all features (disabling tests for particular
formats, missing exceptions, ...); features can be added when needed
to convert tests for a particular function.

It is of course possible also to split up libm-test so that the tests
for each function are run separately (with or without also splitting
up the auto-libm-test-* files).  I don't know if that's desirable.

-- 
Joseph S. Myers
joseph@codesourcery.com