Re: Improving libm-test.inc structure and maintenance

[OndÅej BÃlka <neleai at seznam dot cz>]

On Thu, May 02, 2013 at 10:46:55PM +0000, Joseph S. Myers wrote:
> libm-test.inc is a 14000-line file, containing testcases for most libm
> functions where the inputs were generally manually chosen but the
> expected outputs were computed with MPFR or other tools.  When adding
> libm tests, it is necessary to generate expected outputs and put them
> in the file; there is no standard automation for generating them.  It
> is also necessary to generate the expectations for exceptions and
> errno setting, and similarly put them in the file, and likewise the
> conditionals for which tests are run for which floating-point formats.
> 
> The tests are described by lines containing calls to function-like
> entities such as TEST_f_f.  Those calls are interpreted by
> gen-libm-test.pl, which converts them to actual C code (libm-test.c)
> in the build directory.
> 
> Most tests are run only in the default (round-to-nearest) rounding
> mode, although some are run in other rounding modes.  If a given test
> is to be run in each rounding mode, it must be repeated four times in
> the file in separate functions, with the appropriate results for each
> rounding mode manually placed in each call to TEST_*.
> 
Lack of rounding modes is probably because we did not respect them until
recently. Perhaps we can add flag to detect results that are correct
upto rounding.

> I propose two main changes to how these tests are handled:
> 
snip
>   I envisage two new checked-in files to support this, say
>   auto-libm-test-in and auto-libm-test-out.  The first file would
>   contain a list of testcases, listing the function and the input
>   arguments for each test.  The second, generated by a new checked-in
>   program (using MPFR and MPC), would also contain the expected
>   results (including errno and exception settings) for each such test,
>   for each floating-point format.  The new program could either
>   completely regenerate the file, or (the default) only generate
>   results for newly added tests (full regeneration would take several
>   minutes on a fast system - MPC is *slow* for some of the inverse
>   trig / hyperbolic functions).
>
Why do auto-libm-test-out  have to be checked in? If outputs be computed 
automaticaly we can save them in test directory. I could write utility
that caches mpc results on disk and returns them.

 
>   The test inputs might be listed as decimal or hex floats, or maybe
>   as special values such as pi.  In any case, each input would be
>   converted to a set of floating-point values, by rounding up and down
>   in each floating-point format.  This would produce a set of tests
>   (from which duplicates would be automatically removed).  For each
>   format, there would then be a subset of those tests for which the
>   values are representable in that format, and those would be the
>   tests run for that format.  Other information that would be included
>   with the inputs would be whether to disable some tests for
>   TEST_INLINE; whether to disable for particular formats because of
>   bugs; whether to allow spurious exceptions or permit expected
>   exceptions to be missing because of bugs; similarly, whether to
>   allow errno settings to be missing because of bugs.
> 
>   (There are six supported formats - flt-32, dbl-64, ldbl-96 (Intel),
>   ldbl-96 (m68k), ldbl-128, ldbl-128ibm - though the differences
>   between the two ldbl-96 variants only affect subnormals.
>   ldbl-128ibm would be treated for the purposes of the generators as a
>   format with a fixed 106-bit precision except for subnormals, except
>   that LDBL_MAX is slightly smaller than it would be for an ordinary
>   106-bit format.)
> 
>   gen-libm-test.pl would then process auto-libm-test-out to generate C
>   code inside the tables of tests for given functions, where they say
>   (for example) AUTO_TESTS (cos).  So the tables would combine manual
>   tests (for NaNs, infinities, etc.) and automatic ones (for finite
>   inputs and outputs - though the finite outputs may overflow /
>   underflow for particular formats).  Results would automatically be
>   generated for all rounding modes; the C code in libm-test.inc would
>   at least initially be responsible for running tests in each mode
>   where desired (appropriate macros / functions might be added to make
>   it easy to do so).
> 
> I suggest that the "names" of the tests that go in libm-test-ulps just
> name the function, rounding mode (as applicable) and inputs (as hex
> floats, removing the existing support for prettier names of certain
> test inputs), but not the expected output.  This would of course
> invalidate existing ulps and require them to be regenerated from
> scratch, but apart from x86/x86_64 we try to do such regeneration
> before each release anyway, and it's probably a good idea for
> x86/x86_64 despite the machine-dependence.
> 
> I'd envisage changes such as the above being implemented incrementally
> - first setting up support for tables of tests and moving tests into
> such tables bit by bit, before then setting up support for automatic
> test generation and moving existing tests into it.  This means that
> initial versions of support for tables, or for automatic generation,
> don't need to support all features (disabling tests for particular
> formats, missing exceptions, ...); features can be added when needed
> to convert tests for a particular function.
> 
> It is of course possible also to split up libm-test so that the tests
> for each function are run separately (with or without also splitting
> up the auto-libm-test-* files).  I don't know if that's desirable.
>
One possibility would be use dryrun format which also could allow
automate bug report of inaccurate results. I could combine this with 
program that tries random values and looks for inaccuracies. 
> -- 
> Joseph S. Myers
> joseph@codesourcery.com