Re: [PATCH v3 01/17] Fix and test "checkpoint" in non-stop mode

[Doug Evans <dje at google dot com>]

On Tue, Apr 28, 2015 at 9:06 AM, Doug Evans <dje@google.com> wrote:
> The axis on which I'm worried is whether this will turn out to be
> 1 extra file or 1000. One extra file per test doesn't scale.
> And trying to reign in 100 files, or whatever the threshold is
> before it's clear that we need to change course, will be more
> painful than having a table-driven approach that can be simple
> now and still grow as needed. (*1)
>
> Another way to go would be really just have 1 extra file (or 1 extra
> file per gdb.foo subdir or some such), and this file
> documents all the tests intended to be run in different modes
> within one "make check". One could even extend
> this to be more than just all-stop/non-stop.
>
> Another way, though, is to have two "make check" runs:
> One with all-stop and one with non-stop.
> Tests that can't handle one or the other should be marked
> as such anyway. How a test chooses to handle the choice
> is an internal implementation decision regardless of which
> route is chosen. Yeah, running some tests twice this way
> won't enhance coverage (e.g., no point in running help.exp in
> all-stop and non-stop), but we should be almost within epsilon
> of being able to do this today (I know I've done it in the past),
> and complete test runs only take a few minutes on any reasonably
> beefy system (at least they do on mine).
>
> (*1): A hybrid table driven approach could be to (effectively)
> run the test twice using whatever mechanism that running
> "make check" twice would use. This way only the specified
> set of tests would be run twice (instead of running the whole testsuite
> twice), and one can still manually run individual tests in whatever
> mode (all-stop/non-stop/whatever) one chooses.
>
> My counter-proposal would be to see the extent to which we can
> just run the testsuite in all-stop and non-stop, and do that for now
> while we work towards something that scales better.
> Someone could get a basic table-driven scheme going in a day.

[Sorry for the resend ... cursed text/html.]

Another thought was that if this is just a one-off and you're manually
splitting up non-stop from all-stop to maintain parallelizability of the tests
then I might not mind the new file.

IOW, tests that want to handle both all-stop and non-stop in one
"make check" can do so in one file. Any loss in parallelization
is probably minimal (modulo really big tests).

Btw, checkpoint.exp completes in 9 seconds on my system.
I suspect doubling that and doing both all-stop and non-stop in
the one file won't increase "make check-parallel" noticeably.