This is the mail archive of the systemtap@sourceware.org mailing list for the systemtap project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Systemtap .sum vs .log


Kris Van Hees wrote:
> In followup to our conversation on the conference call...  The summary file
> does not capture the ERROR messages from runtest, and those are significant
> in the processing of test results. [...]

That is a reasonable point, but ...
That ERROR message does not come from runtest but from systemtap.
There are other ERROR messages from systemtap that do not constitute
errors (e.g. parseko/cmdline06.stp).

> When e.g. the staprun executable is not setuid root (or equiv
> privs), some tests print out an ERROR about this, [...]  actually
> reported as XFAIL.  [...]

This should be a rare if any case.  The only time when this setuid
business is tested is for pass-5 tests run by "make installcheck".
There are not many (if any) xfail pass-5 tests.

> Another problem (brought to light by a question from David Wilder)
> is that the dejagnu summary output seems to be inconsistent with the
> actual log messages (both in the summary and verbose logs):

The .sum file contains the judgement of dejagnu.  Some peculiar test
cases took it upon themseves to add stuff like
     puts "PASS: $test"
into the .exp files, in addition to the appropriate
     pass $test
Some of the former have been removed; perhaps you just came across
a straggler with a price on its head.

Or not ...

David Wilder <dwilder@us.ibm.com> writes:

> Here is the difference, the systemtap.sum has one more PASS line that
> is missing from systemtap.log.
> [...]
>  PASS: args search for staprun (/usr/local/bin/staprun)
> +PASS: args compile
>  PASS: args search for probe module (args_3085.ko)
> [...]

Dunno why that line would be missing from the .log file.  I don't see
that discrepancy in some of my recent .log/.sum files I checked.

>  PASS: args search for probe module (/tmp/stapwfd7KX/args_3085.ko)

But parenthesized comments like (....) should probably not be quite so
installation-specific.  That prevents mechanical regression analysis
across installations.

> I don't see any XPASS in ether the .sum or .log.  However there is a
> single KPASS in both logs.    KPASS ?? known successes ???, must be a
> bug.

KPASS means that a test was declared a "known failure" i.e., KFAIL,
but nevertheless passed.  This is considered a failure by dejagnu,
but we should treat it as good news.  The "setup_kfail" predicate
in the dejagnu test case could be narrowed or eliminated for tests
that have graduated from the "known failure" category.

- FChE


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]