This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: Systemtap .sum vs .log
- From: fche at redhat dot com (Frank Ch. Eigler)
- To: David Wilder <dwilder at us dot ibm dot com>
- Cc: Kris Van Hees <kris dot van dot hees at oracle dot com>, systemtap at sourceware dot org
- Date: Thu, 11 Oct 2007 17:58:02 -0400
- Subject: Re: Systemtap .sum vs .log
- References: <20071011193108.GC2054@oracle.com> <470E935A.10500@us.ibm.com>
Kris Van Hees wrote:
> In followup to our conversation on the conference call... The summary file
> does not capture the ERROR messages from runtest, and those are significant
> in the processing of test results. [...]
That is a reasonable point, but ...
That ERROR message does not come from runtest but from systemtap.
There are other ERROR messages from systemtap that do not constitute
errors (e.g. parseko/cmdline06.stp).
> When e.g. the staprun executable is not setuid root (or equiv
> privs), some tests print out an ERROR about this, [...] actually
> reported as XFAIL. [...]
This should be a rare if any case. The only time when this setuid
business is tested is for pass-5 tests run by "make installcheck".
There are not many (if any) xfail pass-5 tests.
> Another problem (brought to light by a question from David Wilder)
> is that the dejagnu summary output seems to be inconsistent with the
> actual log messages (both in the summary and verbose logs):
The .sum file contains the judgement of dejagnu. Some peculiar test
cases took it upon themseves to add stuff like
puts "PASS: $test"
into the .exp files, in addition to the appropriate
pass $test
Some of the former have been removed; perhaps you just came across
a straggler with a price on its head.
Or not ...
David Wilder <dwilder@us.ibm.com> writes:
> Here is the difference, the systemtap.sum has one more PASS line that
> is missing from systemtap.log.
> [...]
> PASS: args search for staprun (/usr/local/bin/staprun)
> +PASS: args compile
> PASS: args search for probe module (args_3085.ko)
> [...]
Dunno why that line would be missing from the .log file. I don't see
that discrepancy in some of my recent .log/.sum files I checked.
> PASS: args search for probe module (/tmp/stapwfd7KX/args_3085.ko)
But parenthesized comments like (....) should probably not be quite so
installation-specific. That prevents mechanical regression analysis
across installations.
> I don't see any XPASS in ether the .sum or .log. However there is a
> single KPASS in both logs. KPASS ?? known successes ???, must be a
> bug.
KPASS means that a test was declared a "known failure" i.e., KFAIL,
but nevertheless passed. This is considered a failure by dejagnu,
but we should treat it as good news. The "setup_kfail" predicate
in the dejagnu test case could be narrowed or eliminated for tests
that have graduated from the "known failure" category.
- FChE