Re: [Q] Number of unit tests in Mauve and of assertions in Classpath?

["PureNative Software" <info at pure-native dot com>]

Sascha,

Regarding your question:

> >> - Do you know any (as responsive :-) java open source community with
> >> some big project and which
> >> uses both, JDK 1.4 (thus theoretically assertions) and unit tests?

Although not a free (as in GNU) software project, NewJ Library for C++, an
independent 100% native implementation of the core Java API, uses both unit
tests and assertions. The test cases are our C++ version of Mauve, which
make use of the gnu::testlet::Testlet and the rest of the testing framework
just like the Java version. The assertions are actually built right into the
Core J2 Library implementation, which may be compiled in or out for debug or
release builds. As you might expect, the use of both unit tests (like Mauve)
and assertions find more coding or logic errors than either one alone.
Furthermore, unit tests and assertions serve slightly different purposes and
are applicable to slightly different circumstances.

For example, unit tests serve to verify that the implementation of the
library meets the agreed-upon specification. These test cases often validate
method inputs with expected outputs. And test cases serve this purpose
rather well if they are broad enough. Often, such test cases probe boundary
conditions, but it is difficult for them to test all possible inputs,
whether valid or invalid, that may be passed to a method in a real-world
situation. This is where assertions come in. Assertions are always there (if
it has been opted to enable them) so they can validate input and outputs for
any (and every) real-world application. Furthermore, they can test things
that test cases cannot, such as, validating inaccessible members and
performing internal consistency checks. We have found that using both test
cases and assertions gives us a higher level of confidence in the quality of
our library than using one testing technique alone.

Regards,
Vargas
http://www.pure-native.com/newj.html

----- Original Message -----
From: "Markus Gälli" <markus.gaelli@iam.unibe.ch>
To: "Sascha Brawer" <brawer@dandelis.ch>
Cc: "Stephen Crawley" <crawley@dstc.edu.au>;
<mauve-discuss@sources.redhat.com>
Sent: Thursday, March 11, 2004 5:51 AM
Subject: Re: [Q] Number of unit tests in Mauve and of assertions in
Classpath?


> Hi Sascha,
>
> >>> roughly 145 [Mauve tests] are currently failing.
> >> I guess this 145 are not independent, meaning, if you fixed one, many
> >> others would be fixed too. Right?
> >
> > In some cases, there exist multiple tests for the same functionality.
> > For
> > example the tests for solving quadratic and cubic equations, where the
> > testee is called with a bunch of somewhat random test equations. You'll
> > have the same situation with any tests that are feeding random input
> > data
> > into a function and check whether the result is as expected. Any
> > failures
> > would be sort of mutually dependent: As soon as you fix any of these
> > failures, the other failures are likely to disappear immediately
> > (because
> > it's usually the same code that is failing). Stated in more abstract
> > terms, the "fixing order" is only a partial, not a total ordering on
> > tests.
> Right. In my experience many tests are on the same level also. There
> you basically don't
> care with which you begin, as any exception will do. I think that you
> can get more
> out of the ones which form an inclusion hierarchy of coverage sets
> meaning
> that one test is concerned with a more abstract view than the other but
> both
> running the same inner parts.
> >
> >> I am just writing a paper stating that it makes sense to sort failing
> >> tests by size of covered methods beginning with the smallest.
> >
> > Interesting... Why would this be an improvement -- because you assume
> > that smaller methods get invoked by larger ones, and the smaller
> > methods
> > would be the specific point of failure? In that case, couldn't you just
> > sort the failed tests based on a (post-order) traversal of the observed
> > call graph? Or, this may be easier to obtain, based on the total number
> > of instructions/bytecodes executed by the testee and its callees?
> That was what I meant. I hope I am not that sloppy in my paper... ;-)
> It should have been: "The size of the sets of covered method
> signatures".
> > Well, I
> > guess the reason for sorting by method size will be stated in your
> > paper.
> > Would you mind posting a link to your paper on this list when you've
> > published it? Thanks in advance!
> You can read an old version here:
> http://www.iam.unibe.ch/~scg/Archive/Papers/
> Gael03bPartialOrderingTestsByCoverageSets.pdf
> Will hopefully publish a published version Real Soon Now (TM)
> >> My motivation for asking here, was that if you had assertions (which I
> >> surely understand that you don't have), more of the failing
> >> tests would fail at the most specific assertion, making my sorting a
> >> bit more useless.
> >
> > This seems like a reasonable assumption, but I wouldn't know how to
> > falsify it.
> >
> >> But hey, you just gave me a wonderful argument: In some big and
> >> important projects, assertions are even not an option!
> >
> > Well, I guess we will eventually use assertions in Classpath. There's
> > no
> > law saying we must not use 1.4 language features. The reason why we
> > haven't been using assertions in the past is the lack of support by
> > free
> > Java compilers. Which would be easy enough to fix (a quick-and-dirty
> > fix
> > would be just to ignore the assert statement).
> Difficult to introduce that to Object? Could be made painless (in
> Smalltalk at least)
> See: http://www.smalltalkchronicles.net/edition2-1/st_compiler.htm
>
> (snip)
> >
> > Mauve simply takes a large text file that lists the names of test
> > classes, loads the respective class (which must implement the
> > gnu.testlet.Testlet interface) and invokes its "test" method. So the
> > "sorting" is just the ordering in the text file. If you sorted this
> > list
> > in any particular way, the output (= failing tests) would be sorted in
> > the same order.
> >
> >> - Do you know any (as responsive :-) java open source community with
> >> some big project and which
> >> uses both, JDK 1.4 (thus theoretically assertions) and unit tests?
> >
> > I guess most JUnit users would fall into that category, but I
> > unfortunately don't know any specific examples.
> >
> Thanks a lot.
>
> Cheers,
>
> Markus
>