unittest vs py.test?

[Raymond Hettinger]

Re: unittest vs py.test?

[Peter Hansen][color=blue]
> If py.test provides a driver utility that does
> effectively this, well, that's nice for users. If
> it doesn't run them as separate processes, it wouldn't
> suit me anyway.
>
> Still, it sounds like it does have a strong following
> of smart people: enough to make me want to take a
> closer look at it to see what the fuss is about. :-)[/color]

FWIW, py.test scales nicely. Also, it takes less time to try
it out or read the docs than discuss it to death on a newsgroup.
The learning curve is minimal.


Raymond Hettinger

[Raymond Hettinger]

Re: unittest vs py.test?

[Peter Hansen][color=blue]
> This is pretty, but I *want* my tests to be contained
> in separate functions or methods.[/color]

In py.test, those would read:

def test1():
assert a == b

def test2():
raises(Error, func, args)

Enclosing classes are optional.


Raymond

[Bengt Richter]

Re: unittest vs py.test?

On Sat, 02 Apr 2005 09:24:30 GMT, "Raymond Hettinger" <vze4rx4y@veriz on.net> wrote:
[color=blue]
>[Peter Hansen][color=green]
>> If py.test provides a driver utility that does
>> effectively this, well, that's nice for users. If
>> it doesn't run them as separate processes, it wouldn't
>> suit me anyway.
>>
>> Still, it sounds like it does have a strong following
>> of smart people: enough to make me want to take a
>> closer look at it to see what the fuss is about. :-)[/color]
>
>FWIW, py.test scales nicely. Also, it takes less time to try
>it out or read the docs than discuss it to death on a newsgroup.
>The learning curve is minimal.
>
>[/color]
Is there a package that is accessible without svn?

Regards,
Bengt Richter

[Roy Smith]

Re: unittest vs py.test?

In article <424e73a3.10078 2477@news.oz.ne t>, bokr@oz.net (Bengt Richter)[color=blue]
> Is there a package that is accessible without svn?[/color]

That seems to be its weak point right now.

Fortunately, you can get pre-built svn clients for many platforms
(http://subversion.tigris.org/project...nary-packages), and
from there you just have to run a single command (svn get URL). Still, the
py.test folks would be doing themselves a favor if they made it available
with more common tools.

[Peter Hansen]

Re: unittest vs py.test?

Roy Smith wrote:[color=blue]
> Actually, I believe it does. I'm just starting to play with this, but it
> looks like you can do:
>
> py.test test_sample.py
>
> and it'll run a single test file.[/color]

Well, my driver script can do that too. I just meant
I could do "test_sample.py " and have it run the test
any time, if I wanted, and I was mainly trying to
show that even the __name__ == '__main__' part of
my example was not essential to the use of unittest.
Comparing apples to apples, so to speak, since it
looks like you don't use an import to get access
to py.test.

As near as I can tell, other than requiring an import
statement, and a class statement, there is zero
additional overhead with unittest versus py.test,
given the way I want to structure my tests (in
functions or methods). Is that true? If so, I stand
by my claim that the difference in "weight" between
the two is much much less than some have claimed.

-Peter

[Peter Hansen]

Re: unittest vs py.test?

Raymond Hettinger wrote:[color=blue]
> [Peter Hansen]
>[color=green]
>>This is pretty, but I *want* my tests to be contained
>>in separate functions or methods.[/color]
>
>
> In py.test, those would read:
>
> def test1():
> assert a == b
>
> def test2():
> raises(Error, func, args)
>
> Enclosing classes are optional.[/color]

So basically py.test skips the import statement,
near as I can tell, at the cost of requiring a
utility to be installed in the PATH.

Where was all that weight that unittest supposedly
has?

(I'm not dissing py.test, and intend to check it
out. I'm just objecting to claims that unittest
somehow is "heavy", when those claiming that it
is seem to think you have to use TestSuites and
TestRunner objects directly... I think they've
overlooked the relatively lightweight approach
that has worked so well for me for four years...)

-Peter

[Raymond Hettinger]

Re: unittest vs py.test?

[Peter Hansen][color=blue]
> (I'm not dissing py.test, and intend to check it
> out.[/color]

Not to be disrepectful, but objections raised by someone
who hasn't worked with both tools equate to hot air.

[color=blue]
> I'm just objecting to claims that unittest
> somehow is "heavy", when those claiming that it
> is seem to think you have to use TestSuites and
> TestRunner objects directly... I think they've
> overlooked the relatively lightweight approach
> that has worked so well for me for four years...)[/color]

Claiming? Overlooked? You do know that I wrote the
example in unittest docs, the tutorial example, and hundreds
of the test cases in the standard library. It is not an
uninformed opinion that the exposed object model for
unittest is more complex.

As for "heaviness" , it is similar to comparing alkaline AA
batteries to lithium AA batteries. The first isn't especially heavy,
but it does weigh twice as much as the latter. It only becomes a
big deal when you have to carry a dozen battery packs on a hiking
trip. My guess is that until you've written a full test suite with
py.test, you won't get it. There is a distinct weight difference between
the packages -- that was their whole point in writing a new testing tool
when we already had two.

When writing a large suite, you quick come to appreciate being able
to use assert statements with regular comparision operators, debugging
with normal print statements, and not writing self.assertEqua l over and
over again. The generative tests are especially nice.

Until you've exercised both packages, you haven't helped the OP
whose original request was: "Is there anybody out there who has
used both packages and can give a comparative review?"


Raymond

[Paul Rubin]

Re: unittest vs py.test?

"Raymond Hettinger" <vze4rx4y@veriz on.net> writes:[color=blue]
> When writing a large suite, you quick come to appreciate being able
> to use assert statements with regular comparision operators, debugging
> with normal print statements, and not writing self.assertEqua l over and
> over again. The generative tests are especially nice.[/color]

But assert statements vanish when you turn on the optimizer. If
you're going to run your application with the optimizer turned on, I
certainly hope you run your regression tests with the optimizer on.

[Roy Smith]

Re: unittest vs py.test?

In article <7x3bu8t7or.fsf @ruckus.brouhah a.com>,
Paul Rubin <http://phr.cx@NOSPAM.i nvalid> wrote:
[color=blue]
> "Raymond Hettinger" <vze4rx4y@veriz on.net> writes:[color=green]
> > When writing a large suite, you quick come to appreciate being able
> > to use assert statements with regular comparision operators, debugging
> > with normal print statements, and not writing self.assertEqua l over and
> > over again. The generative tests are especially nice.[/color]
>
> But assert statements vanish when you turn on the optimizer. If
> you're going to run your application with the optimizer turned on, I
> certainly hope you run your regression tests with the optimizer on.[/color]

That's an interesting thought. In something like C++, I would never think
of shipping anything other than the exact binary I had tested ("test what
you ship, ship what you test"). It's relatively common for turning on
optimization to break something in mysterious ways in C or C++. This is
both because many compilers have buggy optimizers, and because many
programmers are sloppy about depending on uninitialized values.

But, with something like Python (i.e. high-level interpreter), I've always
assumed that turning optimization on or off would be a much safer
operation. It never would have occurred to me that I would need to test
with optimization turned on and off. Is my faith in optimization misguided?

Of course, all of the Python I write is for internal use; I haven't yet
been able to convince an employer that we should be shipping Python to
customers.

[Terry Reedy]

Re: unittest vs py.test?


"Paul Rubin" <"http://phr.cx"@NOSPAM. invalid> wrote in message
news:7x3bu8t7or .fsf@ruckus.bro uhaha.com...[color=blue]
> "Raymond Hettinger" <vze4rx4y@veriz on.net> writes:[color=green]
>> When writing a large suite, you quick come to appreciate being able
>> to use assert statements with regular comparision operators, debugging
>> with normal print statements, and not writing self.assertEqua l over and
>> over again. The generative tests are especially nice.[/color]
>
> But assert statements vanish when you turn on the optimizer. If
> you're going to run your application with the optimizer turned on, I
> certainly hope you run your regression tests with the optimizer on.[/color]

I don't see why you think so. Assertion statements in the test code make
it harder, not easier for the test to pass. Ditto, I believe, for any in
the run code, if indeed there are any.

Terry J. Reedy

[Paul Rubin]

Re: unittest vs py.test?

"Terry Reedy" <tjreedy@udel.e du> writes:[color=blue][color=green]
> > But assert statements vanish when you turn on the optimizer. If
> > you're going to run your application with the optimizer turned on, I
> > certainly hope you run your regression tests with the optimizer on.[/color]
>
> I don't see why you think so. Assertion statements in the test code make
> it harder, not easier for the test to pass. Ditto, I believe, for any in
> the run code, if indeed there are any.[/color]

If the unit tests are expressed as assert statements, and the assert
statements get optimized away, then running the unit tests on the
optimized code can obviously never find any test failures.

[Scott David Daniels]

Re: unittest vs py.test?

Paul Rubin wrote:[color=blue]
> "Terry Reedy" <tjreedy@udel.e du> writes:
>[color=green][color=darkred]
>>>But assert statements vanish when you turn on the optimizer. If
>>>you're going to run your application with the optimizer turned on, I
>>>certainly hope you run your regression tests with the optimizer on.[/color]
>>
>>I don't see why you think so. Assertion statements in the test code make
>>it harder, not easier for the test to pass. Ditto, I believe, for any in
>>the run code, if indeed there are any.[/color]
>
>
> If the unit tests are expressed as assert statements, and the assert
> statements get optimized away, then running the unit tests on the
> optimized code can obviously never find any test failures.[/color]

Any code depending upon __debug__ being 0 won't be tested. Sometimes
test structures update values as a side-effect of tracking the debugging
state. Not massively likely, but it makes for a scary environment when
your tests cannot be run on a non-debug version.

--Scott David Daniels
Scott.Daniels@A cm.Org

[Roy Smith]

Re: unittest vs py.test?

Scott David Daniels <Scott.Daniels@ Acm.Org> wrote:[color=blue]
> Any code depending upon __debug__ being 0 won't be tested. Sometimes
> test structures update values as a side-effect of tracking the debugging
> state. Not massively likely, but it makes for a scary environment when
> your tests cannot be run on a non-debug version.
>
> --Scott David Daniels
> Scott.Daniels@A cm.Org[/color]

What would happen if you defined

def verify (value):
if not value:
throw AssertionError

and then everyplace in your py.test suite where you would normally have
done "assert foo", you now do "verify (foo)"? A quick test shows that it
appears to do the right thing. I made a little test file:

------------------------------
#!/usr/bin/env python

def verify (value):
if not value:
raise AssertionError

class Test_foo:
def test_one (self):
assert 0

def test_two (self):
verify (0)
------------------------------

when I run that with "python py.test", I get two failures. When I run it
with "python -O py.test", I get one pass and one fail, which is what I
expected to get if the assert gets optimized away.

The output is a little more verbose, since it shows the exception raised in
verify(), but it gives you a stack dump, so it's not that hard to look one
frame up and see where verify() was called from.

It's interesting that given the penchant for light-weight-ness in py.test,
that the default output is so verbose (and, to my mind, confusing) compared
to unittest. I guess one could write their own output formatter and cut
down on the verbosity?

[Peter Hansen]

Re: unittest vs py.test?

Raymond Hettinger wrote:[color=blue]
> [Peter Hansen][color=green]
>>(I'm not dissing py.test, and intend to check it
>>out.[/color]
>
> Not to be disrepectful, but objections raised by someone
> who hasn't worked with both tools equate to hot air.[/color]

Not to be disrespectful either, but criticism by
someone who has completely missed my point (and apparently
not read my posts) doesn't seem entirely fair.

At no time in this thread have I objected to py.test.
The sole point of my posts has been to object to those
claiming unittest as "heavy" when in the same
breath they seem to think you have to know all kinds
of details about TestSuite, TestRunner, and TestResult
objects just to use it. I tried to demonstrate that
my way of using it appears to be on the same order of
"lightness" as some of the samples that were being used
to show how much lighter py.test was.
[color=blue]
> Until you've exercised both packages, you haven't helped the OP
> whose original request was: "Is there anybody out there who has
> used both packages and can give a comparative review?"[/color]

It seems possible to me that I might have helped him
solely by pointing out that unittest might not be so
"heavy" as some people claimed. I got the impression
that he might be swayed by some unfounded claims not
even to look further at unittest, which I felt would
be a bad thing. (Not to say your comments are unfounded,
as clearly they are valid... I happen to believe mine
have been, in this thread, as well. I guess you're
free to believe otherwise. Cheers.)

-Peter

[Roy Smith]

Re: unittest vs py.test?

Peter Hansen <peter@engcorp. com> wrote:[color=blue]
>It seems possible to me that I might have helped him
>solely by pointing out that unittest might not be so
>"heavy" as some people claimed. I got the impression
>that he might be swayed by some unfounded claims not
>even to look further at unittest, which I felt would
>be a bad thing.[/color]

I'm the "him" referred to above. I've been using unittest ever since
it was first added to the standard library (actually, now that I think
about it, I believe I may have been using it even before then).

And yes, I think unittest brings along a certain amount of baggage.
There is something attractive about having the same basic framework
work in many languages (PyUnit, JUnit, C++Unit, etc), but on the other
hand, it does add ballast. I use it, I certainly don't hate it, but
on the other hand, there are enough things annoying about it that it's
worth investing the effort to explore alternatives.

From the few days I've been playing with py.test, I think I like what
I see, but it's got other issues. The "optimizati on elides assert"
issue we've been talking about is one.

It's also neat that I can write unittest-style test classes or go the
simplier route of just writing static test functions, but there's a
certain amount of TIMTOWTDI (did I spell that right?) smell to that.

I'm also finding the very terse default output from unittest (basicly
a bunch of dots followed by "all N tests passed") highly preferable to
py.test's verbosity.

In short, I haven't made up my mind yet, but I do appreciate the input
I've gotten.