unittest vs py.test?

[Roy Smith]

I've used the standard unittest (pyunit) module on a few projects in the
past and have always thought it basicly worked fine but was just a little
too complicated for what it did.

I'm starting a new project now and I'm thinking of trying py.test
(http://codespeak.net/py/current/doc/test.html). It looks pretty cool from
the docs. Is there anybody out there who has used both packages and can
give a comparative review?

[Raymond Hettinger]

Re: unittest vs py.test?

[Roy Smith][color=blue]
> I've used the standard unittest (pyunit) module on a few projects in the
> past and have always thought it basicly worked fine but was just a little
> too complicated for what it did.
>
> I'm starting a new project now and I'm thinking of trying py.test
> (http://codespeak.net/py/current/doc/test.html). It looks pretty cool from
> the docs. Is there anybody out there who has used both packages and can
> give a comparative review?[/color]

I've used both and found py.test to be effortless and much less verbose.
For more complex testing strategies, py.test is also a winner. The generative
tests are easier to write than crafting a similar strategy for unittest.

py.test does not currently integrate well with doctest; however, that will
likely
be the next feature to be added (per holger's talk at PyCon).

For output, unittest's TextTestRunner produces good looking, succinct output
on successful tests. For failed tests, it is not bad either. In contrast,
py.test
output is more highly formatted and voluminous -- it takes a while to get used
to.

unittest users have to adapt to the internal structure of the unittest module
and
become familiar with its class structure (test fixture, test case, test suite,
and test
runner objects). py.test does a good job of hiding its implementation.

py.test is relatively new and is continuing to evolve. Support tools like a
GUI test runner are just emerging. In contrast, unittest is based on a proven
model and the code is mature.

unittest module updates come up in distinct releases, often months or years
apart. py.test is subject to constant update by subversion. Personally, I like
the continuous updates, but it could be unsettling if you're depending on it
for production code.


Raymond Hettinger

[Nigel Rowe]

Re: unittest vs py.test?

Roy Smith wrote:
[color=blue]
> I've used the standard unittest (pyunit) module on a few projects in the
> past and have always thought it basicly worked fine but was just a little
> too complicated for what it did.
>
> I'm starting a new project now and I'm thinking of trying py.test
> (http://codespeak.net/py/current/doc/test.html). It looks pretty cool
> from
> the docs. Is there anybody out there who has used both packages and can
> give a comparative review?[/color]

Have you seen Grig Gheorghiu's 3 part comparison of unittest, and py.test?

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-1-unittest.html

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-2-doctest.html

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html

--
Nigel Rowe
A pox upon the spammers that make me write my address like..
rho (snail) swiftdsl (stop) com (stop) au

[bearophileHUGS@lycos.com]

Re: unittest vs py.test?

Nigel Rowe>Have you seen Grig Gheorghiu's 3 part comparison of
unittest, and py.test?<

Very interesting articles, thank you. Testing seems something still in
quick development.

For small functions the doctests are useful, but py.test has some
advantages. Probably something even better that py.test can be
designed, taking some ideas from the doctests, or vice versa :-]
In py.test I see a couple of features useful for the Python language
too:

The raises:
py.test.raises( NameError, "self.alist.sor t(int_compare)" )
py.test.raises( ValueError, self.alist.remo ve, 6)
(A try can probably do something similar)

And the improved error messages:
"When it encounters a failed assertion, py.test prints the lines [3-4
lines?] in the method containing the assertion, up to and including the
failure. It also prints the actual and the expected values involved in
the failed assertion."

http://agiletesting.blogspot.com/2005/01/python-unit-testing-part-3-pytest-tool.html

Such things can help avoid (just for simple situations/functions!)
testing frameworks in the first place, so you can use just the normal
Python code to test other code.

Bye,
Bearophile

[Roy Smith]

Re: unittest vs py.test?

Nigel Rowe <rho@see.signat ure.invalid> wrote:
[color=blue]
> Have you seen Grig Gheorghiu's 3 part comparison of unittest, and py.test?
>
> http://agiletesting.blogspot.com/200...-unittest.html
> http://agiletesting.blogspot.com/200...2-doctest.html
> http://agiletesting.blogspot.com/200...test-tool.html[/color]

Just finished reading them now. Thanks for the pointer, they make an
excellent review of the space.

One thing that worries me a little is that all three seem to have
advantages and disadvantages, yet none is so obviously better than the
others that it stands out as the only reasonable way to do it. This means
some groups will adopt one, some will adopt another, and the world will
become fragmented.

[Grig Gheorghiu]

Re: unittest vs py.test?

In my mind, practicing TDD is what matters most. Which framework you
choose is a function of your actual needs. The fact that there are 3 of
them doesn't really bother me. I think it's better to have a choice
from a small number of frameworks rather than have no choice or have a
single choice that might not be the best for your specific environment
-- provided of course that this doesn't evolve into a PyWebOff-like
nightmare :-)

Grig

[Raymond Hettinger]

Re: unittest vs py.test?

[Roy Smith <roy@panix.co m>][color=blue]
> One thing that worries me a little is that all three seem to have
> advantages and disadvantages, yet none is so obviously better than the
> others that it stands out as the only reasonable way to do it. This means
> some groups will adopt one, some will adopt another, and the world will
> become fragmented.[/color]

Worry is a natural thing for someone with "panix" in their email address ;-)

FWIW, the evolution of py.test is to also work seemlessly with existing tests
from the unittest module.

the world diversifies, the world congeals,


Raymond Hettinger

[Jeremy Bowers]

Re: unittest vs py.test?

On Fri, 01 Apr 2005 16:42:30 +0000, Raymond Hettinger wrote:[color=blue]
> FWIW, the evolution of py.test is to also work seemlessly with existing tests
> from the unittest module.[/color]

Is this true now, or is this planned?

I read(/skimmed) the docs for py.test when you linked to the project, but
I don't recall seeing that. Certainly some of the features made me drool
but I have an investment in unittest. If I can relatively easily port them
over, I'd love to use py.test. (I don't care about a small per-file
change, it'd probably be one I can automate anyhow. But I can't afford to
re-write every test.) I didn't see anything like this in the docs, but I
may have missed it.

That'd be cool.

[Grig Gheorghiu]

Re: unittest vs py.test?

>From what I know, the PyPy guys already have a unittest-to-py.test
translator working, but they didn't check in the code yet. You can send
an email to py-dev at codespeak.net and let them know you're interested
in this functionality.

Grig

[Colin J. Williams]

Re: unittest vs py.test?

Grig Gheorghiu wrote:[color=blue]
> In my mind, practicing TDD is what matters most. Which framework you
> choose is a function of your actual needs. The fact that there are 3 of
> them doesn't really bother me. I think it's better to have a choice
> from a small number of frameworks rather than have no choice or have a
> single choice that might not be the best for your specific environment
> -- provided of course that this doesn't evolve into a PyWebOff-like
> nightmare :-)
>
> Grig
>[/color]
Grig,

Many thanks for your helpful essays.

unittest seems rather heavy. I don't like mixing tests with
documentation, it gives the whole thing a cluttered look.
Py.test is the more appealing but it doesn't appear to be
ready for packaging yet.

Thanks,

Colin W.

[Peter Hansen]

Re: unittest vs py.test?

Colin J. Williams wrote:[color=blue]
> unittest seems rather heavy. I don't like mixing tests with
> documentation, it gives the whole thing a cluttered look.[/color]

unittest can really be rather light. Most of our
test cases are variations on the following, with
primarily application-specific code added rather than
boilerplate or other unittest-related stuff:

import unittest

class TestCase(unitte st.TestCase):
def test01(self):
'''some test....'''
self.assertEqua ls(a, b)

def test02(self):
'''another test'''
self.assertRais es(Error, func, args)


if __name__ == '__main__':
unittest.main()


That's it... add testXX() methods as required and
they will be executed in sorted order (alphabetically )
automatically when you run from the command line.
The above might look excessive in comparison to the
test code, but add some real code and the overhead
quickly dwindles to negligible.

I'm a little puzzled why folks so often consider this
particularly "heavy". No need to deal with suites,
TestResult objects, etc, as others have suggested,
unless you are trying to extend it in some special
way.

-Peter

[Roy Smith]

Re: unittest vs py.test?

Peter Hansen <peter@engcorp. com> wrote:[color=blue]
> unittest can really be rather light. Most of our
> test cases are variations on the following, with
> primarily application-specific code added rather than
> boilerplate or other unittest-related stuff:
>
> import unittest
>
> class TestCase(unitte st.TestCase):
> def test01(self):
> '''some test....'''
> self.assertEqua ls(a, b)[/color]

Well, right there the "extra" stuff you needed to do (vs. py.test) was
import unittest, inherit from it, and do "self.assertEqu als" instead of
just plain assert. But (see below), that's not the big thing that attracts
me to py.test.
[color=blue]
> I'm a little puzzled why folks so often consider this
> particularly "heavy". No need to deal with suites,
> TestResult objects, etc, as others have suggested,
> unless you are trying to extend it in some special
> way.[/color]

In all but the most trivial project, you're going to have lots of tests.
Typically, each class (or small set of closely related classes) will go in
one source file, with a corresponding test file. You'll probably have
stuff scattered about a number of different directories too. That means
you need to build some infrastructure to find and run all those various
tests.

One way would be futzing with suites (which I still haven't completely
figured out). Another way would be building a hierarchical series of
dependencies in Make (or whatever build tool you use) to run your tests.
The latter is what I usually do. The idea that I can just type "python
py.test" at the top level and have it find and run everything for me just
blows me away convenience-wise.

I also like the idea that I just stick print statements into my tests and
the output automagically goes away unless the test fails. I'm a firm
believer that unit tests should NOT PRODUCE ANY OUTPUT unless they fail.
I'm working with a system now where the unit tests not only produce reams
of output, but it's also rigged to keep going in the face of failure.
Trying to find the actual error in the output is a nightmare.

[Raymond Hettinger]

Re: unittest vs py.test?

[Peter Hansen][color=blue]
> unittest can really be rather light. Most of our
> test cases are variations on the following, with
> primarily application-specific code added rather than
> boilerplate or other unittest-related stuff:
>
> import unittest
>
> class TestCase(unitte st.TestCase):
> def test01(self):
> '''some test....'''
> self.assertEqua ls(a, b)
>
> def test02(self):
> '''another test'''
> self.assertRais es(Error, func, args)[/color]
. . .[color=blue]
> I'm a little puzzled why folks so often consider this
> particularly "heavy".[/color]

unittest never felt heavy to me until I used py.test. Only then do you realize
how much boilerplate is needed with unittest. Also, the whole py.test approach
has a much simpler object model.

BTW, the above code simplifies to:

from py.test import raises
assert a == b
raises(Error, func, args)


Raymond Hettinger

[Peter Hansen]

Re: unittest vs py.test?

Raymond Hettinger wrote:[color=blue]
> BTW, the above code simplifies to:
>
> from py.test import raises
> assert a == b
> raises(Error, func, args)[/color]

This is pretty, but I *want* my tests to be contained
in separate functions or methods. The trivial amount
of extra overhead that unittest requires fits with
the way I want to write my tests, so it basically
represents zero overhead for me.

The above doesn't look like it would scale very
well to many tests in terms of maintaining some
semblance of structure and readability.

And once you add some functions or whatever to do
that, I'm still unclear on how the one or two lines
of extra code that unittest requires represents
an amount of code that really merits the label "heavy".

As for Roy's comments: I use a small internally
developed driver script which uses os.walk to find
all the files matching tests/*_unit.py or tests/story*.py
in all subfolders of the project, and which runs them
in separate processes to ensure none can pollute
the environment in which other tests run. I can
dispense with the unittest.main() call, but I like
to be able to run the tests standalone. I guess
with py.test I couldn't do that...

If py.test provides a driver utility that does
effectively this, well, that's nice for users. If
it doesn't run them as separate processes, it wouldn't
suit me anyway.

Still, it sounds like it does have a strong following
of smart people: enough to make me want to take a
closer look at it to see what the fuss is about. :-)

-Peter

[Roy Smith]

Re: unittest vs py.test?

In article <_4-dnaRR8dv8dNDfRV n-tg@powergate.ca >,
Peter Hansen <peter@engcorp. com> wrote:
[color=blue]
> As for Roy's comments: I use a small internally
> developed driver script which uses os.walk to find
> all the files matching tests/*_unit.py or tests/story*.py
> in all subfolders of the project, and which runs them
> in separate processes to ensure none can pollute
> the environment in which other tests run. I can
> dispense with the unittest.main() call, but I like
> to be able to run the tests standalone. I guess
> with py.test I couldn't do that...[/color]

Actually, I believe it does. I'm just starting to play with this, but it
looks like you can do:

py.test test_sample.py

and it'll run a single test file. I imagine you could use your os.walk
fixture in combination with this to run each test in its own process if you
wanted to.