How to use CPPUnit effectively?

[Gianni Mariani]

Re: How to use CPPUnit effectively?

Roland Pibinger wrote:
...

>So, what's so "repelling" ?

>
I've not been aware of your Austria framework (even though I live in
Austria. BTW, why haven't you named it Italia?). My favorite unit test
'framework' still is the Standard 'assert' (admittedly a macro in most
implementations ).

Because it started off as a country that started with "A". The next
project is Borneo. Besides I think the Austrian's are the most fun
people in Europe.

Hmm. AT_TCAssert gives you the line number and file name (and stack
trace if you want) of the failure and allows you to run the rest of the
tests. You get a full report on all the test failures not just the
first one to fail.

Anyhow, if you're happy with assert, more power to you !

[Phlip]

Re: How to use CPPUnit effectively?

Roland Pibinger wrote:

I guess a good C++ unit test 'framework' is yet to be written. Most of
the available frameworks are cluttered up with macros which IMO is
rather repelling.

One Noel Llopis once wrote a book on C++ for game programmers. You should
have seen how he twitched when first I showed him my TEST_() macro (based on
the CppUnitLite TEST() macro). Game programmers are known to abuse macros in
false pursuit of efficiency, and I expect his book ranted against them.

And now his UnitTest++ has exactly the same suite of macros.

In a C language, thou shalt write an assertion with a macro. Even the
various published style guides say so. Further, you can generally write a
better assertion than many vaunted soft languages can support.

In C++, use a macro for any of these techniques:

- conditional compilation
- extracting compiler-defined constants (like __LINE__)
- stringerization
- token pasting

You need all of them for a powerful and flexible unit test rig.

People who are repelled by any language technique, without actually
understanding its uses and costs, are not on the path to wisdom.

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Dave Steffen]

Re: How to use CPPUnit effectively?

"Hooyoo" <zhao_huyong@12 6.comwrites:

How to use CPPUnit effectively? When I want to use CPPUnit, I
always fint it useless and waste my time. I think that's mainly
because I don't know how to use CPPUnit effectively. So if somebody
here has some experience, please share it to me.

Our first cut at a home-grown unit test framework was very much like
JUnit / CPPUnit. The amount of extra verbage needed for even the
simplest tests was a major problem.

We moved to Boost's unit test framework about a year ago, and it's
_much_ better. None of this "unit tests are classes that inherit
from..." stuff, each unit test is a free function.

Roland P. won't like it, because it's replete with macros. :-) But
the "auto unit test" facility is very handy, and does a very, very
important thing: it makes writing unit tests extremely cheap.

I recommend that you check it out.

----------------------------------------------------------------------
Dave Steffen, Ph.D. Disobey this command!
Software Engineer IV - Douglas Hofstadter
Numerica Corporation
dg@steffen a@t numer@ica d@ot us (remove @'s to email me)

[Phlip]

Re: How to use CPPUnit effectively?

Dave Steffen wrote:

We moved to Boost's unit test framework about a year ago, and it's
_much_ better. None of this "unit tests are classes that inherit
from..." stuff, each unit test is a free function.

Fixtures are a unit test Best Practice, so test rigs should enable them.
Both UnitTest++ and my TEST_() enable them without requiring them. Boost's
doesn't. But indeed do check it out; it _is_ Boost, after all!

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Roland Pibinger]

Re: How to use CPPUnit effectively?

On 31 Oct 2006 13:44:44 -0700, Dr. Dave Steffen wrote:

We moved to Boost's unit test framework about a year ago, and it's
_much_ better. None of this "unit tests are classes that inherit
from..." stuff, each unit test is a free function.
>
Roland P. won't like it, because it's replete with macros. :-) But
the "auto unit test" facility is very handy, and does a very, very
important thing: it makes writing unit tests extremely cheap.

If you are fond of boosted stuff TUT might interest you (not that I
recommend it). It uses built-in C++ macros (a.k.a. templates) instead
of preprocessor macros: http://tut-framework.sourceforge.net/

Best regards,
Roland Pibinger

[Dave Steffen]

Re: How to use CPPUnit effectively?

"Phlip" <phlipcpp@yahoo .comwrites:

Dave Steffen wrote:
>

We moved to Boost's unit test framework about a year ago, and it's
_much_ better. None of this "unit tests are classes that inherit
from..." stuff, each unit test is a free function.

>
Fixtures are a unit test Best Practice, so test rigs should enable them.

Yes, good point...

Both UnitTest++ and my TEST_() enable them without requiring them. Boost's
doesn't. But indeed do check it out; it _is_ Boost, after all!

... and using Boost::UnitTest I miss them on occasion. IIRC there
are ways to do _whole test suite_ setups and fixtures, but I haven't
hacked around enough to use them yet.

The thing that makes Boost's unit test framework so near and dear to
my heart is the extreme lack of typing overhead. I credit this for
the large number of unit tests that people in my organization are
writing (on their own, without prodding from anyone) these days; as
opposed to our old JUnit-ish thing, which took _loads_ of typing to
get anywhere, so people just tended to ignore it.

----------------------------------------------------------------------
Dave Steffen, Ph.D. Disobey this command!
Software Engineer IV - Douglas Hofstadter
Numerica Corporation
dg@steffen a@t numerica d@ot us (remove @'s to email me)

[jolz]

Re: How to use CPPUnit effectively?

You should write tests for each new line (or two) of the code. Never write

new code without a failing test.

Allways?
How do I test concurrency?
How do I test that after adding new button my gui looks ugly?
How do I test proper excepion handling after disk failure?
How do I test efficiency of used algorithm?
There is a lot of code that making an unit test for is useless.
Unit test are really nice especially when code changes - if test passes
after the change that's a hint that maybe nothing was broken. But they
are also dangerous. Many times I'v heard that if test passes than
everything works. Actually often it means that there are to little
tests. Or maybe every single function works, but together they don't
work (especially true in multithread environment). Or maybe new
functionality that was added introduces several hundred of new states
in application that wans't considered when existing test was written.
Also probanly everybody who write tests spent a lot of time debuging
working code and finding that the mistake was in the test.
Yes, test are important, but I tkink that they often overrated. I'v
often heard that TDD's goal is to make it green. Beware of red. Make it
as simple as possible. But I don't recall hearing the word "design".

[Gianni Mariani]

Re: How to use CPPUnit effectively?

jolz wrote:

>You should write tests for each new line (or two) of the code. Never write
>new code without a failing test.

>
Allways?
How do I test concurrency?

Easy - run many of them.

How do I test that after adding new button my gui looks ugly?

I don't know what you talk about - it looks good to me :-)

How do I test proper excepion handling after disk failure?

Have you app code read/write to an abstracted layer that you can
simulate those kinds of failures. BTW - This would not be a test that
would be high on my priority list. If the unlikely event the customer
has a disk failure, it's unlikely that my code is going to be the only
one to blow up.

The idea here is be selective of the failure modes you test for. Make
sure they are going to be a priority to the customer.

How do I test efficiency of used algorithm?

Run the test on large data sets and limit the execution time.

There is a lot of code that making an unit test for is useless.
Unit test are really nice especially when code changes - if test passes
after the change that's a hint that maybe nothing was broken. But they
are also dangerous. Many times I'v heard that if test passes than
everything works. Actually often it means that there are to little
tests. Or maybe every single function works, but together they don't
work (especially true in multithread environment).

This one is easy - test them in a multithreaded environment. Many of
the classes I create that have to be thread safe are subject to what I
term a "Monte Carlo" test where I nail them with various random
requests. For example there is a "timer" module. I have tests where
timers are added, removed, fail on and on over a varying number of threads.

In an automated build system, the build run continuously and
occasionally there is a "failure" so we grab the core dumps and viola,
sometimes you nail it.

The motto of the story is, write many tests and run them as often as you
can and have a way to check the results of failure - usually with a core
dump. Many times I have resorted to having the program suspend itself
on failure and going into the automated test system and attaching a
debugger to the suspended code (usually on Windows).

Also, by doing an automated continuous build you see which changes to
the software possibly caused the problem.

Unit tests don't have to be small. I remember once where the whole
application was a class I could instantiate (only ever used const
globals), so I did one of my monte carlo tests and - ooh it failed due
to some very esoteric shut down before being fully initialized bug.
Everyone, including myself, thought this would never happen in real
life. .... Guess again. On the initial release that's exactly one of
the critical bugs that came back, which btw we knew what it was because
we did the test.

... Or maybe new
functionality that was added introduces several hundred of new states
in application that wans't considered when existing test was written.

Write more tests to try to make those states happen.

Also probanly everybody who write tests spent a lot of time debuging
working code and finding that the mistake was in the test.

So ? The tests can take some time to write but it is far less expensive
and much more useful to find the bugs before it gets to the customer.

Yes, test are important, but I tkink that they often overrated. I'v
often heard that TDD's goal is to make it green. Beware of red. Make it
as simple as possible. But I don't recall hearing the word "design".

What is this thing you call "design" ?

What is it's deliverable ? i.e. when you have finishes a design, what
is it that you have ?

If you think TDD, your design is the unit tests + the interface to the
application. BTW - not all the unit tests, just the ones that help
underpin the interface. For instance I would not do a monte carlo
stress test at the design stage, but I would do it for verification.

While you write the code, you're doing to come across issues that may
mean changing the interfaces and hence the unit tests so having a boat
load of tests to fix when your interfaces changes does not help.

[VJ]

Re: How to use CPPUnit effectively?

Phlip wrote:

Hooyoo wrote:
>
>

>>How to use CPPUnit effectively?

>
>
Use UnitTest++. It's much leaner and easier to use:
>

http://unittest-cpp.sourceforge.net/

>
CppUnit is stuffed with features you won't need, and they often get in the
way of the few you do.
>

We are using http://cxxtest.sourceforge.net/guide.html and it is good
and easy to understand

[jolz]

Re: How to use CPPUnit effectively?

How do I test concurrency?

>
Easy - run many of them.

And if 1000 passes succedes that means that everything is ok? I don't
tink so. For example:

#include <boost/thread/thread.hpp>

int counter = 0;
int c1;
int c2;

void t1() {
for (int i = 0; i < 1000; i++);
c1 = counter;
++counter;
}

void t2() {
for (int i = 0; i < 1000; i++);
c2 = counter;
++counter;
}

int main() {
for (int i = 0; i < 100000; i++) {
boost::thread thrd1(&t1);
boost::thread thrd2(&t2);
thrd1.join();
thrd2.join();
assert(c1 != c2);
}
}

passes, but the code is obviously wrong. And even if test woulh have
failed it doen't really mean anything. Real life examples are much more
subtle. Also the same test may work on one comuter + compiler +
operating system (for example test machine) and fail on every other.
It runs about a minute on my computer. If I would have to test this way
all functions in 500000 line application it would take a lot of time.
And usually 1 fuction have more than 1 test.
Notice that single look at the code proves that code is wrong, but
100000 passes of the test gave the illusion that everything is ok.

How do I test proper excepion handling after disk failure?

Have you app code read/write to an abstracted layer that you can
simulate those kinds of failures.

But this way I have to change my code. Well, it may lead to a better
design, but still the test tests the abstract layer and not the actual
problem.

How do I test efficiency of used algorithm?

Run the test on large data sets and limit the execution time.

Remember - first write test, than write code. So how do you guess what
is the correct time? And what happens if test is run on a faster
machine? I guess it will pass every time even is something was broken.

Those were only examples. In real life there are much more of those (so
far I think that we agree that gui is not unit testable, and there are
lots of developers that do only gui).

Also probanly everybody who write tests spent a lot of time debuging
working code and finding that the mistake was in the test.

>
So ? The tests can take some time to write but it is far less expensive
and much more useful to find the bugs before it gets to the customer.

But even less expensive is simply looking into the code and thinking
what to do to make it right, not what to do to make some test pass.
Test will pass anyway.

Yes, test are important, but I tkink that they often overrated. I'v
often heard that TDD's goal is to make it green. Beware of red. Make it
as simple as possible. But I don't recall hearing the word "design".

>
What is this thing you call "design" ?

TDD says - write as little as posible to make something work. And don't
do anything that is not necessary. Than add something to make something
else work. Almoust always change the first code to cooperate with new
code. But often one nows that for example creating cont string will be
eventually necessary. Why don't start with this? UML is one way to
deign. But it usually doesn't focus on details. Unit tests focuses only
on details and nothing else. I think none of them is ultimate solution.

I think that unit test may be dangerous exactly because of everyting
you wrote. I get the impression that you think that if test passes that
it means that application is bug free. That may be true for very
limited number of applications (for example library that does matrix
calculations) but not in many real life applications.

[Phlip]

Re: How to use CPPUnit effectively?

jolz wrote:

Remember - first write test, than write code.

And don't thread. Write an event-driven architecture that time-slices
things. That is, in turn, easy to switch to threading if you find a real
need. That real need will drive the test concerns. And you can't test-in
thread robustness; you must review and model it.

So how do you guess what
is the correct time?

You ask your business analyst, or you set the time comfortably close to what
the code currently does. Tests don't (generally) validate features. They
generally detect if your code strays from its original activities.

And what happens if test is run on a faster
machine? I guess it will pass every time even is something was broken.

But it must pass on every developer machine, too.

Those were only examples. In real life there are much more of those (so
far I think that we agree that gui is not unit testable,

GUIs are always unit testable. They are just a bad place to start learning!

http://www.zeroplayer.com/cgi-bin/wiki?TestFirstUserInterfaces

and there are
lots of developers that do only gui).

Oh, I only do the variables that start with [L-Z], but I don't put that on
my resume!

>So ? The tests can take some time to write but it is far less expensive
>and much more useful to find the bugs before it gets to the customer.

>
But even less expensive is simply looking into the code and thinking
what to do to make it right, not what to do to make some test pass.
Test will pass anyway.

Short-term, tests make development faster by avoiding debugging, including
all the "program in the debugger" that everyone does with modern editors.

Long-term, tests make development faster by recording decisions about
behavior in a place where they are easy to review, and generally automatic.
They shorten the QA pipeline between programmers and users, and they help
turn the decision to release into strictly a business decision.

Yes, test are important, but I tkink that they often overrated. I'v
often heard that TDD's goal is to make it green. Beware of red. Make it
as simple as possible. But I don't recall hearing the word "design".

Uh, TDD used to stand for "Test Driven Design". It now stands for
"Developmen t", because TDDers try to express every aspect of development -
analysis, design, coding, integration, productization, etc. - as a form of
test.

>What is this thing you call "design" ?

>
TDD says - write as little as posible to make something work. And don't
do anything that is not necessary. Than add something to make something
else work. Almoust always change the first code to cooperate with new
code. But often one nows that for example creating cont string will be
eventually necessary. Why don't start with this? UML is one way to
deign. But it usually doesn't focus on details. Unit tests focuses only
on details and nothing else. I think none of them is ultimate solution.

UML is where you draw a bunch of circles and arrows, each one of which
represents an opportunity for a bug. If we treat UML as a methodology (which
it is not), then we might draw a design, then code the design without any
behavior. So it's all bugs. Then we go thru whomping down each bug by adding
the behaviors to each box and arrow.

Software is about behavior. Design is the support for behavior; it's less
important. It should be always easy to change. (I read a nice job listing
recently, that said "Refactorin g is rare due to good design practices". My
clairvoyancy skills are not as accurate as they require!)

So implement the behavior first, then refactor the design while preserving
that behavior.

I think that unit test may be dangerous exactly because of everyting
you wrote. I get the impression that you think that if test passes that
it means that application is bug free. That may be true for very
limited number of applications (for example library that does matrix
calculations) but not in many real life applications.

It shows the program passed tests that it used to pass. Such tests might
have been reviewed. The behaviors they test were certainly reviewed. So
preserving the tests will preserve this investment into reviewing the
behavior. This frees up everyones time, to concentrate on implementing
features and exploratory testing.

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Phlip]

Re: How to use CPPUnit effectively?

jolz wrote:

>You should write tests for each new line (or two) of the code. Never
>write
>new code without a failing test.

Allways?

If you can't think of the test, how will you think of the line of code?

The test doesn't need to perfectly prove the line will work. In the
degenerate case, the test simply checks the line exists. Another line could
fool the test. Don't do that, and add different tests with different attack
angles, to triangulate the code you need.

How do I test concurrency?

Developer tests are only import for rapid development. Not for (shocked
gasp) proving a program works. If you simply must go concurrent, write a
good structure for your semaphores, enforce this _structure_ with tests, and
then soak-test your final application, just to check for CPU bugs and such.

How do I test that after adding new button my gui looks ugly?

If that's important (it usually _isn't_), you make reviewing the GUI very
easy:

http://www.zeroplayer.com/cgi-bin/wiki?TestFlea

How do I test proper excepion handling after disk failure?

Sometimes you let things like this slide. Yes, you occassionally write a
line of code without a test. The difference is Ayou could have written the
test, and Byour code follows community agreements about robustness.

For example, an in-house tool application should simply die and tell the
user to report to the help-desk. A commercial application, by contrast,
shouldn't make those assumptions.

A test on disk failure should use a Mock Object to mock out the file system.
That will return an error on command, so a test can trivially ensure the
error gets handled correctly.

How do I test efficiency of used algorithm?

You don't need to do that to test-first each line of the algorithm. You need
to know what the algorithm is, so you can implement it under test. If you
don't know the algorithm yet, test-first might help discover it, but you
still need other design techniques.

If the algorithm is important, then it should have tests that catch when any
detail of its internal operations change. So only implement important things
via test-first.

There is a lot of code that making an unit test for is useless.

Uh, then why write the code?

If you assume a unit test is expensive, then you will never get over that
curve and make them cheap.

If, instead, you start with unit tests, they accelerate development. For
each new line of code, the unit test is easier to write than debugging that
line would be.

Don't tell my bosses this, but I have programs that I almost never manually
drive. So, for me, there's a lot of code that manual testing is useless!

Unit test are really nice especially when code changes - if test passes
after the change that's a hint that maybe nothing was broken. But they
are also dangerous.

So get them reviewed as often as you review the code. Also, tests make an
excellent way to document the code, and you should help your business
analysts review this code.

Many times I'v heard that if test passes than
everything works.

Nobody should believe that, even when you have 100% statement coverage, and
it appears true!

Actually often it means that there are too little
tests. Or maybe every single function works, but together they don't
work (especially true in multithread environment).

Don't multithread. Do write tests that re-use lots of objects in
combination. (Don't abuse Mock Objects, for example.) Your design should be
so loosely coupled that you can create any object under test without any
other object.

Or maybe new
functionality that was added introduces several hundred of new states
in application that wans't considered when existing test was written.

Yep. But the odds you _know_ you did that are very high.

Also probanly everybody who write tests spent a lot of time debuging
working code and finding that the mistake was in the test.

Uh, "debugging" ?

Under test-first, if the tests fail unexpectedly, you hit Undo until they
pass. That's where most debugging goes.

If you try the feature again, and the tests fail again, you Undo and then
start again, at a smaller scale, with a _smaller_ change. If this passes, do
it again. You might find the error in the tests like this, because it will
relate to the line of code you are trying to change.

This is like debugging, but with one major difference. Each and every time
you get a Green Bar, you could integrate and ship your code.

So a test-rich environment is like having a big button on your editor called
"Kill Bug". Invest in the power of that button.

Yes, test are important, but I tkink that they often overrated. I'v
often heard that TDD's goal is to make it green. Beware of red. Make it
as simple as possible. But I don't recall hearing the word "design".

If you are reading the blogs and such, they often preach to the choir. They
know you know it's about "design".

Google for "emergent design" to learn the full power of the system.

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Phlip]

Re: How to use CPPUnit effectively?

Dave Steffen wrote:

>Fixtures are a unit test Best Practice, so test rigs should enable them.

>
Yes, good point...
>

>Both UnitTest++ and my TEST_() enable them without requiring them.
>Boost's
>doesn't. But indeed do check it out; it _is_ Boost, after all!

>
... and using Boost::UnitTest I miss them on occasion. IIRC there
are ways to do _whole test suite_ setups and fixtures, but I haven't
hacked around enough to use them yet.
>
The thing that makes Boost's unit test framework so near and dear to
my heart is the extreme lack of typing overhead.

Boost made a module that's easy to type??! ;-)

Seriously, all the TEST() macro systems limit typing in some way, better
than raw CppUnit. However...

I credit this for
the large number of unit tests that people in my organization are
writing (on their own, without prodding from anyone) these days; as
opposed to our old JUnit-ish thing, which took _loads_ of typing to
get anywhere, so people just tended to ignore it.

That is /so/ much more important than convenient fixture systems! (And you
could trivially add them, with another TEST_FIXTURE() macro.)

Carry on!

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Phlip]

Re: How to use CPPUnit effectively?

VJ wrote:

We are using http://cxxtest.sourceforge.net/guide.html and it is good and
easy to understand

I don't understand the brief amount of documentation I read.

Given test<3>, do I guess right that we have to increment that 3 by hand,
each time we

I want to smoke crack while coding, and having to remember what number I'm
up to interferes with my lifestyle. I can't even think of a macro that fixes
this (with a compile-time constant).

So does cxxtest really make me increment that number? Or should I, like,
read more of the documentation before passing judgement?

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[jolz]

Re: How to use CPPUnit effectively?

And don't thread.

So how do I write gui application? Do you really write for example
database communicatin from gui thread? If so, how do you allow user to
stop the operation. Or resize window. Baically do anything other than
staring in a gray window. Application logic should be in a network
thread? Or maybe parallel algorithm should be in a single thread?

So how do you guess what
is the correct time?

>
or you set the time comfortably close to what
the code currently does.

What is the purpose of test if I write test so it allways passes?
And what about changing hardware? Rewrite all tests? Change 1 global
settings and pray that it is changed with correct proportion? Or maybe
make new settings so all test passes?

But it must pass on every developer machine, too.

Does really tests that run few hours/days are run on all machines in a
company?

GUIs are always unit testable. They are just a bad place to start learning!

I'v seen tester trying to write a gui test. Heard about others. It was
allways a lot of work with minor advantages. It was only testing
behaviour. Never how application looks. And it was an easier version -
java gui. I hava no idea how can't be it even worse in language without
gui in a standart.

Short-term, tests make development faster by avoiding debugging, including
all the "program in the debugger" that everyone does with modern editors.

Debugger also won't work in any of situations I presented. The fact
that test isn't worse than debugger doesn't mean that it usefull.

UML is where you draw a bunch of circles and arrows, each one of which
represents an opportunity for a bug. If we treat UML as a methodology (which
it is not), then we might draw a design, then code the design without any
behavior. So it's all bugs.

I didn't quite get how uml causes bugs. But let's not start another
off-topic from this one.

It shows the program passed tests that it used to pass.

Yes. And it is the only thing that makes test usefull for me. It means
that I didn't screwed up to much. But It certainly doesn't mean that
application works. It doesn't even mean that it isn't worse that
before.

If you can't think of the test, how will you think of the line of code?

I'v already presented a code that passes test and doesn't work. I know
how to correct the code, but have no idea how to write a test that
validetes the code.

Each and every time
you get a Green Bar, you could integrate and ship your code.

Again the thing that scares me the most. Green = good. Don't think
about anythink else. If it's green so it must work. Well, it doesn't.I
have nothing against tests. Sometimes they are usefull. But they don't
solve all developer's problems.