How to use CPPUnit effectively?

[Gianni Mariani]

Re: How to use CPPUnit effectively?

jolz wrote:

>>How do I test concurrency?

>Easy - run many of them.

>
And if 1000 passes succedes that means that everything is ok? I don't
tink so.

I tink that would be a bad test.

... For example:

example of bad test removed. I can write bad tests too...

...really mean anything.

Many people beg to differ. Sure, you can write a bogus test, bets are
however that if you don't do it you're likely going to find a problem as
a customer report which is far more costly.

... Real life examples are much more
subtle.

My experience is the opposite.

... Also the same test may work on one comuter + compiler +
operating system (for example test machine) and fail on every other.

Exactly, so run it on a bunch of different test machines. I reccomend a
dual platform development for exactly this reason. Linux-posix + Win32
is a great combination.

It runs about a minute on my computer. If I would have to test this way
all functions in 500000 line application it would take a lot of time.

again - bad test. You have to run a dual CPU or better to run
multithreaded tests. Also your test has too much overhead, you create
new threads all the time. I usually create a parameterized number of
threads make them all arrive at a barrier and then unleash them all at
the same time. Then I run the test with those threads throughout the
entire test.

And usually 1 fuction have more than 1 test.
Notice that single look at the code proves that code is wrong, but
100000 passes of the test gave the illusion that everything is ok.

Again, your clever, write a better test.

This argument sounds like: "Doctor doctor, it hurts when I point a gun
at my foot and shoot". Well, duh !

>

>>How do I test proper excepion handling after disk failure?

>Have you app code read/write to an abstracted layer that you can
>simulate those kinds of failures.

>
But this way I have to change my code. Well, it may lead to a better
design, but still the test tests the abstract layer and not the actual
problem.

You can't test every situation all the time, nobody argues that. The
question is, what is the cost/benefit of doing some. Clearly if you
have no tests you're running blind. If you have some tests you're
better off and so on until there is a diminishing return. Maybe you can
use this principle. "If I write a unit test and it finds no bugs, then I
stop and move onto the next project."

>

>>How do I test efficiency of used algorithm?

>Run the test on large data sets and limit the execution time.

>
Remember - first write test, than write code. So how do you guess what
is the correct time? And what happens if test is run on a faster
machine? I guess it will pass every time even is something was broken.

Use common sense at first and be a little conservative. It depends on
the app.

>
Those were only examples. In real life there are much more of those (so
far I think that we agree that gui is not unit testable, and there are
lots of developers that do only gui).

There are frameworks for testing GUI. I'm not familiar with the state
of the art. I was hinting at being careful about being distinct from
function and appearance. I agree that you can't test if something is
aesthetic, but you can test if somthing works.

>

>>Also probanly everybody who write tests spent a lot of time debuging
>>working code and finding that the mistake was in the test.

>So ? The tests can take some time to write but it is far less expensive
>and much more useful to find the bugs before it gets to the customer.

>
But even less expensive is simply looking into the code and thinking
what to do to make it right, not what to do to make some test pass.
Test will pass anyway.

I think you're wrong here. I have yet to meet a programmer that can
write code without bugs.

[Phlip]

Re: How to use CPPUnit effectively?

jolz wrote:

>And don't thread.

>
So how do I write gui application?

Briefly, sometimes you need threads. Fight them, because they are bad. They
are like 'goto'. Don't thread just because "my function runs too long, and I
want to do something else at the same time". Fix the function so you can
time-slice its algorithm. That overwhelmingly improves your design, anyway.

The tutorials for threads often give lame examples that only need simpler
fixes. The tutorials often don't examine the full rationale for threading.
If you drive with one hand and eat a sandwich with another, you might need a
thread. Internet Explorer probably uses threads. If our apps are simpler, we
shouldn't need them, and should resist adding them until we explore all
possible alternatives.

Do you really write for example
database communicatin from gui thread?

The question here is simple: Why and how does communication block your GUIs
thread?

Winsock, for example, fixed that (for 16-bit Windows without much thread
support) by turning network input into a window message. If you have input
communication, you can often _research_ to find the right function that
multiplexes its events together with your GUI events.

If you can't find such a function, then you must use a thread, and you
should use PostMessage() or similar to send events to your GUI thread. If
you thread, make sure your code in both threads is sufficiently event-driven
that you don't need to abuse your semaphores.

Threading violates encapsulation when one thread must remain "aware" of
another thread's intimate details, just to get something done.

If so, how do you allow user to
stop the operation.

I didn't say "lock the GUI up early and often". I just said "don't thread".
There's a lot of middle ground.

So how do you guess what
is the correct time?

>>
>or you set the time comfortably close to what
>the code currently does.

>
What is the purpose of test if I write test so it allways passes?

A test here and there that can't fail is mostly harmless. Note that, under
TDD, you often write tests and show them failing at least once, before
passing them.

The purpose of my time example is to instantly fail if someone "upgrades"
the tested code and exceeds the time limit. They ought to see the failure,
Undo their change, and try again. Even if the test defends nothing, the
upgrades shouldn't be sloppy.

And what about changing hardware? Rewrite all tests?

Nobody said time-test all tests.

Change 1 global
settings and pray that it is changed with correct proportion? Or maybe
make new settings so all test passes?

To continue your digression here, one might ask what we expect to do without
tests when we change 1 global variable (besides not use global variables).
Should we debug every function in our program? Of course not, we just run
out of time. So we debug a few of them, and _then_ we pray.

That is the most common form of implementation, and it is seriously broken.

>But it must pass on every developer machine, too.

>
Does really tests that run few hours/days are run on all machines in a
company?

Yes, because developers are running all the tests before integrating their
code. If any test fails, they don't submit. So the same tests must pass on
every machine, many times a day.

What do you do before integrating? Spot check a few program features,
manually?

>GUIs are always unit testable. They are just a bad place to start
>learning!

>
I'v seen tester trying to write a gui test.

That is QA, and test-last. That means it's a professional tester, trying to
write a test that will control quality. And she or he is using the program
from the outside, not changing its source to write the test.

That's all downhill from the kind of testing we describe here. And without a
unit test layer, it's bloody impossible. But they keep trying!

Heard about others. It was
allways a lot of work with minor advantages. It was only testing
behaviour. Never how application looks. And it was an easier version -
java gui. I hava no idea how can't be it even worse in language without
gui in a standart.

And that is probably describing Capture/Playback testing, which is the worst
of the lot.

Now if I can write a test, in the same language as the tested code, that
tests how our program drives the database, or the file system, why can't I
write a test that checks how we drive the GUI? What's so special about GUIs?

>Short-term, tests make development faster by avoiding debugging,
>including
>all the "program in the debugger" that everyone does with modern editors.

>
Debugger also won't work in any of situations I presented. The fact
that test isn't worse than debugger doesn't mean that it usefull.

That is a different topic. I mean that TDD prevents the need to run the
debugger. Programming teams who switch to TDD routinely report that they
never even turn their debugger on; never set a breakpoint; never inspect a
variable.

I didn't quite get how uml causes bugs. But let's not start another
off-topic from this one.

UML doesn't cause bugs. Drawing a UML diagram, then converting the diagram
to code full of empty classes with no behavior, causes bugs. All the classes
are empty, but they inherit and delegate properly! Then you must debug, over
and over again, to put the behavior in.

>Each and every time
>you get a Green Bar, you could integrate and ship your code.

>
Again the thing that scares me the most. Green = good. Don't think
about anythink else. If it's green so it must work. Well, it doesn't.I
have nothing against tests. Sometimes they are usefull. But they don't
solve all developer's problems.

Nobody said that, so don't extrapolate from it.

Under TDD, sometimes you predict the next Bar will be Red. The point of the
exercise is you constantly predict the next Bar color, and you are almost
always right. Predictability = good. It shows that your understanding of the
code matches what the code actually does.

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Ian Collins]

Re: How to use CPPUnit effectively?

Phlip wrote:

jolz wrote:
>
>

>>>And don't thread.

>>
>>So how do I write gui application?

>
>
Briefly, sometimes you need threads. Fight them, because they are bad. They
are like 'goto'.

Even if you application benefits from concurrency?

I never used goto, but I often use threads and I don't have any issues
with doing TDD with an MT application.

--
Ian Collins.

[Phlip]

Re: How to use CPPUnit effectively?

Ian Collins wrote:

>Briefly, sometimes you need threads. Fight them, because they are bad.
>They
>are like 'goto'.

>
Even if you application benefits from concurrency?

If your app benefits from goto, use it. Various techniques have various
cost-benefit ratios.

Like goto, the cost-benefit ratio for threads is known to be suspicious.
They are hard to test for a reason; that's not the unit tests' fault!!

Fight them, by learning how to avoid them, if at all possible. Such research
will generally lead to a better event-driven design. This design, in turn,
will be easy to thread if you then prove the need.

I never used goto, but I often use threads and I don't have any issues
with doing TDD with an MT application.

That's because TDD is not the same thing as a formal QA effort to determine
your defect rate.

And I suspect there are those who have mocked their kernel and CPU just to
put unit tests on their threads!

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Phlip]

Re: How to use CPPUnit effectively?

Gianni Mariani wrote:

This argument sounds like: "Doctor doctor, it hurts when I point a gun at
my foot and shoot". Well, duh !

Make sure you test the gun first!

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Ian Collins]

Re: How to use CPPUnit effectively?

Phlip wrote:

Ian Collins wrote:
>
>

>>>Briefly, sometimes you need threads. Fight them, because they are bad.
>>>They
>>>are like 'goto'.

>>
>>Even if you application benefits from concurrency?

>
>
If your app benefits from goto, use it. Various techniques have various
cost-benefit ratios.
>

I still think the analogy is too strong, I can't think of any situation
where I'd resort to goto.

Like goto, the cost-benefit ratio for threads is known to be suspicious.
They are hard to test for a reason; that's not the unit tests' fault!!
>
Fight them, by learning how to avoid them, if at all possible. Such research
will generally lead to a better event-driven design. This design, in turn,
will be easy to thread if you then prove the need.
>

Most of the work I do does a lot of processing and I/O. The
applications invariably run on multi-core systems. Keeping all the
cores busy get the job done quicker.

While they can either be abused or used inappropriately , threads can
often simplify a design.

>

>>I never used goto, but I often use threads and I don't have any issues
>>with doing TDD with an MT application.

>
>
That's because TDD is not the same thing as a formal QA effort to determine
your defect rate.
>

I didn't say it was.

And I suspect there are those who have mocked their kernel and CPU just to
put unit tests on their threads!
>

I wouldn't go that far, but mocking the thread library to run the units
tests in a single threaded application is a useful technique. From my
experience, MT unit tests are unnecessary and tend to end in tears.

--
Ian Collins.

[Phlip]

Re: How to use CPPUnit effectively?

Ian Collins wrote:

I still think the analogy is too strong, I can't think of any situation
where I'd resort to goto.

Not even "goto another stack and CPU context"?

That's what a thread does; goes from the middle of one function to the
middle of another.

Most of the work I do does a lot of processing and I/O. The
applications invariably run on multi-core systems. Keeping all the
cores busy get the job done quicker.

My hostility to thread abuse dates from the 1980s, with single-processor
machines. I worked for many years at a shop stuck with a design invented by
a consultant who read the lame tutorials I mention, and used threads where
he should have used good structure. The threads enabled the bad structure,
which should have been fully event-driven. I got to see the drag this
imposed on our product - and all the thread bugs that went with it.

Granted, the bad structure wasn't directly the threads' fault. But a good
structure could have made the threads easier to live with - or replace!

On a modern multi-context CPU, don't thread and then lock everything down.
You will naturally have a single-threaded, multi-CPU application!

Yet if those threads are indeed independent of each other, then you merely
have a multi-process situation. That's very different from the juggling-act
you get if you use threads for trivial reasons, such as GUI concurrency!

>That's because TDD is not the same thing as a formal QA effort to
>determine
>your defect rate.

I didn't say it was.

You can TDD the functions your threads call. I suspect that writing a
TDD-style test case that juggles threads is very hard, so I would slack off
on that. (It's not strictly required to generate the code.) So TDD's
incidental power of QA will suffer in this situation.

>And I suspect there are those who have mocked their kernel and CPU just
>to
>put unit tests on their threads!
>>

I wouldn't go that far, but mocking the thread library to run the units
tests in a single threaded application is a useful technique. From my
experience, MT unit tests are unnecessary and tend to end in tears.

And still not a reason not to write unit tests! ;-)

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[Ian Collins]

Re: How to use CPPUnit effectively?

Phlip wrote:

Ian Collins wrote:
>
>

>>I still think the analogy is too strong, I can't think of any situation
>>where I'd resort to goto.

>
>
Not even "goto another stack and CPU context"?
>

Boy some people like their hairs split pretty fine :)

>
Yet if those threads are indeed independent of each other, then you merely
have a multi-process situation.

With all the benefits and pitfalls of a shared data model.

>

>>>And I suspect there are those who have mocked their kernel and CPU just
>>>to
>>>put unit tests on their threads!
>>>

>>
>>I wouldn't go that far, but mocking the thread library to run the units
>>tests in a single threaded application is a useful technique. From my
>>experience, MT unit tests are unnecessary and tend to end in tears.

>
>
And still not a reason not to write unit tests! ;-)
>

Indeed, but I'd advise against writing MT units tests. As you have
pointed out before, that level of testing doesn't help with TDD and is
best left to the acceptance tests.

--
Ian Collins.

[Gianni Mariani]

Re: How to use CPPUnit effectively?

Ian Collins wrote:
....

Indeed, but I'd advise against writing MT units tests. As you have
pointed out before, that level of testing doesn't help with TDD and is
best left to the acceptance tests.

Unless you're designing something is multi threaded by design. For
example, If the type is performing some kind of asynchronous event
management, it's kind of useless to write a TDD unit test for it that
does not show the MT aspects...

[Ian Collins]

Re: How to use CPPUnit effectively?

Gianni Mariani wrote:

Ian Collins wrote:
....
>

>Indeed, but I'd advise against writing MT units tests. As you have
>pointed out before, that level of testing doesn't help with TDD and is
>best left to the acceptance tests.

>
>
Unless you're designing something is multi threaded by design. For
example, If the type is performing some kind of asynchronous event
management, it's kind of useless to write a TDD unit test for it that
does not show the MT aspects...
>

You can test the logic without running in an MT environment.

Same with testing something like a signal handler or interrupt service
routine, you don't have to send an event to test the code. You know the
underlying OS/hardware will deliver the event, your job is to build the
code that processes it.

--
Ian Collins.

[Gianni Mariani]

Re: How to use CPPUnit effectively?

Ian Collins wrote:

Gianni Mariani wrote:

>Ian Collins wrote:
>....
>>

>>Indeed, but I'd advise against writing MT units tests. As you have
>>pointed out before, that level of testing doesn't help with TDD and is
>>best left to the acceptance tests.

>>
>Unless you're designing something is multi threaded by design. For
>example, If the type is performing some kind of asynchronous event
>management, it's kind of useless to write a TDD unit test for it that
>does not show the MT aspects...
>>

You can test the logic without running in an MT environment.
>
Same with testing something like a signal handler or interrupt service
routine, you don't have to send an event to test the code. You know the
underlying OS/hardware will deliver the event, your job is to build the
code that processes it.

That's the point. In a limited number of cases, you can't.

Show me what your timer test code looks like ?

[Ian Collins]

Re: How to use CPPUnit effectively?

Gianni Mariani wrote:

Ian Collins wrote:
>

>Gianni Mariani wrote:
>>

>>Ian Collins wrote:
>>....
>>>
>>>Indeed, but I'd advise against writing MT units tests. As you have
>>>pointed out before, that level of testing doesn't help with TDD and is
>>>best left to the acceptance tests.
>>>
>>>
>>Unless you're designing something is multi threaded by design. For
>>example, If the type is performing some kind of asynchronous event
>>management, it's kind of useless to write a TDD unit test for it that
>>does not show the MT aspects...
>>>

>You can test the logic without running in an MT environment.
>>
>Same with testing something like a signal handler or interrupt service
>routine, you don't have to send an event to test the code. You know the
>underlying OS/hardware will deliver the event, your job is to build the
>code that processes it.

>
>
That's the point. In a limited number of cases, you can't.
>

Can't what? Build the code, or test it?

Code that provides the *lowest level* threading functionality has to be
tested in an MT environment, but a) I'd argue this doesn't constitute
unit testing and b) how often do you write that type of code, once per
project? Once per platform? Once?

If you are developing an event handler, decouple the processing logic
from whatever connects it to the source of the event. Care to provide
an example of where this can't be done?

Show me what your timer test code looks like ?

I don't have any, but I'm sure the OS developer does.

--
Ian Collins.

[Gianni Mariani]

Re: How to use CPPUnit effectively?

Ian Collins wrote:

Gianni Mariani wrote:

>Ian Collins wrote:
>>

>>Gianni Mariani wrote:
>>>
>>>Ian Collins wrote:
>>>....
>>>>
>>>>Indeed, but I'd advise against writing MT units tests. As you have
>>>>pointed out before, that level of testing doesn't help with TDD and is
>>>>best left to the acceptance tests.
>>>>
>>>Unless you're designing something is multi threaded by design. For
>>>example, If the type is performing some kind of asynchronous event
>>>management , it's kind of useless to write a TDD unit test for it that
>>>does not show the MT aspects...
>>>>
>>You can test the logic without running in an MT environment.
>>>
>>Same with testing something like a signal handler or interrupt service
>>routine, you don't have to send an event to test the code. You know the
>>underlying OS/hardware will deliver the event, your job is to build the
>>code that processes it.

>>
>That's the point. In a limited number of cases, you can't.
>>

Can't what? Build the code, or test it?

Appreciate the implications of the design.

>
Code that provides the *lowest level* threading functionality has to be
tested in an MT environment, but a) I'd argue this doesn't constitute
unit testing and b) how often do you write that type of code, once per
project? Once per platform? Once?

It depends on the asynchronous nature of the problem you're trying to solve.

>
If you are developing an event handler, decouple the processing logic
from whatever connects it to the source of the event. Care to provide
an example of where this can't be done?

Sure, the Austria C++ thread pool design unit test. Maybe the Austria
C++ timer unit test. Both of these tests must show how you deal with
the inherent multi threaded nature of the service you're designing
otherwise you're not really seeing all the design issues.

>
>

>Show me what your timer test code looks like ?

>
I don't have any, but I'm sure the OS developer does.

Actually you make my point, if they had a unit test before they coded
they probably would have changed the design. The O.S. timer facilities
are very difficult to use correctly. For example, most OS timer
services come back with a signal. This means you can't do anything that
takes a resource. About the only thing you can do safely is set a flag
or call some other "signal safe" OS calls (like write a byte to a pipe).

The code below shows what the Austria timer unit test "kernel" looks
like. Event(...) gets called back in the "timer service" thread. It
doesn't look like it but it is inherently multi threaded !


class MyTimerClient
: public TimerClient_Bas ic
{

TimerLog & m_log;
int m_client_number ;

public:

MyTimerClient( TimerLog & io_log, int i_client_number )
: m_log( io_log ),
m_client_number ( i_client_number )
{
}

protected:

virtual bool Reschedule(
const TimeStamp & i_current_time,
TimeStamp & o_reschedule_ti me
) = 0;

private:

virtual bool Event(
const TimeStamp & i_current_time,
TimeStamp & o_reschedule_ti me
)
{
m_log.push_back ( m_client_number , ThreadSystemTim e() );
return Reschedule( i_current_time, o_reschedule_ti me );
}

};

[Phlip]

Re: How to use CPPUnit effectively?

Gianni Mariani wrote:

>Same with testing something like a signal handler or interrupt service
>routine, you don't have to send an event to test the code. You know the
>underlying OS/hardware will deliver the event, your job is to build the
>code that processes it.

>
That's the point. In a limited number of cases, you can't.

CppUnit and its ilk can be used for QA testing, but that's not what we've
been discussing. We are discussing TDD, where we write tests that help force
our lines of code to exist. This helps to refactor our designs as they grow.

If we are not writing a kernel, we don't need to TDD its event systems. We
won't refactor our kernel, so gaps in our test coverage are very low risk.

So you can safely rely on a policy "wait till we have a bug there". (Note
this is the policy that test-free programs already follow - for everything!)

Show me what your timer test code looks like ?

TEST_(TestDialo g, SetTimer)
{
CButton slowButton = m_aDlg.GetDlgIt em(IDC_SLOW_OPE RATION);
slowButton.Send Message(BM_CLIC K);
BOOL thereWasaTimerT oKill = m_aDlg.KillTime r(0);
CPPUNIT_ASSERT( thereWasaTimerT oKill);
}

In Win32 (under WTL), we test that our BM_CLICK handler sets a timer by
killing that timer.

I'm aware that doesn't answer the exact question, but I already had this
answer available!

--
Phlip
http://www.greencheese.us/ZeekLand <-- NOT a blog!!!

[jolz]

Re: How to use CPPUnit effectively?

And if 1000 passes succedes that means that everything is ok? I don't

tink so.

>
I tink that would be a bad test.
>

... For example:

example of bad test removed. I can write bad tests too...

Sure it is bad. But how would look a good test that shows that c1 and
c2 may be equal? My point is that such a test doesn't exist.

You can't test every situation all the time, nobody argues that.

Somebody does:
1. You should write tests for each new line (or two) of the code.
Never write

new code without a failing test.

2. If you can't think of the test, how will you think of the line of
code?

I'm just trying to explain that "You can't test every situation all the
time", not that all tests are evil.
I also think that there are more situations that can't be tested than
those that can, but of course I may be wrong here.