A simple unit test framework

[Ian Collins]

Re: A simple unit test framework

James Kanze wrote:

On May 6, 9:46 pm, Ian Collins <ian-n...@hotmail.co mwrote:

>James Kanze wrote:

>>On May 6, 3:31 am, Ian Collins <ian-n...@hotmail.co mwrote:
>>>Gianni Mariani wrote:

>

>>>>I have met very few customers that know what a spec is even if it
>>>>smacked them up the side of the head.

>

>>>Welcome to the club!

>

>>>>Sad. Inevitably it leads to pissed off customer.

>

>>>Any agile process (XP, Scrum or whatever) is ideal for this situation.

>

>>If your goal is to rip off the customer, yes.

>
[First, a meta-comment: I am, of course, exagerating to make
a point, and I don't really suspect Ian of trying to use any
technique to rip off his customers.]
>

>So by helping them to get what they really wanted, rather than forcing
>them to commit to what they thought the wanted, I'm ripping them off?

>
How does testing help the customer to get what he really wants?

By being an integral part of the process.

>
Roughly speaking, the user interface needs prototyping; the rest
of the code needs specification. But I don't really see a role
for testing.

Not every customer is in a position to provide specifications and those
that think they can often change their minds once the development has
started.

>The person I'm ripping off is me, I'm doing my self out of all the bug
>fixing and rework jobs.

>

>Man you have a strange view of customer focused development.

>
Customer focused means talking to the customer in his language,
not in yours. A test suite doesn't do this. Getting a customer
to sign off a project on the basis of a test that he's not
capable of understanding is not, IMHO, showing him much respect.
>

If you were to take the time to look into XP or Scrum, you would see
that you are to some extent preaching to the choir. That's why there
are (web driven) test suites like FIT or Selenium that are specifically
designed for the customer to understand and in some cases, use them
selves. Don't forget the customer acceptance tests are system
behavioral tests, not code unit tests (unplug module A and module A
missing trap is sent kind of tests). How the application behaves is
what the customer wants to see.

--
Ian Collins.

[James Kanze]

Re: A simple unit test framework

On May 6, 9:59 pm, "Phlip" <phlip...@yahoo .comwrote:

Ian Collins wrote:

Have you tried it? Not having to hold code reviews was one of the
biggest real savings for us.

What you are up against here is Kanze is one of the most aggressive and
competent reviewers out there, and he leads by reviewing. This explains his
devastating accuracy on C++ questions. So by claiming two dumb people can
get by with pairing and TDD, instead of submitting their code to him for
review, you are taking him out of his Comfort Zone.

Thanks for the complement, but it's not personal, and it's not
about me. The important aspect of code review (one of them,
anyway) is the fresh, unjaundiced viewpoint; the person looking
at the code has not seen it before. Another, perhaps even more
important, is that the programmers write their code to be
reviewed. Someone who has never seen the code before must be
capable of understanding it quickly and without undo effort, and
it must be "obvious" why it works, and that it cannot fail. If,
during review, the author has to explain why the code works, the
code fails review.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[James Kanze]

Re: A simple unit test framework

Gianni Mariani wrote:

James Kanze wrote:

On May 6, 3:05 am, Gianni Mariani <gi3nos...@mari ani.wswrote:

James Kanze wrote:

...

>>The latest trends are to write tests first which demonstrates the
>>requirement s, then code (classes+method s).

>The latest trend where? Certainly not in any company concerned
>with good management, or quality software.

Look up TDD.

I'm familiar with the theory. Regretfully, it doesn't work out
in practice.

What part ?

What do you mean, what part? TDD doesn't result in sufficient
quality when used to develop code.

>>In this case you will not
>>have to do a coverage, but it is a plus. This way, the code you write
>>will be minimal and easier to understand and maintain.

>And will not necessarily meet requirements, or even be useful.

Actually, it does meet the requirements by definition since the test
case demonstrates how the requirements should be met.

Bullshit. I've seen just too many cases of code which is wrong,
but for which no test suite is capable of reliably triggering
the errors.

Example ?

Bug #21334 in the std::string implementation in g++.

Most of the problems in DCL.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[James Kanze]

Re: A simple unit test framework

On May 7, 10:02 am, anon <a...@no.nowrot e:

James Kanze wrote:

On May 5, 2:31 am, Ian Collins <ian-n...@hotmail.co mwrote:

Pete Becker wrote:
>Ian Collins wrote:
>>Pete Becker wrote:

>>If you apply TDD correctly, you only write code to pass tests, so all of
>>your code is covered.

>Suppose you're writing test cases for the function log, which calculates
>the logarithm of its argument. Internally, it will use different
>techniques for various ranges of argument values. But the specification
>for log, of course, doesn't tell you this, so your test cases aren't
>likely to hit each of those ranges, and certainly won't make careful
>probes near their boundaries. It's only by looking at the code that you
>can write these tests.

Pete, I think you are missing the point of TDD.

It's easy for those unfamiliar with the process to focus on the "T" and
ignore the "DD". TDD is a tool for delivering better code, the tests
drive the design, they are not driven by it.

Which, of course, is entirely backwards.

It is, but you get better code

Better code than what?

In practice, if a coder wants to write the unit tests first, no
one will stop him; his deliverable consists of both the code and
the unit tests. But there's absolutly no advantage in doing so,
and most people I've worked with feel more comfortable writing
the actual code first.

So if I were tasked with
writing he function log, I'd start with a simple test, say log(10) and
then add more tests to cover the full range of inputs.

That is, of course, one solution. It's theoretically possible
for log, but it will take several hundred centuries of CPU time,
which means that it's not very practical. In practice, the way
you verify that a log function is correct is with code review,
with testing of the border cases (which implies the what Pete is
calling white box testing), to back up the review.

If you were to write a log() function, would you test it
against all floats?

That's precisely my point: you can't. So you have to analyse
the code, to determine where the critical boundaries are, and
test the limit cases. And of course, you have to document that
analysis, because the code reviewers will definitly want to
review it.

These tests
would specify the behavior and drive the internals of the function.

In this case, I think that the behavior is specified before
hand. It is a mathematical function, after all, and we can know
the precise result for every possible input. In practice, of
course, it isn't at all possible to test it, at least not
exhaustively.

Remember, if code isn't required to pass a test, it doesn't get written.

So your log function only has to produce correct results for the
limited set of values you use to test it? I hope I never have
to use a library you wrote.

If you take some random numbers (for example 0.0, 0.007, 0.29, 0.999,
1.0, 1.0001, 1.9900, 555.0, 999999.0) and your log function for these
numbers gives correct results (with small enough error) you can be sure
your log function is good

Obviously, you don't know how floating point arithmetic works,
or you wouldn't make such a stupid statement.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[anon]

Re: A simple unit test framework

James Kanze wrote:

On May 7, 10:02 am, anon <a...@no.nowrot e:

>James Kanze wrote:

>>On May 5, 2:31 am, Ian Collins <ian-n...@hotmail.co mwrote:
>>>Pete Becker wrote:
>>>>Ian Collins wrote:
>>>>>Pete Becker wrote:

>

>>>>>If you apply TDD correctly, you only write code to pass tests, so all of
>>>>>your code is covered.

>

>>>>Suppose you're writing test cases for the function log, which calculates
>>>>the logarithm of its argument. Internally, it will use different
>>>>technique s for various ranges of argument values. But the specification
>>>>for log, of course, doesn't tell you this, so your test cases aren't
>>>>likely to hit each of those ranges, and certainly won't make careful
>>>>probes near their boundaries. It's only by looking at the code that you
>>>>can write these tests.

>

>>>Pete, I think you are missing the point of TDD.

>

>>>It's easy for those unfamiliar with the process to focus on the "T" and
>>>ignore the "DD". TDD is a tool for delivering better code, the tests
>>>drive the design, they are not driven by it.

>

>>Which, of course, is entirely backwards.

>

>It is, but you get better code

>
Better code than what?
>
In practice, if a coder wants to write the unit tests first, no
one will stop him; his deliverable consists of both the code and
the unit tests. But there's absolutly no advantage in doing so,
and most people I've worked with feel more comfortable writing
the actual code first.
>

I can see you are extremely against TDD, and thats your personal preference.
As long as you have good unit tests, and high coverage, I can say your
code is good.

>>>So if I were tasked with
>>>writing he function log, I'd start with a simple test, say log(10) and
>>>then add more tests to cover the full range of inputs.

>

>>That is, of course, one solution. It's theoretically possible
>>for log, but it will take several hundred centuries of CPU time,
>>which means that it's not very practical. In practice, the way
>>you verify that a log function is correct is with code review,
>>with testing of the border cases (which implies the what Pete is
>>calling white box testing), to back up the review.

>

>If you were to write a log() function, would you test it
>against all floats?

>
That's precisely my point: you can't. So you have to analyse
the code, to determine where the critical boundaries are, and
test the limit cases. And of course, you have to document that
analysis, because the code reviewers will definitly want to
review it.
>

>>>These tests
>>>would specify the behavior and drive the internals of the function.

>

>>In this case, I think that the behavior is specified before
>>hand. It is a mathematical function, after all, and we can know
>>the precise result for every possible input. In practice, of
>>course, it isn't at all possible to test it, at least not
>>exhaustivel y.

>

>>>Remember, if code isn't required to pass a test, it doesn't get written.

>

>>So your log function only has to produce correct results for the
>>limited set of values you use to test it? I hope I never have
>>to use a library you wrote.

>

>If you take some random numbers (for example 0.0, 0.007, 0.29, 0.999,
>1.0, 1.0001, 1.9900, 555.0, 999999.0) and your log function for these
>numbers gives correct results (with small enough error) you can be sure
>your log function is good

>
Obviously, you don't know how floating point arithmetic works,
or you wouldn't make such a stupid statement.

Maybe I don't know. So, why is it a stupid statement?

How would you test your log() function?

[Gianni Mariani]

Re: A simple unit test framework

James Kanze wrote:
....

>Really. Well write better code. Again, if you can't test it, by
>definition it's bad code.

>
So you exclude multithreading and floating point. Since neither
can really be tested.

Certainly not.

Try reading that again. The whole purpose of the diatribe was for
testing MT code.

I still maintain that if you can't test your code to an acceptable level
of confidence then your code is no good - needs to be
re-architected/re-written/wahetever.

>

>>>Given cache latencies, pre-emption from other threads, program
>>>randomness (like memory allocation variances) you can achieve
>>>pretty close to full coverage of every possible race condition
>>>in about 10 seconds of testing. There are some systematic
>>>start-up effects that may not be found, but you mitigate that
>>>by running automated testing. (In my shop, we run unit tests
>>>on the build machine around the clock - all the time.)

>

>>So perhaps, after a couple of centuries, you can say that your
>>code is reliable.

>

>In practice it usually finds problems in milliseconds.

>
So you write bad code to begin with. Or simply prefer to ignore
real problems that don't show up trivially.

That statement reminds me of a question I sent to support at a software
vendor for a protocol stack in '86. My question was basically "how does
your stack handle corrupt packets". The question came back, "why would
you want to send it corrupt packets ?" .... :-(

[James Kanze]

Re: A simple unit test framework

On May 7, 10:20 am, Ian Collins <ian-n...@hotmail.co mwrote:

James Kanze wrote:

On May 6, 9:46 pm, Ian Collins <ian-n...@hotmail.co mwrote:

James Kanze wrote:
>On May 6, 3:31 am, Ian Collins <ian-n...@hotmail.co mwrote:
>>Gianni Mariani wrote:

>>>I have met very few customers that know what a spec is even if it
>>>smacked them up the side of the head.

>>Welcome to the club!

>>>Sad. Inevitably it leads to pissed off customer.

>>Any agile process (XP, Scrum or whatever) is ideal for this situation.

>If your goal is to rip off the customer, yes.

[First, a meta-comment: I am, of course, exagerating to make
a point, and I don't really suspect Ian of trying to use any
technique to rip off his customers.]

So by helping them to get what they really wanted, rather than forcing
them to commit to what they thought the wanted, I'm ripping them off?

How does testing help the customer to get what he really wants?

By being an integral part of the process.

Testing is, and always has been, an integral part of the
process. Just as is design, coding, code review, integration...
Obviously, having a process does help the customer get what he
wants. My question was more along the lines of: how does
involving the customer in all this (most of which he doesn't
care about, nor understand) help him get what he wants?

Roughly speaking, the user interface needs prototyping; the rest
of the code needs specification. But I don't really see a role
for testing.

Not every customer is in a position to provide specifications and those
that think they can often change their minds once the development has
started.

You need some sort of specification to begin coding. If the
customer can't provide it, you provide it for him, but you do
get him to sign it off.

And of course, specifications have never been cast in stone.
They evolve. But it's not open ended, either. If the customer
suddenly decides that he wants the program to do ten times more
than was originally considered, that will have an impact on the
cost and the delay. The customer asks for the change, I work
out how much it will cost, in time and money, and he decides
whether he wants it or not.

I've seen companies who never accepted any changes. They very
soon find themselves without any customers. I've also seen
companies who accepted all changes, without consideration of the
additional time or cost. They very soon go bankrupt. It's a
two way street.

But I'm sure you know all this. What I want to know is what
relationship this has to testing, per se? These have been
self-evident truths since my first contact with computer
science, back in the 1960's.

The person I'm ripping off is me, I'm doing my self out of all the bug
fixing and rework jobs.

Man you have a strange view of customer focused development.

Customer focused means talking to the customer in his language,
not in yours. A test suite doesn't do this. Getting a customer
to sign off a project on the basis of a test that he's not
capable of understanding is not, IMHO, showing him much respect.

If you were to take the time to look into XP or Scrum, you would see
that you are to some extent preaching to the choir.

I wonder if part of our differences don't have to do with the
type of applications we write. In another posting, you
mentionned that you did a program for a Web. In other words,
most of your requirements are visible behavior, and the
requirements for reliability and longivity aren't really very
high. I'm not too sure what you understand by "tests" here,
because I consider unit tests something that runs automatically,
with no visible output unless there is an error, but certainly,
in such a context, showing the customer a more or less working
program very early in the process, and getting feed back from
it, is important.

I work mainly on large scale servers. The programs run 24 hours
a day, 7 days a week. Deploying a new version may mean stopping
the system, at considerable cost. Nothing that the program does
is immediately visible (but if it doesn't work, a lot of other
things suddenly stop working, and that IS visible). An error in
the code costs real money.

That's why there are (web driven) test suites like FIT or
Selenium that are specifically designed for the customer to
understand and in some cases, use them selves. Don't forget
the customer acceptance tests are system behavioral tests, not
code unit tests (unplug module A and module A missing trap is
sent kind of tests). How the application behaves is what the
customer wants to see.

Or doesn't want to see:-). (Much of my work in the past has
been involved with network routing. Which normally should
happen silently, behind the scenes.)

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[Gianni Mariani]

Re: A simple unit test framework

James Kanze wrote:

On May 7, 2:02 am, Branimir Maksimovic <b...@hotmail.c omwrote:

>On May 6, 3:03 am, Gianni Mariani <gi3nos...@mari ani.wswrote:

>

>>James Kanze wrote:

>

>>...

>

>>>Yes, but nobody but an idiot would pay you for such a thing.
>>>Thread safety, to site but the most obvious example, isn't
>>>testable, so you just ignore it?

>

>>Common misconception.

>

>>1. Testability of code is a primary objective. (i.e. code that can't be
>>tested is unfit for purpose)

>

>>2. Any testing (MT or not) is about a level of confidence, not absoluteness.

>

>>I have discovered that MT test cases that push the limits of the code
>>using random input does provide sufficient coverage to produce a level
>>of confidence that makes the target "testable".

>

>I have exactly opposite experience. Multi threading program failures
>show up only with specially crafted input.
>That is I have to look into code first and then make tests to
>specifically target places in code .
>But, such errors are usually most simplistic ones, those are common
>threading errors, and can be usually catched by eye.

>
Even when you know exactly where the error is, it's sometimes
impossible to write code which reliably triggers it. Consider
std::string, in g++: if you have an std::string object shared by
two threads, and one thread copies it, and the other does [] or
grabs an iterator at exactly the same time, you can end up with
a dangling pointer---an std::string object whose implementation
memory has been freed. The probability of doing so, however, is
very small, and even knowing exactly where the error is, I've
yet to be able to write a program which will reliably trigger
it.

Are you sure. I ran into that problem in an earlier version of gcc's
std::string. I have never seen it since and I did review the
std::string code at 3.0 and I was satisfied that all was well.

std::string is not thread safe but this should work:

std::string global1( "A" );

std::string global2( global1 );

void thread1()
{
global1 += "1";
}

void thread2()
{
global2 += "A";
}

>
Consider some variations on DCL. In at least one case, you only
get into trouble if one of the processors has read memory in the
same cache line as the pointer just before executing the
critical code. Which means that the function can work perfectly
in one application, and fail when you link it into another
application.

Yes, but it's exactly this type of test that finds bugs like that.

>
As I mentionned in a response to another poster, it may just
depend on what you consider acceptable quality. I've worked on
critical systems a lot in the past. With contractual penalties
for downtime. So I tend to set my standards high.
Interestingly enough, however, it turns out that developing code
to such high standards is actually cheaper than just churning it
out.

Yes. Better tested code is easier to develop with.

... As a rough, back of the envelope figure: correcting an
error found in code review costs one tenth of correcting the
same error found in unit tests,

What are you smoking ? Sure you can find some obvious bugs in a code
review, but I would have already run the unit tests before I do the code
review.

... which costs one tenth of
correcting the same error found in integration tests, which
costs one tenth of correcting the same error found in the field.

We can haggle on how many orders of magnitude field errors cost but I'll
agree it's at least 2 orders above unit tests.

Not to mention the advantages of the transfer of knowledge which
takes place in code review. The result is that any time an
integration test finds an error, it is considered first and
foremost an error in the process; we analyse the process to find
out where it failed. And there is even a tendancy to do this
for unit tests, although perhaps not as systematically.
>
[...]

>>It does require a true multi processor system to test adequately.

>

>Not just mp system, but *all* possible mp systems, which is
>of course impossible.

>
It's perhaps worth pointing out that most current low-end MP
systems use a more or less synchronized memory model. This is
not true, however, for Alpha processors, nor the Itanium, nor, I
think, top of the line Sparcs, nor, probably future desktop
processors. That means that the fact that code works on a
current four processor system based on Intel 32 bits means
nothing with regards to future processors.

Only in low level code. Code written above the iron layer should work
just fine.

[James Kanze]

Re: A simple unit test framework

On May 7, 10:08 am, anon <a...@no.nowrot e:

James Kanze wrote:

[...]

In this case you will not
have to do a coverage, but it is a plus. This way, the code you write
will be minimal and easier to understand and maintain.

And will not necessarily meet requirements, or even be useful.

If you write tests according to the requirements, you can be 100% sure
that statement is correct

A requirement for the function log() is that it return the
correct value, at 1 LSB, for all positive floating point values.
If you write a test directly according to that, it will take
something over 3000 centuries to run, supposing that you can
test one value per microsecond (which is beyond the capabilities
of many modern machines).

If the requirements involve thread safety, it may not even be
possible to theoretically guarantee coverage, since a number of
aleas are involved that you cannot control. The same thing is
true about any number of other types of applciations: real time
process control, network management, etc.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[anon]

Re: A simple unit test framework

James Kanze wrote:

On May 7, 10:08 am, anon <a...@no.nowrot e:

>James Kanze wrote:

>
[...]

>>>In this case you will not
>>>have to do a coverage, but it is a plus. This way, the code you write
>>>will be minimal and easier to understand and maintain.

>

>>And will not necessarily meet requirements, or even be useful.

>

>If you write tests according to the requirements, you can be 100% sure
>that statement is correct

>
A requirement for the function log() is that it return the
correct value, at 1 LSB, for all positive floating point values.
If you write a test directly according to that, it will take
something over 3000 centuries to run, supposing that you can
test one value per microsecond (which is beyond the capabilities
of many modern machines).

So, what you are saying here is that every math function is tested the
way you suggested?

>
If the requirements involve thread safety, it may not even be
possible to theoretically guarantee coverage, since a number of
aleas are involved that you cannot control. The same thing is
true about any number of other types of applciations: real time
process control, network management, etc.

There are things which are not possible to be unit tested, but if a code
can be unit tested, then there are no reasons not to do so.
Off course, I am talking about serious projects - not examples and hello
worlds.

[Ian Collins]

Re: A simple unit test framework

James Kanze wrote:

On May 6, 10:01 pm, Ian Collins <ian-n...@hotmail.co mwrote:
>

>I don't have the exact before figures,

>
Which is typical for most process improvements:-). At least at
the beginning; one of the problems in the initial process (or
lack of it) is that it doesn't generate figures.
>

>but there were dozens of bugs in
>the system for the previous version of the product and they took a
>significant amount of developer and test time. The lack of unit tests
>made the code extremely hard to fix without introducing new bugs.
>Comprehensiv e unit tests are the only way to break out of this cycle.

>
Comprehensive unit tests are important for many reasons. I'm
not arguing against them. I'm saying that they aren't
sufficient, if they are the only measure. For that matter, how
do you know the tests are comprehensive? The only means I know
is to review them at the same time you review the code.
>

Because there were written first. We've been there before!

The code was well reviewed there were six or eight on the team, we
rotated pairs and everyone worked on every bit of code.

Comprehensive unit tests are most important for maintenance.
New code should always be reviewed. But a review is not without
costs, and reviewing an entire module because someone has
changed just a couple of characters in just one line isn't
really cost effective. Comprehensive unit tests, on the other
hand, are very effective at catching slips of the finger in the
editor.
>

Again, that's where paring and continuous integration come in, the code
is under constant scrutiny from many pairs of eyes.

>

>We didn't bother tracking bugs for the replacement product, there were
>so few of them and due to their minor nature, they could be fixed within
>a day of being reported. We had about 6 in the first year.

>
And why weren't they being tracked? 6 is, IMHO, a lot, and can
certainly be reduced. Find out why the bug crept in there to
begin with, and modify the process to eliminate it. (I've
released code where the only bug, for the lifetime of the
product, was a spelling error in a log message. That happened
to be the product where there weren't any unit tests, but
admittedly, it was a very small project---not even 100 KLoc in
all.)
>

The cause was too obvious, a couple of spelling mistakes and a couple of
bugs in a bit of code that crept in without tests. The latter was an
object lesson in following the process, the former a prime demonstration
of engineer's collective inability to spell! The remainder were in an
obscure protocol implementation we lacked the equipment to test. We had
a perfect implementation of our misunderstandin g of the specification!

>

>Sounds like you weren't pairing.

>
We tried that, but found that it's no where near as cost
effective as good code review. The code review brings in an
"outside" view---someone who's not implicated in the code. In
practice, it turns out that it's this external viewpoint which
really finds the most bugs. And of course, code review costs
less than pairing (although not by a large amount).
>

Pairing is not just about review, but it brings a unique energy into the
programming process. Rotating pairs gives the "outside" view, in our
case, there wasn't anyone out side the team to review the code.

>

>So you practiced full on XP for a few months and measured the results?

>
One team used full XP for about six months. We stopped the
effort when we saw their error rate shooting up significantly.
(We were at about 1 error per 100 KLoc, going into integration,
at that time. The team using full XP ended up at about ten
times that.)
>

Did they hire an experienced coach? That is usually the result of not
doing thing correctly.

--
Ian Collins.

[James Kanze]

Re: A simple unit test framework

On May 7, 10:55 am, Gianni Mariani <gi3nos...@mari ani.wswrote:

James Kanze wrote:

On May 7, 2:02 am, Branimir Maksimovic <b...@hotmail.c omwrote:

On May 6, 3:03 am, Gianni Mariani <gi3nos...@mari ani.wswrote:

[...]

Even when you know exactly where the error is, it's sometimes
impossible to write code which reliably triggers it. Consider
std::string, in g++: if you have an std::string object shared by
two threads, and one thread copies it, and the other does [] or
grabs an iterator at exactly the same time, you can end up with
a dangling pointer---an std::string object whose implementation
memory has been freed. The probability of doing so, however, is
very small, and even knowing exactly where the error is, I've
yet to be able to write a program which will reliably trigger
it.

Are you sure. I ran into that problem in an earlier version of gcc's
std::string. I have never seen it since and I did review the
std::string code at 3.0 and I was satisfied that all was well.

I'm sure. I wouldn't swear that it hasn't been corrected in the
latest version, but it was definitely present in 3.4.3. The bug
report is still open; from what I understand, a totally new
string class is in the works, so the current one will not be
corrected.

std::string is not thread safe

Everything in the standard library in g++, from 3.0 on, is
supposed to be thread safe. There is some uncertainty, for some
classes, as to how this is defined, and I hesitated to post the
bug, because I wasn't sure that std::string was supposed to meet
the Posix requirements (although all of the other g++ containers
meet them). That is, however, a bit irrelevant to the
discussion here. Posix compliant thread safety is a reasonable
choice, the current implementation of std::string in g++ doesn't
meet it, and I cannot imagine any possible test which would
display this defect in a reliable fashion.

but this should work:

std::string global1( "A" );

std::string global2( global1 );

void thread1()
{
global1 += "1";
}

void thread2()
{
global2 += "A";
}

That definitly should (and as far as I know, does) work. The
problem is more along the lines of:

std::string global( "a" ) ;

void thread1()
{
std::string s1( global ) ;
}

void thread2()
{
std::string s2( global.begin(), global.end() ) ;
}

Since I'm not modifying global, Posix says that the above should
work (or rather, that it should work if the involved types were
known to Posix). Code review shows that it can fail in the
current implementation in G++. I would very much like to see a
test program where it reliably fails, however.

Consider some variations on DCL. In at least one case, you only
get into trouble if one of the processors has read memory in the
same cache line as the pointer just before executing the
critical code. Which means that the function can work perfectly
in one application, and fail when you link it into another
application.

Yes, but it's exactly this type of test that finds bugs like that.

What type of test? You mean that for each test, you wrap the
code in a different environment, recompile, and relink?

As I mentionned in a response to another poster, it may just
depend on what you consider acceptable quality. I've worked on
critical systems a lot in the past. With contractual penalties
for downtime. So I tend to set my standards high.
Interestingly enough, however, it turns out that developing code
to such high standards is actually cheaper than just churning it
out.

Yes. Better tested code is easier to develop with.

The issue hasn't been raised to date, but...

Good code is easy to understand. Code that isn't easy to
understand fails code review. How does testing verify this?

(In some ways, easy to understand is more important than an
absence of errors.)

... As a rough, back of the envelope figure: correcting an
error found in code review costs one tenth of correcting the
same error found in unit tests,

What are you smoking ? Sure you can find some obvious bugs in
a code review, but I would have already run the unit tests
before I do the code review.

That's doing things the hard way, and is hardly cost effective.
In code review, you are told that you forgot to initialize
variable i in line 1234 of abc.cc. The unit test tells you that
tests 25 through 30 all fail. Which information makes it easier
to find and fix the error?

And a well run code review doesn't find just the obvious bugs.
It also finds those which no test will find. (I found the bug
in the g++ implementation of std::string by reviewing the code.)

... which costs one tenth of
correcting the same error found in integration tests, which
costs one tenth of correcting the same error found in the field.

We can haggle on how many orders of magnitude field errors
cost but I'll agree it's at least 2 orders above unit tests.

My figures are "back of the envelope". The actual values will
vary enormously. The goal is only to give a rough idea.

[...]

>It does require a true multi processor system to test adequately.

Not just mp system, but *all* possible mp systems, which is
of course impossible.

It's perhaps worth pointing out that most current low-end MP
systems use a more or less synchronized memory model. This is
not true, however, for Alpha processors, nor the Itanium, nor, I
think, top of the line Sparcs, nor, probably future desktop
processors. That means that the fact that code works on a
current four processor system based on Intel 32 bits means
nothing with regards to future processors.

Only in low level code. Code written above the iron layer should work
just fine.

Dream on.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[Michael DOUBEZ]

Re: A simple unit test framework

anon a écrit :

James Kanze wrote:

>On May 7, 10:08 am, anon <a...@no.nowrot e:

>>James Kanze wrote:

>>
> [...]

>>>>In this case you will not
>>>>have to do a coverage, but it is a plus. This way, the code you write
>>>>will be minimal and easier to understand and maintain.

>>

>>>And will not necessarily meet requirements, or even be useful.

>>

>>If you write tests according to the requirements, you can be 100% sure
>>that statement is correct

>>
>A requirement for the function log() is that it return the
>correct value, at 1 LSB, for all positive floating point values.
>If you write a test directly according to that, it will take
>something over 3000 centuries to run, supposing that you can
>test one value per microsecond (which is beyond the capabilities
>of many modern machines).

>
So, what you are saying here is that every math function is tested the
way you suggested?

Please, lookup "numerical analysis". The stability of an algorithm can
be proven and TDD is useless in this case.
Test are then only validation tests (test of special value, overflow
....) which are well defined.

>If the requirements involve thread safety, it may not even be
>possible to theoretically guarantee coverage, since a number of
>aleas are involved that you cannot control. The same thing is
>true about any number of other types of applciations: real time
>process control, network management, etc.

>
There are things which are not possible to be unit tested, but if a code
can be unit tested, then there are no reasons not to do so.
Off course, I am talking about serious projects - not examples and hello
worlds.

That doesn't bring you a "100% sure that statement is correct" as you
said. The only thing you can do is increase correctness and robustness.

If the "100% sure" existed, security systems such as fail-back (recovery
upon unexpected error) or comity (same algorithm implement by 3
different teams - the correct value is chosen at the majority) wouldn't
exists in critical systems.

Michael

[Ian Collins]

Re: A simple unit test framework

James Kanze wrote:

On May 7, 10:20 am, Ian Collins <ian-n...@hotmail.co mwrote:

>James Kanze wrote:
>>

>>Customer focused means talking to the customer in his language,
>>not in yours. A test suite doesn't do this. Getting a customer
>>to sign off a project on the basis of a test that he's not
>>capable of understanding is not, IMHO, showing him much respect.

>

>If you were to take the time to look into XP or Scrum, you would see
>that you are to some extent preaching to the choir.

>
I wonder if part of our differences don't have to do with the
type of applications we write. In another posting, you
mentionned that you did a program for a Web. In other words,
most of your requirements are visible behavior, and the
requirements for reliability and longivity aren't really very
high. I'm not too sure what you understand by "tests" here,
because I consider unit tests something that runs automatically,
with no visible output unless there is an error, but certainly,
in such a context, showing the customer a more or less working
program very early in the process, and getting feed back from
it, is important.
>

I work in two differing domains, AJAX web applications (JavaScript
/PHP/MySQL) and embedded C++, mainly DC power system controllers. If
the web application goes down, just blame IE. If the power system
fails, the customer (Telcos) can loose millions of dollars in lost call
revenue.

Very different reliability criteria and run times, but I use the same
process to ensure quality for both.

Ian.

--
Ian Collins.

[anon]

Re: A simple unit test framework

Michael DOUBEZ wrote:

anon a écrit :

>James Kanze wrote:

>>On May 7, 10:08 am, anon <a...@no.nowrot e:
>>>James Kanze wrote:
>>>
>> [...]
>>>>>In this case you will not
>>>>>have to do a coverage, but it is a plus. This way, the code you write
>>>>>will be minimal and easier to understand and maintain.
>>>
>>>>And will not necessarily meet requirements, or even be useful.
>>>
>>>If you write tests according to the requirements, you can be 100% sure
>>>that statement is correct
>>>
>>A requirement for the function log() is that it return the
>>correct value, at 1 LSB, for all positive floating point values.
>>If you write a test directly according to that, it will take
>>something over 3000 centuries to run, supposing that you can
>>test one value per microsecond (which is beyond the capabilities
>>of many modern machines).

>>
>So, what you are saying here is that every math function is tested the
>way you suggested?

>
Please, lookup "numerical analysis". The stability of an algorithm can
be proven and TDD is useless in this case.
Test are then only validation tests (test of special value, overflow
...) which are well defined.
>

Thanks. I will take a look

Steps I would follow in this explicit case of a log algorithm are:
1) find a good numerical mathematical algorithm which estimates good
enough result of a log function for passed value. This result would have
to be good enough for its use.
2) implement it
3) write several tests (maybe 10 tests) only to validate

I thought the stability of a numerical algorithm should be proven in
mathematical theory.

>>If the requirements involve thread safety, it may not even be
>>possible to theoretically guarantee coverage, since a number of
>>aleas are involved that you cannot control. The same thing is
>>true about any number of other types of applciations: real time
>>process control, network management, etc.

>>
>There are things which are not possible to be unit tested, but if a
>code can be unit tested, then there are no reasons not to do so.
>Off course, I am talking about serious projects - not examples and
>hello worlds.

>
That doesn't bring you a "100% sure that statement is correct" as you
said. The only thing you can do is increase correctness and robustness.
>
If the "100% sure" existed, security systems such as fail-back (recovery
upon unexpected error) or comity (same algorithm implement by 3
different teams - the correct value is chosen at the majority) wouldn't
exists in critical systems.

I never said it is possible to cover every possible case. But having a
good test is much better then having no tests at all