A simple unit test framework

[Pete Becker]

Re: A simple unit test framework

Gianni Mariani wrote:

>
I find the distinction of being a "programmer " or a "test programmer" to
be counter productive.
>

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

[Pete Becker]

Re: A simple unit test framework

Gianni Mariani wrote:

Pete Becker wrote:
...

>When I was a QA manager I'd bristle whenever any developer said they'd
>"let testers" do something. That's simply wrong. Testing is not an
>adjunct to development. Testing is a profession, with technical
>challenges that differ from, and often exceed in complexity, those
>posed by development. The skills required to do it well are vastly
>different, but no less sophisticated, than those needed to write the
>product itself. Most developers think they can write good tests, but
>in reality, they simply don't have the right skills, and test suites
>written by developers are usually naive. A development manager who
>gives testers "complete freedom" is missing the point: that's not
>something the manager can give or take, it's an essential part of
>effective testing.
>>

>
The most successful teams I worked with were teams that wrote their own
tests.
>
I find the distinction of being a "programmer " or a "test programmer" to
be counter productive.

I do, too, because those particular terms suggest a false hierarchy. A
better distinction might be between an application programmer and a test
programmer. The fact remains that developers rarely have the skills to
write good tests or the mindset to write good tests.

>
Testability is a primary objective. (one of my tenets for good software
development).
>
Without somthing being testable, it can't be maintained. Maintaining
and developing are the same thing. Hence, can't be tested means can't
be developed.
>
Yes, there are edge cases where the developer is unable to fully test
the code. That's ok, tests don't have to be perfect first time, as
problems are found the developer gets to learn and develop better tests.
>
>
I almost always now resort to at least one monte-carlo test.
(construct/randomly twiddle/destruct) large numbers of times in random
orders with random data.
>

Yup. Typical developer-written test: I don't understand testing well
enough to do it right, so I'll do something random and hope to hit a
problem. <g>

As I've said several times, developing and testing involve two distinct
sets of skills. Developers think they're good at testing, but any
professional tester will tell you that they aren't.

--

-- Pete
Roundhouse Consulting, Ltd. (www.versatilecoding.com)
Author of "The Standard C++ Library Extensions: a Tutorial and
Reference." (www.petebecker.com/tr1book)

[James Kanze]

Re: A simple unit test framework

On May 4, 3:01 pm, anon <a...@no.nowrot e:

Pete Becker wrote:

Ian Collins wrote:

Pete Becker wrote:
>nw wrote:
>>I previously asked for suggestions on teaching testing in C++. Based
>>on some of the replies I received I decided that best way to proceed
>>would be to teach the students how they might write their own unit
>>test framework, and then in a lab session see if I can get them to
>>write their own. To give them an example I've created the following
>>UTF class (with a simple test program following). I would welcome and
>>suggestions on how anybody here feels this could be improved:

>A fool with a tool is still a fool. The challenge in testing is not test
>management, but designing test cases to cover the possible failures in
>the code under test. That's something that most developers don't do
>well, because their focus is on getting the code to run.
Unless the test are written first!

You can't do coverage analysis or any other form of white-box testing on
code that hasn't been written. There is a big difference between a
tester's minset and a develper's mindset, and it's very hard for one
person to do both.

The latest trends are to write tests first which demonstrates the
requirements, then code (classes+method s).

The latest trend where? Certainly not in any company concerned
with good management, or quality software.

In this case you will not
have to do a coverage, but it is a plus. This way, the code you write
will be minimal and easier to understand and maintain.

And will not necessarily meet requirements, or even be useful.

--
James Kanze (Gabi Software) email: james.kanze@gma il.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[James Kanze]

Re: A simple unit test framework

On May 4, 10:51 pm, Ian Collins <ian-n...@hotmail.co mwrote:

anon wrote:

Pete Becker wrote:

Ian Collins wrote:
>Pete Becker wrote:
>>nw wrote:
>>>I previously asked for suggestions on teaching testing in C++. Based
>>>on some of the replies I received I decided that best way to proceed
>>>would be to teach the students how they might write their own unit
>>>test framework, and then in a lab session see if I can get them to
>>>write their own. To give them an example I've created the following
>>>UTF class (with a simple test program following). I would welcome and
>>>suggestion s on how anybody here feels this could be improved:

>>A fool with a tool is still a fool. The challenge in
>>testing is not test management, but designing test cases
>>to cover the possible failures in the code under test.
>>That's something that most developers don't do well,
>>because their focus is on getting the code to run.
>Unless the test are written first!

You can't do coverage analysis or any other form of white-box testing
on code that hasn't been written. There is a big difference between a
tester's minset and a develper's mindset, and it's very hard for one
person to do both.

The latest trends are to write tests first which demonstrates the
requirements, then code (classes+method s). In this case you will not
have to do a coverage, but it is a plus. This way, the code you write
will be minimal and easier to understand and maintain.
I agree this way looks harder (and I am not using it), but I am sure
once you get use to it - your programming skills will improve drastically

I find it makes the design and code process easier and way more fun. No
more debugging sessions!

Until the code starts actually being used.

I'll admit that it's probably more fun to just write whatever
you want, rather than to have a hard specification which has to
be met, but the companies I work for want something useful.

--
James Kanze (Gabi Software) email: james.kanze@gma il.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[James Kanze]

Re: A simple unit test framework

On May 5, 1:56 am, Ian Collins <ian-n...@hotmail.co mwrote:

Pete Becker wrote:

anon wrote:

Pete Becker wrote:

>You can't do coverage analysis or any other form of white-box testing
>on code that hasn't been written. There is a big difference between a
>tester's minset and a develper's mindset, and it's very hard for one
>person to do both.

The latest trends are to write tests first which demonstrates the
requirements, then code (classes+method s). In this case you will not
have to do a coverage, but it is a plus.

That's not what coverage analysis refers to. Coverage analysis takes the
test cases that you've written and measures (speaking a bit informally)
how much of the code is actually tested by the test set. You can't make
that measurement until you've written the code.

If you apply TDD correctly, you only write code to pass tests, so all of
your code is covered.

Yes, but nobody but an idiot would pay you for such a thing.
Thread safety, to site but the most obvious example, isn't
testable, so you just ignore it?

My customers want to know what the code will do, and how much
development will cost, before they allocate the resources to
develope it. Which means that I have a requirements
specification which has to be met.

--
James Kanze (Gabi Software) email: james.kanze@gma il.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[James Kanze]

Re: A simple unit test framework

On May 5, 2:31 am, Ian Collins <ian-n...@hotmail.co mwrote:

Pete Becker wrote:

Ian Collins wrote:

Pete Becker wrote:

If you apply TDD correctly, you only write code to pass tests, so all of
your code is covered.

Suppose you're writing test cases for the function log, which calculates
the logarithm of its argument. Internally, it will use different
techniques for various ranges of argument values. But the specification
for log, of course, doesn't tell you this, so your test cases aren't
likely to hit each of those ranges, and certainly won't make careful
probes near their boundaries. It's only by looking at the code that you
can write these tests.

Pete, I think you are missing the point of TDD.
>
It's easy for those unfamiliar with the process to focus on the "T" and
ignore the "DD". TDD is a tool for delivering better code, the tests
drive the design, they are not driven by it.

Which, of course, is entirely backwards.

So if I were tasked with
writing he function log, I'd start with a simple test, say log(10) and
then add more tests to cover the full range of inputs.

That is, of course, one solution. It's theoretically possible
for log, but it will take several hundred centuries of CPU time,
which means that it's not very practical. In practice, the way
you verify that a log function is correct is with code review,
with testing of the border cases (which implies the what Pete is
calling white box testing), to back up the review.

These tests
would specify the behavior and drive the internals of the function.

In this case, I think that the behavior is specified before
hand. It is a mathematical function, after all, and we can know
the precise result for every possible input. In practice, of
course, it isn't at all possible to test it, at least not
exhaustively.

Remember, if code isn't required to pass a test, it doesn't get written.

So your log function only has to produce correct results for the
limited set of values you use to test it? I hope I never have
to use a library you wrote.

--
James Kanze (Gabi Software) email: james.kanze@gma il.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[James Kanze]

Re: A simple unit test framework

On May 5, 3:08 pm, "Bo Persson" <b...@gmb.dkwro te:

Ok, then. If you were to develop a log() function, how many
test cases would you need for a black box testing?

In this case, it's not really just a question of black box
versus white box. To effectively test log(), you need something
around 1,8 E19 tests. At one test per microsecond (and I doubt
that you can do them that fast), that's over 6000 centuries.

If you want to be relatively sure of the correctness of your
function, and still deliver it on schedule, you will take two
steps:

-- you will have it reviewed by people who know numeric
processing, for possible errors, and

-- you will write tests based on knowledge of the algorithm
used, to verify the limit cases for that algorithm.

Pete's point is, I think, that you cannot do the latter until
you've decided what algorithm(s) to use. In the case of a
function like log(), you can't simply test 100% of the possible
input, and you don't know what is important to test until you've
actually written the code.

Note that the two steps above work together. The people doing
the code review also review the tests, and verify that your
choice of test cases was appropriate for the code you wrote.

--
James Kanze (Gabi Software) email: james.kanze@gma il.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[Ian Collins]

Re: A simple unit test framework

James Kanze wrote:

On May 4, 10:51 pm, Ian Collins <ian-n...@hotmail.co mwrote:

>anon wrote:

>

>>The latest trends are to write tests first which demonstrates the
>>requirement s, then code (classes+method s). In this case you will not
>>have to do a coverage, but it is a plus. This way, the code you write
>>will be minimal and easier to understand and maintain.
>>I agree this way looks harder (and I am not using it), but I am sure
>>once you get use to it - your programming skills will improve drastically

>

>I find it makes the design and code process easier and way more fun. No
>more debugging sessions!

>
Until the code starts actually being used.
>

No, The first embedded product my team developed using TDD has had about
half a dozen defects reported in 5 years and several thousand units of
field use. They were all in bits of the code with poor tests.

I'll admit that it's probably more fun to just write whatever
you want, rather than to have a hard specification which has to
be met, but the companies I work for want something useful.
>

What a load of bollocks. Who said anything about "write whatever you
want"? If you are going to slag something off, please understand it and
try it first.

--
James Kanze (Gabi Software) email: james.kanze@gma il.com

Please fix this!

--
Ian Collins.

[Ian Collins]

Re: A simple unit test framework

James Kanze wrote:

On May 5, 3:08 pm, "Bo Persson" <b...@gmb.dkwro te:
>

>Ok, then. If you were to develop a log() function, how many
>test cases would you need for a black box testing?

>
In this case, it's not really just a question of black box
versus white box. To effectively test log(), you need something
around 1,8 E19 tests. At one test per microsecond (and I doubt
that you can do them that fast), that's over 6000 centuries.
>
If you want to be relatively sure of the correctness of your
function, and still deliver it on schedule, you will take two
steps:
>
-- you will have it reviewed by people who know numeric
processing, for possible errors, and
>
-- you will write tests based on knowledge of the algorithm
used, to verify the limit cases for that algorithm.
>

Better still, sit down with someone who knows numeric processing and
write the tests and the function, together.

Pete's point is, I think, that you cannot do the latter until
you've decided what algorithm(s) to use.

It can also work the other way, you are unsure of the algorithm, so you
start be writing a few simple test cases and as these become more
complex, the algorithm finds its self. I did this with polynomial fit a
while back, stating with a test for a two point fit, then three, then a
quadratic and so on. I reached the general case in about a dozen tests.

--
Ian Collins.

[Alf P. Steinbach]

Re: A simple unit test framework

* Ian Collins:

>

>--
>James Kanze (Gabi Software) email: james.kanze@gma il.com

>
Please fix this!

James is (for good reasons) posting via Google, which ingeniously strips
off the space at the end of the signature delimiter.

The problem with Google: just like Microsoft there seems to be a
high-level management decision to gather as much feedback as possible,
countered by low-level management decisions to sabotage that so that it
looks like it's in place, but actually impossible for anyone to report
anything (you're redirected to irrelevant places, submit buttons don't
work, nothing is actually received, no relevant category is listed, no
mail address available, and so on ad nauseam).

Hence, Google Earth places Norway in Sweden, Google Groups strips off
significant spaces, and so on and so forth, and even though thousands
and tens of thousands /try/ to report this, Google's as unwise as ever
about its failings. The price of becoming a behemoth company, and what
I'm speculating is probably the reason: lying cheating weasels are
attracted like moths to a flame, and form the lower management echelons.
Oh, sorry, this' off-topic in clc++m, but it sure felt good to get
that off my chest!

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

[Alf P. Steinbach]

Re: A simple unit test framework

* Alf P. Steinbach:

* Ian Collins:

>>

>>--
>>James Kanze (Gabi Software) email: james.kanze@gma il.com

>>
>Please fix this!

>
James is (for good reasons) posting via Google, which ingeniously strips
off the space at the end of the signature delimiter.
>
The problem with Google: just like Microsoft there seems to be a
high-level management decision to gather as much feedback as possible,
countered by low-level management decisions to sabotage that so that it
looks like it's in place, but actually impossible for anyone to report
anything (you're redirected to irrelevant places, submit buttons don't
work, nothing is actually received, no relevant category is listed, no
mail address available, and so on ad nauseam).
>
Hence, Google Earth places Norway in Sweden, Google Groups strips off
significant spaces, and so on and so forth, and even though thousands
and tens of thousands /try/ to report this, Google's as unwise as ever
about its failings. The price of becoming a behemoth company, and what
I'm speculating is probably the reason: lying cheating weasels are
attracted like moths to a flame, and form the lower management echelons.
Oh, sorry, this' off-topic in clc++m, but it sure felt good to get that
off my chest!

(James: no, I didn't follow up on you-know-what.)

--
A: Because it messes up the order in which people normally read text.
Q: Why is it such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?

[Ian Collins]

Re: A simple unit test framework

James Kanze wrote:

On May 4, 3:01 pm, anon <a...@no.nowrot e:
>

>The latest trends are to write tests first which demonstrates the
>requirements , then code (classes+method s).

>
The latest trend where? Certainly not in any company concerned
with good management, or quality software.
>

Have you ever been in charge of a company's software development? I
have and the best thing I ever did to improve both the productivity of
the teams and quality of the code was to introduce eXtreme Programming,
which includes TDD as a core practice.

Our delivery times and field defect rates more than vindicated the change.

--
Ian Collins.

[Ian Collins]

Re: A simple unit test framework

Pete Becker wrote:

Gianni Mariani wrote:

>Pete Becker wrote:
>...

>>When I was a QA manager I'd bristle whenever any developer said
>>they'd "let testers" do something. That's simply wrong. Testing is
>>not an adjunct to development. Testing is a profession, with
>>technical challenges that differ from, and often exceed in
>>complexity, those posed by development. The skills required to do it
>>well are vastly different, but no less sophisticated, than those
>>needed to write the product itself. Most developers think they can
>>write good tests, but in reality, they simply don't have the right
>>skills, and test suites written by developers are usually naive. A
>>development manager who gives testers "complete freedom" is missing
>>the point: that's not something the manager can give or take, it's an
>>essential part of effective testing.
>>>

>>
>The most successful teams I worked with were teams that wrote their
>own tests.
>>
>I find the distinction of being a "programmer " or a "test programmer"
>to be counter productive.

>
I do, too, because those particular terms suggest a false hierarchy. A
better distinction might be between an application programmer and a test
programmer. The fact remains that developers rarely have the skills to
write good tests or the mindset to write good tests.
>

So we do agree! That's pretty much the job title I gave my test
developers. They were just as much developers as those who developed
the application code. Their key skills were designing good tests and
knowing what tools to use to write, run and report on those tests.

--
Ian Collins.

[Ian Collins]

Re: A simple unit test framework

Alf P. Steinbach wrote:

* Ian Collins:

>>

>>--
>>James Kanze (Gabi Software) email: james.kanze@gma il.com

>>
>Please fix this!

>
James is (for good reasons) posting via Google, which ingeniously strips
off the space at the end of the signature delimiter.
>
The problem with Google: just like Microsoft there seems to be a
high-level management decision to gather as much feedback as possible,
countered by low-level management decisions to sabotage that so that it
looks like it's in place, but actually impossible for anyone to report
anything (you're redirected to irrelevant places, submit buttons don't
work, nothing is actually received, no relevant category is listed, no
mail address available, and so on ad nauseam).
>
Hence, Google Earth places Norway in Sweden, Google Groups strips off
significant spaces, and so on and so forth, and even though thousands
and tens of thousands /try/ to report this, Google's as unwise as ever
about its failings. The price of becoming a behemoth company, and what
I'm speculating is probably the reason: lying cheating weasels are
attracted like moths to a flame, and form the lower management echelons.
Oh, sorry, this' off-topic in clc++m, but it sure felt good to get that
off my chest!
>

A rant a day keeps the ulcers away!

--
Ian Collins.

[Gianni Mariani]

Re: A simple unit test framework

Pete Becker wrote:

Gianni Mariani wrote:

>Bo Persson wrote:
>...

>>Ok, then. If you were to develop a log() function, how many test
>>cases would you need for a black box testing?

>>
>Ah - that's a design question.
>>
>What are your design objectives ?
>>
>I assume you want to log various information.
>>

>
Well, no. log() in this case is the logarithm function that I used as an
example earlier in this thread.
>

Oh - well that's why you need a spec !!!!