Mars Rover Controlled By Java

[Yoyoma_2]

Re: Mars Rover Not Responding

Tony Hill wrote:[color=blue]
> On 29 Jan 2004 16:20:29 -0800, brucebo@my-deja.com (Bruce Bowen)
> wrote:[color=green]
>>Uncle Al <UncleAl0@hate. spam.net> wrote in message news:<4016A579. 397B55A@hate.sp am.net>...
>>
>>This is a trivial engineering problem to address. Encase the HD in a
>>larger sealed and presurize container, with enough surface area and/or
>>internal air circulation to keep it cool. A low power 2.5" HD
>>shouldn't take that much larger of a container. What about the flash
>>sized microdrives?[/color]
>
>
> Yeah, and the 4+ Gs that the drive would experience during take-off
> would do wonders for that drive! Not to mention the high levels of
> radiation in space would probably fry any drive (to the best of my
> knowledge, no one makes rad-hardened hard drives).[/color]

I actually know of some rad-proofing drives. Actually i hear there are
alot of them, they are mostly used in the military (mil-spec drives).
For example, hard drives on an aircraft carrier have to be able to take
a direct nuclear assult and still function. Little piece of cold war
trivia for you :)

Ran proofing isn't a very big deal.

According to this page i just pulled up at
random(http://www.westerndigital.com/en/pro...sp?DriveID=41),
normal desktop "run of the mill" drives can take impulses of up to 250G
when non-operating, and 60G while operating at a delta-t of 2 seconds.
That's pretty good.

So if that's for ordinary hard drives, immagine a mil-spec drive. I
doubt carriers kept all of their data on flash memory in the 1970s.
Even if they use mag tape reel, it implies that they have developed some
sort of rad-proofing for it.
[color=blue]
>
> Just stick everything on a disk-on-chip, much easier and cheaper than
> trying to jerry-rig some sort of hard drive contraption.
>
> -------------
> Tony Hill
> hilla <underscore> 20 <at> yahoo <dot> ca[/color]

[BarryNL]

Re: Mars Rover Not Responding



Yoyoma_2 wrote:[color=blue]
> Tony Hill wrote:
>[color=green]
>> On 29 Jan 2004 16:20:29 -0800, brucebo@my-deja.com (Bruce Bowen)
>> wrote:
>>[color=darkred]
>>> Uncle Al <UncleAl0@hate. spam.net> wrote in message
>>> news:<4016A579. 397B55A@hate.sp am.net>...
>>>
>>> This is a trivial engineering problem to address. Encase the HD in a
>>> larger sealed and presurize container, with enough surface area and/or
>>> internal air circulation to keep it cool. A low power 2.5" HD
>>> shouldn't take that much larger of a container. What about the flash
>>> sized microdrives?[/color]
>>
>>
>>
>> Yeah, and the 4+ Gs that the drive would experience during take-off
>> would do wonders for that drive! Not to mention the high levels of
>> radiation in space would probably fry any drive (to the best of my
>> knowledge, no one makes rad-hardened hard drives).[/color]
>
>
> I actually know of some rad-proofing drives. Actually i hear there are
> alot of them, they are mostly used in the military (mil-spec drives).
> For example, hard drives on an aircraft carrier have to be able to take
> a direct nuclear assult and still function. Little piece of cold war
> trivia for you :)
>
> Ran proofing isn't a very big deal.
>
> According to this page i just pulled up at
> random(http://www.westerndigital.com/en/pro...sp?DriveID=41),
> normal desktop "run of the mill" drives can take impulses of up to 250G
> when non-operating, and 60G while operating at a delta-t of 2 seconds.
> That's pretty good.
>
> So if that's for ordinary hard drives, immagine a mil-spec drive. I
> doubt carriers kept all of their data on flash memory in the 1970s. Even
> if they use mag tape reel, it implies that they have developed some sort
> of rad-proofing for it.[/color]

And the 4Gs thing is a non-issue. Most normal ATA drives can take around
300Gs when not operating or 30Gs when running without damage.

[Stanley Krute]

Re: Mars Rover Not Responding

Howdy Edward
[color=blue]
> I am almost flabbergasted into textlessness. The fact that a system
> ... any system, not just a computer ... may work correctly in some one
> or two delta range yet fail in some 10 or 20 delta range ... some
> newbie tyro university graduate wet behind the ears neophyte kid might
> make this mistake in a small project, and the old seasoned pro salt
> seen it all manager would take this as a teaching opportunity. But in
> an entire organization, a huge project putting a robot on a distant
> planet, and not once did this occur to anybody!?[/color]

Yep, you nailed it.

My 5-word software testing book: Run 'er Hard & Long

-- stan

[Double-A]

Re: Mars Rover Not Responding

Re: Mars Rover Not Responding!


Maybe it's your cologne, you Martian perv!

[Nick Maclaren]

Re: Mars Rover Not Responding

In article <I4ySb.22620$Lr 6.1991717975@tw ister2.starband .net>,
Stanley Krute <Stan@StanKrute .com> wrote:[color=blue]
>Howdy Edward
>[color=green]
>> I am almost flabbergasted into textlessness. The fact that a system
>> ... any system, not just a computer ... may work correctly in some one
>> or two delta range yet fail in some 10 or 20 delta range ... some
>> newbie tyro university graduate wet behind the ears neophyte kid might
>> make this mistake in a small project, and the old seasoned pro salt
>> seen it all manager would take this as a teaching opportunity. But in
>> an entire organization, a huge project putting a robot on a distant
>> planet, and not once did this occur to anybody!?[/color]
>
>Yep, you nailed it.
>
>My 5-word software testing book: Run 'er Hard & Long[/color]

Sigh. That is very likely the CAUSE of the problem :-(

Any particular test schedule (artificial or natural) will create a
distribution of circumstances over the space of all that are handled
differently by the program. And, remember, we are talking about a
space of cardinality 10^(10^4) to 10^(10^8). Any particular, broken
logic (usually a combination of sections of code and data) may be
invoked only once ever millennium, or perhaps never.

Now, change the test schedule in an apparently trivial way, or use
the program for real, and that broken logic may be invoked once a
day. Ouch. Incidentally, another way of looking at this is the
probability of distinguishing two finite element automata by feeding
in test strings and comparing the results. It was studied some
decades back, and the conclusions are not pretty.

The modern, unsystematic approach to testing is hopeless as an
egnineering technique, though fine as a political or marketing one.
For high-reliability codes, we need to go back to the approaches
used in the days when most computer people were also mathematicians,
engineers or both.


Regards,
Nick Maclaren.

[Stanley Krute]

Re: Mars Rover Not Responding

Howdy Nick
[color=blue]
> The modern, unsystematic approach to testing is hopeless as an
> egnineering technique, though fine as a political or marketing one.
> For high-reliability codes, we need to go back to the approaches
> used in the days when most computer people were also mathematicians,
> engineers or both.[/color]

Deep agreement as to importance of math-smart testing.

-- stan

[Edward Green]

Re: Mars Rover Not Responding

nmm1@cus.cam.ac .uk (Nick Maclaren) wrote in message news:<bvgar8$4k s$1@pegasus.csx .cam.ac.uk>...[color=blue]
> In article <I4ySb.22620$Lr 6.1991717975@tw ister2.starband .net>,
> Stanley Krute <Stan@StanKrute .com> wrote:
>[color=green]
> >My 5-word software testing book: Run 'er Hard & Long[/color]
>
> Sigh. That is very likely the CAUSE of the problem :-(
>
> Any particular test schedule (artificial or natural) will create a
> distribution of circumstances over the space of all that are handled
> differently by the program. And, remember, we are talking about a
> space of cardinality 10^(10^4) to 10^(10^8). Any particular, broken
> logic (usually a combination of sections of code and data) may be
> invoked only once ever millennium, or perhaps never.
>
> Now, change the test schedule in an apparently trivial way, or use
> the program for real, and that broken logic may be invoked once a
> day. Ouch. Incidentally, another way of looking at this is the
> probability of distinguishing two finite element automata by feeding
> in test strings and comparing the results. It was studied some
> decades back, and the conclusions are not pretty.
>
> The modern, unsystematic approach to testing is hopeless as an
> egnineering technique, though fine as a political or marketing one.
> For high-reliability codes, we need to go back to the approaches
> used in the days when most computer people were also mathematicians,
> engineers or both.[/color]

Going back to the case at hand: it still sounds to me like the stated
cause of the bug -- more files were written to flash memory than were
anticipated in design -- is at least restrospectivel y obviously a
possibility which should have been addressed at the design stage, and
maybe should have been prospectively obvious also.

I'm not quite sure how to jive this with your theoretical insight that
we are searching a space of a cardinality of 10 followed by many, many
zeroes, and similar observations which make the problem sound
hopeless: maybe I'm naively wrong, or not (well that covers all the
possibilities ;-).

Is asking that when the program expends some resource it handles the
problem in some minimally damaging way really an impossibly hard
problem in the space of all the impossibly hard problems with which
computer science abounds, or is it merely a challenging but tractable
engineering problem?

For example, suppose we had a machine running around some play pen,
the the space of possible joint states of the machine and the play pen
were of cardinality 10 followed by some humungous number of zeroes.
And suppose that when the machine leaves the play pen, that is a
"crash". Now, we might ask why the machine crashed, and the designer
might respond with language about the cardinality of the joint state
space, and the impossibility of complete testing. But now we might
ask why he did not put a _fence_ around the play pen, and this answer
is no longer sufficient, and the answer "well, we let it run around
for a while, and it didn't seem likely to cross the boundaries, so we
didn't bother with a fence", is marginal.

Is the problem of building a number of internal fences in complex
systems sufficient to provided timely alert to unanticipated operating
conditions itself an intractably hard problem, or merely hard?

[Nick Maclaren]

Re: Mars Rover Not Responding

In article <2a0cceff.04013 11411.47a8cd13@ posting.google. com>,
Edward Green <nulldev00@aol. com> wrote:[color=blue]
>
>Going back to the case at hand: it still sounds to me like the stated
>cause of the bug -- more files were written to flash memory than were
>anticipated in design -- is at least restrospectivel y obviously a
>possibility which should have been addressed at the design stage, and
>maybe should have been prospectively obvious also.[/color]

Perhaps. Without investigating the problem carefully, it is impossible
to tell.
[color=blue]
>I'm not quite sure how to jive this with your theoretical insight that
>we are searching a space of a cardinality of 10 followed by many, many
>zeroes, and similar observations which make the problem sound
>hopeless: maybe I'm naively wrong, or not (well that covers all the
>possibilitie s ;-).
>
>Is asking that when the program expends some resource it handles the
>problem in some minimally damaging way really an impossibly hard
>problem in the space of all the impossibly hard problems with which
>computer science abounds, or is it merely a challenging but tractable
>engineering problem?[/color]

There are several aspects here. Minimising damage IS an impossibly
hard problem (not just exponentially expensive, but insoluble). But
that is often used as excuse to avoid even attempting to constrain
the consequent damages. Think of it this way.

Identifying and logging resource exhaustion takes a fixed time, so
there is no excuse not to do it. Yes, that can exhaust space in the
logging files, so there is a recursive issue, but there are known
partial solutions to that.

Identifying the cause is simple if there is one factor, harder if
there are two, and so on. To a great extent, that is also true of
predicting the resources needed, but that can be insoluble even with
one factor. This is confused by the fact that, the fewer the factors,
the more likely a bug is to be removed in initial testing.

Most of my bug-tracking time is spent on ones with 3-10 factors, on
a system with (say) 100-1,000 relevant factors. It isn't surprising
that the vendors' testing has failed to detect them. There are only
two useful approaches to such issues:

1) To design the system using a precise mathematical model, so
that you can eliminate, minimise or constrain interactions. This
also needs PRECISE interface specifications, of course, not the sloppy
rubbish that is almost universal.

2) To provide good detection and diagnostic facilities, to help
locating the causes and effects of such problems. This is even more
neglected nowadays, and is of limited help for systems like Mars
missions.


Regards,
Nick Maclaren.

[A. G. McDowell]

Re: Mars Rover Not Responding

In article <bvgar8$4ks$1@p egasus.csx.cam. ac.uk>, Nick Maclaren
<nmm1@cus.cam.a c.uk> writes[color=blue]
>In article <I4ySb.22620$Lr 6.1991717975@tw ister2.starband .net>,
>Stanley Krute <Stan@StanKrute .com> wrote:[color=green]
>>Howdy Edward
>>[color=darkred]
>>> I am almost flabbergasted into textlessness. The fact that a system
>>> ... any system, not just a computer ... may work correctly in some one
>>> or two delta range yet fail in some 10 or 20 delta range ... some
>>> newbie tyro university graduate wet behind the ears neophyte kid might
>>> make this mistake in a small project, and the old seasoned pro salt
>>> seen it all manager would take this as a teaching opportunity. But in
>>> an entire organization, a huge project putting a robot on a distant
>>> planet, and not once did this occur to anybody!?[/color]
>>
>>Yep, you nailed it.
>>
>>My 5-word software testing book: Run 'er Hard & Long[/color]
>
>Sigh. That is very likely the CAUSE of the problem :-(
>
>Any particular test schedule (artificial or natural) will create a
>distribution of circumstances over the space of all that are handled
>differently by the program. And, remember, we are talking about a
>space of cardinality 10^(10^4) to 10^(10^8). Any particular, broken
>logic (usually a combination of sections of code and data) may be
>invoked only once ever millennium, or perhaps never.
>
>Now, change the test schedule in an apparently trivial way, or use
>the program for real, and that broken logic may be invoked once a
>day. Ouch. Incidentally, another way of looking at this is the
>probability of distinguishing two finite element automata by feeding
>in test strings and comparing the results. It was studied some
>decades back, and the conclusions are not pretty.
>
>The modern, unsystematic approach to testing is hopeless as an
>egnineering technique, though fine as a political or marketing one.
>For high-reliability codes, we need to go back to the approaches
>used in the days when most computer people were also mathematicians,
>engineers or both.
>
>
>Regards,
>Nick Maclaren.[/color]
I agree that more could be done before thorough testing, but I would not
attempt to replace even random and soak testing. Formal methods alone
will never be enough because they prove only the correctness of a
specification implemented by a model, not that the specification or the
model are accurate enough representations of the real world. The speed
with which NASA have claimed to replicate the problem suggests something
widely enough spread through the state space to be replicable by soak
testing. Furthermore, a planetary probe is a pretty good match to even
random testing, because (given the relative costs of putting something
on Mars and of conducting automatic testing in simulation or in a
warehouse) it may be possible to run for longer in test than in real
life, reducing the chance of bugs that show up only in real life.
Example: the priority inversion bug that hit a previous probe had
apparently shown up in testing but been ignored, because it wasn't what
they were looking for at the time. My impression of current good
practice is that black box testing, white box testing, and code review
are good at finding different sorts of bugs, and so should be used
together. I would lump pre-testing approaches into code review.
--
A. G. McDowell

[Nick Maclaren]

Re: Mars Rover Not Responding

In article <XBnrnhAHQOHAFw hg@mcdowella.de mon.co.uk>,
A. G. McDowell <nospam@nospam. co.uk> wrote:[color=blue]
>
>I agree that more could be done before thorough testing, but I would not
>attempt to replace even random and soak testing. Formal methods alone
>will never be enough because they prove only the correctness of a
>specificatio n implemented by a model, not that the specification or the
>model are accurate enough representations of the real world. The speed
>with which NASA have claimed to replicate the problem suggests something
>widely enough spread through the state space to be replicable by soak
>testing. Furthermore, a planetary probe is a pretty good match to even
>random testing, because (given the relative costs of putting something
>on Mars and of conducting automatic testing in simulation or in a
>warehouse) it may be possible to run for longer in test than in real
>life, reducing the chance of bugs that show up only in real life.
>Example: the priority inversion bug that hit a previous probe had
>apparently shown up in testing but been ignored, because it wasn't what
>they were looking for at the time. My impression of current good
>practice is that black box testing, white box testing, and code review
>are good at finding different sorts of bugs, and so should be used
>together. I would lump pre-testing approaches into code review.[/color]

That is a classic example of what I say is mistaken methodology!
Yes, pretty well everything that you say is true, but you have missed
the fact that interactions WILL change the failure syndromes in ways
that mean untargetted testing will miss even the most glaringly
obvious errors. There are just TOO MANY possible combinations of
conditions to rely on random testing.

For a fairly simple or pervasive problem, unintelligent 'soak' testing
will work. For a more complex one, it won't. Unless you target the
testing fairly accurately, certain syndromes will either not occur or
be so rare as not to happen in a month of Sundays. I see this problem
on a daily basis :-(

An aspect of a mathematical design that I did not say explicitly (but
only hinted at) is that you can identify areas to check that the code
matches the mode and other areas where the analysis descends from
mathematics to hand-waving. You can then design precise tests for the
former, and targetted soak tests for the latter. It isn't uncommon
for such an approach to increase the effectiveness of testing beyond
all recognition.


Regards,
Nick Maclaren.

[A. G. McDowell]

Re: Mars Rover Not Responding

In article <bvj7h0$ahh$1@p egasus.csx.cam. ac.uk>, Nick Maclaren
<nmm1@cus.cam.a c.uk> writes
(trimmed)[color=blue]
>
>That is a classic example of what I say is mistaken methodology!
>Yes, pretty well everything that you say is true, but you have missed
>the fact that interactions WILL change the failure syndromes in ways
>that mean untargetted testing will miss even the most glaringly
>obvious errors. There are just TOO MANY possible combinations of
>conditions to rely on random testing.
>[/color]
(trimmed)[color=blue]
>
>An aspect of a mathematical design that I did not say explicitly (but
>only hinted at) is that you can identify areas to check that the code
>matches the mode and other areas where the analysis descends from
>mathematics to hand-waving. You can then design precise tests for the
>former, and targetted soak tests for the latter. It isn't uncommon
>for such an approach to increase the effectiveness of testing beyond
>all recognition.
>
>
>Regards,
>Nick Maclaren.[/color]
I would be very interested to hear more about increasing the
effectiveness of testing beyond all recognition. I am a professional
programmer in an area where we routinely estimate the testing effort as
about equal to the programming effort (in terms of staff time, but not
necessarily staff cost). Do you have references? As a token of sincerity
I will provide references for what we seem to agree is commercial
practice (whether it should be or not):

The main reference establishing commercial practice that I can find
online seems to be "A controlled Experiment in Program Testing and Code
Walkthroughs/Inspections" by Myers, CACM Volume 21, Number 9. The date
shows that some technology transfer might indeed be overdue - Volume 21
translates to 1978! However, I think automated testing, especially
regression testing, has become a lot easier, or at least more popular,
than it was (JUnit, Rational Robot, etc.). The notion of test coverage
tools seems to be of similar vintage and actually became less accessible
for a while as no version of tcov appeared for setups other than K&R C
on Unix. I spent part of last week trying out Hansel, an open source
test coverage tool for Java, available on www.sourceforge.net.

References from within books to hand:
Black box and white box complementary: "Testing Computer Software", by
Kamer, Falk, and Nguyen, Chapter 12 P271.
Code Review invaluable (but few details on how to do one): "Software
Assessments, Benchmarks, and Best Practices", by Capers Jones, e.g.
Chapter on Best Practices for Systems Software, P367
Mars Pathfinder bug ignored during pre-launch tests: "The Practice of
Programming", by Kernighan and Pike, Section 5.2 P121. (The next chapter
is a good short overview of commercial-practice test design circa 1999).
--
A. G. McDowell

[Anne & Lynn Wheeler]

Re: Mars Rover Not Responding

"A. G. McDowell" <nospam@nospam. co.uk> writes:[color=blue]
> I would be very interested to hear more about increasing the
> effectiveness of testing beyond all recognition. I am a professional
> programmer in an area where we routinely estimate the testing effort as
> about equal to the programming effort (in terms of staff time, but not
> necessarily staff cost). Do you have references? As a token of sincerity
> I will provide references for what we seem to agree is commercial
> practice (whether it should be or not):[/color]

when we were doing the original payment gateway

http://www.garlic.com/~lynn/aadsm5.htm#asrn2

http://www.garlic.com/~lynn/aadsm5.htm#asrn3

we set up a test matrix ... not for the sotware ... but for the
service. nominal payment infrastructure trouble desk did 5 minute
problem first level problem determination ... however that was for an
infrastructure that was almost exclusively circuit based.

while it was possible to translate the (payment) message formats from
a circuit based infrastructure to a packet based infrastructure ...
translating the circuit-based service operation to a packet-based
infrastructure was less clear cut (merchant/webhost complains that
payments aren't working ... expects internet/packet connection to be
much less expensive than direct circuit ... but at the same time
expects compareable availability).

The claim has been that coding for a service operation is 4-10 times
that of a straight application implementation and ten times the effort
because of needing to understand all possible failure modes
.... regardless of whether they are characteristic of the software or
hardware or some totally unrelated environmental characteristic.

in any case, one of the issues was detailed analysis of existing
trouble desk circuit-based problem determination procedures and being
able to translate that into a packet-based (internet) environment and
still attempt to come close to the goal of being able to perform first
level problem determination in five minutes. When we started there
were cases of trouble ticket being close NTF (no trouble found) after
3hrs of manual investigation.

of course this was also at a time ... when it was difficult to find
any ISP that even knew how to spell "service level agreement".

aka ... it is possible for software to perform flawlessly and still be
useless.

some of this came from doing ha/cmp

http://www.garlic.com/~lynn/subtopic.html#hacmp

misc. related past threads
http://www.garlic.com/~lynn/2000g.html#50 Egghead cracked, MS IIS again
http://www.garlic.com/~lynn/2001e.html#48 Where are IBM z390 SPECint2000 results?
http://www.garlic.com/~lynn/2001f.html#75 Test and Set (TS) vs Compare and Swap (CS)
http://www.garlic.com/~lynn/2001k.html#18 HP-UX will not be ported to Alpha (no surprise)exit
http://www.garlic.com/~lynn/2001n.html#85 The demise of compaq
http://www.garlic.com/~lynn/2001n.html#91 Buffer overflow
http://www.garlic.com/~lynn/2001n.html#93 Buffer overflow
http://www.garlic.com/~lynn/2002.html#28 Buffer overflow
http://www.garlic.com/~lynn/2002.html#29 Buffer overflow
http://www.garlic.com/~lynn/2002e.html#73 Blade architectures
http://www.garlic.com/~lynn/2002f.html#24 Computers in Science Fiction
http://www.garlic.com/~lynn/2002h.html#11 Why did OSI fail compared with TCP-IP?
http://www.garlic.com/~lynn/2002h.html#12 Why did OSI fail compared with TCP-IP?
http://www.garlic.com/~lynn/2002n.html#11 Wanted: the SOUNDS of classic computing
http://www.garlic.com/~lynn/2003b.html#53 Microsoft worm affecting Automatic Teller Machines
http://www.garlic.com/~lynn/2003g.html#62 IBM says AMD dead in 5yrs ... -- Microsoft Monopoly vs. IBM
http://www.garlic.com/~lynn/2003j.html#15 A Dark Day
http://www.garlic.com/~lynn/2003l.html#49 Thoughts on Utility Computing?
http://www.garlic.com/~lynn/2003p.html#37 The BASIC Variations

--
Anne & Lynn Wheeler | http://www.garlic.com/~lynn/
Internet trivia 20th anv http://www.garlic.com/~lynn/rfcietff.htm

[arne thormodsen]

Re: Mars Rover Not Responding


"A. G. McDowell" <nospam@nospam. co.uk> wrote in message
news:AZXuhVAHKW HAFwjb@mcdowell a.demon.co.uk.. .[color=blue]
> In article <bvj7h0$ahh$1@p egasus.csx.cam. ac.uk>, Nick Maclaren
> <nmm1@cus.cam.a c.uk> writes
> (trimmed)[color=green]
> >
> >That is a classic example of what I say is mistaken methodology[/color][/color]

I hate to barge in and short-circuit a really very interesting
discussion, but
I think a point has been missed.

The SW on the Rover was designed from the start to be remotely
debugged
and patched. This is a truely wonderful thing. Why spend the effort
it takes
to design bullet-proof SW when you can spend it designing fixable SW?

--arne (I'll take my answer off the air... ;-)

[Jan C. Vorbrüggen]

Re: Mars Rover Not Responding

> Example: the priority inversion bug that hit a previous probe

....Mars Pathfinder...
[color=blue]
> had apparently shown up in testing but been ignored, because it
> wasn't what they were looking for at the time.[/color]

Of course, that project was run on a shoe-string budget (relatively),
the priority-inversion-caused resets occured sporadically (only a few
times during maybe a year's worth of testing) and unreproducibly, and
most importantly: those guys had a strict deadline, and a lot of other
problems to solve. A rasonable trade-off, all things considered, IMO.

Jan

[Nick Maclaren]

Re: Mars Rover Not Responding


In article <AZXuhVAHKWHAFw jb@mcdowella.de mon.co.uk>,
"A. G. McDowell" <nospam@nospam. co.uk> writes:
|> >
|> I would be very interested to hear more about increasing the
|> effectiveness of testing beyond all recognition. I am a professional
|> programmer in an area where we routinely estimate the testing effort as
|> about equal to the programming effort (in terms of staff time, but not
|> necessarily staff cost). Do you have references? As a token of sincerity
|> I will provide references for what we seem to agree is commercial
|> practice (whether it should be or not):

Regrettably not :-( I have seen references, yes, but they were
usually incidental remarks and I can't now remember exactly where I
saw them. The trouble is that they dated from the days when such
things were 'just done' and the techniques were often passed by
word of mouth, or were regarded as so obvious as to be not worth
describing.

One aspect you may be able to find is in the testing of numeric
code. One standard form of targetting is to increase the coverage
of the 'difficult' or boundary areas, because a relatively small
number of tests is adequate for the main sections. I know that
something similar has also been done in compiler test suites.

The only 'theoretical' reference I know of is to a related area:
Monte-Carlo methods. But, if you regard the problem as estimating
the number of bugs, then that is immediately applicable. I use
Hammersley and Handscomb, but that may be out of print.

Thanks for the references. There is certainly stuff dating from
the 1960s and early 1970s, but it could be the devil to track down.
NAG has certainly been doing it since it started (early 1970s),
in the numeric context.


Regards,
Nick Maclaren.