Numerics, NaNs, IEEE 754 and C99

[Simon Brunning]

Re: Numerics, NaNs, IEEE 754 and C99

On 14 Jun 2006 10:33:21 -0700, Sébastien Boisgérault
<Sebastien.Bois gerault@gmail.c om> wrote:[color=blue]
> Jeez, 12 posts in this IEEE 754 thread, and still
> no message from uncle timmy ? ;)[/color]

Somebody reboot the timbot, please. Seems to have hung.

--
Cheers,
Simon B,
simon@brunningo nline.net,

http://www.brunningonline.net/simon/blog/

[Grant Edwards]

Re: Numerics, NaNs, IEEE 754 and C99

On 2006-06-14, Sébastien Boisgérault <Sebastien.Bois gerault@gmail.c om> wrote:
[color=blue]
> Jeez, 12 posts in this IEEE 754 thread, and still no message
> from uncle timmy ? ;)
>
> Please, we need enlightenment here and *now* :)[/color]

What we need is fewer people like me who do nothing but
complain about it...

--
Grant Edwards grante Yow! I am covered with
at pure vegetable oil and I am
visi.com writing a best seller!

[Grant Edwards]

Re: Numerics, NaNs, IEEE 754 and C99

On 2006-06-14, Nick Maclaren <nmm1@cus.cam.a c.uk> wrote:[color=blue]
>
> In article <mailman.7040.1 150302496.27775 .python-list@python.org >,
> Gary Herron <gherron@island training.com> writes:
>|>
>|> The IEEE standard specifies (plus or minus) infinity as the result of
>|> division by zero. This makes sense since such is the limit of division
>|> by a quantity that goes to zero. The IEEE standard then goes on to
>|> define reasonable results for arithmetic between infinities and real
>|> values. The production of, and arithmetic on, infinities is a choice
>|> that any application may want allow or not.
>
> The mistake you have made (and it IS a mistake) is in assuming
> that the denominator approaches zero from the direction
> indicated by its sign.[/color]

I assume the "you" in that sentence refers to the IEEE FP
standards group. I just try to follow the standard, but I have
found that the behavior required by the IEEE standard is
generally what works best for my applications.
[color=blue]
> There are many reasons why it is likely to not be, but let's give only
> two:
>
> It may be a true zero - i.e. a count that is genuinely zero, or
> the result of subtracting a number from itself.[/color]

I do real-world engineering stuff with measured physical
quatities. There generally is no such thing as "true zero".
[color=blue]
> I fully agree that infinity arithmetic is fairly well-defined for
> most operations, but it most definitely is not in this case. It should
> be reserved for when the operations have overflowed.[/color]

All I can say is that 1/0 => Inf sure seems to work well for
me.

--
Grant Edwards grante Yow! World War Three can
at be averted by adherence
visi.com to a strictly enforced
dress code!

[Nick Maclaren]

Re: Numerics, NaNs, IEEE 754 and C99


In article <12917f6jp7u90f 8@corp.supernew s.com>,
Grant Edwards <grante@visi.co m> writes:
|>
|> I assume the "you" in that sentence refers to the IEEE FP
|> standards group. I just try to follow the standard, but I have
|> found that the behavior required by the IEEE standard is
|> generally what works best for my applications.

Well, it could be, but actually it was a reference to the sentence "This
makes sense since such is the limit of division by a quantity that goes
to zero."

|> I do real-world engineering stuff with measured physical
|> quatities. There generally is no such thing as "true zero".

It is extremely unusual for even such programs to use ONLY continuous
interval scale quantities, but they might dominate your usage, I agree.
Such application areas are very rare, but do exist. For example, I can
tell that you don't use statistics in your work, and therefore do not
handle events (including the analysis of failure rates).

|> > I fully agree that infinity arithmetic is fairly well-defined for
|> > most operations, but it most definitely is not in this case. It should
|> > be reserved for when the operations have overflowed.
|>
|> All I can say is that 1/0 => Inf sure seems to work well for me.

Now, can you explain why 1/0 => -Inf wouldn't work as well? I.e. why
are ALL of your zeroes, INCLUDING those that arise from subtractions,
are known to be positive?

If you can, then you have a case (and an EXTREMELY unusual application
domain. If you can't, then I am afraid that your calculations are
unreliable, at best.

The point here is that +infinity is the correct answer when the zero is
known to be a positive infinitesimal, just as -infinity is when it is
known to be a negative one. NaN is the only numerically respectable
result if the sign is not known, or it might be a true zero.


Regards,
Nick Maclaren.

[Christophe]

Re: Numerics, NaNs, IEEE 754 and C99

Nick Maclaren a écrit :[color=blue]
> Now, can you explain why 1/0 => -Inf wouldn't work as well? I.e. why
> are ALL of your zeroes, INCLUDING those that arise from subtractions,
> are known to be positive?[/color]

I would say that the most common reason people assume 1/0 = Inf is
probably because they do not make use of negative numbers or they forgot
they exist at all.

[Nick Maclaren]

Re: Numerics, NaNs, IEEE 754 and C99


In article <44912058$0$296 74$636a55ce@new s.free.fr>,
Christophe <chris.cavalari a@free.fr> writes:
|>
|> > Now, can you explain why 1/0 => -Inf wouldn't work as well? I.e. why
|> > are ALL of your zeroes, INCLUDING those that arise from subtractions,
|> > are known to be positive?
|>
|> I would say that the most common reason people assume 1/0 = Inf is
|> probably because they do not make use of negative numbers or they forgot
|> they exist at all.

Usually the latter :-(

Yes, if you are working with the non-negative real numbers (or even
non-negative integers), then the best result of 1/0 is +infinity.
Thus, if you are working with events, a count of zero is a true zero,
but its inverse can reasonably be said to be +infinity.

However, it is common for non-experts to assume that they are working
in the non-negative real domain, because numbers are conventionally
measured that way, but in fact that is not so. The use of elapsed
times is one such "gotcha". I have seen quite a few programs that
have assumed that elapsed times were always positive, and which have
blown up when applied to real problems, where the measurement of an
elapsed time may be negative.

Hence, the SAFE approach is to make the inverse of all zeros a NaN.


Regards,
Nick Maclaren.

[Grant Edwards]

Re: Numerics, NaNs, IEEE 754 and C99

On 2006-06-15, Nick Maclaren <nmm1@cus.cam.a c.uk> wrote:[color=blue]
>
> In article <12917f6jp7u90f 8@corp.supernew s.com>,
> Grant Edwards <grante@visi.co m> writes:
>|>
>|> I assume the "you" in that sentence refers to the IEEE FP
>|> standards group. I just try to follow the standard, but I have
>|> found that the behavior required by the IEEE standard is
>|> generally what works best for my applications.
>
> Well, it could be, but actually it was a reference to the
> sentence "This makes sense since such is the limit of division
> by a quantity that goes to zero."[/color]

That sentence was written by the IEEE FP standards committee
explaining why they chose the behavior they did.
[color=blue]
> The point here is that +infinity is the correct answer when the zero is
> known to be a positive infinitesimal, just as -infinity is when it is
> known to be a negative one. NaN is the only numerically respectable
> result if the sign is not known, or it might be a true zero.[/color]

OK, you're right. I still prefer that Python follow the
standard they everything else does. Part of what I use Python
for is to simulate devices that obey the IEEE 754 floating
point standards. I need Python to follow the standard.

--
Grant Edwards grante Yow! Yow! Now we can
at become alcoholics!
visi.com

[Grant Edwards]

Re: Numerics, NaNs, IEEE 754 and C99

On 2006-06-15, Nick Maclaren <nmm1@cus.cam.a c.uk> wrote:
[color=blue]
> Hence, the SAFE approach is to make the inverse of all zeros a
> NaN.[/color]

OK. You're right. I'm wrong about what my Python programs
should do.

In any case, that ship sailed.

Maybe you can argue convincingly that theoretically your
approach is better than IEEE 754. You're not going to get the
standard changed. You're not going to get all of the computers
on the planet re-worked. Making Python incompatible with IEEE
754 is a bad idea.

I might be able to come up with a convincing arguement that
driving on the right-hand side of the road is better than
driving on the left, but that doesn't make a good idea to do so
when in England, Japan, or Australia.

--
Grant Edwards grante Yow! This PIZZA symbolizes
at my COMPLETE EMOTIONAL
visi.com RECOVERY!!

[rainbow.cougar@gmail.com]

Re: Numerics, NaNs, IEEE 754 and C99

Nick Maclaren wrote:[color=blue]
> ...
> Now, I should like to improve this, but there are two problems. The
> first is political, and is whether it would be acceptable in Python to
> restore the semantics that were standard up until about 1980 in the
> numerical programming area. I.e. one where anything that is numerically
> undefined or at a singularity which can deliver more than one value is
> an error state (e.g. raises an an exception or returns a NaN). This
> is heresy in the C99 and Java camps, and is none too acceptable in the
> IEEE 754R one.
>
>...[/color]

As one of the few people on this group who will admit to programming
before 1980, let alone doing numerical programming then (I presume
writing code to process gigabytes of seismic data via Fast Fourier
equations meet your criteria), the only semantics that existed then
were for the program to *CRASH* hard when a singularity or undefined
state occurred. When using the various flavors of FORTRAN and vector
processors the undefined states could take a few microseconds to
manifest, but crash and burn they would, with some difficulty to
analyze the burning wreckage.

C is not a mathematically based language as FORTRAN is, so it is funny
to criticize it for being what it never was, however IEEE754 and NaN's
and the other standardization s put into place make back analyzing what
is wrong with your program and/or data far easier then what we had
before.


Curtis

[Nick Maclaren]

Re: Numerics, NaNs, IEEE 754 and C99


In article <1292p7s46ef5of f@corp.supernew s.com>,
Grant Edwards <grante@visi.co m> writes:
|> On 2006-06-15, Nick Maclaren <nmm1@cus.cam.a c.uk> wrote:
|>
|> > Hence, the SAFE approach is to make the inverse of all zeros a
|> > NaN.
|>
|> OK. You're right. I'm wrong about what my Python programs
|> should do.
|>
|> In any case, that ship sailed.

And sank, spawning a zillion lifeboats heading in different directions :-)

|> Maybe you can argue convincingly that theoretically your
|> approach is better than IEEE 754. You're not going to get the
|> standard changed. You're not going to get all of the computers
|> on the planet re-worked. Making Python incompatible with IEEE
|> 754 is a bad idea.

No, that is wrong, on many counts.

Firstly, a FAR more common assumption is that integers wrap in twos'
complement - Python does not do that.

Secondly, it is NOT incompatible with IEEE 754, which is a pure hardware
standard. All it does is to trap the exception and take appropriate
action (as permitted by that standard).

Thirdly, virtually no hardware sticks strictly to IEEE 754, and no
language that I know of has EVER attempted to make it the strict basis
for its arithmetic model.

Fourthly, I am not proposing to change any hardware, and could even
provide a Python option to deliver mathematically incorrect results
when you want them.


Regards,
Nick Maclaren.

[Nick Maclaren]

Re: Numerics, NaNs, IEEE 754 and C99


In article <1150379424.085 598.4130@u72g20 00cwu.googlegro ups.com>,
"rainbow.cougar @gmail.com" <rainbow.cougar @gmail.com> writes:
|>
|> As one of the few people on this group who will admit to programming
|> before 1980, let alone doing numerical programming then (I presume
|> writing code to process gigabytes of seismic data via Fast Fourier
|> equations meet your criteria), the only semantics that existed then
|> were for the program to *CRASH* hard when a singularity or undefined
|> state occurred. When using the various flavors of FORTRAN and vector
|> processors the undefined states could take a few microseconds to
|> manifest, but crash and burn they would, with some difficulty to
|> analyze the burning wreckage.

Yes, it does, but I had used some dozens of systems, Fortrans and other
languages by then. I did not make myself entirely clear, which partly
accounts for your response, but I am afraid that you are mistaken.

What I was referring to was that the consensus was that it was a
program error to calculate a value at a singularity or otherwise when
the result was mathematically undefined.

Yes, the languages specified that to be undefined behaviour (and usually
still do), but it was clear that a good implementation was ALOLOWED to
trap and diagnose the failures. Many did and some still do - Python is
pretty good, but not in this area.

Examples of numerically near-bulletproof compilers included Egtran, XFAT
(if I recall after 35+ years), WATFIV, Delft Algol, SPITBOL, FLACC and
many others (and some in the 1980s, too). MOST of the better Fortran
run-time systems (even IBM's Mod II library) used to detect and flag
invalid arguments to library routines.

==>> Why should Python regard that, provably achievable, level of
robustness as positevely undesirable?

|> C is not a mathematically based language as FORTRAN is, so it is funny
|> to criticize it for being what it never was, however IEEE754 and NaN's
|> and the other standardization s put into place make back analyzing what
|> is wrong with your program and/or data far easier then what we had
|> before.

As someone who has done it more-or-less continually since the 1960s,
I am afraid that is not so. Yes, NaNs COULD do that, if the (fixable)
flaws in IEEE 754 were sorted out, but Java and C99 have picked up on the
flaws and used them as justification for ignoring 90% of the principles
of IEEE 754.

==>> But the flaws in those are not the point here. I am asking
whether people would positively OBJECT (as the did in C99) to
errors being detected, possibly as an overridable option.


Regards,
Nick Maclaren.

[Grant Edwards]

Re: Numerics, NaNs, IEEE 754 and C99

On 2006-06-15, Nick Maclaren <nmm1@cus.cam.a c.uk> wrote:
[color=blue]
>|> Making Python incompatible with IEEE |> 754 is a bad idea.
>
> No, that is wrong, on many counts.
>
> Firstly, a FAR more common assumption is that integers wrap in twos'
> complement - Python does not do that.[/color]

It used to, and I sure wish there was still a way to get that
behavior back. Now I have to do all sorts of bitwise ands that
I didn't use to.
[color=blue]
> Secondly, it is NOT incompatible with IEEE 754, which is a
> pure hardware standard. All it does is to trap the exception
> and take appropriate action (as permitted by that standard).[/color]

There are two allowed (selectable?) behaviors for 1/0: trap and
Inf.
[color=blue]
> Thirdly, virtually no hardware sticks strictly to IEEE 754,
> and no language that I know of has EVER attempted to make it
> the strict basis for its arithmetic model.[/color]

All the HW and libraries I've used returned Inf for 1/0.
[color=blue]
> Fourthly, I am not proposing to change any hardware,[/color]

IMO, having Python's FP results disagree with common HW FP
results would be a pretty bad thing.
[color=blue]
> and could even provide a Python option to deliver
> mathematically incorrect results when you want them.[/color]

As long as I can get IEEE 754 results, that's cool. I'd prefer
if Python just let the HW do it's thing when possible.

--
Grant Edwards grante Yow! How do I get HOME?
at
visi.com

[Nick Maclaren]

Re: Numerics, NaNs, IEEE 754 and C99


In article <1292tvei30ahhd 4@corp.supernew s.com>,
Grant Edwards <grante@visi.co m> writes:
|> >
|> > Firstly, a FAR more common assumption is that integers wrap in twos'
|> > complement - Python does not do that.
|>
|> It used to, and I sure wish there was still a way to get that
|> behavior back. Now I have to do all sorts of bitwise ands that
|> I didn't use to.

Given that it is perhaps THE most common cause of severe failure
in large scientific codes, I don't agree that it should be the default.
A 'twos complement' class would be quite reasonable.

|> > Secondly, it is NOT incompatible with IEEE 754, which is a
|> > pure hardware standard. All it does is to trap the exception
|> > and take appropriate action (as permitted by that standard).
|>
|> There are two allowed (selectable?) behaviors for 1/0: trap and
|> Inf.

Er, no. The second is REQUIRED to set an exception flag, which IEEE 754
assumes that the code will test and take appropriate action (which can
be anything, including aborting the program and replacing it by a NaN).
See http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf

|> > Thirdly, virtually no hardware sticks strictly to IEEE 754,
|> > and no language that I know of has EVER attempted to make it
|> > the strict basis for its arithmetic model.
|>
|> All the HW and libraries I've used returned Inf for 1/0.

Well, I have clearly used a much wider range than you have, then.
But it isn't just that aspect I was referring to. You will also find
that a high proportion of hardware traps into the kernel on overflow,
and it is software that returns the infinity.

|> > Fourthly, I am not proposing to change any hardware,
|>
|> IMO, having Python's FP results disagree with common HW FP
|> results would be a pretty bad thing.

No matter WHAT results are returned, they will do that! I was triggered
into this project by attempting to write NaN-safe code and finding that
the first few versions of Python I tried all did different things.
When I saw that it just mapped what C does, all became clear.

|> > and could even provide a Python option to deliver
|> > mathematically incorrect results when you want them.
|>
|> As long as I can get IEEE 754 results, that's cool. I'd prefer
|> if Python just let the HW do it's thing when possible.

Including crash and burn?

Seriously. I don't think that you realise just how many layers of
fixup there are on a typical "IEEE 754" system, and how many variations
there are even for a single platform.


Regards,
Nick Maclaren.

[Fredrik Lundh]

Re: Numerics, NaNs, IEEE 754 and C99

Grant Edwards wrote:
[color=blue][color=green]
>> Firstly, a FAR more common assumption is that integers wrap in twos'
>> complement - Python does not do that.[/color]
>
> It used to[/color]

for integers ? what version was that ?

</F>

[Grant Edwards]

Re: Numerics, NaNs, IEEE 754 and C99

On 2006-06-15, Nick Maclaren <nmm1@cus.cam.a c.uk> wrote:
[color=blue]
>|>> Firstly, a FAR more common assumption is that integers wrap in twos'
>|>> complement - Python does not do that.
>|>
>|> It used to, and I sure wish there was still a way to get that
>|> behavior back. Now I have to do all sorts of bitwise ands that
>|> I didn't use to.
>
> Given that it is perhaps THE most common cause of severe
> failure in large scientific codes, I don't agree that it
> should be the default. A 'twos complement' class would be
> quite reasonable.[/color]

Since Python went to non-fixed-length integers, a couple of
those classes have been written. I tried one of them and it
was rather clumsy. There was no conversion/coercion between
different widths or between fixed-width integers and "regular"
integers. That mean you had to call a constructor even if all
you wanted to do was add 2 to a value.
[color=blue]
>|> > Secondly, it is NOT incompatible with IEEE 754, which is a
>|> > pure hardware standard. All it does is to trap the exception
>|> > and take appropriate action (as permitted by that standard).
>|>
>|> There are two allowed (selectable?) behaviors for 1/0: trap and
>|> Inf.
>
> Er, no. The second is REQUIRED to set an exception flag,[/color]

But it's not required to generate a trap according to my
reading of the spec.
[color=blue]
> which IEEE 754 assumes that the code will test and take
> appropriate action (which can be anything, including aborting
> the program and replacing it by a NaN). See
> http://www.cs.berkeley.edu/~wkahan/JAVAhurt.pdf[/color]

IEEE Std 754-1985, subclause 7.2 - Division by Zero

"If the divisor is zero and the dividend is a finite nonzero
number, then the division by zero shall be signaled. The
result, when no trap occurs, shall be a correctly signed
(infinity symbol)(6.3)."

To me it looks like the spec specifically allows for a case
where "no trap occurrs" and the result is Inf.
[color=blue]
>|> As long as I can get IEEE 754 results, that's cool. I'd prefer
>|> if Python just let the HW do it's thing when possible.
>
> Including crash and burn?[/color]

No. :)
[color=blue]
> Seriously. I don't think that you realise just how many
> layers of fixup there are on a typical "IEEE 754" system, and
> how many variations there are even for a single platform.[/color]

Probably not.

--
Grant Edwards grante Yow! My face is new, my
at license is expired, and I'm
visi.com under a doctor's care!!!!