struct doesn't handle NaN values?

**John Roth** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

"Grant Edwards" <grante@visi.co m> wrote in message
news:slrnca7k1l .an4.grante@gra nte.rivatek.com ...[color=blue]
> Perhaps I'm doing something wrong: the struct module docs say
> it's IEE 754, but I can't figure out how to get it to handle
> NaN values correctly (either packing or unpacking).
>[color=green][color=darkred]
> >>> x = float('nan')
> >>> struct.pack("<f ",x)[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> SystemError: frexp() result out of range
>[color=green][color=darkred]
> >>> struct.unpack(" <f",'\xff\xff\x ff\xff')[/color][/color]
> (-6.8056469327705 772e+38,)
>
> I don't have my copy of 754 at hand, but I'm pretty sure that
> 0xffffffff is a NaN (printf on IA32 Linux agrees) and not
> -6.8056469327705 772e+38 as claimed by struct.unpack() .[/color]

I believe that struct uses the c library as is, like the
float support in general. Unfortunately, that leaves
everyone at the mercy of the rather inconsistent
and ideosyncratic implementations of corner cases
in existing c libraries. See PEP 754 for a discussion
of the issues.

John Roth[color=blue]
>
> --
> Grant Edwards grante Yow! Of course, you
> at UNDERSTAND about the[/color]
PLAIDS[color=blue]
> visi.com in the SPIN CYCLE --[/color]

**David M. Cooke** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

At some point, Grant Edwards <grante@visi.co m> wrote:
[color=blue]
> Perhaps I'm doing something wrong: the struct module docs say
> it's IEE 754, but I can't figure out how to get it to handle
> NaN values correctly (either packing or unpacking).
>[color=green][color=darkred]
>>>> x = float('nan')
>>>> struct.pack("<f ",x)[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> SystemError: frexp() result out of range[/color]

Doing this with the native format is no trouble:[color=blue][color=green][color=darkred]
>>> struct.pack('f' , x)[/color][/color][/color]
'\x00\x00\xc0\x 7f'

....which is a NaN since the exponent part is all 1's and the
significand is non-zero.
[color=blue][color=green][color=darkred]
>>>> struct.unpack(" <f",'\xff\xff\x ff\xff')[/color][/color]
> (-6.8056469327705 772e+38,)[/color]

Again,[color=blue][color=green][color=darkred]
>>> struct.unpack(' f', '\xff\xff\xff\x ff')[/color][/color][/color]
(nan,)

Of course, if you want to worry about endian issues here, I think
you're SOL with using struct.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca

**Grant Edwards** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

On 2004-05-13, John Roth <newsgroups@jhr othjr.com> wrote:
[color=blue][color=green]
>> Perhaps I'm doing something wrong: the struct module docs say
>> it's IEE 754, but I can't figure out how to get it to handle
>> NaN values correctly (either packing or unpacking).[/color][/color]
[color=blue]
> I believe that struct uses the c library as is,[/color]

That's not what the docs say. The struct docs says it converts
between native Python values and IEEE 754. If that's not the
case and it converts between Python float format and "native C
library" format, then the docs need to be changed [and I'll
need to write something that converts to/from IEEE 754 format.]
[color=blue]
> like the float support in general. Unfortunately, that leaves
> everyone at the mercy of the rather inconsistent and
> ideosyncratic implementations of corner cases in existing c
> libraries.[/color]

But, my c library seems to handle it correctly:

1) 0xffffffff is treated correctly as a NaN.
2) NaN's are detected and converted to strings as 'nan'.

Which part of the C library is broken?

--
Grant Edwards grante Yow! AIEEEEE! I am having
at an UNDULATING EXPERIENCE!
visi.com

**Grant Edwards** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

On 2004-05-13, David M. Cooke <cookedm+news@p hysics.mcmaster .ca> wrote:
[color=blue][color=green]
>> Perhaps I'm doing something wrong: the struct module docs say
>> it's IEE 754, but I can't figure out how to get it to handle
>> NaN values correctly (either packing or unpacking).[/color][/color]
[color=blue]
> Doing this with the native format is no trouble:[/color]
[color=blue]
> Of course, if you want to worry about endian issues here, I think
> you're SOL with using struct.[/color]

Yes, I want to worry about endian issues. I'm communicating
via DeviceNet, and DeviceNet specifies little-endian. I don't
want to assume the host is little-endian, so I guess I'm SOL. :(

OTOH, I suppose if I assume that 32-bit integers and floats
have the same byte order, I can use 32-bit native integers as
an intermediate format:

nanString = struct.pack("f" ,float('nan'))
nanInt, = struct.unpack(" I",nanString )
extNan = struct.pack("<I ",nanInt)

That's ugly, and I'm sure will break someday, but I guess it
works for now.

I suppose I should take a look at the sources for struct and
see if I can fix it...

--
Grant Edwards grante Yow! How's it going in
at those MODULAR LOVE UNITS??
visi.com

**John Roth** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

"Grant Edwards" <grante@visi.co m> wrote in message
news:slrnca7q3u .bdf.grante@gra nte.rivatek.com ...[color=blue]
> On 2004-05-13, John Roth <newsgroups@jhr othjr.com> wrote:
>[color=green][color=darkred]
> >> Perhaps I'm doing something wrong: the struct module docs say
> >> it's IEE 754, but I can't figure out how to get it to handle
> >> NaN values correctly (either packing or unpacking).[/color][/color]
>[color=green]
> > I believe that struct uses the c library as is,[/color]
>
> That's not what the docs say. The struct docs says it converts
> between native Python values and IEEE 754. If that's not the
> case and it converts between Python float format and "native C
> library" format, then the docs need to be changed [and I'll
> need to write something that converts to/from IEEE 754 format.][/color]

Actually, it doesn't. It does say IEEE, but it doesn't say
IEEE 754. While that's a nit, it also doesn't say it converts
anything other than byte order. When you look at the source,
I suspect that what you'll find is a straight copy from the buffer
into wherever the float object keeps the actual value, or the
reverse, of course.[color=blue]
>[color=green]
> > like the float support in general. Unfortunately, that leaves
> > everyone at the mercy of the rather inconsistent and
> > ideosyncratic implementations of corner cases in existing c
> > libraries.[/color]
>
> But, my c library seems to handle it correctly:
>
> 1) 0xffffffff is treated correctly as a NaN.
> 2) NaN's are detected and converted to strings as 'nan'.
>
> Which part of the C library is broken?[/color]

Which C library? Python runs on 20 different systems,
many of which have multiple operating systems, each
of which has its own C library with its own problems.
It's not even a standards issue: the older standards didn't
specify what the C library's conversion routines should do.

I referred you to PEP 754 for a reason. That PEP contains
a thorough discussion of the issues in the treatment of special
values in Python floating point.

PEP 754 – IEEE 754 Floating Point Special Values | peps.python.org

http://www.python.org/peps/pep-0754.html

This PEP proposes an API and a provides a reference module that generates and tests for IEEE 754 double-precision special values: positive infinity, negative infinity, and not-a-number (NaN).

It also contains a reference to a module that handles
the matter.

John Roth[color=blue]
>
> --
> Grant Edwards grante Yow! AIEEEEE! I am[/color]
having[color=blue]
> at an UNDULATING[/color]
EXPERIENCE![color=blue]
> visi.com[/color]

**Grant Edwards** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

On 2004-05-13, John Roth <newsgroups@jhr othjr.com> wrote:
[color=blue][color=green]
>> Which part of the C library is broken?[/color]
>
> Which C library?[/color]

That's what I'm asked? I was told the C library was broken. I
wanted to know what C library.
[color=blue]
> Python runs on 20 different systems, many of which have
> multiple operating systems, each of which has its own C
> library with its own problems. It's not even a standards
> issue: the older standards didn't specify what the C library's
> conversion routines should do.[/color]

I presumed that struct was doing the conversion itself. The
doc specified IEEE format, it would have to do it's own
conversion since it couldn't assume that the host used IEEE
format, and as you say, there's no portable library support
that can be relied upon.
[color=blue]
> I referred you to PEP 754 for a reason. That PEP contains
> a thorough discussion of the issues in the treatment of special
> values in Python floating point.
>
> http://www.python.org/peps/pep-0754.html
>
> It also contains a reference to a module that handles
> the matter.[/color]

I read it. It only handles double-precision values, and I'm
working with single-precision values.

--
Grant Edwards grante Yow! I just had my entire
at INTESTINAL TRACT coated
visi.com with TEFLON!

**John Roth** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

"Grant Edwards" <grante@visi.co m> wrote in message
news:slrnca7slj .bsf.grante@gra nte.rivatek.com ...[color=blue]
> On 2004-05-13, John Roth <newsgroups@jhr othjr.com> wrote:
>[color=green][color=darkred]
> >> Which part of the C library is broken?[/color]
> >
> > Which C library?[/color]
>
> That's what I'm asked? I was told the C library was broken. I
> wanted to know what C library.[/color]

The breakage has to do with the character values they
use to denote infinities and NaN, not (as far as
I know) with the detection of them. Whether they are
"broken" is a matter of interpretation, since the C standard
didn't specify what the library should expect and return
when converting from string to float and vice versa.
What is true is that they aren't consistent.[color=blue]
>[color=green]
> > Python runs on 20 different systems, many of which have
> > multiple operating systems, each of which has its own C
> > library with its own problems. It's not even a standards
> > issue: the older standards didn't specify what the C library's
> > conversion routines should do.[/color]
>
> I presumed that struct was doing the conversion itself. The
> doc specified IEEE format, it would have to do it's own
> conversion since it couldn't assume that the host used IEEE
> format, and as you say, there's no portable library support
> that can be relied upon.[/color]

I don't think so. I believe that the reference to IEEE means
that it generally expects bit compatible IEEE representations .
There may be differences, but I doubt if struct is aware of
them. As usual, the source would be definitive.
[color=blue][color=green]
> > I referred you to PEP 754 for a reason. That PEP contains
> > a thorough discussion of the issues in the treatment of special
> > values in Python floating point.
> >
> > http://www.python.org/peps/pep-0754.html
> >
> > It also contains a reference to a module that handles
> > the matter.[/color]
>
> I read it. It only handles double-precision values, and I'm
> working with single-precision values.[/color]

I expect it would be easy enough to convert. I just read
it, and it doesn't look at all complicated.

John Roth

[color=blue]
>
> --
> Grant Edwards grante Yow! I just had my[/color]
entire[color=blue]
> at INTESTINAL TRACT coated
> visi.com with TEFLON![/color]

**Grant Edwards** · Jul 18 '05, 10:53 AM

Re: struct doesn't handle NaN values?

In article <10a88p7deudehc a@news.supernew s.com>, John Roth wrote:
[color=blue][color=green][color=darkred]
>>>> Which part of the C library is broken?
>>>
>>> Which C library?[/color]
>>
>> That's what I'm askeing? I was told the C library was broken.
>> I wanted to know what C library.[/color]
>
> The breakage has to do with the character values they use to
> denote infinities and NaN,[/color]

Right. That's got nothing to do with what I'm currently
whining about, which was struct's failure when byte ordering is
specified to convert properly between a native NaN and IEEE 754
NaN (even when native format is IEEE 754) . I'm quite happy
with the way string<->native-float handles NaNs on the host I'm
using. Not that it wouldn't be nice for it to be consistent
across platforms.
[color=blue]
> not (as far as I know) with the detection of them. Whether
> they are "broken" is a matter of interpretation, since the C
> standard didn't specify what the library should expect and
> return when converting from string to float and vice versa.
> What is true is that they aren't consistent.[/color]

Though I realize there are consistency issues, they work quite
well enough for my application at the moment. Struct, however,
doesn't.
[color=blue][color=green]
>> I presumed that struct was doing the conversion itself. The
>> doc specified IEEE format, it would have to do it's own
>> conversion since it couldn't assume that the host used IEEE
>> format, and as you say, there's no portable library support
>> that can be relied upon.[/color]
>
> I don't think so.[/color]

Eh? So you think that struct is using a C library to do the
conversion, or that there is portable C library support for
converting between native FP format and IEEE 754?
[color=blue]
> I believe that the reference to IEEE means that it generally
> expects bit compatible IEEE representations .[/color]

Hmm, what would be the difference would be between standrd IEEE
representation and a "bit-compatible IEEE representation" .
[color=blue]
> There may be differences, but I doubt if struct is aware of
> them. As usual, the source would be definitive.
>[color=green][color=darkred]
>>> It also contains a reference to a module that handles the
>>> matter.[/color]
>>
>> I read it. It only handles double-precision values, and I'm
>> working with single-precision values.[/color]
>
> I expect it would be easy enough to convert. I just read it,
> and it doesn't look at all complicated.[/color]

Probably not. I've done it before (in C) but having struct
convert the values properly would be right thing to do rather
expect the user to check for certain values that struct doesn't
convert correctly.

--
Grant Edwards grante Yow! My LESLIE GORE record
at is BROKEN...
visi.com

struct doesn't handle NaN values?

struct doesn't handle NaN values?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment