interpret 4 byte as 32-bit float (IEEE-754)

[franzkowiak]

Hello,

I've read some bytes from a file and just now I can't interpret 4 bytes
in this dates like a real value.

An extract from my program

def l32(c):
return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) + (ord(c[3])<<24)

....

value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11



Anybody an answer ?



regards
gf

[Scott David Daniels]

Re: interpret 4 byte as 32-bit float (IEEE-754)

franzkowiak wrote:[color=blue]
> I've read some bytes from a file and just now I can't interpret 4 bytes
> in this dates like a real value. An extract from my program:
> def l32(c):
> return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) + (ord(c[3])<<24)
> ...
> value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11
>[/color]
OK, here's the skinny (I used blocks & views to get the answer):

import struct
bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
struct.unpack(' >f', bytes)

I was suspicious of that first byte, thought it might be an exponent,
since it seemed to have too many on bits in a row to be part of 1.11.

-Scott David Daniels
Scott.Daniels@A cm.Org

[Grant Edwards]

Re: interpret 4 byte as 32-bit float (IEEE-754)

On 2005-01-15, Scott David Daniels <Scott.Daniels@ Acm.Org> wrote:
[color=blue][color=green]
>> I've read some bytes from a file and just now I can't interpret 4 bytes
>> in this dates like a real value.[/color][/color]
[color=blue]
> OK, here's the skinny (I used blocks & views to get the answer):
>
> import struct
> bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
> struct.unpack(' >f', bytes)[/color]

Just be careful. That doesn't work for all 32-bit IEEE floating
point values:
[color=blue][color=green][color=darkred]
>>> import struct
>>> bytes = '\xff\xff\xff\x ff'
>>> print struct.unpack(' >f',bytes)[/color][/color][/color]
(-6.8056469327705 772e+38,)

0xffffff is _not_ -6.8...e38. It's a NaN.

IIRC, it doesn't work for infinities either. I haven't tried
denormals.

--
Grant Edwards grante Yow! It's hard being
at an ARTIST!!
visi.com

[G.Franzkowiak]

Re: interpret 4 byte as 32-bit float (IEEE-754)

Scott David Daniels schrieb:[color=blue]
> franzkowiak wrote:
>[color=green]
>> I've read some bytes from a file and just now I can't interpret 4
>> bytes in this dates like a real value. An extract from my program:
>> def l32(c):
>> return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) +
>> (ord(c[3])<<24)
>> ...
>> value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11
>>[/color]
> OK, here's the skinny (I used blocks & views to get the answer):
>
> import struct
> bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
> struct.unpack(' >f', bytes)
>
> I was suspicious of that first byte, thought it might be an exponent,
> since it seemed to have too many on bits in a row to be part of 1.11.
>
> -Scott David Daniels
> Scott.Daniels@A cm.Org[/color]

Ok, I the string exist with "mystr = f.read(4)" and the solution for
this case is in your line "struct.unpack( '>f', bytes)"
But what can I do when I want the interpret the content from the Integer
myInt (*myInt = 0x3F8CCCCD) like 4-byte-real ?
This was stored with an othes system in a binary file to
CD CC 8C 3F and now is it in python in value. The conversion is not
possible. It's right... one of this bytes is an exponent.
I want copy the memory content from the "value address" to "myReal
address" and use print "%f" %myReal.
Is myReal then the right format ?
What can I do with python, in FORTH is it simple
( >f f. )

gf

[Scott David Daniels]

Re: interpret 4 byte as 32-bit float (IEEE-754)

G.Franzkowiak wrote:[color=blue]
> Scott David Daniels schrieb:
>[color=green]
>> franzkowiak wrote:
>>[color=darkred]
>>> I've read some bytes from a file and just now I can't interpret 4
>>> bytes in this dates like a real value. An extract from my program:
>>> def l32(c):
>>> return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) +
>>> (ord(c[3])<<24)
>>> ...
>>> value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11
>>>[/color]
>> OK, here's the skinny (I used blocks & views to get the answer):
>>
>> import struct
>> bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
>> struct.unpack(' >f', bytes)
>>
>> I was suspicious of that first byte, thought it might be an exponent,
>> since it seemed to have too many on bits in a row to be part of 1.11.
>>
>> -Scott David Daniels
>> Scott.Daniels@A cm.Org[/color]
>
>
> Ok, I the string exist with "mystr = f.read(4)" and the solution for
> this case is in your line "struct.unpack( '>f', bytes)"
> But what can I do when I want the interpret the content from the Integer
> myInt (*myInt = 0x3F8CCCCD) like 4-byte-real ?
> This was stored with an othes system in a binary file to
> CD CC 8C 3F and now is it in python in value. The conversion is not
> possible. It's right... one of this bytes is an exponent.
> I want copy the memory content from the "value address" to "myReal
> address" and use print "%f" %myReal.
> Is myReal then the right format ?
> What can I do with python, in FORTH is it simple
> ( >f f. )
>
> gf
>
>
>[/color]
If you really want to do this kind of byte fiddling:

http://members.dsl-only.net/~daniels/block.html

Then:
from block import Block, View
b = Block(4) # enough space for one float (more is fine)
iv = View('i', b) # getting to it as an integer
fv = View('f', b) # same memory as floating point
iv[0] = 0x3F8CCCCD # Here is a sample just using the integer
print fv[0]

On an Intel/Amd/Generic "PC" machine, you should get 1.1

-Scott David Daniels
Scott.Daniels@A cm.Org

[Terry Reedy]

Re: interpret 4 byte as 32-bit float (IEEE-754)


"Scott David Daniels" <Scott.Daniels@ Acm.Org> wrote in message
news:41e952b9$1 @nntp0.pdx.net. ..[color=blue]
> franzkowiak wrote:[color=green]
>> I've read some bytes from a file and just now I can't interpret 4 bytes
>> in this dates like a real value. An extract from my program:
>> def l32(c):
>> return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) +
>> (ord(c[3])<<24)
>> ...
>> value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11
>>[/color]
> OK, here's the skinny (I used blocks & views to get the answer):
>
> import struct
> bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
> struct.unpack(' >f', bytes)
>
> I was suspicious of that first byte, thought it might be an exponent,
> since it seemed to have too many on bits in a row to be part of 1.11.[/color]

I believe exponents are typically stored as a positive offset from the
largest negative exponent. 3F8 is about half of 7FF, so that seems about
right for an actual exponent of 0.

Terry J. Reedy

[G.Franzkowiak]

Re: interpret 4 byte as 32-bit float (IEEE-754)

Scott David Daniels schrieb:[color=blue]
> G.Franzkowiak wrote:
>[color=green]
>> Scott David Daniels schrieb:
>>[color=darkred]
>>> franzkowiak wrote:
>>>
>>>> I've read some bytes from a file and just now I can't interpret 4
>>>> bytes in this dates like a real value. An extract from my program:
>>>> def l32(c):
>>>> return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) +
>>>> (ord(c[3])<<24)
>>>> ...
>>>> value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11
>>>>
>>> OK, here's the skinny (I used blocks & views to get the answer):
>>>
>>> import struct
>>> bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
>>> struct.unpack(' >f', bytes)
>>>
>>> I was suspicious of that first byte, thought it might be an exponent,
>>> since it seemed to have too many on bits in a row to be part of 1.11.
>>>
>>> -Scott David Daniels
>>> Scott.Daniels@A cm.Org[/color]
>>
>>
>>
>> Ok, I the string exist with "mystr = f.read(4)" and the solution for
>> this case is in your line "struct.unpack( '>f', bytes)"
>> But what can I do when I want the interpret the content from the
>> Integer myInt (*myInt = 0x3F8CCCCD) like 4-byte-real ?
>> This was stored with an othes system in a binary file to
>> CD CC 8C 3F and now is it in python in value. The conversion is not
>> possible. It's right... one of this bytes is an exponent.
>> I want copy the memory content from the "value address" to "myReal
>> address" and use print "%f" %myReal.
>> Is myReal then the right format ?
>> What can I do with python, in FORTH is it simple
>> ( >f f. )
>>
>> gf
>>
>>
>>[/color]
> If you really want to do this kind of byte fiddling:
> http://members.dsl-only.net/~daniels/block.html
>
> Then:
> from block import Block, View
> b = Block(4) # enough space for one float (more is fine)
> iv = View('i', b) # getting to it as an integer
> fv = View('f', b) # same memory as floating point
> iv[0] = 0x3F8CCCCD # Here is a sample just using the integer
> print fv[0]
>
> On an Intel/Amd/Generic "PC" machine, you should get 1.1
>
> -Scott David Daniels
> Scott.Daniels@A cm.Org[/color]

That's good :-))
I'm missing the makefile ;-)
I'm using the other world... right

Thank you

[Scott David Daniels]

Re: interpret 4 byte as 32-bit float (IEEE-754)

G.Franzkowiak wrote:[color=blue]
> Scott David Daniels schrieb:[color=green]
>> If you really want to do this kind of byte fiddling:
>> http://members.dsl-only.net/~daniels/block.html
>> Then:
>> from block import Block, View
>> b = Block(4) # enough space for one float (more is fine)
>> iv = View('i', b) # getting to it as an integer
>> fv = View('f', b) # same memory as floating point
>> iv[0] = 0x3F8CCCCD # Here is a sample just using the integer
>> print fv[0]
>> On an Intel/Amd/Generic "PC" machine, you should get 1.1[/color]
>
> That's good :-))
> I'm missing the makefile ;-)
> I'm using the other world... right[/color]
There's a lot more than one other world. distlib is your friend.

There is no makefile. If you are not on a windows box,
get the source, extract the files from the zip, and run:

python setup.py install

-Scott David Daniels
Scott.Daniels@A cm.Org

[Bengt Richter]

Re: interpret 4 byte as 32-bit float (IEEE-754)

On Sat, 15 Jan 2005 11:00:36 -0800, Scott David Daniels <Scott.Daniels@ Acm.Org> wrote:
[color=blue]
>G.Franzkowia k wrote:[color=green]
>> Scott David Daniels schrieb:
>>[color=darkred]
>>> franzkowiak wrote:
>>>
>>>> I've read some bytes from a file and just now I can't interpret 4
>>>> bytes in this dates like a real value. An extract from my program:
>>>> def l32(c):
>>>> return ord(c[0]) + (ord(c[1])<<8) + (ord(c[2])<<16) +
>>>> (ord(c[3])<<24)
>>>> ...
>>>> value = l32(f.read(4)) <--- 3F 8C CC CD should be 1.11
>>>>
>>> OK, here's the skinny (I used blocks & views to get the answer):
>>>
>>> import struct
>>> bytes = ''.join(chr(int (txt, 16)) for txt in '3F 8C CC CD'.split())
>>> struct.unpack(' >f', bytes)
>>>
>>> I was suspicious of that first byte, thought it might be an exponent,
>>> since it seemed to have too many on bits in a row to be part of 1.11.
>>>
>>> -Scott David Daniels
>>> Scott.Daniels@A cm.Org[/color]
>>
>>
>> Ok, I the string exist with "mystr = f.read(4)" and the solution for
>> this case is in your line "struct.unpack( '>f', bytes)"
>> But what can I do when I want the interpret the content from the Integer
>> myInt (*myInt = 0x3F8CCCCD) like 4-byte-real ?
>> This was stored with an othes system in a binary file to
>> CD CC 8C 3F and now is it in python in value. The conversion is not
>> possible. It's right... one of this bytes is an exponent.
>> I want copy the memory content from the "value address" to "myReal
>> address" and use print "%f" %myReal.
>> Is myReal then the right format ?
>> What can I do with python, in FORTH is it simple
>> ( >f f. )
>>
>> gf
>>
>>
>>[/color]
>If you really want to do this kind of byte fiddling:
> http://members.dsl-only.net/~daniels/block.html
>
>Then:
> from block import Block, View
> b = Block(4) # enough space for one float (more is fine)
> iv = View('i', b) # getting to it as an integer
> fv = View('f', b) # same memory as floating point
> iv[0] = 0x3F8CCCCD # Here is a sample just using the integer
> print fv[0]
>
>On an Intel/Amd/Generic "PC" machine, you should get 1.1
>[/color]
Ok, for most bit patterns (except QNANs), you can do it in pure python:

----< i32as_single.py >------------------------------------
class NoQNAN(ValueErr or): pass # use to flag inability to return QNANs

def i32as_single(i) :
"""
(UIAM in my interpretation of a 1995 Pentium Processor Developer's Manual)
This converts bits in a 32-bit integer as if they were bits
of a single-precision IEEE 754 floating point number to python float

+---+---+---+---+---+---+---+---+---+---+---+---+ ... +---+---+---+---+
| s | e e e e e e e eb| b b b ... b b b b0 |
+---+---+---+---^---+---+---+---^---+---+---+---^ ... ^---+---+---+---+
31 30 29 28 27 26 25 24 23 22 21 20 3 2 1 0

where s is the sign bit, and is the only thing that changes between + and -
and e..eb is an 8-bit biased (by 127) exponent, and eb is the hidden
unit 1 bit followed by 23 b..b0 significant "fraction" bits",
normalized so that the most significant bit is always 1 and therefore
doesn't have to be stored at eb, except that when all but the sign bit
are zero, eb is ignored and the value is then a signed zero value.
The binary fraction starting bit is after the hidden '1' bit eb at 23,
so viewing bits 0..23 as an integer, we have to divide by 2**23 (or
adjust the exponent) to get the 1.xxxx values when the official unbiased
exponent is zero (offset value 127). Thus 0x3f800000 is zero unbiased
exponent and no other bits, for a value of 1.0

A biased exponent of 255 signifies a NaN, and a biased exponent of
zero signifies a denormal (which doesn't have a hidden bit, and whose
unit bit is bit 22). Single precision denormals can be normalized
in python (double) float format, which is done for the return value.
"""
signbit = i&0x80000000
if not i&0x7fffffff: return signbit and -0.0 or 0.0 # if -0.0 available
biased_exp = (i>>23) & 0xff # bits 30-23
unitbit = biased_exp and 0x800000 or 0 # bit 23, or 0 for denormals
significand = i & 0x7fffff # bits 22-0
if biased_exp == 255:
if significand:
raise NoQNAN, "Sorry, can't generate QNAN from %08x" % i
return signbit and -1e9999 or 1e9999 # XXX s/b INF's for sure??
adjexp = (biased_exp or 1) - 127 - 23 # adjusted for denormal
posvalue = (significand + unitbit)*2.0**a djexp
return signbit and -posvalue or posvalue

def test():
import struct
num = 0
for sign in (0, 2*(-2**30)):
for i3 in xrange(128):
num3 = i3<<24
for i2 in xrange(256):
print '\r%08x'%(num,) , # show progress
num2 = i2<<16
# skip mid byte of significand, make 0
# and twiddle only a few bits at the bottom
for num0 in xrange(8):
num = sign+num3+num2+ num0
s = ''.join(map(chr ,(num0, 0, i2,((sign and 0x80 or 0)+i3))))
try: ti32as = i32as_single(nu m)
except NoQNAN: continue
tstruct = struct.unpack(' f', s)[0] # XXX '<f' => no INF ??
if ti32as != tstruct:
print '\n%x =>%r\n%r => %r' % (num, ti32as, s, tstruct)

if __name__ == '__main__':
test()
-----------------------------------------------------------


[21:47] C:\pywk\clp>i32 as_single.py
C:\pywk\clp\i32 as_single.py:31 : FutureWarning: hex/oct constants > sys.maxint will return positi
ve values in Python 2.4 and up
signbit = i&0x80000000
7fff0007C:\pywk \clp\i32as_sing le.py:51: FutureWarning: %u/%o/%x/%X of negative int will return a
signed string in Python 2.4 and up
print '\r%08x'%(num,) , # show progress
ff7f0007C:\pywk \clp\i32as_sing le.py:38: FutureWarning: %u/%o/%x/%X of negative int will return a
signed string in Python 2.4 and up
raise NoQNAN, "Sorry, can't generate QNAN from %08x" % i
fffe0007

Of course, the test() gives a clue how you might write the whole thing using struct
by just summing the four chr-ed extracted bytes from the input i ;-)

Anyway, a couple things (how to make a QNAN in python without calling struct, and '<' in struct):
[color=blue][color=green][color=darkred]
>>> from i32as_single import i32as_single as i32f
>>> i32f(0x3f8ccccd )[/color][/color][/color]
1.1000000238418 579[color=blue][color=green][color=darkred]
>>> i32f(0x00000000 )[/color][/color][/color]
0.0[color=blue][color=green][color=darkred]
>>> i32f(0x7f800000 )[/color][/color][/color]
1.#INF[color=blue][color=green][color=darkred]
>>> i32f(0xff800000 )[/color][/color][/color]
-1.#INF

But I don't know how to build QNaNs:
[color=blue][color=green][color=darkred]
>>> i32f(0x7f800001 )[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "i32as_single.p y", line 38, in i32as_single
raise NoQNAN, "Sorry, can't generate QNAN from %08x" % i
i32as_single.No QNAN: Sorry, can't generate QNAN from 7f800001

Whereas struct does:
[color=blue][color=green][color=darkred]
>>> import struct
>>> struct.unpack(' f','\x00\x00\x8 0\x7f')[0][/color][/color][/color]
1.#INF[color=blue][color=green][color=darkred]
>>> struct.unpack(' f','\x01\x00\x8 0\x7f')[0][/color][/color][/color]
1.#QNAN

BTW, your example:[color=blue][color=green][color=darkred]
>>> struct.unpack(' f','\xcd\xcc\x8 c\x3f')[0][/color][/color][/color]
1.1000000238418 579[color=blue][color=green][color=darkred]
>>> i32f(0x3f8ccccd )[/color][/color][/color]
1.1000000238418 579

But what does this mean? (I wanted to test with little endian unpacking, since
that is what I knew I created, but that isn't what '<' means?? ...)
[color=blue][color=green][color=darkred]
>>> struct.unpack(' f','\x00\x00\x8 0\x7f')[0][/color][/color][/color]
1.#INF[color=blue][color=green][color=darkred]
>>> struct.unpack(' <f','\x00\x00\x 80\x7f')[0][/color][/color][/color]
3.4028236692093 846e+038

Regards,
Bengt Richter

[Tim Peters]

Re: interpret 4 byte as 32-bit float (IEEE-754)

[Bengt Richter]
....[color=blue]
> But I don't know how to build QNaNs:[/color]

You can subtract infinity from infinity. While all Python behavior in
the presence of NaNs, infinities, and signed zeroes is a
platform-dependent accident, it you're on a box that has such things,
and figure out some (accidental!) way to spell infinity, then inf-inf
should return a NaN (although on a WinTel box, it's most likely to be
spelled "-1.#IND" when converted to string).

....
[color=blue]
> Whereas struct does:[/color]

No, it doesn't. All Python behavior in the presence of NaNs,
infinities, and signed zeroes is a platform-dependent accident.
[color=blue][color=green][color=darkred]
> >>> import struct
> >>> struct.unpack(' f','\x00\x00\x8 0\x7f')[0][/color][/color]
> 1.#INF[/color]

An accident (both that you got back an infinity, and the string
representation of an infinity).
[color=blue][color=green][color=darkred]
> >>> struct.unpack(' f','\x01\x00\x8 0\x7f')[0][/color][/color]
> 1.#QNAN[/color]

Ditto.

Those specific accidents are reliable on the box you're using.
[color=blue]
> BTW, your example:[color=green][color=darkred]
> >>> struct.unpack(' f','\xcd\xcc\x8 c\x3f')[0][/color][/color]
> 1.1000000238418 579[color=green][color=darkred]
> >>> i32f(0x3f8ccccd )[/color][/color]
> 1.1000000238418 579
>
> But what does this mean?[/color]

Mostly just that behavior is predictable when you're *not* using
infinities, NaNs or signed zeroes from Python.
[color=blue]
> (I wanted to test with little endian unpacking, since
> that is what I knew I created, but that isn't what '<' means?? ...)[/color]

Little-endian is part of what '<' means. '<' also means "use standard
alignment" and "use standard data size" and "use standard bit
representation" . As soon as you force anything like that, Python has
to try to interpret the bits itself. In the absence of "<", ">" and
"!", plain "f" asks for a wholly platform-native result. In that
case, Python doesn't have to know anything about what the bits might
mean: it stores the bytes into a native sizeof(float)-byte memory
area natively aligned for a float, and you get back whatever the
native C thinks that means.
[color=blue][color=green][color=darkred]
> >>> struct.unpack(' f','\x00\x00\x8 0\x7f')[0][/color][/color]
> 1.#INF[/color]

Entirely determined by the platform C compiler and hardware.
[color=blue][color=green][color=darkred]
> >>> struct.unpack(' <f','\x00\x00\x 80\x7f')[0][/color][/color]
> 3.4028236692093 846e+038[/color]

Entirely determined by what Python thinks the bits should mean, but
Python doesn't know anything about infinities, NaNs, or signed zeroes,
and C89 library routines Python calls to construct the value (like
ldexp()) have no portably defined behavior in their presence either.