How to efficiently read binary files?

**Grant Edwards** · Apr 30 '06, 10:05 PM

Re: How to efficiently read binary files?

On 2006-04-30, David Lees <debl2NoSpam@ve rizon.net> wrote:[color=blue]
> I want to process large binary files (>2GB) in Python. I have played
> around with prototypes in pure Python and profiled the code. Most of
> the time seems to be spent converting back and forth to and from strings
> using the struct module. Is there a way to directly read into an array
> of integers in Python?[/color]

Perhaps the numarray module?

--
Grant Edwards
grante@visi.com

**Robert Kern** · Apr 30 '06, 10:15 PM

Re: How to efficiently read binary files?

Grant Edwards wrote:[color=blue]
> On 2006-04-30, David Lees <debl2NoSpam@ve rizon.net> wrote:
>[color=green]
>>I want to process large binary files (>2GB) in Python. I have played
>>around with prototypes in pure Python and profiled the code. Most of
>>the time seems to be spent converting back and forth to and from strings
>>using the struct module. Is there a way to directly read into an array
>>of integers in Python?[/color]
>
> Perhaps the numarray module?[/color]

numpy for new code, please. In particular, numarray is limited by 32-bit APIs
even on 64-bit platforms, so a >2GB file will be difficult to process even when
using mmap. numpy removed this restriction on 64-bit platforms. 32-bit users
will still have to split up the file into <2GB chunks, though.

http://numeric.scipy.org/

numpy-discussion List Signup and Options

https://lists.sourceforge.net/lists/listinfo/numpy-discussion

A package for scientific computing with Python

--
Robert Kern
robert.kern@gma il.com

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

**Alex Martelli** · Apr 30 '06, 10:45 PM

Re: How to efficiently read binary files?

David Lees <debl2NoSpam@ve rizon.net> wrote:
[color=blue]
> I want to process large binary files (>2GB) in Python. I have played
> around with prototypes in pure Python and profiled the code. Most of
> the time seems to be spent converting back and forth to and from strings
> using the struct module. Is there a way to directly read into an array
> of integers in Python?[/color]
[color=blue][color=green][color=darkred]
>>> import array
>>> x=array.array(' l')
>>> help(x.fromfile )[/color][/color][/color]

Help on built-in function fromfile:

fromfile(...)
fromfile(f, n)

Read n objects from the file object f and append them to the end of
the array. Also called as read.

Alex

**David Lees** · May 1 '06, 01:45 AM

Re: How to efficiently read binary files?

Alex Martelli wrote:[color=blue]
> David Lees <debl2NoSpam@ve rizon.net> wrote:
>[color=green]
>> I want to process large binary files (>2GB) in Python. I have played
>> around with prototypes in pure Python and profiled the code. Most of
>> the time seems to be spent converting back and forth to and from strings
>> using the struct module. Is there a way to directly read into an array
>> of integers in Python?[/color]
>[color=green][color=darkred]
>>>> import array
>>>> x=array.array(' l')
>>>> help(x.fromfile )[/color][/color]
>
> Help on built-in function fromfile:
>
> fromfile(...)
> fromfile(f, n)
>
> Read n objects from the file object f and append them to the end of
> the array. Also called as read.
>
>
> Alex[/color]

Thank you. This is exactly what I was looking for. I just tried it and
it works great.

David Lees

**Grant Edwards** · May 1 '06, 03:15 PM

Re: How to efficiently read binary files?

On 2006-04-30, Robert Kern <robert.kern@gm ail.com> wrote:[color=blue]
> Grant Edwards wrote:[color=green]
>> On 2006-04-30, David Lees <debl2NoSpam@ve rizon.net> wrote:
>>[color=darkred]
>>>I want to process large binary files (>2GB) in Python. I have played
>>>around with prototypes in pure Python and profiled the code. Most of
>>>the time seems to be spent converting back and forth to and from strings
>>>using the struct module. Is there a way to directly read into an array
>>>of integers in Python?[/color]
>>
>> Perhaps the numarray module?[/color]
>
> numpy for new code, please.[/color]

So numarray and numpy were both written to replace numeric?

--
Grant Edwards grante Yow! I want to kill
at everyone here with a cute
visi.com colorful Hydrogen Bomb!!

**Robert Kern** · May 1 '06, 04:55 PM

Re: How to efficiently read binary files?

Grant Edwards wrote:[color=blue]
> On 2006-04-30, Robert Kern <robert.kern@gm ail.com> wrote:
>[color=green]
>>Grant Edwards wrote:[/color][/color]
[color=blue][color=green][color=darkred]
>>>Perhaps the numarray module?[/color]
>>
>>numpy for new code, please.[/color]
>
> So numarray and numpy were both written to replace numeric?[/color]

numpy was written to replace both Numeric and numarray. There is a good
explanation in Chapter 1 of _The Guide to NumPy_ included in the sample chapters:

http://numeric.scipy.org/scipybooksample.pdf

There were a number of features in Numeric that could not be replicated in
numarray, among them the ufunc C API and the speed for small arrays. numpy's
code base is closer to Numeric's but it incorporates nearly all (if not all) of
the features of numarray. The numarray developers fully support the migration to
numpy as the one array package.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

**Grant Edwards** · May 1 '06, 05:15 PM

Re: How to efficiently read binary files?

On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:
[color=blue][color=green][color=darkred]
>>>>Perhaps the numarray module?
>>>
>>>numpy for new code, please.[/color]
>>
>> So numarray and numpy were both written to replace numeric?[/color]
>
> numpy was written to replace both Numeric and numarray.[/color]

too many batteries...

;)

--
Grant Edwards grante Yow! ... bleakness...
at desolation... plastic
visi.com forks...

**Robert Kern** · May 1 '06, 07:05 PM

Re: How to efficiently read binary files?

Grant Edwards wrote:[color=blue]
> On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:
>[color=green][color=darkred]
>>>>>Perhaps the numarray module?
>>>>
>>>>numpy for new code, please.
>>>
>>>So numarray and numpy were both written to replace numeric?[/color]
>>
>>numpy was written to replace both Numeric and numarray.[/color]
>
> too many batteries...[/color]

Too many cryptic complaints...

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

**Grant Edwards** · May 1 '06, 07:15 PM

Re: How to efficiently read binary files?

On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:[color=blue]
> Grant Edwards wrote:[color=green]
>> On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:
>>[color=darkred]
>>>>>>Perhaps the numarray module?
>>>>>
>>>>>numpy for new code, please.
>>>>
>>>>So numarray and numpy were both written to replace numeric?
>>>
>>>numpy was written to replace both Numeric and numarray.[/color]
>>
>> too many batteries...[/color]
>
> Too many cryptic complaints...[/color]

Too easy to miss the intended humor on Usenet...

--
Grant Edwards grante Yow! Where's my SOCIAL
at WORKER?
visi.com

**Robert Kern** · May 1 '06, 07:35 PM

Re: How to efficiently read binary files?

Grant Edwards wrote:[color=blue]
> On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:
>[color=green]
>>Grant Edwards wrote:
>>[color=darkred]
>>>On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:
>>>
>>>>>>>Perhap s the numarray module?
>>>>>>
>>>>>>numpy for new code, please.
>>>>>
>>>>>So numarray and numpy were both written to replace numeric?
>>>>
>>>>numpy was written to replace both Numeric and numarray.
>>>
>>>too many batteries...[/color]
>>
>>Too many cryptic complaints...[/color]
>
> Too easy to miss the intended humor on Usenet...[/color]

Oh, I saw the smiley. I knew it was meant to be humorous. I just didn't
understand it. The only "batteries" reference I know of in this context is the
"batteries included" philosophy of the stdlib. Of course, none of Numeric,
numarray, or numpy have anything to do with the stdlib, so again I am confused.

--
Robert Kern

"I have come to believe that the whole world is an enigma, a harmless enigma
that is made terrible by our own mad attempt to interpret it as though it had
an underlying truth."
-- Umberto Eco

**Grant Edwards** · May 1 '06, 08:15 PM

Re: How to efficiently read binary files?

On 2006-05-01, Robert Kern <robert.kern@gm ail.com> wrote:
[color=blue]
> Oh, I saw the smiley. I knew it was meant to be humorous. I
> just didn't understand it. The only "batteries" reference I
> know of in this context is the "batteries included" philosophy
> of the stdlib. Of course, none of Numeric, numarray, or numpy
> have anything to do with the stdlib, so again I am confused.[/color]

Sorry. I forgot about the distinction between the standard
library and the "third-party" packages like scientific
packages.

--
Grant Edwards grante Yow! I think I'll make
at SCRAMBLED EGGS!! They're
visi.com each in LITTLE SHELLS...

How to efficiently read binary files?

How to efficiently read binary files?

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment