memory error with zipfile module

[Hari Sekhon]

I do

import zipfile
zip=zipfile.Zip File('d:\somepa th\cdimage.zip' )
zip.namelist()
['someimage.iso']

then either of the two:

A) file('someimage .iso','w').writ e(zip.read('som eimage.iso'))
or
B) content=zip.rea d('someimage.is o')

but both result in the same error:

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "D:\u\Python24\ lib\zipfile.py" , line 357, in read
bytes = dc.decompress(b ytes)
MemoryError

I thought python was supposed to handle memory for you?

The python zipfile module is obviously broken...


Any advise?

[bruno at modulix]

Re: memory error with zipfile module

Hari Sekhon wrote:[color=blue]
> I do
>
> import zipfile
> zip=zipfile.Zip File('d:\somepa th\cdimage.zip' )
> zip.namelist()
> ['someimage.iso']
>
> then either of the two:
>
> A) file('someimage .iso','w').writ e(zip.read('som eimage.iso'))
> or
> B) content=zip.rea d('someimage.is o')
>
> but both result in the same error:
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "D:\u\Python24\ lib\zipfile.py" , line 357, in read
> bytes = dc.decompress(b ytes)
> MemoryError[/color]

<ot>Is that the *full* traceback ?</ot>
[color=blue]
> I thought python was supposed to handle memory for you?[/color]

Err... This doesn't mean that it bypasses system's memory management.

http://pyref.infogami.com/MemoryError

http://mail.zope.org/pipermail/zope/2004-October/153882.html

"""
MemoryError is raised by Python when an underlying (OS-level) allocation
fails.
(...)
Normally this would mean that you were out of even virtual memory
(swap), but it could also be a symptom of a libc bug, a bad RAM chip, etc.
"""

What do you think will append if you try to allocate a huge block when
you've already eaten all available memory ? Do you really hope that
Python will give you extra ram for free ?-)

Please try this code:

import zipfile
zip=zipfile.Zip File('d:\somepa th\cdimage.zip' )
info = zip.getinfo('so meimage.iso')
csize = info.compress_s ize
fsize = info.file_size
print "someimage compressed size is : %s" % csize
print "someimage real file size is : %s" % fsize
print """
So, knowing how zipfile.read() is actually implemented,
total needed ram is : %s
""" % (csize + fsize)

print "Well... Do I have that much memory available ???"

[color=blue]
> The python zipfile module is obviously broken...[/color]

s/is obviously broken/could be improved to handle huge files/

Making such statements may not be the best way to make friends...
[color=blue]
> Any advise?[/color]
Yes : Python is free software ('free' as in 'free speach' *and* as in
'free beer'), mostly written by benevolent contributors. So try and
improve the zipfile module by yourself, and submit your enhancements.
Then we all will be very grateful, and your name will be forever in the
Python Hall of Fame.

Or choose to behave as a whiny-whiny clueless luser making dumb
statements, and your name will be buried for a long long time in a lot
of killfiles.

It's up to you !-)

NB : If you go the first route, this may help:

http://www.python.org/doc/2.4.2/lib/module-zlib.html

with particular attention to the decompressobj.

HTH
--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[jordan.taylor2@gmail.com]

Re: memory error with zipfile module

Take a look at the pywin32 extension, which I believe has some lower
level memory allocation and file capabilities that might help you in
this situation. If I'm completely wrong, someone please tell me XD.
Of course, you could just make the read() a step process, reading, O
lets say 8192 bytes at a time (could be bigger if u want), writes them
to the new file, and then reads the next portion. This will be slower
(not sure how much) than if you had some AMD X2 64 with 3 gigs of ram
and could just read the file all at once, but it should work.

[bruno at modulix]

Re: memory error with zipfile module

jordan.taylor2@ gmail.com wrote:[color=blue]
> Take a look at the pywin32 extension, which I believe has some lower
> level memory allocation and file capabilities that might help you in
> this situation.[/color]

But then the solution would not be portable, which would be a shame
since the zlib module (on which ZipFile relies for compression /
decompression) already has everything needed to handle streams.

--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[Sion Arrowsmith]

Re: memory error with zipfile module

Hari Sekhon <sekhon.hari@go oglemail.com> wrote:[color=blue]
>import zipfile
>zip=zipfile.Zi pFile('d:\somep ath\cdimage.zip ')
>zip.namelist ()
>['someimage.iso'][/color]
[ ... ][color=blue]
>B) content=zip.rea d('someimage.is o')
>
>Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "D:\u\Python24\ lib\zipfile.py" , line 357, in read
> bytes = dc.decompress(b ytes)
>MemoryError
>
>I thought python was supposed to handle memory for you?[/color]

Yes, but it can't handle more memory than the operating system
is prepared to give it. How big is cdimage.zip? How big is the
uncompressed someimage.iso? How much memory do you have?
[color=blue]
>The python zipfile module is obviously broken...[/color]

This isn't at all obvious to me.

--
\S -- siona@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

[Sion Arrowsmith]

Re: memory error with zipfile module

bruno at modulix <onurb@xiludom. gro> wrote:[color=blue]
>http://mail.zope.org/pipermail/zope/...er/153882.html
>"""
>MemoryError is raised by Python when an underlying (OS-level) allocation
>fails.
>(...)
>Normally this would mean that you were out of even virtual memory
>(swap), but it could also be a symptom of a libc bug, a bad RAM chip, etc.
>"""[/color]

There's another possibility, which I ran into recently. Which is a
problem with physical+virtua l memory exceding the space addressable by
a process. So I've got 2G physical + 4G swap and I'm getting just such
a memory error -- I'm sure the compressed + uncompressed data isn't
going to eat all of that. But on a 32bit OS, it doesn't need to of
course. 2G is quite enough to cause problems....

--
\S -- siona@chiark.gr eenend.org.uk -- http://www.chaos.org.uk/~sion/
___ | "Frankly I have no feelings towards penguins one way or the other"
\X/ | -- Arthur C. Clarke
her nu becomeþ se bera eadward ofdun hlæddre heafdes bæce bump bump bump

[bruno at modulix]

Re: memory error with zipfile module

Sion Arrowsmith wrote:[color=blue]
> Hari Sekhon <sekhon.hari@go oglemail.com> wrote:[/color]
(snip)[color=blue][color=green]
>>The python zipfile module is obviously broken...[/color]
>
> This isn't at all obvious to me.[/color]

zipfile.read() does not seem to take full advantage of zlib's
decompressobj's features. This could perhaps be improved (left as an
exercice to the OP, who is obviously very good at detecting broken
memory management <g>).

Also, there's a known bug with file headers beginning past 2+GB - which
is not a very common case...
http://sourceforge.net/tracker/index...70&atid=105470

So yes, there is actually something broken - but this has nothing to do
with the OP problem - *and* there are actually some limitions (FWIW, the
main goal of zipfile was mostly to implement support for zipped python
packages, not to replace Winzip). But what, the OP is going to fix this,
isn't he ?-)


--
bruno desthuilliers
python -c "print '@'.join(['.'.join([w[::-1] for w in p.split('.')]) for
p in 'onurb@xiludom. gro'.split('@')])"

[Roger Miller]

Re: memory error with zipfile module

The basic problem is that the zipfile interface only reads and writes
whole files, so it may perform poorly or fail on huge files. At one
time I implemented a patch to allow reading files in chunks. However I
believe that the current interface has too many problems to solve by
incremental patching, and that a zipfile2 module is probably warranted.
(Is anyone working on this?)

In the meantime I think the best solution is often to just run an
external zip/unzip utility to do the heavy lifting.

[Bruno Desthuilliers]

Re: memory error with zipfile module

Roger Miller a écrit :[color=blue]
> The basic problem is that the zipfile interface only reads and writes
> whole files, so it may perform poorly or fail on huge files. At one
> time I implemented a patch to allow reading files in chunks. However I
> believe that the current interface has too many problems to solve by
> incremental patching,[/color]

Yeps, that was the general tone of some thread on python-dev. And from
what I saw of the source code, it may effectively not be the cleanest
part of the stdlib. But what, it does what it was written for at first :
provide working support for zipped packages.
[color=blue]
> and that a zipfile2 module is probably warranted.
> (Is anyone working on this?)[/color]

Seems like Bob Ippolito was on the rank, but I guess you'll get better
answers on python-dev.
[color=blue]
> In the meantime I think the best solution is often to just run an
> external zip/unzip utility to do the heavy lifting.[/color]

Indeed !-)

But while having zip/unzip installed OOTB on a unix-like system is close
to warrented, it may not be the case on Windows.

[Fredrik Lundh]

Re: memory error with zipfile module

Hari Sekhon wrote:
[color=blue]
> Is it me or is having to use os.system() all the time symtomatic of a
> deficiency/things which are missing from python as a language?[/color]

it's you.

</F>

[Fredrik Lundh]

Re: memory error with zipfile module

Hari Sekhon wrote:
[color=blue]
> I take it that it's still a work in progress to be able to pythonify
> everything, and until then we're just gonna have to rely on shell and
> those great C coded coreutils and stuff like that. Ok, I'm rather fond
> of Bash+coreutils, highest ratio of code lines to work I've ever
> seen.... it's the real strength of Linux. Shame about Windows...[/color]

you make very little sense.

</F>

[Fredrik Lundh]

Re: memory error with zipfile module

Hari Sekhon wrote:
[color=blue]
> I've seen people using everything from zip to touch, either out of
> laziness or out of the fact it wouldn't work very well in python, this
> zip case is a good example.[/color]

so based on a limitation in one library, and some random code you've
seen on the internet, you're making generalizations about the language ?

the zip case is a pretty lousy example, btw; after all, using the
existing API, it's not that hard to implement an *incremental* read
function if the provided read-into-string version isn't sufficient:

import zipfile, zlib

##
# Given a 'zip' instance, copy data from the 'name' to the
# 'out' stream.

def explode(out, zip, name):

zinfo = zip.getinfo(nam e)

if zinfo.compress_ type == zipfile.ZIP_STO RED:
decoder = None
elif zinfo.compress_ type == zipfile.ZIP_DEF LATED:
decoder = zlib.decompress obj(-zlib.MAX_WBITS)
else:
raise zipfile.BadZipF ile("unsupporte d compression method")

zip.fp.seek(zin fo.file_offset)

size = zinfo.compress_ size

while 1:
data = zip.fp.read(min (size, 8192))
if not data:
break
size -= len(data)
if decoder:
data = decoder.decompr ess(data)
out.write(data)

if decoder:
out.write(decod er.decompress(' Z'))
out.write(decod er.flush())

</F>