Python IMAP4 Memory Error

[Dody Suria Wijaya]

Hi, I encountered a Memory Error Exception on using IMAP4 just like in
Python documentation example, on a specially large email (10 MB). Any
idea how to fix/circumvent this?
[color=blue][color=green][color=darkred]
>>> typ, data = M.fetch(89, '(RFC822)')[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "C:\Python24\li b\imaplib.py", line 426, in fetch
typ, dat = self._simple_co mmand(name, message_set, message_parts)
File "C:\Python24\li b\imaplib.py", line 1028, in _simple_command
return self._command_c omplete(name, self._command(n ame, *args))
File "C:\Python24\li b\imaplib.py", line 858, in _command_comple te
typ, data = self._get_tagge d_response(tag)
File "C:\Python24\li b\imaplib.py", line 959, in _get_tagged_res ponse
self._get_respo nse()
File "C:\Python24\li b\imaplib.py", line 921, in _get_response
data = self.read(size)
File "C:\Python24\li b\imaplib.py", line 1123, in read
data = self.sslobj.rea d(size-read)
MemoryError

--
dsw

[Dody Suria Wijaya]

Re: Python IMAP4 Memory Error

Mode details, this occurs on Python 2.4.2 windows, but not on Python
2.3.4 cygwin or Python 2.3.5 windows binary.

Dody Suria Wijaya wrote:[color=blue]
>
> Hi, I encountered a Memory Error Exception on using IMAP4 just like in
> Python documentation example, on a specially large email (10 MB). Any
> idea how to fix/circumvent this?
>[color=green][color=darkred]
> >>> typ, data = M.fetch(89, '(RFC822)')[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "C:\Python24\li b\imaplib.py", line 426, in fetch
> typ, dat = self._simple_co mmand(name, message_set, message_parts)
> File "C:\Python24\li b\imaplib.py", line 1028, in _simple_command
> return self._command_c omplete(name, self._command(n ame, *args))
> File "C:\Python24\li b\imaplib.py", line 858, in _command_comple te
> typ, data = self._get_tagge d_response(tag)
> File "C:\Python24\li b\imaplib.py", line 959, in _get_tagged_res ponse
> self._get_respo nse()
> File "C:\Python24\li b\imaplib.py", line 921, in _get_response
> data = self.read(size)
> File "C:\Python24\li b\imaplib.py", line 1123, in read
> data = self.sslobj.rea d(size-read)
> MemoryError
>
> --
> dsw[/color]

[Noah]

Re: Python IMAP4 Memory Error

This looks like a bug in your build of Python 2.4.2 for Windows.
Basically it means that C's malloc() function in the Python interpreter
failed.

You can catch this exception to try to recover. Here is an example:

try:
typ, data = M.fetch(num, '(RFC822)')
exception MemoryError, e:
print "Could not get message number", num
print "IMAP4.fetc h failed due to a MemoryError"
print str(e)
import platform
if platform.versio n() == "2.4.2":
print "This is Python 2.4.2"
print "You have a buggy build of Python. Try another."

You could also try IMAP4.partial() which looks like it might allow
you to retrieve part of the message (I can't be sure because the
docs are not very detailed). Loop through and get 1 MByte at a time.
I have no idea if partial() is intended to be used this way.
It doesn't even say what happens if there is no more data or
if the amount of data you request is longer than what is available.
You will have to experiment.

Yours,
Noah

[Fredrik Lundh]

Re: Python IMAP4 Memory Error

Dody Suria Wijaya wrote:
[color=blue]
> Hi, I encountered a Memory Error Exception on using IMAP4 just like in
> Python documentation example, on a specially large email (10 MB). Any
> idea how to fix/circumvent this?
>[color=green][color=darkred]
> >>> typ, data = M.fetch(89, '(RFC822)')[/color][/color]
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "C:\Python24\li b\imaplib.py", line 426, in fetch
> typ, dat = self._simple_co mmand(name, message_set, message_parts)
> File "C:\Python24\li b\imaplib.py", line 1028, in _simple_command
> return self._command_c omplete(name, self._command(n ame, *args))
> File "C:\Python24\li b\imaplib.py", line 858, in _command_comple te
> typ, data = self._get_tagge d_response(tag)
> File "C:\Python24\li b\imaplib.py", line 959, in _get_tagged_res ponse
> self._get_respo nse()
> File "C:\Python24\li b\imaplib.py", line 921, in _get_response
> data = self.read(size)
> File "C:\Python24\li b\imaplib.py", line 1123, in read
> data = self.sslobj.rea d(size-read)
> MemoryError[/color]

try adding a print statement to lib/imaplib.py, just before that read
statement,

print size, read, size-read
data = self.sslobj.rea d(size-read)

and let us know what it prints.

</F>

[Dody Suria Wijaya]

Re: Python IMAP4 Memory Error

Fredrik Lundh wrote:[color=blue]
> try adding a print statement to lib/imaplib.py, just before that read
> statement,
>
> print size, read, size-read
> data = self.sslobj.rea d(size-read)
>
> and let us know what it prints.[/color]

14130601 0 14130601
14130601 16353 14114248
14130601 32737 14097864
14130601 49121 14081480
14130601 65505 14065096
14130601 81889 14048712
14130601 94705 14035896
14130601 111089 14019512
14130601 127473 14003128
14130601 143857 13986744
14130601 160241 13970360
14130601 176625 13953976
14130601 192525 13938076
14130601 208909 13921692
14130601 221725 13908876
14130601 238109 13892492
14130601 254493 13876108
14130601 270877 13859724
14130601 287261 13843340
14130601 303645 13826956
14130601 319053 13811548
14130601 335437 13795164
14130601 340953 13789648
14130601 357337 13773264
14130601 373721 13756880
14130601 390105 13740496
14130601 406489 13724112
14130601 422873 13707728
14130601 429685 13700916
14130601 444285 13686316
14130601 460669 13669932
14130601 477053 13653548
14130601 493437 13637164
14130601 509821 13620780
14130601 526205 13604396
14130601 531557 13599044
14130601 547941 13582660
14130601 553457 13577144
14130601 569841 13560760
14130601 586225 13544376
14130601 602609 13527992
14130601 618993 13511608
14130601 635377 13495224
14130601 651761 13478840
14130601 668145 13462456
14130601 677229 13453372
14130601 693613 13436988
14130601 706429 13424172
14130601 722813 13407788
14130601 739197 13391404
14130601 755581 13375020
14130601 771965 13358636
14130601 788349 13342252
14130601 793701 13336900
14130601 801001 13329600
14130601 817385 13313216
14130601 833769 13296832
14130601 850153 13280448
14130601 866537 13264064
14130601 882921 13247680
14130601 895737 13234864
14130601 912121 13218480
14130601 928505 13202096
14130601 944889 13185712
14130601 961273 13169328
14130601 977657 13152944
14130601 983009 13147592
14130601 999393 13131208
14130601 1004909 13125692
14130601 1021293 13109308
14130601 1037677 13092924
14130601 1054061 13076540
14130601 1070445 13060156
14130601 1086829 13043772
14130601 1098185 13032416
14130601 1114569 13016032
14130601 1130953 12999648
14130601 1147337 12983264
14130601 1163721 12966880
14130601 1180105 12950496
14130601 1191625 12938976
14130601 1208009 12922592
14130601 1213525 12917076
14130601 1229909 12900692
14130601 1246293 12884308
14130601 1262677 12867924
14130601 1279061 12851540
14130601 1295445 12835156
14130601 1309393 12821208
14130601 1325777 12804824
14130601 1342161 12788440
14130601 1358545 12772056
14130601 1374929 12755672
14130601 1391313 12739288
14130601 1407697 12722904
14130601 1424081 12706520
14130601 1440465 12690136
14130601 1456849 12673752
14130601 1462365 12668236
14130601 1478749 12651852
14130601 1495133 12635468
14130601 1511517 12619084
14130601 1527901 12602700
14130601 1544285 12586316
14130601 1556937 12573664
14130601 1573321 12557280
14130601 1589705 12540896
14130601 1606089 12524512
14130601 1622473 12508128
14130601 1638857 12491744
14130601 1645669 12484932
14130601 1660269 12470332
14130601 1676653 12453948
14130601 1693037 12437564
14130601 1699689 12430912
14130601 1716073 12414528
14130601 1732457 12398144
14130601 1748841 12381760
14130601 1765225 12365376
14130601 1781609 12348992
14130601 1790045 12340556
14130601 1806429 12324172
14130601 1822813 12307788
14130601 1839197 12291404
14130601 1855581 12275020
14130601 1871965 12258636
14130601 1884617 12245984
14130601 1901001 12229600
14130601 1917385 12213216
14130601 1933769 12196832
14130601 1950153 12180448
14130601 1966537 12164064
14130601 1978057 12152544
14130601 1994441 12136160
14130601 1999957 12130644
14130601 2016341 12114260
14130601 2032725 12097876
14130601 2049109 12081492
14130601 2065493 12065108
14130601 2081877 12048724
14130601 2095825 12034776
14130601 2112209 12018392
14130601 2125025 12005576
14130601 2141409 11989192
14130601 2157793 11972808
14130601 2174177 11956424
14130601 2190561 11940040
14130601 2206945 11923656
14130601 2217005 11913596
14130601 2228193 11902408
14130601 2244577 11886024
14130601 2260961 11869640
14130601 2277345 11853256
14130601 2293729 11836872
Traceback (most recent call last):
File "imap_tikikuli_ monitor.py", line 15, in ?
typ, data = M.fetch(num, '(RFC822)')
File "c:\python24\li b\imaplib.py", line 426, in fetch
typ, dat = self._simple_co mmand(name, message_set, message_parts)
File "c:\python24\li b\imaplib.py", line 1028, in _simple_command
return self._command_c omplete(name, self._command(n ame, *args))
File "c:\python24\li b\imaplib.py", line 858, in _command_comple te
typ, data = self._get_tagge d_response(tag)
File "c:\python24\li b\imaplib.py", line 959, in _get_tagged_res ponse
self._get_respo nse()
File "c:\python24\li b\imaplib.py", line 921, in _get_response
data = self.read(size)
File "c:\python24\li b\imaplib.py", line 1124, in read
data = self.sslobj.rea d(size-read)
MemoryError




If I change the code to use non SSL IMAP4, and put similar printing code
here's the exception:

14130601 8161 14122440
14130601 11649 14118952
14130601 13109 14117492
14130601 16029 14114572
14130601 18949 14111652
14130601 21869 14108732
14130601 24789 14105812
14130601 26249 14104352
14130601 29169 14101432
14130601 32089 14098512
14130601 35009 14095592
14130601 37929 14092672
14130601 42309 14088292
14130601 45229 14085372
14130601 48149 14082452
14130601 51069 14079532
14130601 53989 14076612
14130601 55449 14075152
14130601 58369 14072232
14130601 61289 14069312
14130601 64209 14066392
14130601 65505 14065096
14130601 74265 14056336
14130601 139965 13990636
14130601 142885 13987716
14130601 145805 13984796
14130601 148725 13981876
14130601 151645 13978956
14130601 153105 13977496
14130601 156025 13974576
14130601 158945 13971656
14130601 161701 13968900
14130601 164621 13965980
14130601 167541 13963060
14130601 170461 13960140
14130601 174841 13955760
14130601 179221 13951380
14130601 182141 13948460
14130601 185061 13945540
14130601 189441 13941160
14130601 192361 13938240
14130601 195281 13935320
14130601 198201 13932400
14130601 202581 13928020
14130601 205501 13925100
14130601 208421 13922180
14130601 212801 13917800
14130601 215721 13914880
14130601 218641 13911960
14130601 221561 13909040
14130601 227401 13903200
14130601 230321 13900280
14130601 234701 13895900
14130601 237621 13892980
14130601 240541 13890060
14130601 243461 13887140
14130601 247841 13882760
14130601 250761 13879840
14130601 253681 13876920
14130601 258061 13872540
14130601 260981 13869620
14130601 263901 13866700
14130601 266821 13863780
14130601 271201 13859400
14130601 275581 13855020
14130601 278501 13852100
14130601 281421 13849180
14130601 285801 13844800
14130601 288721 13841880
14130601 293101 13837500
14130601 296021 13834580
14130601 298941 13831660
14130601 301861 13828740
14130601 306241 13824360
14130601 309161 13821440
14130601 313541 13817060
14130601 316461 13814140
14130601 322301 13808300
14130601 325221 13805380
14130601 329601 13801000
14130601 332521 13798080
14130601 335441 13795160
14130601 339821 13790780
14130601 342741 13787860
14130601 347121 13783480
14130601 350041 13780560
14130601 354421 13776180
14130601 357341 13773260
14130601 360261 13770340
14130601 363181 13767420
14130601 369021 13761580
14130601 373401 13757200
14130601 376321 13754280
14130601 380373 13750228
14130601 383293 13747308
14130601 387673 13742928
14130601 393513 13737088
14130601 397893 13732708
14130601 400813 13729788
14130601 405193 13725408
14130601 408113 13722488
14130601 412493 13718108
14130601 416873 13713728
14130601 421253 13709348
14130601 424173 13706428
14130601 427093 13703508
14130601 431473 13699128
14130601 434393 13696208
14130601 438773 13691828
14130601 441693 13688908
14130601 444449 13686152
14130601 448829 13681772
14130601 451749 13678852
14130601 456129 13674472
14130601 459049 13671552
14130601 463429 13667172
14130601 466349 13664252
14130601 470729 13659872
14130601 473649 13656952
14130601 476569 13654032
14130601 480949 13649652
14130601 483869 13646732
14130601 488249 13642352
14130601 491169 13639432
14130601 494089 13636512
14130601 498469 13632132
14130601 501389 13629212
14130601 505769 13624832
14130601 508689 13621912
14130601 514529 13616072
14130601 517449 13613152
14130601 521829 13608772
14130601 526209 13604392
14130601 529129 13601472
14130601 532049 13598552
14130601 536429 13594172
14130601 539349 13591252
14130601 543729 13586872
14130601 546649 13583952
14130601 549569 13581032
14130601 553949 13576652
14130601 558329 13572272
14130601 562709 13567892
14130601 565629 13564972
14130601 570009 13560592
Traceback (most recent call last):
File "imap_tikikuli_ monitor.py", line 15, in ?
typ, data = M.fetch(num, '(RFC822)')
File "c:\python24\li b\imaplib.py", line 426, in fetch
typ, dat = self._simple_co mmand(name, message_set, message_parts)
File "c:\python24\li b\imaplib.py", line 1028, in _simple_command
return self._command_c omplete(name, self._command(n ame, *args))
File "c:\python24\li b\imaplib.py", line 858, in _command_comple te
typ, data = self._get_tagge d_response(tag)
File "c:\python24\li b\imaplib.py", line 959, in _get_tagged_res ponse
self._get_respo nse()
File "c:\python24\li b\imaplib.py", line 921, in _get_response
data = self.read(size)
File "c:\python24\li b\imaplib.py", line 234, in read
return self.file.read( size)
File "c:\python24\li b\socket.py", line 304, in read
data = self._sock.recv (recv_size)
MemoryError



the print code is "print size, buf_len, left" on the line just before
data = self._sock.recv (recv_size)

[Dody Suria Wijaya]

Re: Python IMAP4 Memory Error

Noah wrote:[color=blue]
> This looks like a bug in your build of Python 2.4.2 for Windows.
> Basically it means that C's malloc() function in the Python interpreter
> failed.
>[/color]

On a second trial, it's also failed on Python 2.3.5 for Windows, Python
2.3.3 for Windows, and Python 2.2.3 for Windows. So this seems to me as
a Windows system related bug, not a particular version of Python bug.

[Fredrik Lundh]

Re: Python IMAP4 Memory Error

Jean-Paul Calderone wrote:
[color=blue][color=green]
> >On a second trial, it's also failed on Python 2.3.5 for Windows, Python
> >2.3.3 for Windows, and Python 2.2.3 for Windows. So this seems to me as
> >a Windows system related bug, not a particular version of Python bug.[/color]
>
> Arguably, it's a bug in Python's imaplib module. Sure, the Windows memory
> allocator is feeble and falls over when asked to do perfectly reasonable things.[/color]

if you look at the debug output (which you may already have done),
it's an obvious case of fragmentation-inducing behaviour. any malloc-
based system may choke on the sequence

for a large number of attempts:
allocate a 15 megabyte block
shrink block to a couple of kilobytes
occasionally allocate a medium-sized block

from what I can tell, replacing the

data = self.sslobj.rea d(size-read)

line with

data = self.sslobj.rea d(min(size-read, 16384))

should do the trick.

</F>

[Dody Suria Wijaya]

Re: Python IMAP4 Memory Error

like magic it did the trick :D

This should be applied to future Python release. Thanks.

Fredrik Lundh wrote:[color=blue]
> Jean-Paul Calderone wrote:
> if you look at the debug output (which you may already have done),
> it's an obvious case of fragmentation-inducing behaviour. any malloc-
> based system may choke on the sequence
>
> for a large number of attempts:
> allocate a 15 megabyte block
> shrink block to a couple of kilobytes
> occasionally allocate a medium-sized block
>
> from what I can tell, replacing the
>
> data = self.sslobj.rea d(size-read)
>
> line with
>
> data = self.sslobj.rea d(min(size-read, 16384))
>
> should do the trick.[/color]