Memory problem

[Yi Xing]

I tried the following code:

>>i=0
>>n=2600*2600*3 0
>>a=array.array ("f")
>>while (i<=n):

... i=i+1
... a.append(float( i))
...
Traceback (most recent call last):
File "<stdin>", line 3, in ?
MemoryError

to see the size of the array at the time of memory error:

>>>len(a)

8539248.

I use Windows XP x64 with 4GB RAM.

[John Machin]

Re: Memory problem

Yi Xing wrote:

I tried the following code:
>

>i=0
>n=2600*2600* 30
>a=array.array( "f")
>while (i<=n):

.. i=i+1
.. a.append(float( i))

Not a good idea. The array has to be resized, which may mean that a
realloc won't work because of fragmentation, you're out of luck because
plan B is to malloc another chunk, but that's likely to fail as well.

..
Traceback (most recent call last):
File "<stdin>", line 3, in ?
MemoryError
>
to see the size of the array at the time of memory error:

>>len(a)

8539248.

Incredible. That's only 34 MB. What is the size of your paging file?
What memory guzzlers were you running at the same time? What was the
Task Manager "Performanc e" pane showing while your test was running?
What version of Python?

FWIW I got up to len(a) == 122998164 (that's 14 times what you got) on
a machine with only 1GB of memory and a 1523MB paging file, with
Firefox & ZoneAlarm running (the pagefile was showing approx 300MB in
use at the start of the test).

I use Windows XP x64 with 4GB RAM.

Maybe there's a memory allocation problem with the 64-bit version.
Maybe MS just dropped in the old Win95 memory allocator that the timbot
used to fulminate about :-(

Cheers,
John

[Yi Xing]

Re: Memory problem

Is there a way that I can define a two-dimensional array in
array.array()? Thanks.
On Aug 14, 2006, at 2:28 PM, John Machin wrote:

Yi Xing wrote:

>I tried the following code:
>>

>>>>i=0
>>>>n=2600*2600 *30
>>>>a=array.arr ay("f")
>>>>while (i<=n):

>.. i=i+1
>.. a.append(float( i))

>
Not a good idea. The array has to be resized, which may mean that a
realloc won't work because of fragmentation, you're out of luck because
plan B is to malloc another chunk, but that's likely to fail as well.

>..
>Traceback (most recent call last):
> File "<stdin>", line 3, in ?
>MemoryError
>>
>to see the size of the array at the time of memory error:

>>>>len(a)

>8539248.

>
Incredible. That's only 34 MB. What is the size of your paging file?
What memory guzzlers were you running at the same time? What was the
Task Manager "Performanc e" pane showing while your test was running?
What version of Python?
>
FWIW I got up to len(a) == 122998164 (that's 14 times what you got) on
a machine with only 1GB of memory and a 1523MB paging file, with
Firefox & ZoneAlarm running (the pagefile was showing approx 300MB in
use at the start of the test).
>

>I use Windows XP x64 with 4GB RAM.

>
Maybe there's a memory allocation problem with the 64-bit version.
Maybe MS just dropped in the old Win95 memory allocator that the timbot
used to fulminate about :-(
>
Cheers,
John
>
--

http://mail.python.org/mailman/listinfo/python-list

>

[Martin v. Löwis]

Re: Memory problem

John Machin wrote:

Incredible. That's only 34 MB. What is the size of your paging file?
What memory guzzlers were you running at the same time? What was the
Task Manager "Performanc e" pane showing while your test was running?
What version of Python?

He didn't say Windows (so far). AFAICT, his system might be Linux, and
he might have an ulimit of 1GB (or some such).

Regards,
Martin

[John Machin]

Re: Memory problem


Martin v. Löwis wrote:

John Machin wrote:

Incredible. That's only 34 MB. What is the size of your paging file?
What memory guzzlers were you running at the same time? What was the
Task Manager "Performanc e" pane showing while your test was running?
What version of Python?

>
He didn't say Windows (so far).

Yes he did -- reread his message, which I quoted in the message which
you have *partially* quoted above.
"""
I use Windows XP x64 with 4GB RAM.
"""

Cheers,
John

[Marc 'BlackJack' Rintsch]

Re: Memory problem

In <mailman.9332.1 155594261.27775 .python-list@python.org >, Yi Xing wrote:

Is there a way that I can define a two-dimensional array in
array.array()? Thanks.

If you need more than one dimension you really should take a look at
`numarray` or `numpy`. What are you going to do with the data once it's
loaded into memory?

Ciao,
Marc 'BlackJack' Rintsch