On 21 Ott, 17:19, Rolf Wester <rolf.wes...@il t.fraunhofer.de wrote:
To be sure that the deallocated memory is not cached at some level to
be reused, you could try
someting like this:
while 1:
l = [dict() for i in range(4000000)]
l = None # no need of gc and del
For what is worth, on my PC ( Windows XP and Python 2.5.2) the memory
usage of the process
monitored with the Task manager grows up to 600 MB before the memory
is actually released.
Note that in your example, as in mine, you do not need to call
gc.collect(), because
the huge list object is already deleted when you do "del a" ( or in my
case when I reassign "l" and the
huge list drops to 0 reference counts ). The basic memory garbage
collector in CPython
is based on reference counts; gc is only used to find and break
circular reference chains,
which your example do not create. As a proof of that, if ypu print the
retuirn value of
gc.collect (whic is the number of collected objects) you should get 0.
Ciao
------
FB
Hi,
>
I have the problem that with long running Python scripts (many loops)
memory consumption increases until the script crashes. I used the
following small script to understand what might happen:
>
import gc
>
print len(gc.get_obje cts())
>
a = []
for i in range( 4000000 ):
a.append( None )
for i in range( 4000000 ):
a[i] = {}
>
print len(gc.get_obje cts())
>
ret = raw_input("Retu rn:")
>
del a
gc.collect()
>
print len(gc.get_obje cts())
>
ret = raw_input("Retu rn:")
>
The output is:
4002706
Return:
2705
Return:
>
When I do ps aux | grep python before the first "Return" I get:
wester 5255 51.2 16.3 1306696 1286828 pts/4 S+ 17:59 0:30 python
memory_prob2.py
>
and before the second one:
wester 5255 34.6 15.9 1271784 1255580 pts/4 S+ 17:59 0:31 python
memory_prob2.py
>
This indicates that although the garbage collector freed 4000001 objects
memory consumption does not change accordingly.
>
I tried the C++ code:
>
#include <iostream>
using namespace std;
>
int main()
{
int i;
cout << ":";
//ps 1
cin >i;
>
double * v = new double[40000000];
cout << ":";
//ps 2
cin >i;
>
for(int i=0; i < 40000000; i++)
v[i] = i;
>
cout << v[40000000-1] << ":";
//ps 3
cin >i;
>
delete [] v;
>
cout << ":";
//ps 4
cin >i;
>
}
>
and got from ps:
>
ps 1: 11184
ps 1: 323688
ps 1: 323688
ps 1: 11184
>
which means that the memery which is deallocated is no longer used by
the C++ program.
>
Do I miss something or is this a problem with Python? Is there any means
to force Python to release the memory that is not used any more?
>
I would be very appreciative for any help.
>
With kind regards
>
Rolf
>
I have the problem that with long running Python scripts (many loops)
memory consumption increases until the script crashes. I used the
following small script to understand what might happen:
>
import gc
>
print len(gc.get_obje cts())
>
a = []
for i in range( 4000000 ):
a.append( None )
for i in range( 4000000 ):
a[i] = {}
>
print len(gc.get_obje cts())
>
ret = raw_input("Retu rn:")
>
del a
gc.collect()
>
print len(gc.get_obje cts())
>
ret = raw_input("Retu rn:")
>
The output is:
4002706
Return:
2705
Return:
>
When I do ps aux | grep python before the first "Return" I get:
wester 5255 51.2 16.3 1306696 1286828 pts/4 S+ 17:59 0:30 python
memory_prob2.py
>
and before the second one:
wester 5255 34.6 15.9 1271784 1255580 pts/4 S+ 17:59 0:31 python
memory_prob2.py
>
This indicates that although the garbage collector freed 4000001 objects
memory consumption does not change accordingly.
>
I tried the C++ code:
>
#include <iostream>
using namespace std;
>
int main()
{
int i;
cout << ":";
//ps 1
cin >i;
>
double * v = new double[40000000];
cout << ":";
//ps 2
cin >i;
>
for(int i=0; i < 40000000; i++)
v[i] = i;
>
cout << v[40000000-1] << ":";
//ps 3
cin >i;
>
delete [] v;
>
cout << ":";
//ps 4
cin >i;
>
}
>
and got from ps:
>
ps 1: 11184
ps 1: 323688
ps 1: 323688
ps 1: 11184
>
which means that the memery which is deallocated is no longer used by
the C++ program.
>
Do I miss something or is this a problem with Python? Is there any means
to force Python to release the memory that is not used any more?
>
I would be very appreciative for any help.
>
With kind regards
>
Rolf
To be sure that the deallocated memory is not cached at some level to
be reused, you could try
someting like this:
while 1:
l = [dict() for i in range(4000000)]
l = None # no need of gc and del
For what is worth, on my PC ( Windows XP and Python 2.5.2) the memory
usage of the process
monitored with the Task manager grows up to 600 MB before the memory
is actually released.
Note that in your example, as in mine, you do not need to call
gc.collect(), because
the huge list object is already deleted when you do "del a" ( or in my
case when I reassign "l" and the
huge list drops to 0 reference counts ). The basic memory garbage
collector in CPython
is based on reference counts; gc is only used to find and break
circular reference chains,
which your example do not create. As a proof of that, if ypu print the
retuirn value of
gc.collect (whic is the number of collected objects) you should get 0.
Ciao
------
FB