python 2.5a2, gcc 4.1 and memory problems

[Michele Petrazzo]

Hi list,
I'm doing some tests on my debian testing and I see a very strange
memory problem with py 2.5a2 (just downloaded) and compiled with gcc
4.1.0, but not with the gcc 3.3.5:

My test are:

#--test.py
import sys
if sys.version.sta rtswith("2.3"):
from sets import Set as set
b=set(range(500 0000))
a=set(range(100 000))
c = b-a

for i in 3 4 5; do echo "python 2.$i" && time python2.$i test.py; done

My configure args for compile py 2.5a2 are:

michele:~/Python-2.5a2$ export CC=gcc-4.1
michele:~/Python-2.5a2$ export CXX=g++-4.1
michele:~/Python-2.5a2$ ./configure --prefix=/usr --enable-shared
michele:~/Python-2.5a2$ make
michele:~/Python-2.5a2$ make install

michele:~/Python-2.5a2$ export CC=gcc-3.3
michele:~/Python-2.5a2$ export CXX=g++-3.3
michele:~/Python-2.5a2$ ./configure --prefix=/usr --enable-shared
michele:~/Python-2.5a2$ make
michele:~/Python-2.5a2$ make install

Then I execute my test. The memory usage of 2.5a2 and gcc 3.3 that I see
with "top", is the same (about VIRT: 260 MB and RES: 250MB ) that with
the py 2.3 and 2.4, but then I recompile with 4.1 and execute the same
test, my system "stop to work"... with "top" I can see that it use VIRT:
2440 MB and RES: 640MB RAM (I think all that I have into my pc)

I haven't tried to recompile py 2.4 myself with gcc 4.1 because it is
already compiled with it (4.0.3), so I think (only think) that is a py
2.5 problem.
I'm right? or I have to compile it with something other switches?

Thanks,
Michele

[nnorwitz@gmail.com]

Re: python 2.5a2, gcc 4.1 and memory problems

Michele Petrazzo wrote:[color=blue]
>
> I haven't tried to recompile py 2.4 myself with gcc 4.1 because it is
> already compiled with it (4.0.3), so I think (only think) that is a py
> 2.5 problem.
> I'm right? or I have to compile it with something other switches?[/color]

Sounds like a gcc problem to me. Try adding --with-pydebug in the
configure options. My guess is that it will work. This option enables
asserts, but more importantly disables optimization. My guess is that
this is an optimization problem with gcc. I assume
-fno-strict-aliasing is one of the gcc flags. It should be as this is
required for building python.

Cheers,
n

[Tim Peters]

Re: python 2.5a2, gcc 4.1 and memory problems

[Michele Petrazzo][color=blue]
> I'm doing some tests on my debian testing and I see a very strange
> memory problem with py 2.5a2 (just downloaded) and compiled with gcc
> 4.1.0, but not with the gcc 3.3.5:
>
> My test are:
>
> #--test.py
> import sys
> if sys.version.sta rtswith("2.3"):
> from sets import Set as set
> b=set(range(500 0000))
> a=set(range(100 000))
> c = b-a
>
> for i in 3 4 5; do echo "python 2.$i" && time python2.$i test.py; done
>
> My configure args for compile py 2.5a2 are:
> ...
> Then I execute my test. The memory usage of 2.5a2 and gcc 3.3 that I see
> with "top", is the same (about VIRT: 260 MB and RES: 250MB ) that with
> the py 2.3 and 2.4, but then I recompile with 4.1 and execute the same
> test, my system "stop to work"... with "top" I can see that it use VIRT:
> 2440 MB and RES: 640MB RAM (I think all that I have into my pc)[/color]

FYI, here are stats I saw running the test program on Windows:

Python \ VM size at end (KB) peak mem use (KB)
------ ------------------- -----------------
2.3.5 262 313
2.4.3 262 313
trunk 196 230

"trunk" should be very close to 2.5a2 in memory use. "How much
memory" the OS thinks you're using greatly depends on the platform C's
malloc/free implementation, so I'd suspect a relevant difference in
glibc (assuming that's the C library you're using). As the results on
Windows show, it's unlikely that Python is _directly_ "consuming more
memory" here; it's quite possible that the platform free() is hanging
on to temp memory where in a different implementation it returned that
temp memory to the OS.

[Giovanni Bajo]

Re: python 2.5a2, gcc 4.1 and memory problems

Michele Petrazzo wrote:
[color=blue]
> Then I execute my test. The memory usage of 2.5a2 and gcc 3.3 that I
> see with "top", is the same (about VIRT: 260 MB and RES: 250MB ) that
> with the py 2.3 and 2.4, but then I recompile with 4.1 and execute
> the same test, my system "stop to work"... with "top" I can see that
> it use VIRT: 2440 MB and RES: 640MB RAM (I think all that I have into
> my pc)
>
> I haven't tried to recompile py 2.4 myself with gcc 4.1 because it is
> already compiled with it (4.0.3), so I think (only think) that is a py
> 2.5 problem.
> I'm right? or I have to compile it with something other switches?[/color]


Make sure -fno-strict-aliasing is specified. Python does not conform to ISO C
type aliasing rules.
--
Giovanni Bajo

[Michele Petrazzo]

Re: python 2.5a2, gcc 4.1 and memory problems

nnorwitz@gmail. com wrote:[color=blue]
> Michele Petrazzo wrote:[color=green]
>> I haven't tried to recompile py 2.4 myself with gcc 4.1 because it
>> is already compiled with it (4.0.3), so I think (only think) that
>> is a py 2.5 problem. I'm right? or I have to compile it with
>> something other switches?[/color]
>
> Sounds like a gcc problem to me. Try adding --with-pydebug in the
> configure options. My guess is that it will work. This option
> enables asserts, but more importantly disables optimization.[/color]

Yes, with this option it work, but has very bad performances.
[color=blue]
> My guess is that this is an optimization problem with gcc. I assume
> -fno-strict-aliasing is one of the gcc flags. It should be as this
> is required for building python.
>[/color]

Yes, I specify that flag for compiling, but still the same... Always
problems. Also tried with --without-pymalloc and no changes.
[color=blue]
> Cheers, n
>[/color]

I think that I'll wait for the firsts debian packages, or compile it
with gcc 3.3.

Thanks,
Michele

[nnorwitz@gmail.com]

Re: python 2.5a2, gcc 4.1 and memory problems

Michele Petrazzo wrote:[color=blue]
> nnorwitz@gmail. com wrote:[color=green]
> > Michele Petrazzo wrote:[color=darkred]
> >> I haven't tried to recompile py 2.4 myself with gcc 4.1 because it
> >> is already compiled with it (4.0.3), so I think (only think) that
> >> is a py 2.5 problem. I'm right? or I have to compile it with
> >> something other switches?[/color]
> >
> > Sounds like a gcc problem to me. Try adding --with-pydebug in the
> > configure options. My guess is that it will work. This option
> > enables asserts, but more importantly disables optimization.[/color]
>
> Yes, with this option it work, but has very bad performances.[/color]

Sure, you didn't say if you tried just disabling optimization. That is
often a problem. configure normally, but change the optimization from
-O3 to -O0, -O1, and -O2. My guess is that -O1 will work and -O2
won't. Even -O0 will be faster than --with-pydebug since assertions
won't be enabled (you may need to pass --without-pydebug to disable the
option).

n