make test segfaults with "--enable-shared" on Python 2.3.3

**Berthold Hoellmann** · Jul 18 '05, 07:08 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

bhoel@web.de (Berthold Hoellmann) writes:
[color=blue]
> Hello,
>
> When I use
>
> ./configure --with-thread --with-fpectl --with-signal-module \
> --with-pymalloc --enable-shared --with-cxx=g++
>
> make test
>
> on 2.3.3 I get
>
> ...
> test_queue
> test_quopri
> test_random
> test_re
> make: *** [test] Speicherzugriff sfehler (Speicherauszug erstellt)[/color]

Everything works fine if I remove the "--enable-shared" flag from
configure.

Regards

Berthold

--
bhoel@web.de / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.

**Michael Hudson** · Jul 18 '05, 07:08 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

bhoel@web.de (Berthold Hoellmann) writes:
[color=blue]
> Hello,
>
> When I use
>
> ./configure --with-thread --with-fpectl --with-signal-module \
> --with-pymalloc --enable-shared --with-cxx=g++
>
> make test[/color]

What platform, compiler, and versions thereof?
[color=blue]
>
> on 2.3.3 I get
>
> ...
> test_queue
> test_quopri
> test_random
> test_re
> make: *** [test] Speicherzugriff sfehler (Speicherauszug erstellt)
>
> gdb ./python core.29964
> GNU gdb 5.3.93
> Copyright 2003 Free Software Foundation, Inc.
> ...
> Reading symbols from /home/devel/compile/Python-2.3.3/build/lib.linux-i686-2.3/pyexpat.so...do ne.
> Loaded symbols for /home/devel/compile/Python-2.3.3/build/lib.linux-i686-2.3/pyexpat.so[/color]

[snippety]
[color=blue]
> Some kind of "infinite" recursion?[/color]

Well, there's a test in tes_sre that uses quite a lot of C stack. I
guess it's possible that something about shared library code uses more
stack. Can you run 'ulimit -s <something larger>' and try again?

Cheers,
mwh

--
Lisp does badly because we refuse to lie. When people ask us if
we can solve insoluble problems we say that we can't, and because
they expect us to lie to them, they find some other language
where the truth is less respected. -- Tim Bradshaw, comp.lang.lisp

**Berthold Hoellmann** · Jul 18 '05, 07:08 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

Michael Hudson <mwh@python.net > writes:
[color=blue]
> bhoel@web.de (Berthold Hoellmann) writes:
>[color=green]
>> Hello,
>>
>> When I use
>>
>> ./configure --with-thread --with-fpectl --with-signal-module \
>> --with-pymalloc --enable-shared --with-cxx=g++
>>
>> make test[/color]
>
> What platform, compiler, and versions thereof?[/color]

Sorry, of course:

SuSE Linux 9.0,
gcc (GCC) 3.3.1 (SuSE Linux)

I always forget about my *FLAGS:

CXXFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
-ffast-math -fno-math-errno -funsafe-math-optimizations \
-fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow

CFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
-ffast-math -fno-math-errno -funsafe-math-optimizations \
-fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow

(My Processor is a AMD Athlon(tm) XP 2000+, 512MB RAM)
[color=blue]
> ...
> Well, there's a test in tes_sre that uses quite a lot of C stack. I
> guess it's possible that something about shared library code uses more
> stack. Can you run 'ulimit -s <something larger>' and try again?
>[/color]

Well, stacksize is unlimited (somehow)
[color=blue]
>limit[/color]
cputime unlimited
filesize unlimited
datasize unlimited
stacksize unlimited
coredumpsize unlimited
memoryuse unlimited
vmemoryuse unlimited
descriptors 1024
memorylocked unlimited
maxproc 4095

Regards
Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.

**Andrew MacIntyre** · Jul 18 '05, 07:08 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

On Sun, 28 Dec 2003, Berthold Hoellmann wrote:
[color=blue][color=green]
> > When I use
> >
> > ./configure --with-thread --with-fpectl --with-signal-module \
> > --with-pymalloc --enable-shared --with-cxx=g++
> >
> > make test
> >
> > on 2.3.3 I get
> >
> > ...
> > test_queue
> > test_quopri
> > test_random
> > test_re
> > make: *** [test] Speicherzugriff sfehler (Speicherauszug erstellt)[/color]
>
> Everything works fine if I remove the "--enable-shared" flag from
> configure.[/color]

You don't mention what platform you are seeing this on. The output of a
verbose test run (python Lib/test/regrtest.py -v test_re), preferably with
error messages translated to English, may help diagnose the issue.

I know that there are platforms where the amount of stack space available
to a threaded process is not easily controlled, and recent versions of gcc
are creating much larger stack frames than earlier versions. The sre
module in 2.3.x (and earlier) is recursive and thus sensitive to stack
space availability. A core dump is a likely indicator of this. Read the
Modules/_sre.c source file for more info. sre in 2.4 will be
significantly improved in this regard.

FYI, since 2.3 PyMalloc is a default option so you don't need
--with-pymalloc, and most recent Linux and BSD systems will default to
building with --with-thread and --with-signal-module.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

**Andrew MacIntyre** · Jul 18 '05, 07:08 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

On Sun, 28 Dec 2003, Berthold Hoellmann wrote:
[color=blue]
> CXXFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
> -ffast-math -fno-math-errno -funsafe-math-optimizations \
> -fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow
>
> CFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
> -ffast-math -fno-math-errno -funsafe-math-optimizations \
> -fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow[/color]

Build Python with the configure defaults before you start getting creative
with optimisation.

Problems resulting from trying to use more aggressive optimisations than
the default should be taken up with the compiler vendor.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

**Berthold Höllmann** · Jul 18 '05, 07:09 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

Andrew MacIntyre <andymac@bullse ye.apana.org.au > writes:
[color=blue]
> On Sun, 28 Dec 2003, Berthold Hoellmann wrote:
>[color=green]
>> CXXFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
>> -ffast-math -fno-math-errno -funsafe-math-optimizations \
>> -fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow
>>
>> CFLAGS=-O3 -fstrict-aliasing -funroll-loops -fschedule-insns2 \
>> -ffast-math -fno-math-errno -funsafe-math-optimizations \
>> -fno-trapping-math -march=athlon-xp -mfpmath=sse -msse -mmmx -m3dnow[/color]
>
> Build Python with the configure defaults before you start getting creative
> with optimisation.
>
> Problems resulting from trying to use more aggressive optimisations than
> the default should be taken up with the compiler vendor.[/color]

Hello,

make distclean
env CFLAGS= CXXFLAGS= ./configure --with-thread --with-fpectl \
--with-signal-module --with-pymalloc --enable-shared --with-cxx=g++
env LANG=C CFLAGS= CXXFLAGS= make test

gave:

test_quopri
test_random
test_re
make: *** [test] Segmentation fault

aggain.

Regards

Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.

**Berthold Höllmann** · Jul 18 '05, 07:09 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

Andrew MacIntyre <andymac@bullse ye.apana.org.au > writes:
[color=blue]
> On Sun, 28 Dec 2003, Berthold Hoellmann wrote:
>[color=green][color=darkred]
>> > When I use
>> >
>> > ./configure --with-thread --with-fpectl --with-signal-module \
>> > --with-pymalloc --enable-shared --with-cxx=g++
>> >
>> > make test
>> >
>> > on 2.3.3 I get
>> >
>> > ...
>> > test_queue
>> > test_quopri
>> > test_random
>> > test_re
>> > make: *** [test] Speicherzugriff sfehler (Speicherauszug erstellt)[/color][/color][/color]

make: *** [test] Segmentation fault (core dumped)
[color=blue][color=green]
>>
>> Everything works fine if I remove the "--enable-shared" flag from
>> configure.[/color]
>
> You don't mention what platform you are seeing this on. The output of a
> verbose test run (python Lib/test/regrtest.py -v test_re), preferably with
> error messages translated to English, may help diagnose the issue.[/color]

env LANG=C LD_LIBRARY_PATH =/home/devel/compile/Python-2.3.3:/usr/local/v/lib:/usr/lib/qt3/lib:/usr/local/pgsql/lib:/usr/teTeX/lib./python Lib/test/regrtest.py -v test_re

(LD_LIBRARY_PAT H is taken from the "make test" output) gives:

test_re
test_anyall (test.test_re.R eTests) ... ok
test_basic_re_s ub (test.test_re.R eTests) ... ok
test_bigcharset (test.test_re.R eTests) ... ok
test_bug_113254 (test.test_re.R eTests) ... ok
test_bug_114660 (test.test_re.R eTests) ... ok
test_bug_117612 (test.test_re.R eTests) ... ok
test_bug_418626 (test.test_re.R eTests) ... ok
test_bug_448951 (test.test_re.R eTests) ... ok
test_bug_449000 (test.test_re.R eTests) ... ok
test_bug_449964 (test.test_re.R eTests) ... ok
test_bug_462270 (test.test_re.R eTests) ... ok
test_bug_527371 (test.test_re.R eTests) ... ok
test_bug_545855 (test.test_re.R eTests) ... ok
test_bug_612074 (test.test_re.R eTests) ... ok
test_bug_725106 (test.test_re.R eTests) ... ok
test_bug_725149 (test.test_re.R eTests) ... ok
test_bug_764548 (test.test_re.R eTests) ... ok
test_category (test.test_re.R eTests) ... ok
test_constants (test.test_re.R eTests) ... ok
test_expand (test.test_re.R eTests) ... ok
test_finditer (test.test_re.R eTests) ... ok
test_flags (test.test_re.R eTests) ... ok
test_getattr (test.test_re.R eTests) ... ok
test_getlower (test.test_re.R eTests) ... ok
test_groupdict (test.test_re.R eTests) ... ok
test_ignore_cas e (test.test_re.R eTests) ... ok
test_non_consum ing (test.test_re.R eTests) ... ok
test_not_litera l (test.test_re.R eTests) ... ok
test_pickling (test.test_re.R eTests) ... ok
test_qualified_ re_split (test.test_re.R eTests) ... ok
test_qualified_ re_sub (test.test_re.R eTests) ... ok
test_re_escape (test.test_re.R eTests) ... ok
test_re_findall (test.test_re.R eTests) ... ok
test_re_groupre f (test.test_re.R eTests) ... ok
test_re_groupre f_exists (test.test_re.R eTests) ... ok
test_re_match (test.test_re.R eTests) ... ok
test_re_split (test.test_re.R eTests) ... ok
test_re_subn (test.test_re.R eTests) ... ok
test_repeat_min max (test.test_re.R eTests) ... ok
test_scanner (test.test_re.R eTests) ... ok
test_search_cov erage (test.test_re.R eTests) ... ok
test_search_sta r_plus (test.test_re.R eTests) ... ok
test_special_es capes (test.test_re.R eTests) ... ok
test_sre_charac ter_literals (test.test_re.R eTests) ... ok
test_stack_over flow (test.test_re.R eTests) ... ok
test_symbolic_r efs (test.test_re.R eTests) ... ok

----------------------------------------------------------------------
Ran 46 tests in 0.401s

OK
Running re_tests test suite
1 test OK.
CAUTION: stdout isn't compared in verbose mode:
a test that passes in verbose mode may fail without it.

so it seems that the other modules loaded in the "make test" run
decrease the avaliable stack so that the test does not succeed but
calling the test alone finds enough stack space avaliable?
[color=blue]
> I know that there are platforms where the amount of stack space available
> to a threaded process is not easily controlled, and recent versions of gcc
> are creating much larger stack frames than earlier versions. The sre
> module in 2.3.x (and earlier) is recursive and thus sensitive to stack
> space availability. A core dump is a likely indicator of this. Read the
> Modules/_sre.c source file for more info. sre in 2.4 will be
> significantly improved in this regard.
>
> FYI, since 2.3 PyMalloc is a default option so you don't need
> --with-pymalloc, and most recent Linux and BSD systems will default to
> building with --with-thread and --with-signal-module.[/color]

I really should know, but I usually keep "good" configure settings for
reuse and normally don't think much about the settings when everything
works.

Regards

Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.

**Andrew MacIntyre** · Jul 18 '05, 07:10 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

On Mon, 29 Dec 2003, Berthold Höllmann wrote:
[color=blue]
> make distclean
> env CFLAGS= CXXFLAGS= ./configure --with-thread --with-fpectl \
> --with-signal-module --with-pymalloc --enable-shared --with-cxx=g++
> env LANG=C CFLAGS= CXXFLAGS= make test
>
> gave:
>
> test_quopri
> test_random
> test_re
> make: *** [test] Segmentation fault[/color]

1. Just to confirm that the configure generated Makefile is what it
should be, what are the OPT and BASECFLAGS values?

2. Do a verbose run of the test_re test with the interpreter you built
above (ie with the default optimisations):
./python Lib/test/regrtest.py -v test_re >test_re.log 2>&1

3. If it coredumps (which I expect it to based on your advice above), use
gdb to get the backtrace.

The test log and the gdb backtrace should help isolate where things are
going astray.

BTW, I'm not sure why you're explicitly using --with-thread,
--with-signal-module and --with-pymalloc as they are all enabled by
default. I'm also unsure as to whether you really need to specify
--with-cxx; it might be worth a try configure'ing without it.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

**Andrew MacIntyre** · Jul 18 '05, 07:10 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

On Mon, 29 Dec 2003, Berthold Höllmann wrote:
[color=blue]
> so it seems that the other modules loaded in the "make test" run
> decrease the avaliable stack so that the test does not succeed but
> calling the test alone finds enough stack space avaliable?[/color]

The above message crossed with my other reply.

This looks very much like the stack problem I referred to, which I am
familiar with on FreeBSD. I have been able to build with
--enable-shared on FreeBSD (after some configure tweaking), and see a
similar effect - ie test_re coredumps with --enable-shared and passes
without (and this with gcc 2.95.4). From the backtrace, I see that a
USE_RECURSION_L IMIT of 7200 would have worked, compared to the 7500 that
works for the standard build.

That it passes the individual test but not the full test suite for you
suggests that your build is right on the stacksize limit, and lowering
USE_RECURSION_L IMIT by only a small amount would get you through.

Note that there are some subtleties here - even though your ulimit says
unlimited stackspace, in the presence of threads this may not hold.
It certainly doesn't on FreeBSD 4.x where there the pthreads support has a
hardcoded 1MB stack for the primary thread (which is what is running the
regression test and is distinct from the stack created for each
instantiated thread). The LinuxThreads port on FreeBSD seems not to have
the same restriction. Exactly what is happening on your Linux system I
don't know; you would have to do some research to find out.

Depending on how you wish to proceed, you have a couple of options:
- build normally, then blow away Modules/_sre.o and recompile
Modules/_sre.c with -Os (just temporarily doctor the Makefile).
From what I've seen -Os cuts the stack consumption enough to get
you through the full test suite, and doesn't seriously impact sre's
performance.
- modify the USE_RECURSION_L IMIT macro in Modules/_sre.c to a lower value
and rebuild. Linux would be using the default definition of 10000 at
line 98 (for 2.3.3).
- find a way to change the default stack size for the primary thread.

If you play with the second or third options, a report to the Python bug
tracker on SF with the conclusion you reach would be useful. Note that
such a bug report should include details of OS & compiler, so that
suitable #ifdef'ery can be written, and should be based on Python's
default compiler switches. As I noted previously, sre in Python 2.4 will
not have this issue, but Anthony Baxter has said that he expects to do a
2.3.4 release (April/May O4 as I recall).

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

**Berthold Höllmann** · Jul 18 '05, 07:16 AM

Re: make test segfaults with "--enable-shared" on Python 2.3.3

Andrew MacIntyre <andymac@bullse ye.apana.org.au > writes:
[color=blue]
> On Mon, 29 Dec 2003, Berthold Höllmann wrote:
>[color=green]
>> so it seems that the other modules loaded in the "make test" run
>> decrease the avaliable stack so that the test does not succeed but
>> calling the test alone finds enough stack space avaliable?[/color][/color]
....[color=blue]
> Note that there are some subtleties here - even though your ulimit says
> unlimited stackspace, in the presence of threads this may not hold.
> It certainly doesn't on FreeBSD 4.x where there the pthreads support has a
> hardcoded 1MB stack for the primary thread (which is what is running the[/color]

I found a document stating it is 2MB for each thread on Linux using Google
[color=blue]
> regression test and is distinct from the stack created for each
> instantiated thread). The LinuxThreads port on FreeBSD seems not to have
> the same restriction. Exactly what is happening on your Linux system I
> don't know; you would have to do some research to find out.
>
> Depending on how you wish to proceed, you have a couple of options:
> - build normally, then blow away Modules/_sre.o and recompile
> Modules/_sre.c with -Os (just temporarily doctor the Makefile).
> From what I've seen -Os cuts the stack consumption enough to get
> you through the full test suite, and doesn't seriously impact sre's
> performance.[/color]

This does work for me
[color=blue]
> - modify the USE_RECURSION_L IMIT macro in Modules/_sre.c to a lower value
> and rebuild. Linux would be using the default definition of 10000 at
> line 98 (for 2.3.3).[/color]

#define USE_RECURSION_L IMIT 6835

does succeed,

#define USE_RECURSION_L IMIT 6840

gives segfault for me. Will submit bugreport.
[color=blue]
> - find a way to change the default stack size for the primary thread.[/color]

To be honest: I don't have an idea where to look at for this.

Regards

Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.

make test segfaults with "--enable-shared" on Python 2.3.3

make test segfaults with "--enable-shared" on Python 2.3.3

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment