Test error with Python 2.3.4c1

[Berthold Höllmann]

Hello,

The error desribed in SF error report No. 870120 (segmentation fault
in test_re) still persists for me in 2.3.4c1.

Regards
Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/

[Aahz]

Re: Test error with Python 2.3.4c1

In article <m2fza1h6ka.fsf @pchoel.psh>, Berthold Höllmann <bhoel@web.de > wrote:[color=blue]
>
>The error desribed in SF error report No. 870120 (segmentation fault
>in test_re) still persists for me in 2.3.4c1.[/color]

Is there a patch for it? Bugfix releases are mostly low-effort
collections of patches that already exist -- there's not much attempt
made to call for new patches before the release goes out. All that
bugfix release candidates are intended to do is make sure that no new
bugs have been created.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!

[Berthold Höllmann]

Re: Test error with Python 2.3.4c1

aahz@pythoncraf t.com (Aahz) writes:
[color=blue]
> In article <m2fza1h6ka.fsf @pchoel.psh>, Berthold Höllmann <bhoel@web.de > wrote:[color=green]
>>
>>The error desribed in SF error report No. 870120 (segmentation fault
>>in test_re) still persists for me in 2.3.4c1.[/color]
>
> Is there a patch for it? Bugfix releases are mostly low-effort
> collections of patches that already exist -- there's not much attempt
> made to call for new patches before the release goes out. All that
> bugfix release candidates are intended to do is make sure that no new
> bugs have been created.[/color]

The patch working for me is to reduce the default USE_RECURSION_L IMIT

diff Modules/_sre.c~ Modules/_sre.c
98c98
< #define USE_RECURSION_L IMIT 10000
---[color=blue]
> #define USE_RECURSION_L IMIT 6835[/color]


Regards
Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/

[Aahz]

Re: Test error with Python 2.3.4c1

In article <m27jvdh5k6.fsf @pchoel.psh>, Berthold Höllmann <bhoel@web.de > wrote:[color=blue]
>aahz@pythoncra ft.com (Aahz) writes:[color=green]
>> In article <m2fza1h6ka.fsf @pchoel.psh>, Berthold H=F6llmann <bhoel@web.de =
>> wrote:[color=darkred]
>>>
>>>The error desribed in SF error report No. 870120 (segmentation fault
>>>in test_re) still persists for me in 2.3.4c1.[/color]
>>
>> Is there a patch for it? Bugfix releases are mostly low-effort
>> collections of patches that already exist -- there's not much attempt
>> made to call for new patches before the release goes out. All that
>> bugfix release candidates are intended to do is make sure that no new
>> bugs have been created.[/color]
>
>The patch working for me is to reduce the default USE_RECURSION_L IMIT
>
>diff Modules/_sre.c~ Modules/_sre.c
>98c98
>< #define USE_RECURSION_L IMIT 10000
>---[color=green]
>> #define USE_RECURSION_L IMIT 6835[/color][/color]

In that case, you should check against the CVS version of Python; I
believe there've been some changes that affect this kind of thing. If
that fixes your problem, please close the bug report.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!

[Berthold Höllmann]

Re: Test error with Python 2.3.4c1

aahz@pythoncraf t.com (Aahz) writes:
[color=blue]
> In article <m27jvdh5k6.fsf @pchoel.psh>, Berthold Höllmann <bhoel@web.de > wrote:[color=green]
>>aahz@pythoncr aft.com (Aahz) writes:[color=darkred]
>>> In article <m2fza1h6ka.fsf @pchoel.psh>, Berthold H=F6llmann <bhoel@web.de =
>>> wrote:
>>>>[/color][/color][/color]
....>[color=blue]
> In that case, you should check against the CVS version of Python; I
> believe there've been some changes that affect this kind of thing. If
> that fixes your problem, please close the bug report.[/color]

OK, I just checked the latest CVS and test_re succeeds. But should't
this fixed for 2.3.4 as well?

Another problem not there in the CVS version is an error in test_sax,
that fails for me on 2.3.4c1 with
[color=blue]
>echo test_sax > tests
>make test TESTOPTS="-l -ftests -v"[/color]
case $MAKEFLAGS in \
*-s*) LD_LIBRARY_PATH =/home/devel/compile/Python-2.3.4c1:/usr/local/pgsql/lib:/usr/teTeX/lib CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python -E ./setup.py -q build;; \
*) LD_LIBRARY_PATH =/home/devel/compile/Python-2.3.4c1:/usr/local/pgsql/lib:/usr/teTeX/lib CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python -E ./setup.py build;; \
esac
running build
running build_ext
running build_scripts
find ./Lib -name '*.py[co]' -print | xargs rm -f
LD_LIBRARY_PATH =/home/devel/compile/Python-2.3.4c1:/usr/local/pgsql/lib:/usr/teTeX/lib ./python -E -tt ./Lib/test/regrtest.py -l -ftests -v
test_sax
Failed test_attrs_empt y
Failed test_attrs_watt r
Failed test_double_quo teattr
Failed test_escape_all
Failed test_escape_bas ic
Failed test_escape_ext ra
Failed test_expat_attr s_empty
Failed test_expat_attr s_wattr
Failed test_expat_dtdh andler
Failed test_expat_enti tyresolver
Failed test_expat_file
Failed test_expat_inco mplete
Failed test_expat_incr emental
Failed test_expat_incr emental_reset
Failed test_expat_inps ource_filename
Failed test_expat_inps ource_location
Failed test_expat_inps ource_stream
Failed test_expat_inps ource_sysid
Failed test_expat_loca tor_noinfo
Failed test_expat_loca tor_withinfo
Failed test_expat_nsat trs_empty
Failed test_expat_nsat trs_wattr
Failed test_filter_bas ic
Failed test_make_parse r
Failed test_make_parse r2
Failed test_nsattrs_em pty
Failed test_nsattrs_wa ttr
Failed test_quoteattr_ basic
test test_sax crashed -- exceptions.Type Error: int argument required
Traceback (most recent call last):
File "./Lib/test/regrtest.py", line 394, in runtest
the_package = __import__(abst est, globals(), locals(), [])
File "/home/devel/compile/Python-2.3.4c1/Lib/test/test_sax.py", line 689,in ?
confirm(value() , name)
File "/home/devel/compile/Python-2.3.4c1/Lib/test/test_sax.py", line 502,in test_sax_parse_ exception_str
DummyLocator(No ne, 1)))
File "/usr/local/lib/python2.3/site-packages/_xmlplus/sax/_exceptions.py" , line 94, in __str__
return "%s:%d:%d: %s" % (sysid, self.getLineNum ber(),
TypeError: int argument required
1 test failed:
test_sax
make: [test] Error 1 (ignored)
....

self.getLineNum ber() is None if I check using pdb, but what
disconcerts me is the

File "/usr/local/lib/python2.3/site-packages/_xmlplus/sax/_exceptions.py" , line 94, in __str__

line. Why is sax taken from the installed version instead of the
2.3.4c1 version.

Regards
Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/

[Aahz]

Re: Test error with Python 2.3.4c1

In article <m2vfixfnc3.fsf @pchoel.psh>, Berthold Höllmann <bhoel@web.de > wrote:[color=blue]
>aahz@pythoncra ft.com (Aahz) writes:[color=green]
>>
>> In that case, you should check against the CVS version of Python; I
>> believe there've been some changes that affect this kind of thing. If
>> that fixes your problem, please close the bug report.[/color]
>
>OK, I just checked the latest CVS and test_re succeeds. But should't
>this fixed for 2.3.4 as well?[/color]

Nope. Bugfix releases (see PEP 6) are required to have essentially zero
behavioral changes, and IIRC the series of fixes that fixed your bug
caused some changes to the way sre operates, so it was a non-starter for
the 2.3.x series.

Anyway, please close your bug with a note about it being fixed in 2.4
CVS. As for the other issue, I don't have enough time/energy to even
understand what the issue is. Given that it's about SAX, I suggest you
start a new thread and hopefully someone else will pick it up.
--
Aahz (aahz@pythoncra ft.com) <*> http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!

[Richard Townsend]

Re: Test error with Python 2.3.4c1

> self.getLineNum ber() is None if I check using pdb, but what[color=blue]
> disconcerts me is the
>
> File[/color]
"/usr/local/lib/python2.3/site-packages/_xmlplus/sax/_exceptions.py" , line
94, in __str__[color=blue]
>
> line. Why is sax taken from the installed version instead of the
> 2.3.4c1 version.[/color]


I raised a bug report (864379) back in December or a similar problem with
2.3.3 on HP-UX - except there it caused a core dump.

I too was surprised that it was accessing code in the current installation.

The last response to the bug report suggested an incompatibility with
PyXML-0.8.3, but since then there have been no further responses.

[Berthold Höllmann]

Re: Test error with Python 2.3.4c1

aahz@pythoncraf t.com (Aahz) writes:
[color=blue]
> In article <m2vfixfnc3.fsf @pchoel.psh>, Berthold Höllmann <bhoel@web.de > wrote:[color=green]
>>aahz@pythoncr aft.com (Aahz) writes:[color=darkred]
>>>
>>> In that case, you should check against the CVS version of Python; I
>>> believe there've been some changes that affect this kind of thing. If
>>> that fixes your problem, please close the bug report.[/color]
>>
>>OK, I just checked the latest CVS and test_re succeeds. But should't
>>this fixed for 2.3.4 as well?[/color]
>
> Nope. Bugfix releases (see PEP 6) are required to have essentially zero
> behavioral changes, and IIRC the series of fixes that fixed your bug
> caused some changes to the way sre operates, so it was a non-starter for
> the 2.3.x series.
>
> Anyway, please close your bug with a note about it being fixed in 2.4
> CVS. As for the other issue, I don't have enough time/energy to even
> understand what the issue is. Given that it's about SAX, I suggest you
> start a new thread and hopefully someone else will pick it up.[/color]

OK, I closed the bug, but still would like to see USE_RECURSION_L IMIT
default reduced to something like 6500 (3000 seems to be OK for
FreeBSD on sparc64, so I guess this can't be a behavioral change).

Regards
Berthold
--
bhoel@web.de / http://starship.python.net/crew/bhoel/

[Andrew MacIntyre]

Re: Test error with Python 2.3.4c1

On Sun, 16 May 2004, Berthold Höllmann wrote:
[color=blue][color=green]
> > caused some changes to the way sre operates, so it was a non-starter for
> > the 2.3.x series.
> >
> > Anyway, please close your bug with a note about it being fixed in 2.4
> > CVS. As for the other issue, I don't have enough time/energy to even
> > understand what the issue is. Given that it's about SAX, I suggest you
> > start a new thread and hopefully someone else will pick it up.[/color]
>
> OK, I closed the bug, but still would like to see USE_RECURSION_L IMIT
> default reduced to something like 6500 (3000 seems to be OK for
> FreeBSD on sparc64, so I guess this can't be a behavioral change).[/color]

Prior to 2.4, sre is recursive. Recent versions of gcc (in particular)
at higher optimisation levels are generating much larger stack frames than
used to be the case. This is even moreso the case with 64-bit platforms.

In the particular case of FreeBSD 4.x, in the presence of threads (the
default build choice, using libc_r) there is a hard coded stack size of
1MB, which leaves this environment especially sensitive to stack frame
size. The alternative threads libraries (libkse/libthr) available on 5.x
are less affected, but I don't know to what extent (& Python's configure
script doesn't know about them that I've heard).

I've seen reference to the problem on Linux, but I don't know why that
platform is (apparently) short on stack.

This is a highly platform/compiler dependant issue, which is why the maze
of #ifdef'ery for USE_RECURSION_L IMIT - something that has only happened
since about Python 2.3.1 as I recall.

The most effective solution with recent gcc versions is to recompile sre.c
with -Os (instead of -O3). It is unfortunately awkward to implement with
autoconf. The non-recursive sre in 2.4 is a complete fix, but is too
extensive/invasive a change for back-porting for a bug-fix release.

--
Andrew I MacIntyre "These thoughts are mine alone..."
E-mail: andymac@bullsey e.apana.org.au (pref) | Snail: PO Box 370
andymac@pcug.or g.au (alt) | Belconnen ACT 2616
Web: http://www.andymac.org/ | Australia

[Berthold Höllmann]

Re: Test error with Python 2.3.4c1

Andrew MacIntyre <andymac@bullse ye.apana.org.au > writes:
[color=blue]
> On Sun, 16 May 2004, Berthold Höllmann wrote:
>[color=green][color=darkred]
>> > caused some changes to the way sre operates, so it was a non-starter for
>> > the 2.3.x series.
>> >
>> > Anyway, please close your bug with a note about it being fixed in 2.4
>> > CVS. As for the other issue, I don't have enough time/energy to even
>> > understand what the issue is. Given that it's about SAX, I suggest you
>> > start a new thread and hopefully someone else will pick it up.[/color]
>>
>> OK, I closed the bug, but still would like to see USE_RECURSION_L IMIT
>> default reduced to something like 6500 (3000 seems to be OK for
>> FreeBSD on sparc64, so I guess this can't be a behavioral change).[/color]
>
> Prior to 2.4, sre is recursive. Recent versions of gcc (in particular)
> at higher optimisation levels are generating much larger stack frames than
> used to be the case. This is even moreso the case with 64-bit platforms.
>
> In the particular case of FreeBSD 4.x, in the presence of threads (the
> default build choice, using libc_r) there is a hard coded stack size of
> 1MB, which leaves this environment especially sensitive to stack frame
> size. The alternative threads libraries (libkse/libthr) available on 5.x
> are less affected, but I don't know to what extent (& Python's configure
> script doesn't know about them that I've heard).
>
> I've seen reference to the problem on Linux, but I don't know why that
> platform is (apparently) short on stack.
>
> This is a highly platform/compiler dependant issue, which is why the maze
> of #ifdef'ery for USE_RECURSION_L IMIT - something that has only happened
> since about Python 2.3.1 as I recall.
>
> The most effective solution with recent gcc versions is to recompile sre.c
> with -Os (instead of -O3). It is unfortunately awkward to implement with
> autoconf. The non-recursive sre in 2.4 is a complete fix, but is too
> extensive/invasive a change for back-porting for a bug-fix release.[/color]

But the current value for USE_RECURSION_L IMIT is a problem on (some)
Linux Platforms. Why is reducing the default value a problem, when
setting it to a smaller value on some platforms is not a problem. I
thought everything causing segmentation faults from Python is to be
avoided.

Regards
Berthold
--
Dipl.-Ing. Berthold Höllmann __ Address:
hoel@GL-Group.com G / \ L Germanischer Lloyd
phone: +49-40-36149-7374 -+----+- Vorsetzen 32/35 P.O.Box 111606
fax : +49-40-36149-7320 \__/ D-20459 Hamburg D-20416 Hamburg