Strange re behavior: normal?

**David C. Fox** · Jul 18 '05, 01:36 AM

Re: Strange re behavior: normal?

Robin Munn wrote:[color=blue]
> How is re.split supposed to work? This wasn't at all what I expected:
>
> [rmunn@localhost ~]$ python
> Python 2.2.2 (#1, Jan 12 2003, 12:07:20)
> [GCC 3.2] on linux2
> Type "help", "copyright" , "credits" or "license" for more information.
>[color=green][color=darkred]
>>>>import re
>>>>re.split(r' \W+', 'a b c d')[/color][/color]
>
> ['a', 'b', 'c', 'd']
>[color=green][color=darkred]
>>>># Expected result.[/color][/color]
>
> ...
>[color=green][color=darkred]
>>>>re.split(r' \b', 'a b c d')[/color][/color]
>
> ['a b c d']
>[color=green][color=darkred]
>>>># Huh?[/color][/color]
>
>
> Since \b matches the empty string, but only at the beginning and end of
> a word, I would have expected re.split(r'\b', 'a b c d' to produce
> either:
>
> ['', 'a', ' ', 'b', ' ', 'c', ' ', 'd', '']
>
> or:
>
> ['a', ' ', 'b', ' ', 'c', ' ', 'd']
>
> But I didn't expect that re.split(r'\b', 'a b c d') would yield no splits
> whatsoever. The module doc says "split(patt ern, string[, maxsplit = 0]):
> split string by the occurrences of pattern". re.findall() seems to think
> that \b occurs eight times in 'a b c d':
>
>[color=green][color=darkred]
>>>>re.findall( r'\b', 'a b c d')[/color][/color]
>
> ['', '', '', '', '', '', '', '']
>
> So why doesn't re.split() think so? I'm puzzled.
>[/color]

It looks like re.split is not splitting on zero-length matches. I get
similar behavior (in Python 2.2.2) if I try:

re.split('x*', 'a b c d')

or

re.split('(?=c) ', 'a b c d')

I don't have the Python source handy to verify this hypothesis, though.

If this is correct, it should at least be documented.

David

**Mike Rovner** · Jul 18 '05, 01:36 AM

Re: Strange re behavior: normal?

Fredrik Lundh wrote:[color=blue]
> Mike Rovner wrote:[color=green]
>> IMHO that split behavior is a bug although technicaly it is not.
>> (From re manual:
>> "This module provides regular expression matching operations similar
>> to those found in Perl.")[/color]
>
> is "split" really a matching operation?
>
> fact is, all methods have Python-specific behaviour. it's just the RE
> language itself that's based on Perl.[/color]

With all due respect to Python and not trying to bend it to any other
language
I believe it trys to do what user expects.

string.split() splits on (clearly documented nonempty) substring.
re.split() splits on RE.
splut(r'\b',... ) clearly means (at least for me) 'split on word boundary'
It doesn't reject it (as in r'\b?') nor provide expected behavior.

I understand why it does what it does, but don't agree with it.

Regards,
Mike

**Fredrik Lundh** · Jul 18 '05, 01:36 AM

Re: Strange re behavior: normal?

Mike Rovner wrote:
[color=blue]
> With all due respect to Python and not trying to bend it to any other
> language I believe it trys to do what user expects.[/color]

it does exactly what I expect it to do.

</F>

**John J. Lee** · Jul 18 '05, 01:37 AM

Re: Strange re behavior: normal?

"Fredrik Lundh" <fredrik@python ware.com> writes:
[color=blue]
> Mike Rovner wrote:
>[color=green]
> > With all due respect to Python and not trying to bend it to any other
> > language I believe it trys to do what user expects.[/color]
>
> it does exactly what I expect it to do.
>
> </F>[/color]

You don't count. ;-)

John

**Robin Munn** · Jul 18 '05, 01:41 AM

Re: Strange re behavior: normal?

Michael Janssen <Janssen@rz.u ni-frankfurt.de> wrote:[color=blue]
>
> What's the good of splitting by boundaries? Someone else wanted this a
> few days ago on tutor and I can't figure out a reason by now.[/color]

Heh. I bet I know the name of the person who was asking about this on
the tutor list. He's a friend of mine, and I've been helping him learn
Python. He E-mailed me about trying to split on word boundaries with
re.split(r'\b', 'some text'), and it was his E-mail that caused me to
discover that splitting by boundaries didn't do what I expected.

What's the good of it? As someone else pointed out, it allows you to
fetch the words and the separating text, yielding:

['See', ' ', 'Spot', '. ', 'See', ' ', 'Spot', ' ', 'run', '.']

which may be useful in certain English-language-parsing situations,
since it would allow you to look "ahead" or "back" from a word to see
what punctuation precedes or follows it.

Anyway, the re.split behavior I described isn't particularly bothering
me, but I do think it should be better documented. Time to submit a doc
patch, methinks...

--
Robin Munn <rmunn@pobox.co m> | http://www.rmunn.com/ | PGP key 0x6AFB6838
-----------------------------+-----------------------+----------------------
"Remember, when it comes to commercial TV, the program is not the product.
YOU are the product, and the advertiser is the customer." - Mark W. Schumann

Strange re behavior: normal?

Strange re behavior: normal?

Comment

Comment

Comment

Comment

Comment