A bug for unicode strings in Python 2.4?

[Thomas Moore]

Hi:

Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on
win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> u=u'\u9019\u662 f\u4e2d\u6587\u 5b57\u4e32'
>>> u.split()[/color][/color][/color]
[u'\u9019\u662f\ u4e2d\u6587\u5b 57\u4e32'][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

I think u should get split.

--Frank

[Neil Hodgson]

Re: A bug for unicode strings in Python 2.4?

Thomas Moore:
[color=blue][color=green][color=darkred]
>>>>u=u'\u9019\ u662f\u4e2d\u65 87\u5b57\u4e32'
>>>>u.split()[/color][/color]
>
> [u'\u9019\u662f\ u4e2d\u6587\u5b 57\u4e32']
>
>
> I think u should get split.[/color]

Where do you think "這是ä¸æ–‡å —串" should be split and why?

Neil

[Thomas Moore]

Re: A bug for unicode strings in Python 2.4?

> Thomas Moore:[color=blue]
>[color=green][color=darkred]
> >>>>u=u'\u9019\ u662f\u4e2d\u65 87\u5b57\u4e32'
> >>>>u.split()[/color]
> >
> > [u'\u9019\u662f\ u4e2d\u6587\u5b 57\u4e32']
> >
> >
> > I think u should get split.[/color]
>
> Where do you think "這是ä¸æ–‡å —串" should be split and why?[/color]

Isn't a unicode string character by character?

-Frank

[Fredrik Lundh]

Re: A bug for unicode strings in Python 2.4?

Thomas Moore wrote:
[color=blue]
> Python 2.4.1 (#65, Mar 30 2005, 09:13:57) [MSC v.1310 32 bit (Intel)] on
> win32
> Type "help", "copyright" , "credits" or "license" for more information.[color=green][color=darkred]
> >>> u=u'\u9019\u662 f\u4e2d\u6587\u 5b57\u4e32'
> >>> u.split()[/color][/color]
> [u'\u9019\u662f\ u4e2d\u6587\u5b 57\u4e32'][color=green][color=darkred]
> >>>[/color][/color]
>
> I think u should get split.[/color]

why? split splits on whitespace (basically unicode category Zs), and
there are no whitespace symbols in there:
[color=blue][color=green][color=darkred]
>>> u=u'\u9019\u662 f\u4e2d\u6587\u 5b57\u4e32'
>>> [c.isspace() for c in u][/color][/color][/color]
[False, False, False, False, False, False]

there's no universal "split on words in all languages" function in the
standard python library. You may be able to roll your own using the
information in http://www.unicode.org/reports/tr29/ plus functions
in the unicodedata module (which currently doesn't include the
BreakTest tables; patches are welcome). Or maybe google can
help you find an existing implementation.

</F>

[Thomas Moore]

Re: A bug for unicode strings in Python 2.4?

Hi:

Thanks. I'll write my own split().

Frank

[Szabolcs Nagy]

Re: A bug for unicode strings in Python 2.4?

> Thanks. I'll write my own split().

do you want to split character by character?
then use
list(u'\u9019\u 662f\u4e2d\u658 7\u5b57\u4e32')