RE + UTF-8

[cepl@surfbest.net]

Working on extension of genericwiki.py plugin for PyBlosxom and I have
problems with UTF-8 and RE. When I have this wiki line, it does break
URL too early:

[http://en.wikipedia.org/wiki/Petr_Chelcický Petr Chelcický's]
work(s) into English.

and creates

[<a
href="http://en.wikipedia.or g/wiki/Petr_Chel">http ://en.wikipedia.or g/wiki/Petr_Chel</a>cický
Petr Chelcický's]

The RE genericwiki uses for parsing this:

# WikiName pattern used in your wiki
wikinamepattern = r'\b(([A-Z]\w+){2,})\b' # original
mailurlpattern = r'mailto\:[\"\-\_\.\w]+\@[\-\_\.\w]+\w'
newsurlpattern = r'news\:(?:\w+\ .){1,}\w+'
fileurlpattern =
r'(?:http|https |file|ftp):[/-_.\w-]+[\/\w][?&+=%\w/-_.#]*'

[...]

# Turn '[xxx:address label]' into labeled link
body = re.sub(r'\[(' +
fileurlpattern + '|' +
mailurlpattern + '|' +
newsurlpattern + ')\s+(.+?)\]',
r'<a href="\1">\2</a>', body,re.U)

I have tried to test RE and UTF-8 in Python generally and the results
are even more confusing (done with locale cs_CZ.UTF-8 in konsole):
[color=blue][color=green]
>> locale.getprefe rredencoding()[/color][/color]
'UTF-8'[color=blue][color=green][color=darkred]
>>> print re.sub("(\w*)", "X","[Chelcický]",re.L)[/color][/color][/color]
X[X?Xý][color=blue][color=green][color=darkred]
>>> print re.sub("(\w*)", "X","[Chelcický]",re.UNICOD E)[/color][/color][/color]
X[X?X?X]X[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

I would expect that both print commands should give just plain X, but
apparently Python doesn't undestand that. What's the problem?

Thanks for any reply,

Matej

[Michael Ströder]

Re: RE + UTF-8

cepl@surfbest.n et wrote:[color=blue]
>
> I have tried to test RE and UTF-8 in Python generally and the results
> are even more confusing (done with locale cs_CZ.UTF-8 in konsole):
>[color=green][color=darkred]
>>>locale.getpr eferredencoding ()[/color][/color]
>
> 'UTF-8'
>[color=green][color=darkred]
>>>>print re.sub("(\w*)", "X","[Chelcický]",re.L)[/color][/color][/color]

You first have to turn the raw strings into Unicode strings. It seems on
your console it should be:

unicode('[Chelcický]','utf-8')

Note that you have to set HTTP headers and <form accept-charset=...> in
web applications.

Ciao, Michael.