Locale confusion

[Jorgen Grahn]

[Long posting due to the examples, but pretty simple question.]

I'm sitting here with a Debian Linux 'Woody' system with the default Python
2.2 installation, and I want the re module to understand that
re.compile(r'\W +'. re.LOCALE) doesn't match my national, accented
characters.

I don't quite understand how the locale module reasons about these things,
and Python doesn't seem to act as other programs on my system. Bug or my
mistake? Here's my environment:

frailea> env |grep -e LC -e LANG
LC_MESSAGES=C
LC_TIME=C
LANG=sv_SE
LC_NUMERIC=C
LC_MONETARY=C
frailea> locale
LANG=sv_SE
LC_CTYPE="sv_SE "
LC_NUMERIC=C
LC_TIME=C
LC_COLLATE="sv_ SE"
LC_MONETARY=C
LC_MESSAGES=C
LC_PAPER="sv_SE "
LC_NAME="sv_SE"
LC_ADDRESS="sv_ SE"
LC_TELEPHONE="s v_SE"
LC_MEASUREMENT= "sv_SE"
LC_IDENTIFICATI ON="sv_SE"
LC_ALL=

This seems to indicate that $LANG acts as a fallback when other things (e.g.
LC_CTYPE isn't defined) and that's also what the glibc setlocale(3) man page
says. Works well for me in general, too. However, consider this tiny Python
program:

frailea> cat foo
import locale
print locale.getlocal e()
locale.setlocal e(locale.LC_CTY PE)
print locale.getlocal e()

When I paste it into an interactive Python session, the locale is already
set up correctly (which is what I suppose interactive mode /should/ do):
[color=blue][color=green][color=darkred]
>>> import locale
>>> print locale.getlocal e()[/color][/color][/color]
['sv_SE', 'ISO8859-1'][color=blue][color=green][color=darkred]
>>> locale.setlocal e(locale.LC_CTY PE)[/color][/color][/color]
'sv_SE'[color=blue][color=green][color=darkred]
>>> print locale.getlocal e()[/color][/color][/color]
['sv_SE', 'ISO8859-1'][color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

When I run it as a script it isn't though, and the setlocale() call does not
appear to fall back to looking at $LANG as it's supposed to(?), so my
LC_CTYPE remains in the POSIX locale:

frailea> python foo
(None, None)
(None, None)

The corresponding program written in C works as expected:

frailea> cat foot.c
#include <stdio.h>
#include <locale.h>
int main(void) {
printf("%s\n", setlocale(LC_CT YPE, 0));
printf("%s\n", setlocale(LC_CT YPE, ""));
printf("%s\n", setlocale(LC_CT YPE, 0));
return 0;
}
frailea> ./foot
C
sv_SE
sv_SE

So, is this my fault or Python's? I realize I could just adapt and set
$LC_CTYPE explicitly in my environment, but I don't want to capitulate for a
Python bug, if that's what this is.

BR,
Jorgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!

[Serge.Orlov@gmail.com]

Re: Locale confusion

Jorgen Grahn wrote:
[snip]
[color=blue]
>
> frailea> cat foo
> import locale
> print locale.getlocal e()
> locale.setlocal e(locale.LC_CTY PE)
> print locale.getlocal e()
>
> When I paste it into an interactive Python session, the locale is[/color]
already[color=blue]
> set up correctly (which is what I suppose interactive mode /should/[/color]
do):[color=blue]
>[color=green][color=darkred]
> >>> import locale
> >>> print locale.getlocal e()[/color][/color]
> ['sv_SE', 'ISO8859-1'][color=green][color=darkred]
> >>> locale.setlocal e(locale.LC_CTY PE)[/color][/color]
> 'sv_SE'[color=green][color=darkred]
> >>> print locale.getlocal e()[/color][/color]
> ['sv_SE', 'ISO8859-1'][color=green][color=darkred]
> >>>[/color][/color]
>
> When I run it as a script it isn't though, and the setlocale() call[/color]
does not[color=blue]
> appear to fall back to looking at $LANG as it's supposed to(?), so my
> LC_CTYPE remains in the POSIX locale:
>
> frailea> python foo
> (None, None)
> (None, None)
>
> The corresponding program written in C works as expected:
>
> frailea> cat foot.c
> #include <stdio.h>
> #include <locale.h>
> int main(void) {
> printf("%s\n", setlocale(LC_CT YPE, 0));
> printf("%s\n", setlocale(LC_CT YPE, ""));
> printf("%s\n", setlocale(LC_CT YPE, 0));
> return 0;
> }
> frailea> ./foot
> C
> sv_SE
> sv_SE
>
> So, is this my fault or Python's? I realize I could just adapt and[/color]
set[color=blue]
> $LC_CTYPE explicitly in my environment, but I don't want to[/color]
capitulate for a[color=blue]
> Python bug, if that's what this is.[/color]

Try locale.setlocal e(locale.LC_CTY PE,"") as in your C program. It would
be great if locale.setlocal e with one parameter would be deprecated,
because it suddenly acts like getlocale. It's unpythonic.

By the way, since you took time to setup various LC_* variables there
is no need to play with LC_CTYPE category. Just use the standard idiom.
import locale
locale.setlocal e(LC_ALL,"")

Serge.

[Jorgen Grahn]

Re: Locale confusion

On 11 Jan 2005 05:49:32 -0800, Serge.Orlov@gma il.com <Serge.Orlov@gm ail.com> wrote:[color=blue]
> Jorgen Grahn wrote:
> [snip]
>[color=green]
>>
>> frailea> cat foo
>> import locale
>> print locale.getlocal e()
>> locale.setlocal e(locale.LC_CTY PE)
>> print locale.getlocal e()[/color][/color]

....
[color=blue][color=green]
>> When I run it as a script it isn't though, and the setlocale() call[/color]
> does not[color=green]
>> appear to fall back to looking at $LANG as it's supposed to(?), so my
>> LC_CTYPE remains in the POSIX locale:[/color][/color]
....[color=blue][color=green]
>> So, is this my fault or Python's? I realize I could just adapt and[/color]
> set[color=green]
>> $LC_CTYPE explicitly in my environment, but I don't want to[/color]
> capitulate for a[color=green]
>> Python bug, if that's what this is.[/color]
>
> Try locale.setlocal e(locale.LC_CTY PE,"") as in your C program.[/color]

Oops, you are right. locale.setlocal e(locale.LC_CTY PE,"") sets the locale
from my environment (and gets it right!) while
locale.setlocal e(locale.LC_CTY PE) /returns/ the current locale. I don't know
how I could have missed that, since it's clearly documented and also maps
directly to C usage.
[color=blue]
> It would
> be great if locale.setlocal e with one parameter would be deprecated,
> because it suddenly acts like getlocale. It's unpythonic.[/color]

I dislike the term "unpythonic ", but I tend to agree with you in practice
here. Even better, but maybe not feasible, would be an approach to locales
which doesn't involve changing a global state in this fashion.
[color=blue]
> By the way, since you took time to setup various LC_* variables there
> is no need to play with LC_CTYPE category. Just use the standard idiom.
> import locale
> locale.setlocal e(LC_ALL,"")[/color]

Thanks for pointing that out. I picked out LC_CTYPE for my small program
because I was in a hurry and didn't want to risk non-standard sorting
elsewhere in the program. I hate what the LC_COLLATE=C does to swedish
national characters, but I hate what LC_COLLATE=sv_S E does to non-alphabetic
characters even more.

To paraphrase Barbie: "i18n is hard". ;-)

/Jorgen

--
// Jorgen Grahn <jgrahn@ Ph'nglui mglw'nafh Cthulhu
\X/ algonet.se> R'lyeh wgah'nagl fhtagn!