pep 277, Unicode filenames & mbcs encoding &c.

[Edward K. Ream]

Am I reading pep 277 correctly? On Windows NT/XP, should filenames always
be converted to Unicode using the mbcs encoding? For example,

myFile = unicode(__file_ _, "mbcs", "strict")

This seems to work, and I'm wondering whether there are any other details to
consider.

My experiments with Idle for Python 2.2 indicate that os.path.join doesn't
work as I expect when one of the args is a Unicode string. Everything
before the Unicode string gets thrown away. But this is probably moot: pep
277 implies Python 2.3...

Am I correct that conversions to Unicode (using "mbcs" on Windows) should be
done before passing arguments to os.path.join, os.path.split,
os.path.normpat h, etc. ? Presumably os.path functions use the default
system encoding to convert strings to Unicode, which isn't likely to be
"mbcs" or anything else useful :-)

Are there any situations where some other encoding should be used instead on
Windows? What about other platforms? For instance, does Linux allow
non-ascii file names? If so, what encoding should be specified when
converting to Unicode? Thanks.

Edward
--------------------------------------------------------------------
Edward K. Ream email: edreamleo@chart er.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------

[vincent wehren]

Re: pep 277, Unicode filenames & mbcs encoding &c.

"Edward K. Ream" <edreamleo@char ter.net> schrieb im Newsbeitrag
news:vpagr71f8f udc7@corp.super news.com...
| Am I reading pep 277 correctly? On Windows NT/XP, should filenames always
| be converted to Unicode using the mbcs encoding? For example,
|
| myFile = unicode(__file_ _, "mbcs", "strict")

No and no. You can *still* use regular byte strings. Python will do the
conversion to Unicode for you using "mbcs" as encoding.

|
| This seems to work, and I'm wondering whether there are any other details
to
| consider.
|
| My experiments with Idle for Python 2.2 indicate that os.path.join doesn't
| work as I expect when one of the args is a Unicode string. Everything
| before the Unicode string gets thrown away. But this is probably moot:
pep
| 277 implies Python 2.3...

Exactly. Python Unicode file name support has arrived with 2.3.

|
....
|
| Are there any situations where some other encoding should be used instead
on
| Windows? What about other platforms? For instance, does Linux allow
| non-ascii file names?

You can use "os.path.suppor ts_unicode_file names" to check...


HTH

Vincent Wehren

If so, what encoding should be specified when
| converting to Unicode? Thanks.
Propably the default encoding, on Linux

|
| Edward
| --------------------------------------------------------------------
| Edward K. Ream email: edreamleo@chart er.net
| Leo: Literate Editor with Outlines
| Leo: http://webpages.charter.net/edreamleo/front.html
| --------------------------------------------------------------------
|
|

[Just]

Re: pep 277, Unicode filenames &amp; mbcs encoding &amp;c.

In article <bn3jfo$df$1@ne ws4.tilbu1.nb.h ome.nl>,
"vincent wehren" <vincent@visual trans.de> wrote:
[color=blue]
> | Are there any situations where some other encoding should be used instead
> on
> | Windows? What about other platforms? For instance, does Linux allow
> | non-ascii file names?
>
> You can use "os.path.suppor ts_unicode_file names" to check...[/color]

Actually, you can't, see:

http://python.org/sf/767645

The only two platforms that currently support unicode filenames properly
are Windows NT/XP and MacOSX, and for one of them
os.path.support s_unicode_filen ames returns False :(

Just

[Martin v. Löwis]

Re: pep 277, Unicode filenames &amp; mbcs encoding &amp;c.

"Edward K. Ream" <edreamleo@char ter.net> writes:
[color=blue]
> Am I reading pep 277 correctly? On Windows NT/XP, should filenames always
> be converted to Unicode using the mbcs encoding?[/color]

What do you mean with "should"? "Should Python always..." or "Should
the application always"?

PEP 277 actually answers neither question. As Vincent explains,
nothing changes with respect to using byte strings on the API. The
changes only affect Unicode strings passed to functions expecting file names.
[color=blue]
> For example,
>
> myFile = unicode(__file_ _, "mbcs", "strict")
>
> This seems to work[/color]

And it has nothing to do with PEP 277: You are not passing myFile to
any API function.

If you mean to use myFile as a file name, then yes: this is intended
to work. However, using plain __file__ directly should also work.
[color=blue]
> Am I correct that conversions to Unicode (using "mbcs" on Windows) should be
> done before passing arguments to os.path.join, os.path.split,
> os.path.normpat h, etc. ?[/color]

You should either use only Unicode strings, or only byte strings. The
functions of os.path are not all affected by the PEP 277
implementation (although they probably should).
[color=blue]
> Presumably os.path functions use the default
> system encoding to convert strings to Unicode, which isn't likely to be
> "mbcs" or anything else useful :-)[/color]

Right. This is actually unfortunate.
[color=blue]
> Are there any situations where some other encoding should be used instead on
> Windows?[/color]

If you get data from a cmd.exe Window.
[color=blue]
> What about other platforms? For instance, does Linux allow non-ascii
> file names?[/color]

Yes, it does.
[color=blue]
> If so, what encoding should be specified when converting to Unicode?[/color]

Nobody knows, but the convention is to use the locale's encoding, as
returned by locale.getprefe rredencoding().

Regards,
Martin

[Edward K. Ream]

Re: pep 277, Unicode filenames &amp; mbcs encoding &amp;c.

Many thanks, Martin, for these comments. They are so helpful...
[color=blue]
> You should either use only Unicode strings, or only byte strings. The
> functions of os.path are not all affected by the PEP 277
> implementation (although they probably should).[/color]

My working assumption is that all strings in my app must be Unicode strings.
For example, the crashes happening right now trying to support Unicode
filenames occur when a string is converted to Unicode in situations like:

if fileName1 == fileName2:

where one fileName is a unicode string and the other isn't yet. That's why
I wanted to do:

myFile = unicode(__file_ _, "mbcs", "strict")

The challenge in my app is to make sure the proper encoding is used in the
more than 30 situations where a filename gets created somehow. Naturally,
that's not your problem, nor PEP 277's problem either :-)
[color=blue][color=green]
> > If so, what encoding should be specified when converting to Unicode?[/color]
>
> Nobody knows, but the convention is to use the locale's encoding, as
> returned by locale.getprefe rredencoding().[/color]

Thanks for this.

Edward
--------------------------------------------------------------------
Edward K. Ream email: edreamleo@chart er.net
Leo: Literate Editor with Outlines
Leo: http://webpages.charter.net/edreamleo/front.html
--------------------------------------------------------------------

[Martin v. Löwis]

Re: pep 277, Unicode filenames &amp; mbcs encoding &amp;c.

"Edward K. Ream" <edreamleo@char ter.net> writes:
[color=blue]
> if fileName1 == fileName2:
>
> where one fileName is a unicode string and the other isn't yet. That's why
> I wanted to do:
>
> myFile = unicode(__file_ _, "mbcs", "strict")[/color]

Ah, I see. Instead of "mbcs", you should use
sys.getfilesyst emencoding(). This is what Python will use when
converting the Unicode strings back to byte strings before passing
them to the system (in case it converts back at all, which it doesn't
on Windows thanks to PEP 277).

Regards,
Martin