unicode codecs

[Ivan Voras]

When concatenating strings (actually, a constant and a string...) i get
the following error:

UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xfc in position 1:
ordinal not in range(128)

Now I don't think either string is unicode, but I'm working with
win32api so it might be... :) The point is: I know all values will fit
in a particular code page (iso-8859-2), so how do I change the 'ascii'
codec in the above error into something that will work?

[Christopher Koppler]

Re: unicode codecs

On Mon, 09 Feb 2004 21:59:36 +0100, Ivan Voras
<ivoras@__geri. cc.fer.hr> wrote:
[color=blue]
>When concatenating strings (actually, a constant and a string...) i get
>the following error:
>
>UnicodeDecodeE rror: 'ascii' codec can't decode byte 0xfc in position 1:
>ordinal not in range(128)
>
>Now I don't think either string is unicode, but I'm working with
>win32api so it might be... :) The point is: I know all values will fit
>in a particular code page (iso-8859-2), so how do I change the 'ascii'
>codec in the above error into something that will work?[/color]

To get a real solution, you should also post the offending code, but
you might try to convert your values to unicode with the built-in
unicode() and the string method decode(). See the library reference
sections 2.1 and 2.2.6.

--
Christopher

[Ivan Voras]

Re: unicode codecs

Christopher Koppler wrote:
[color=blue]
> To get a real solution, you should also post the offending code, but
> you might try to convert your values to unicode with the built-in
> unicode() and the string method decode(). See the library reference
> sections 2.1 and 2.2.6.[/color]

I tried that, without luck. It is somewhat difficult to reproduce the
problem, but here's how I see it:

- win32api function returns a string (8bit) with some of the characters
from the upper half of code page, let's call it s1
- a statement such as a='x'+s1 fails with the above error.

I don't really know why should concatenation check if characters are
7-bit clean (or indeed if they represent anything in whatever code page).

Since win32api functions exist also in unicode version, I tried this:

- call the unicode version of function. Returned is a unicode string
(checked, it really is unicode) like u'R\xfcgenwald. txt', let's call it s2
- a statement a='x'+s2.encode ('iso-8859-2') also fails with the exact
same error.

It is strange that if I execute similar code in Idle (e.g. manually
assigning string constants to variables and concatenating), everything
works!

The exact error is:
File "E:\develop\pyn etdb\netdbcreat e.py", line 32, in walkdirs
fullname = root+'\\'+filen ame
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc3 in position 1:
ordinal not in range(128)

The filename variable contains (in my latest effort) utf-8 encoded value
'R\xc3\xbcgenwa ld.mp3', and root variable contains a normal non-unicode
string.

I tried various combinations of unicode and non-unicode types, and thay
all fail sooner or later when they meet with a non-unicode string that
is not 7-bit clean.

[Martin v. Löwis]

Re: unicode codecs

Ivan Voras wrote:[color=blue]
> When concatenating strings (actually, a constant and a string...) i get
> the following error:
>
> UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xfc in position 1:
> ordinal not in range(128)
>
> Now I don't think either string is unicode[/color]

This statement must be false. When concatenating two byte strings, no
codec is ever used. So, either
1. one of the strings is a Unicode objects, or
2. you are not performing concatenation, or you get the exception
from an operation that is not concatenation, or
3. you are not getting this exception.

Most likely, it is 1)
[color=blue]
> The point is: I know all values will fit
> in a particular code page (iso-8859-2), so how do I change the 'ascii'
> codec in the above error into something that will work?[/color]

Explicitly encode the Unicode string in your concatenation as
iso-8859-2.

Regards,
Martin

[Martin v. Löwis]

Re: unicode codecs

Ivan Voras wrote:[color=blue]
> - win32api function returns a string (8bit) with some of the characters
> from the upper half of code page, let's call it s1[/color]

Are you absolutely certain that type(s1) is str?
[color=blue]
> - a statement such as a='x'+s1 fails with the above error.[/color]

Are you absolutely certain the constant is the literal string 'x'?
[color=blue]
> I don't really know why should concatenation check if characters are
> 7-bit clean (or indeed if they represent anything in whatever code page).[/color]

As you have shown, there would be no need, and indeed, Python will not
check code pages in this case. So you must be doing something else.
[color=blue]
> - call the unicode version of function. Returned is a unicode string
> (checked, it really is unicode) like u'R\xfcgenwald. txt', let's call it s2
> - a statement a='x'+s2.encode ('iso-8859-2') also fails with the exact
> same error.[/color]

How do you know it is the concatenation that causes the exception?
[color=blue]
> The exact error is:
> File "E:\develop\pyn etdb\netdbcreat e.py", line 32, in walkdirs
> fullname = root+'\\'+filen ame
> UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xc3 in position 1:
> ordinal not in range(128)
>
> The filename variable contains (in my latest effort) utf-8 encoded value
> 'R\xc3\xbcgenwa ld.mp3', and root variable contains a normal non-unicode
> string.[/color]

Which string precisely (what is its repr())?

Regards,
Martin

[Peter Otten]

Re: unicode codecs

Ivan Voras wrote:
[color=blue]
> When concatenating strings (actually, a constant and a string...) i get
> the following error:
>
> UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xfc in position 1:
> ordinal not in range(128)
>
> Now I don't think either string is unicode, but I'm working with
> win32api so it might be... :) The point is: I know all values will fit
> in a particular code page (iso-8859-2), so how do I change the 'ascii'
> codec in the above error into something that will work?[/color]

You can either convert all strings to unicode or to iso-8859-2.
A hands on approach:
[color=blue][color=green][color=darkred]
>>> u,s[/color][/color][/color]
(u'R\xfcbe', 'R\xfcbe')[color=blue][color=green][color=darkred]
>>> u+s[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xfc in position 1:
ordinal not in range(128)

This error is prevented by an explicit conversion:
[color=blue][color=green][color=darkred]
>>> u.encode("iso-8859-1") + s[/color][/color][/color]
'R\xfcbeR\xfcbe '

or
[color=blue][color=green][color=darkred]
>>> u + s.decode("iso-8859-1")[/color][/color][/color]
u'R\xfcbeR\xfcb e'

If you aren't sure which string is unicode and which is not:
[color=blue][color=green][color=darkred]
>>> def toiso(s):[/color][/color][/color]
.... if isinstance(s, unicode):
.... return u.encode("iso-8859-1")
.... return s
....[color=blue][color=green][color=darkred]
>>> toiso(u) + toiso(s)[/color][/color][/color]
'R\xfcbeR\xfcbe '

Peter

[Peter Otten]

Re: unicode codecs

Peter Otten wrote:
[color=blue][color=green][color=darkred]
>>>> def toiso(s):[/color][/color]
> ... if isinstance(s, unicode):
> ... return u.encode("iso-8859-1")
> ... return s
> ...[color=green][color=darkred]
>>>> toiso(u) + toiso(s)[/color][/color]
> 'R\xfcbeR\xfcbe '[/color]

Oops, that should be:
[color=blue][color=green][color=darkred]
>>> def toiso(t):[/color][/color][/color]
.... if isinstance(t, unicode):
.... return t.encode("iso-8859-1")
.... return t
....[color=blue][color=green][color=darkred]
>>> toiso(u) + toiso(s)[/color][/color][/color]
'R\xfcbeR\xfcbe '

[Ivan Voras]

Re: unicode codecs

Martin v. Löwis wrote:[color=blue]
> Ivan Voras wrote:
>[color=green]
>> - win32api function returns a string (8bit) with some of the
>> characters from the upper half of code page, let's call it s1[/color]
>
>
> Are you absolutely certain that type(s1) is str?[/color]

Yes. Plain string.
[color=blue][color=green]
>> - a statement such as a='x'+s1 fails with the above error.[/color]
>
>
> Are you absolutely certain the constant is the literal string 'x'?[/color]

Um, what else could it be? This is an example, in the real case the
literal string is something else (but of the same format).

[color=blue][color=green]
>> - call the unicode version of function. Returned is a unicode string
>> (checked, it really is unicode) like u'R\xfcgenwald. txt', let's call
>> it s2
>> - a statement a='x'+s2.encode ('iso-8859-2') also fails with the exact
>> same error.[/color]
>
> How do you know it is the concatenation that causes the exception?[/color]

What else could cause it? It's a simple command, nothing fancy - an
concatenation and assignment.

I've tried converting everything to use unicode and I'm getting *really*
weird results now - it may be a bug in the win32api library.

[Ivan Voras]

Re: unicode codecs

Peter Otten wrote:
[color=blue]
> You can either convert all strings to unicode or to iso-8859-2.
> A hands on approach:
>
>[color=green][color=darkred]
>>>>u,s[/color][/color]
>
> (u'R\xfcbe', 'R\xfcbe')
>[color=green][color=darkred]
>>>>u+s[/color][/color]
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> UnicodeDecodeEr ror: 'ascii' codec can't decode byte 0xfc in position 1:
> ordinal not in range(128)
>
> This error is prevented by an explicit conversion:[/color]

Thank you - I eventually found that out the hard way :) It was a mix of
some bugs from my code and the win32api library code, and I was seeing
exeptions pop up from both of them depending on what conditions were
met. Eventually I seem to have found a workaround for the library bugs
but I don't like it - it's a mixup of using unicode and code-page and
converting around when necessary. The good thing is that it doesn't seem
to influence performance a lot...

(Apparently, win32file.FindF ilesW does something with its parameter that
breaks with above error when the parameter is unicode.)

Thanks for the help, all!