Python nuube needs Unicode help

[gheissenberger@gmail.com]

HELP!
Guy who was here before me wrote a script to parse files in Python.

Includes line:
print u
where u is a line from a file we are parsing.
However, we have started recieving data from Brazil. If I open file to
parse in VI, looks like:

<Utt id="3" transcribe="yes " audioRoot="A1"
audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
recValue="{data :CHOICE=NO;}" conf="970" rawText2="" conf2="0"
transcribedText ="n&#227;o" parsableText="n &#227;o"/

Clearly those "n&#227" are some non-Ascii characters, but how do I get
print to understand that?

I keep getting:
"UnicodeEncodeE rror: 'ascii' codec can't encode character u'\xe3' in
position 40:
ordinal not in range(128)"

[Diez B. Roggisch]

Re: Python nuube needs Unicode help

gheissenberger@ gmail.com schrieb:

HELP!
Guy who was here before me wrote a script to parse files in Python.
>
Includes line:
print u
where u is a line from a file we are parsing.
However, we have started recieving data from Brazil. If I open file to
parse in VI, looks like:
>
<Utt id="3" transcribe="yes " audioRoot="A1"
audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
recValue="{data :CHOICE=NO;}" conf="970" rawText2="" conf2="0"
transcribedText ="n&#227;o" parsableText="n &#227;o"/
>
Clearly those "n&#227" are some non-Ascii characters, but how do I get
print to understand that?
>
I keep getting:
"UnicodeEncodeE rror: 'ascii' codec can't encode character u'\xe3' in
position 40:
ordinal not in range(128)"
>

Does the error happen at the

print u

line? If yes, what happens is that you try and print a unicode object.
Which means that it has to be converted (actually the right term is
encoded) to a byte-string. If you don't do that explicitely, it will be
done implicitly, using the default encoding - which is ascii.

If you have non-ascii characters, you end up with the error you see.

What to do? Use something like this:

print u.encode('utf-8')

instead.

Diez

[gheissenberger@gmail.com]

Re: Python nuube needs Unicode help

Progress! You managed to change the error message.

File "./acc_test_script _generator.py", line 106, in loadData
print u.encode('utf-8')
AttributeError: Utterance instance has no attribute 'encode'

I'm missing somethign really obvious here, but I don't know what it
is...


Diez B. Roggisch wrote:

gheissenberger@ gmail.com schrieb:

HELP!
Guy who was here before me wrote a script to parse files in Python.

Includes line:
print u
where u is a line from a file we are parsing.
However, we have started recieving data from Brazil. If I open file to
parse in VI, looks like:

<Utt id="3" transcribe="yes " audioRoot="A1"
audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
recValue="{data :CHOICE=NO;}" conf="970" rawText2="" conf2="0"
transcribedText ="n&#227;o" parsableText="n &#227;o"/

Clearly those "n&#227" are some non-Ascii characters, but how do I get
print to understand that?

I keep getting:
"UnicodeEncodeE rror: 'ascii' codec can't encode character u'\xe3' in
position 40:
ordinal not in range(128)"

>
Does the error happen at the
>
print u
>
line? If yes, what happens is that you try and print a unicode object.
Which means that it has to be converted (actually the right term is
encoded) to a byte-string. If you don't do that explicitely, it will be
done implicitly, using the default encoding - which is ascii.
>
If you have non-ascii characters, you end up with the error you see.
>
What to do? Use something like this:
>
print u.encode('utf-8')
>
instead.
>
Diez

[Gabriel Genellina]

Re: Python nuube needs Unicode help

At Thursday 11/1/2007 18:27, gheissenberger@ gmail.com wrote:

>HELP!
>Guy who was here before me wrote a script to parse files in Python.
>
>Includes line:
>print u
>where u is a line from a file we are parsing.
>However, we have started recieving data from Brazil. If I open file to
>parse in VI, looks like:
>
><Utt id="3" transcribe="yes " audioRoot="A1"
>audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
>recValue="{dat a:CHOICE=NO;}" conf="970" rawText2="" conf2="0"
>transcribedTex t="n&#227;o" parsableText="n &#227;o"/

Is this part of an XML document? You should use a
true XML parser instead of doing that by hand.

>Clearly those "n&#227" are some non-Ascii characters, but how do I get
>print to understand that?

Understanding how Unicode works may be very
useful: http://www.amk.ca/python/howto/unicode

>I keep getting:
>"UnicodeEncode Error: 'ascii' codec can't encode character u'\xe3' in
>position 40:
ordinal not in range(128)"

pyu = u"áéíóú"
pyprint u, repr(u)
áéíóú u'\xe1\xe9\xed\ xf3\xfa'
pyprint str(u)
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeEncodeEr ror: 'ascii' codec can't encode
characters in position 0-4: ordin
al not in range(128)
pyprint u.encode('cp850 ')
áéíóú

(cp850 is my console encoding)


--
Gabriel Genellina
Softlab SRL






_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!

http://www.yahoo.com.ar/respuestas

[Gabriel Genellina]

Re: Python nuube needs Unicode help

At Thursday 11/1/2007 20:42, gheissenberger@ gmail.com wrote:

Progress! You managed to change the error message.
>
File "./acc_test_script _generator.py", line 106, in loadData
print u.encode('utf-8')
>AttributeError : Utterance instance has no attribute 'encode'
>
>I'm missing somethign really obvious here, but I don't know what it
>is...

Then you're not "printing a line from a file we are parsing", which
should be a string or unicode object. You're printing some
"Utterance" instance; probably it has a __str__ method, and there,
you're mixing unicode+strings .


--
Gabriel Genellina
Softlab SRL






_______________ _______________ _______________ _____
Preguntá. Respondé. Descubrí.
Todo lo que querías saber, y lo que ni imaginabas,
está en Yahoo! Respuestas (Beta).
¡Probalo ya!

http://www.yahoo.com.ar/respuestas