Python nuube needs Unicode help

[gheissenberger@gmail.com]

HELP!
Guy who was here before me wrote a script to parse files in Python.

Includes line:
print u
where u is a line from a file we are parsing.
However, we have started recieving data from Brazil. If I open file to
parse in VI, looks like:

<Utt id="3" transcribe="yes " audioRoot="A1"
audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
recValue="{data :CHOICE=NO;}" conf="970" rawText2="" conf2="0"
transcribedText ="n&#227;o" parsableText="n &#227;o"/

Clearly those "n&#227" are some non-Ascii characters, but how do I get
print to understand that?

I keep getting:
"UnicodeEncodeE rror: 'ascii' codec can't encode character u'\xe3' in
position 40:
ordinal not in range(128)"

[Chris Mellon]

Re: Python nuube needs Unicode help

On 11 Jan 2007 13:28:14 -0800, gheissenberger@ gmail.com
<gheissenberger @gmail.comwrote :

HELP!
Guy who was here before me wrote a script to parse files in Python.
>
Includes line:
print u
where u is a line from a file we are parsing.
However, we have started recieving data from Brazil. If I open file to
parse in VI, looks like:
>
<Utt id="3" transcribe="yes " audioRoot="A1"
audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
recValue="{data :CHOICE=NO;}" conf="970" rawText2="" conf2="0"
transcribedText ="n&#227;o" parsableText="n &#227;o"/
>
Clearly those "n&#227" are some non-Ascii characters, but how do I get
print to understand that?
>
I keep getting:
"UnicodeEncodeE rror: 'ascii' codec can't encode character u'\xe3' in
position 40:
ordinal not in range(128)"
>

Find out what encoding the files are in and modify the script to use it.

[gheissenberger@gmail.com]

Re: Python nuube needs Unicode help

Yeah, that's what i don't know how to do.

Chris Mellon wrote:

On 11 Jan 2007 13:28:14 -0800, gheissenberger@ gmail.com
<gheissenberger @gmail.comwrote :

HELP!
Guy who was here before me wrote a script to parse files in Python.

Includes line:
print u
where u is a line from a file we are parsing.
However, we have started recieving data from Brazil. If I open file to
parse in VI, looks like:

<Utt id="3" transcribe="yes " audioRoot="A1"
audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
recValue="{data :CHOICE=NO;}" conf="970" rawText2="" conf2="0"
transcribedText ="n&#227;o" parsableText="n &#227;o"/

Clearly those "n&#227" are some non-Ascii characters, but how do I get
print to understand that?

I keep getting:
"UnicodeEncodeE rror: 'ascii' codec can't encode character u'\xe3' in
position 40:
ordinal not in range(128)"

>
Find out what encoding the files are in and modify the script to use it.

[Marc 'BlackJack' Rintsch]

Re: Python nuube needs Unicode help

In <mailman.2600.1 168552888.32031 .python-list@python.org >, Chris Mellon
wrote:

On 11 Jan 2007 13:28:14 -0800, gheissenberger@ gmail.com
<gheissenberger @gmail.comwrote :
>

><Utt id="3" transcribe="yes " audioRoot="A1"
>audio="313-20070102144528. wav" grammarSet="G3" rawText="n&#227 ;o"
>recValue="{dat a:CHOICE=NO;}" conf="970" rawText2="" conf2="0"
>transcribedTex t="n&#227;o" parsableText="n &#227;o"/
>>
>Clearly those "n&#227" are some non-Ascii characters, but how do I get
>print to understand that?
>>
>I keep getting:
>"UnicodeEncode Error: 'ascii' codec can't encode character u'\xe3' in
>position 40:
> ordinal not in range(128)"
>>

>
Find out what encoding the files are in and modify the script to use it.

The problem is not the encoding of the files as you see they are decoded
to unicode strings by the XML reading part already.

Ciao,
Marc 'BlackJack' Rintsch

[Peter Otten]

Re: Python nuube needs Unicode help

gheissenberger@ gmail.com wrote:

HELP!
Guy who was here before me wrote a script to parse files in Python.
>
Includes line:
print u

According to your other posts 'u' seems to be an instance of a custom
Utterance class with a __str__() method that accidentally returns unicode.
Try changing the print statement to

print unicode(u)

If you're lucky, it works. Otherwise we need a piece of the actual code. To
give you an idea what a self-contained demonstration of your problem might
look like:

>>class Utterance(objec t):

.... def __str__(self): return u"äöü"
....

>>u = Utterance()
>>print u

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position 0-2:
ordinal not in range(128)

>>print unicode(u)

äöü

Peter

[gheissenberger@gmail.com]

Re: Python nuube needs Unicode help

Can you attach files in this forum? Couldn't find the option. Oh well,
here's the file.

#!/usr/bin/python
# Version: 1.1
# Author: Steve Losh

from sets import Set
from optparse import OptionParser
from xml.dom.minidom import parse

AudioPath = 'audio/'
DatafilePath = 'utterances.trm xml'
CONFIDENCE_LOW = None #'500'
CONFIDENCE_HIGH = None #'500'

utterancesFile = None


class Utterance:
def __init__(self, audio, grammarSet, text):
self.audio = audio
self.grammarSet = grammarSet
self.text = text

def __str__(self):
return "SWIrecAcoustic StateReset\ntra nscription " + self.text \
+ "\nrecogniz e " + AudioPath + self.audio


def getGrammarPaths ():
"""Get the paths of all the grammars needed. Returns a Set
containing the results.
If a grammar is listed more than once in the transcription manifest it
will only appear once in these results.

TODO:
Find a less fragile way to split off the server half of the URIs."""
grammarTags = utterancesFile. getElementsByTa gName('Grammar' )
grammarURIs = [tag.getAttribut e('uri') for tag in grammarTags]
grammarPaths = [uri.split('serv let/CA/')[1] for uri in grammarURIs]
return Set(grammarPath s)


def createGrammarNa meFromPath(path ):
"""Convert a given path into an appropriate name for the
grammar."""
path = path.replace('/', '-') # Strip the directory slashes
path = path.replace('. ', '_') # and the dot before the extension.
return path


def loadGrammars():
"""Output the statements that will load the required grammars."""
grammarPaths = list(getGrammar Paths())
grammarsToLoad = {}
for path in grammarPaths:
grammarName = createGrammarNa meFromPath(path )
grammarsToLoad[grammarName] = "grammars/" + path
for grammarName in grammarsToLoad:
print "SWIrecGrammarL oad", grammarName,
grammarsToLoad[grammarName]


def loadGrammarSets ():
"""Output the statements that will define the grammar
sets/contexts. Returns a list of the grammar set names."""
grammarSetList =
utterancesFile. getElementsByTa gName('GrammarS ets')
grammarSets =
grammarSetList[0].getElementsByT agName('Grammar Set')
grammarSetNames = []
for gs in grammarSets:
grammarSetName = gs.getAttribute ('id')
print "context_define ", grammarSetName, CONFIDENCE_LOW,
CONFIDENCE_HIGH
for g in gs.getElementsB yTagName('Gramm ar'):
path = g.getAttribute( 'uri').split('s ervlet/CA/')[1]
print "context_ad d", createGrammarNa meFromPath(path ),
'1000'
print "context_en d\n"
grammarSetNames .append(grammar SetName)
return grammarSetNames


def buildUtterances (call):
"""This function takes a call tag, builds the utterances belonging
to it and returns a list containing them."""
utts = call.getElement sByTagName('Utt ')
utterances = [Utterance( utt.getAttribut e('audio'), \
utt.getAttribut e('grammarSet') ,
utt.getAttribut e('transcribedT ext') ) \
for utt in utts]
return utterances


def getUtterances() :
"""Returns a list of all the utterances we want to test."""
callList = utterancesFile. getElementsByTa gName('Calls')[0]
calls = callList.getEle mentsByTagName( 'Call')
utterances = []
for c in calls:
utterances.exte nd(buildUtteran ces(c))
return utterances


def loadData(uttera nces):
"""Outputs the statements that will tell rec_test what to test."""
contexts = {}
for u in utterances:
if u.grammarSet not in contexts:
contexts[u.grammarSet] = []
contexts[u.grammarSet].append(u)
for c in contexts:
print "open errors " + c + ".errors"
print "open utd " + c + ".utd"
print "context_us e", c
for u in contexts[c]:
print u
print "close utd"
print "close errors"
print "\n"


def makeParser():
parser = OptionParser( "usage: %prog -l LOWER CONFIDENCE -h UPPER
CONFIDENCE [-f FILTER1 -f FILTER2 ...] file" )
parser.add_opti on("-l", "--low-confidence", dest="lower", \
help="The lower confidence level to test at.",
metavar="CONFID ENCE")
parser.add_opti on("-u", "--upper-confidence", dest="upper", \
help="The upper confidence level to test at.",
metavar="CONFID ENCE")
parser.add_opti on("-f", "--filter", dest="filter", action="append" ,
\
help="Only test utterances transcribed as WORD.",
metavar="WORD")
return parser


def main():
global utterancesFile, CONFIDENCE_HIGH , CONFIDENCE_LOW
parser = makeParser()
(options, args) = parser.parse_ar gs()
if len(args) != 1:
parser.error("O ne data file must be specified.")
elif options.lower == None:
parser.error("A lower confidence level must be specified.")
elif options.upper == None:
parser.error("A n upper confidence level must be specified.")

DatafilePath = args[0]
CONFIDENCE_LOW = options.lower
CONFIDENCE_HIGH = options.upper

utterancesFile = parse(DatafileP ath)

print ':ACC\n\n'
loadGrammars()
print "\n\n"

grammarSetNames = loadGrammarSets ()

utterances = getUtterances()
if options.filter != None:
utterances = [u for u in utterances if u.text in
options.filter]

print "\n\n"
loadData(uttera nces)

print "report summary summary.txt"
print "report oov oov.txt"
print "report words words.txt"


if __name__ == '__main__':
main()




Peter Otten wrote:

gheissenberger@ gmail.com wrote:
>

HELP!
Guy who was here before me wrote a script to parse files in Python.

Includes line:
print u

>
According to your other posts 'u' seems to be an instance of a custom
Utterance class with a __str__() method that accidentally returns unicode.
Try changing the print statement to
>
print unicode(u)
>
If you're lucky, it works. Otherwise we need a piece of the actual code. To
give you an idea what a self-contained demonstration of your problem might
look like:
>

>class Utterance(objec t):

... def __str__(self): return u"äöü"
...

>u = Utterance()
>print u

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeEr ror: 'ascii' codec can't encode characters in position 0-2:
ordinal not in range(128)

>print unicode(u)

äöü

Peter

[Diez B. Roggisch]

Re: Python nuube needs Unicode help

gheissenberger@ gmail.com wrote:

Can you attach files in this forum? Couldn't find the option. Oh well,
here's the file.
>
#!/usr/bin/python
# Version: 1.1
# Author: Steve Losh
>
from sets import Set
from optparse import OptionParser
from xml.dom.minidom import parse
>
AudioPath = 'audio/'
DatafilePath = 'utterances.trm xml'
CONFIDENCE_LOW = None #'500'
CONFIDENCE_HIGH = None #'500'
>
utterancesFile = None
>
>
class Utterance:
def __init__(self, audio, grammarSet, text):
self.audio = audio
self.grammarSet = grammarSet
self.text = text
>
def __str__(self):
return "SWIrecAcoustic StateReset\ntra nscription " + self.text \
+ "\nrecogniz e " + AudioPath + self.audio

There your __str__-method is. self.text and self.audio come from the
xml-parsing and are unicode objects - so they need to be encoded, like
this:

def __str__(self):
return "SWIrecAcoustic StateReset\ntra nscription " +
self.text.encod e('utf-8') + "\nrecogniz e " + AudioPath +
self.audio.enco de('utf-8')


Diez

[gheissenberger@gmail.com]

Re: Python nuube needs Unicode help

Diez, you are awesome! This works! Thank you so much!!! You are so
awesome!


Diez B. Roggisch wrote:

gheissenberger@ gmail.com wrote:
>

Can you attach files in this forum? Couldn't find the option. Oh well,
here's the file.

#!/usr/bin/python
# Version: 1.1
# Author: Steve Losh

from sets import Set
from optparse import OptionParser
from xml.dom.minidom import parse

AudioPath = 'audio/'
DatafilePath = 'utterances.trm xml'
CONFIDENCE_LOW = None #'500'
CONFIDENCE_HIGH = None #'500'

utterancesFile = None


class Utterance:
def __init__(self, audio, grammarSet, text):
self.audio = audio
self.grammarSet = grammarSet
self.text = text

def __str__(self):
return "SWIrecAcoustic StateReset\ntra nscription " + self.text \
+ "\nrecogniz e " + AudioPath + self.audio

>
There your __str__-method is. self.text and self.audio come from the
xml-parsing and are unicode objects - so they need to be encoded, like
this:
>
def __str__(self):
return "SWIrecAcoustic StateReset\ntra nscription " +
self.text.encod e('utf-8') + "\nrecogniz e " + AudioPath +
self.audio.enco de('utf-8')
>
>
Diez