the tostring and XML methods in ElementTree

**Serge Orlov** · May 7 '06, 10:25 PM

Re: the tostring and XML methods in ElementTree

mirandacascade@ yahoo.com wrote:[color=blue]
> Question 1: assuming the following:
> a) beforeCtag.text gets assigned a value of 'I\x92m confused'
> b) afterRoot is built using the XML() method where the input to the
> XML() method is the results of a tostring() method from beforeRoot
> Are there any settings/arguments that could have been modified that
> would have resulted in afterCtag.text being of type <type 'str'> and
> afterCtag.text when printed displays:
> I'm confused
>
> ?[/color]

str type (also known as byte string) is only suitable for ascii text.
chr(0x92) is outside of ascii so you should use unicode strings or
you\x92ll be confused :)
[color=blue][color=green][color=darkred]
>>> print u"I\u2019m not confused"[/color][/color][/color]
I'm not confused

[color=blue]
> Question 2: Does the fact that resultToStr is equal to resultToStr2
> mean that an encoding of utf-8 is the defacto default when no encoding
> is passed as an argument to the tostring method, or does it only mean
> that in this particular example, they happened to be the same?[/color]

No. Dejure default encoding is ascii, defacto people try to change it,
but it's not a good idea. I'm not sure how you got the strings to be
the same, but it's definately host-specific result, when I repeat your
interactive session I get different resultToStr at this point:
[color=blue][color=green][color=darkred]
>>> afterRoot = ElementTree.XML (resultToStr)
>>> resultToStr[/color][/color][/color]
'<beforeRoot><C >I’m confused</C></beforeRoot>'

[color=blue]
> 3) would it be possible to construct a statement of the form
>
> newResult = afterCtag.text. encode(?? some argument ??)
>
> where newResult was the same as beforeCtag.text ? If so, what should
> the argument be to the encode method?[/color]

Dealing with unicode doesn't require you to pollute your code with
encode methods, just open the file using codecs module and then write
unicode strings directly:

import codecs
fileHandle = codecs.open('c:/output1.text', 'w',"utf-8")
fileHandle.writ e(u"I\u2019m not confused, because I'm using unicode")
[color=blue]
> 4) what is the second character in encodedCtagtext (the character with
> an ordinal value of 194)?[/color]

That is byte with value 194, it's not a character. It is part of
unicode code point U+0092 when it is encoded in utf-8
[color=blue][color=green][color=darkred]
>>> '\xc2\x92'.deco de("utf-8")[/color][/color][/color]
u'\x92'

This code point actually has no name, so you shouldn't produce it:
[color=blue][color=green][color=darkred]
>>> import unicodedata
>>> unicodedata.nam e('\xc2\x92'.de code("utf-8"))[/color][/color][/color]

Traceback (most recent call last):
File "<pyshell#4 0>", line 1, in -toplevel-
unicodedata.nam e('\xc2\x92'.de code("utf-8"))
ValueError: no such name

**Serge Orlov** · May 7 '06, 10:55 PM

Re: the tostring and XML methods in ElementTree

mirandacascade@ yahoo.com wrote:[color=blue]
> O/S: Windows XP Home
> Vsn of Python: 2.4[/color]

[snip fighting with unicode character U+2019 (RIGHT SINGLE QUOTATION
MARK) ]

I don't know what console you use but if it is IDLE you'll get confused
even more because it is buggy and improperly handles that character:
[color=blue][color=green][color=darkred]
>>> print repr(u'I'm confused')[/color][/color][/color]
u'I\x92m confused'

I'm using Lightning Compiler
<http://cheeseshop.pyth on.org/pypi/Lightning%20Com piler> to run
snippets of code and in the editor tab it handles that character fine:
[color=blue][color=green][color=darkred]
>>> print repr(u'I'm confused')[/color][/color][/color]
u'I\u2019m confused'

But in the console tab it produces the same buggy result :) It looks
like handling unicode is like rocket science :)

**Fredrik Lundh** · May 8 '06, 08:55 AM

Re: the tostring and XML methods in ElementTree

mirandacascade@ yahoo.com wrote:
[color=blue]
> I wanted to see what would happen if one used the results of a tostring
> method as input into the XML method. What I observed is this:
> a) beforeCtag.text is of type <type 'str'>
> b) beforeCtag.text when printed displays: I'm confused
> c) afterCtag.text is of type <type 'unicode'>
> d) afterCtag.text when printed displays: I?m confused[/color]

the XML file format isn't a Python string serialization format, it's an XML infoset
serialization format.

as stated in the documentation, ET always uses Unicode strings for text that
contain non-ASCII characters. for text that *only* contains ASCII, it may use
either Unicode strings or 8-bit strings, depending on the implementation.

the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
(which means that you shouldn't do that; it's not portable).

to learn more about Unicode in Python, google for "python unicode".

</F>

**George Sakkis** · May 17 '06, 08:15 AM

Re: the tostring and XML methods in ElementTree

Fredrik Lundh wrote:
[color=blue]
> mirandacascade@ yahoo.com wrote:
>[color=green]
> > I wanted to see what would happen if one used the results of a tostring
> > method as input into the XML method. What I observed is this:
> > a) beforeCtag.text is of type <type 'str'>
> > b) beforeCtag.text when printed displays: I'm confused
> > c) afterCtag.text is of type <type 'unicode'>
> > d) afterCtag.text when printed displays: I?m confused[/color]
>
> the XML file format isn't a Python string serialization format, it's an XML infoset
> serialization format.
>
> as stated in the documentation, ET always uses Unicode strings for text that
> contain non-ASCII characters. for text that *only* contains ASCII, it may use
> either Unicode strings or 8-bit strings, depending on the implementation.
>
> the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
> (which means that you shouldn't do that; it's not portable).[/color]

I was about to post a similar question when I found this thread.
Fredrik, can you explain why this is not portable ? I'm currently using
(a variation of) the workaround below instead of ET.tostring and it
works fine for me:

def tostring(elemen t, encoding=None):
text = element.text
if text:
if not isinstance(text , basestring):
text2 = str(text)
elif isinstance(text , str) and encoding:
text2 = text.decode(enc oding)
element.text = text2
s = ET.tostring(ele ment, encoding)
element.text = text
return s

Why isn't this the standard behaviour ?

Thanks,
George

**Stefan Behnel** · May 19 '06, 02:25 PM

Re: the tostring and XML methods in ElementTree

George Sakkis wrote:[color=blue]
> Fredrik Lundh wrote:
>[color=green]
>> mirandacascade@ yahoo.com wrote:
>>[color=darkred]
>>> I wanted to see what would happen if one used the results of a tostring
>>> method as input into the XML method. What I observed is this:
>>> a) beforeCtag.text is of type <type 'str'>
>>> b) beforeCtag.text when printed displays: I'm confused
>>> c) afterCtag.text is of type <type 'unicode'>
>>> d) afterCtag.text when printed displays: I?m confused[/color]
>> the XML file format isn't a Python string serialization format, it's an XML infoset
>> serialization format.
>>
>> as stated in the documentation, ET always uses Unicode strings for text that
>> contain non-ASCII characters. for text that *only* contains ASCII, it may use
>> either Unicode strings or 8-bit strings, depending on the implementation.
>>
>> the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
>> (which means that you shouldn't do that; it's not portable).[/color]
>
> I was about to post a similar question when I found this thread.
> Fredrik, can you explain why this is not portable ?[/color]

Because there is no such things as a default encoding for 8-bit strings.

[color=blue]
> I'm currently using
> (a variation of) the workaround below instead of ET.tostring and it
> works fine for me:
>
> def tostring(elemen t, encoding=None):
> text = element.text
> if text:
> if not isinstance(text , basestring):
> text2 = str(text)
> elif isinstance(text , str) and encoding:
> text2 = text.decode(enc oding)
> element.text = text2
> s = ET.tostring(ele ment, encoding)
> element.text = text
> return s
>
>
> Why isn't this the standard behaviour ?[/color]

Because it wouldn't work. What if you wanted to serialize a different encoding
than that of the strings you put into the .text fields? How is ET supposed to
know what encoding your strings have? And how should it know that you didn't
happily mix various different byte encodings in your strings?

Use unicode, that works *and* is portable.

Stefan

**George Sakkis** · May 19 '06, 03:35 PM

Re: the tostring and XML methods in ElementTree

Stefan Behnel wrote:
[color=blue]
> George Sakkis wrote:[color=green]
> > Fredrik Lundh wrote:
> >[color=darkred]
> >> mirandacascade@ yahoo.com wrote:
> >>
> >>> I wanted to see what would happen if one used the results of a tostring
> >>> method as input into the XML method. What I observed is this:
> >>> a) beforeCtag.text is of type <type 'str'>
> >>> b) beforeCtag.text when printed displays: I'm confused
> >>> c) afterCtag.text is of type <type 'unicode'>
> >>> d) afterCtag.text when printed displays: I?m confused
> >> the XML file format isn't a Python string serialization format, it's an XML infoset
> >> serialization format.
> >>
> >> as stated in the documentation, ET always uses Unicode strings for text that
> >> contain non-ASCII characters. for text that *only* contains ASCII, it may use
> >> either Unicode strings or 8-bit strings, depending on the implementation.
> >>
> >> the behaviour if you're passing in non-ASCII text as 8-bit strings is undefined
> >> (which means that you shouldn't do that; it's not portable).[/color]
> >
> > I was about to post a similar question when I found this thread.
> > Fredrik, can you explain why this is not portable ?[/color]
>
> Because there is no such things as a default encoding for 8-bit strings.
>
>[color=green]
> > I'm currently using
> > (a variation of) the workaround below instead of ET.tostring and it
> > works fine for me:
> >
> > def tostring(elemen t, encoding=None):
> > text = element.text
> > if text:
> > if not isinstance(text , basestring):
> > text2 = str(text)
> > elif isinstance(text , str) and encoding:
> > text2 = text.decode(enc oding)
> > element.text = text2
> > s = ET.tostring(ele ment, encoding)
> > element.text = text
> > return s
> >
> >
> > Why isn't this the standard behaviour ?[/color]
>
>
> Because it wouldn't work. What if you wanted to serialize a different encoding
> than that of the strings you put into the .text fields? How is ET supposed to
> know what encoding your strings have? And how should it know that you didn't
> happily mix various different byte encodings in your strings?[/color]

If you're mixing different encodings, no tool can help you clean up the
mess, you're on your own. This is very different though from having
nice utf-8 strings everywhere, asking ET.tostring explicitly to print
them in utf-8 and getting back garbage. Isn't the most reasonable
assumption that the input's encoding is the same with the output, or
does this fall under the "refuse the temptation to guess" motto ? If
this is the case, ET could at least accept an optional input encoding
parameter and convert everything to unicode internally.
[color=blue]
> Use unicode, that works *and* is portable.[/color]

*and* it's not supported by all the 3rd party packages, databases,
middleware, etc. you have to or want to use.
[color=blue]
> Stefan[/color]

George

**Serge Orlov** · May 19 '06, 10:25 PM

Re: the tostring and XML methods in ElementTree

George Sakkis wrote:[color=blue][color=green][color=darkred]
> > > I'm currently using
> > > (a variation of) the workaround below instead of ET.tostring and it
> > > works fine for me:
> > >
> > > def tostring(elemen t, encoding=None):
> > > text = element.text
> > > if text:
> > > if not isinstance(text , basestring):
> > > text2 = str(text)
> > > elif isinstance(text , str) and encoding:
> > > text2 = text.decode(enc oding)
> > > element.text = text2
> > > s = ET.tostring(ele ment, encoding)
> > > element.text = text
> > > return s
> > >
> > >
> > > Why isn't this the standard behaviour ?[/color]
> >
> >
> > Because it wouldn't work. What if you wanted to serialize a different encoding
> > than that of the strings you put into the .text fields? How is ET supposed to
> > know what encoding your strings have? And how should it know that you didn't
> > happily mix various different byte encodings in your strings?[/color]
>
> If you're mixing different encodings, no tool can help you clean up the
> mess, you're on your own. This is very different though from having
> nice utf-8 strings everywhere, asking ET.tostring explicitly to print
> them in utf-8 and getting back garbage. Isn't the most reasonable
> assumption that the input's encoding is the same with the output, or
> does this fall under the "refuse the temptation to guess" motto ? If
> this is the case, ET could at least accept an optional input encoding
> parameter and convert everything to unicode internally.[/color]

This is an optimization. Basically you're delaying decoding. First of
all have you measured the impact on your program if you delay decoding?
I'm sure for many programs it doesn't matter, so what you're proposing
will just pollute their source code with optimization they don't need.
That doesn't mean it's a bad idea in general. I'd prefer it implemented
in python core with minimal impact on such programs, decoding delayed
until you try to access individual characters. The code below can be
implemented without actual decoding:

utf8_text_file. write("abc".dec ode("utf-8") + " def".decode("ut f-8"))

But this example will require decoding done during split method:

a = ("abc".decode(" utf-8") + " def".decode("ut f-8")).split()

[color=blue][color=green]
> > Use unicode, that works *and* is portable.[/color]
>
> *and* it's not supported by all the 3rd party packages, databases,
> middleware, etc. you have to or want to use.[/color]

You can always call .encode method. Granted that could be a waste of
CPU and memory, but it works.

the tostring and XML methods in ElementTree

the tostring and XML methods in ElementTree

Comment

Comment

Comment

Comment

Comment

Comment

Comment