Problem with cPickle and cElementTree

[Barry]

I recently tried switching from ElementTree to cElementTree. My
application parses a collection of large XML files and creates indexes
based on certain attributes. This entire collection is saved as an
instance of my Database class. Using ElementTree and cPickle has
allowed me to save these instances and use them later.

Using cElementTree significantly reduces parse time (~50%) and memory
~(15%) but cPickle refuses to pickle the database object. I receive:

TypeError: expecting string or Unicode object, NoneType found

The offending line of code simple shows my invocation of cPickle,
which is not helpful.

Doing exactly the same thing with ElementTree works fine.

It appears that the objects returned by cElementTree do not pickle
correctly. Is this a know issue? I was unable to find any reports of
this problem.

Any info would be appreciated!

Barry

[Stefan Behnel]

Re: Problem with cPickle and cElementTree

Barry wrote:

I recently tried switching from ElementTree to cElementTree. My
application parses a collection of large XML files and creates indexes
based on certain attributes. This entire collection is saved as an
instance of my Database class. Using ElementTree and cPickle has
allowed me to save these instances and use them later.
>
Using cElementTree significantly reduces parse time (~50%) and memory
~(15%) but cPickle refuses to pickle the database object. I receive:
>
TypeError: expecting string or Unicode object, NoneType found
>
The offending line of code simple shows my invocation of cPickle,
which is not helpful.
>
Doing exactly the same thing with ElementTree works fine.
>
It appears that the objects returned by cElementTree do not pickle
correctly. Is this a know issue? I was unable to find any reports of
this problem.

Pickling is (almost always) built-in for pure Python classes, but it needs to
be implemented explicitly for C classes. cElementTree simply doesn't support this.

If you need a fast and memory friendly XML engine *and* want to pickle
elements, take a look at lxml.objectify. Note that it's only partially
compatible with ElementTree, so you will have to change your code (lxml.etree
is mostly compatible, but it doesn't support pickling). It's a very
easy-to-use XML library, though.

Stefan