How to better pickle an extension type

[dgdev]

I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:

class C:
# for pickling
__getstate__(se lf):
... # make 'state_obj'
return state_obj

__reduce__(self ):
return C,(args,to,__in it__),me.__gets tate__()

# for unpickling
__setstate__(se lf,state_obj):
self.x=state_ob j.x
...


This gets the class pickling and unpickling.

However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.

I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.

Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

[Alex Martelli]

Re: How to better pickle an extension type

dgdev <dgdev3141@yaho o.comwrote:

I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:
>
class C:
# for pickling
__getstate__(se lf):
... # make 'state_obj'
return state_obj
>
__reduce__(self ):
return C,(args,to,__in it__),me.__gets tate__()
>
# for unpickling
__setstate__(se lf,state_obj):
self.x=state_ob j.x
...
>
>
This gets the class pickling and unpickling.
>
However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.
>
I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.
>
Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

I don't understand why you have a problem -- __init__ is NOT called by
default upon loading an object w/__setstate__. Witness:

>>class C(object):

.... def __init__(self, *a): print 'init', a
.... def __getstate__(se lf): print 'gs'; return {}
.... def __setstate__(se lf, *a): print 'ss', a
....

>>c = C()

init ()

>>s = cPickle.dumps(c , 2)

gs

>>s

'\x80\x02c__mai n__\nC\nq\x01)\ x81q\x02}b.'

>>z = cPickle.loads(s )

ss ({},)


Perhaps you're not using protocol 2? You should be...


Alex

[Ziga Seilnacht]

Re: How to better pickle an extension type

dgdev wrote:

I would like to pickle an extension type (written in pyrex). I have
it working thus far by defining three methods:
>
class C:
# for pickling
__getstate__(se lf):
... # make 'state_obj'
return state_obj
>
__reduce__(self ):
return C,(args,to,__in it__),me.__gets tate__()
>
# for unpickling
__setstate__(se lf,state_obj):
self.x=state_ob j.x
...
>
This gets the class pickling and unpickling.
>
However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.
>
I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.
>
Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

Why are you overwriting the __reduce__() method? The default
object.__reduce __() method, inherited by all new style classes,
already does what you want. If you really must overwrite it, and
you don't want __init__() to get called, then you should return a
reconstructor named __newobj__() as the first item of reduce
tuple. Something like this:

>>def __newobj__(cls, *args):

.... return cls.__new__(cls , *args)
....

>>class C(object):

.... def __init__(self):
.... print "I shouldn't be called at reconstruction"
.... def __reduce__(self ):
.... try:
.... getnewargs = self.__getnewar gs__
.... except AttributeError:
.... newargs = (self.__class__ ,)
.... else:
.... newargs = (self.__class__ ,) + getnewargs()
.... try:
.... getstate = self.__getstate __
.... except AttributeError:
.... # this ignores __slots__ complications
.... state = self.__dict__
.... else:
.... state = getstate()
.... # this ignores list and dict subclasses
.... return __newobj__, newargs, state
....

>>c = C()

I shouldn't be called at reconstruction

>>import pickle
>>for proto in range(3):

.... assert isinstance(pick le.loads(pickle .dumps(c, proto)), C)
....

>>>

Ziga

[dgdev]

Re: How to better pickle an extension type

Thanks for your replies.

The code I showed above was pyrex code, not python code. You are
correct that python objects do not require .__reduce__() to be
picklable, but apparently c extension types do (makes sense, they must
be more opaque to the python machinery).

I'll try the .__newobj__(), see if I can get it to do what I want...

On Apr 17, 2:50 am, Ziga Seilnacht <ziga.seilna... @gmail.comwrote :

dgdev wrote:

I would like topicklean extension type (written inpyrex). I have
it working thus far by defining three methods:

>

class C:
# for pickling
__getstate__(se lf):
... # make 'state_obj'
return state_obj

>

__reduce__(self ):
return C,(args,to,__in it__),me.__gets tate__()

>

# for unpickling
__setstate__(se lf,state_obj):
self.x=state_ob j.x
...

>

This gets the class pickling and unpickling.

>

However, I'd like to not specify arguments for __init__ (as I do now
in __reduce__), and so not have __init__ invoked during unpickling.

>

I would like to have the pickling machinery somehow create an
uninitialized object, and then call its __setstate__, where I can re-
create it from 'state_obj'.

>

Is there a kosher way to do so, that is without me having to have a
special mode in the constructor for when the object is being created
by the unpickler?

>
Why are you overwriting the __reduce__() method? The default
object.__reduce __() method, inherited by all new style classes,
already does what you want. If you really must overwrite it, and
you don't want __init__() to get called, then you should return a
reconstructor named __newobj__() as the first item of reduce
tuple. Something like this:
>

>def __newobj__(cls, *args):

>
... return cls.__new__(cls , *args)
...>>class C(object):
>
... def __init__(self):
... print "I shouldn't be called at reconstruction"
... def __reduce__(self ):
... try:
... getnewargs = self.__getnewar gs__
... except AttributeError:
... newargs = (self.__class__ ,)
... else:
... newargs = (self.__class__ ,) + getnewargs()
... try:
... getstate = self.__getstate __
... except AttributeError:
... # this ignores __slots__ complications
... state = self.__dict__
... else:
... state = getstate()
... # this ignores list and dict subclasses
... return __newobj__, newargs, state
...>>c = C()
>
I shouldn't be called at reconstruction> >importpickle

>for proto in range(3):

>
... assert isinstance(pick le.loads(pickle .dumps(c, proto)), C)
...
>
>
>
Ziga