pickling extension class

**Alex Martelli** · Jul 18 '05, 08:25 PM

Re: pickling extension class

harold fellermann <harold.fellerm ann@upf.edu> wrote:
[color=blue]
> File "/sw/lib/python2.4/pickle.py", line 760, in save_global
> raise PicklingError(
> pickle.Pickling Error: Can't pickle <type 'hyper.Periodic Grid'>: it's
> not found as hyper.PeriodicG rid[color=green][color=darkred]
> >>> dir(hyper)[/color][/color]
> ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
> '__file__', '__name__', 'refcount'][color=green][color=darkred]
> >>> hyper.PeriodicG rid[/color][/color]
> <type 'hyper.Periodic Grid'>
>
> So pickle complains about the class PeriodicGrid not being found in the
> module hyper, but a dir()
> proves that python can find it. Has anyone an idea what's going wrong
> here?[/color]

These symptomps are pretty weird -- let's try to pin things down a bit
more. The relevant few lines of pickle.py are:

try:
__import__(modu le)
mod = sys.modules[module]
klass = getattr(mod, name)
except (ImportError, KeyError, AttributeError) :
raise PicklingError(

so, could you please edit your pickle.py to provide VASTLY more info,
say:

try:
print 'Here it goes...:'
_xx = __import__(modu le)
print ' __import__ says: %r' % (_xx,)
mod = sys.modules[module]
print ' in sys.modules: %r' % (mod,)
klass = getattr(mod, name)
print ' klass is: %r' % (klass,)
except (ImportError, KeyError, AttributeError) , _xx:
print ' OOPS, error (%s): %s' % (_xx.__class__, _xx)
raise PicklingError(

and let us know exactly what his modified pickle.py outputs...?

Thanks,

Alex

**harold fellermann** · Jul 18 '05, 08:25 PM

Re: pickling extension class

On 18.01.2005, at 20:31, Alex Martelli wrote:
[color=blue]
> harold fellermann <harold.fellerm ann@upf.edu> wrote:
>[color=green]
>> File "/sw/lib/python2.4/pickle.py", line 760, in save_global
>> raise PicklingError(
>> pickle.Pickling Error: Can't pickle <type 'hyper.Periodic Grid'>: it's
>> not found as hyper.PeriodicG rid[color=darkred]
>>>>> dir(hyper)[/color]
>> ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
>> '__file__', '__name__', 'refcount'][color=darkred]
>>>>> hyper.PeriodicG rid[/color]
>> <type 'hyper.Periodic Grid'>
>>
>> So pickle complains about the class PeriodicGrid not being found in
>> the
>> module hyper, but a dir()
>> proves that python can find it. Has anyone an idea what's going wrong
>> here?[/color]
>
> These symptomps are pretty weird -- let's try to pin things down a bit
> more. The relevant few lines of pickle.py are:
>
> try:
> __import__(modu le)
> mod = sys.modules[module]
> klass = getattr(mod, name)
> except (ImportError, KeyError, AttributeError) :
> raise PicklingError(
>
> so, could you please edit your pickle.py to provide VASTLY more info,[/color]

[...]
[color=blue]
> and let us know exactly what his modified pickle.py outputs...?[/color]

Here it goes...:
OOPS, error (exceptions.Imp ortError): No module named hyper
Traceback (most recent call last):
File "pickle_test.py ", line 5, in ?
pickle.dump(g,f ile("test","w") )
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 1387,
in dump
Pickler(file, protocol, bin).dump(obj)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 231,
in dump
self.save(obj)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 338,
in save
self.save_reduc e(obj=obj, *rv)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 414,
in save_reduce
save(func)
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 293,
in save
f(self, obj) # Call unbound method with explicit self
File "/Volumes/space/Users/harold/uni/pace/ono/pickle.py", line 765,
in save_global
raise PicklingError(
pickle.Pickling Error: Can't pickle <type 'hyper.Periodic Grid'>: it's
not found as hyper.PeriodicG rid

I have noticed that the error does not occur, when the imported module
('hyper') is in the same directory as the script that pickles. When it
is imported from a subpackage (like in the code
I sent you) it goes wrong.

- harold -

--
Reality is for people who lack imagination.

**Alex Martelli** · Jul 18 '05, 08:25 PM

Re: pickling extension class

harold fellermann <harold.fellerm ann@upf.edu> wrote:
...[color=blue]
> Here it goes...:
> OOPS, error (exceptions.Imp ortError): No module named hyper[/color]

So, the __import__ in pickle fails -- indeed, __import__('foo ') when
'foo' ``is imported from a subpackage'' is _supposed_ to fail, as
'hyper' is not the name you SHOULD be importing. For example,

__import__('ema il.Encoders')

is fine, but just

__import__('Enc oders')

fails with ImportError -- there IS no toplevel module by that name!

[color=blue]
> I have noticed that the error does not occur, when the imported module
> ('hyper') is in the same directory as the script that pickles. When it
> is imported from a subpackage (like in the code
> I sent you) it goes wrong.[/color]

If you can't fix the modulename given by your type, you can perhaps
kludge things up by (e.g.)

import the.real.hyper
sys.modules['hyper']=the.real.hyper

before you pickle; but I suspect UNpickling would fail in that case.

Using the standard library copy_reg module to register your way to
pickle and recover instances of your type might work better.

Alex

**David M. Cooke** · Jul 18 '05, 08:25 PM

Re: pickling extension class

harold fellermann <harold.fellerm ann@upf.edu> writes:
[color=blue]
> Hi all,
>
> I have a problem pickling an extension class. As written in the
> Extending/Embedding Manual, I
> provided a function __reduce__ that returns the appropreate tuple.
> This seams to work fine,
> but I still cannot pickle because of the following error:
>[color=green][color=darkred]
> >>> from model import hyper
> >>> g = hyper.PeriodicG rid(4,4,1)
> >>> g.__reduce__()[/color][/color]
> (<type 'hyper.Periodic Grid'>,(4.,4.,1 .))[color=green][color=darkred]
> >>> import pickle
> >>> pickle.dump(g,f ile("test","w") )[/color][/color]
> Traceback (most recent call last):
> File "pickle_test.py ", line 5, in ?
> pickle.dump(g,f ile("test","w") )
> File "/sw/lib/python2.4/pickle.py", line 1382, in dump
> Pickler(file, protocol, bin).dump(obj)
> File "/sw/lib/python2.4/pickle.py", line 231, in dump
> self.save(obj)
> File "/sw/lib/python2.4/pickle.py", line 338, in save
> self.save_reduc e(obj=obj, *rv)
> File "/sw/lib/python2.4/pickle.py", line 414, in save_reduce
> save(func)
> File "/sw/lib/python2.4/pickle.py", line 293, in save
> f(self, obj) # Call unbound method with explicit self
> File "/sw/lib/python2.4/pickle.py", line 760, in save_global
> raise PicklingError(
> pickle.Pickling Error: Can't pickle <type 'hyper.Periodic Grid'>: it's
> not found as hyper.PeriodicG rid[color=green][color=darkred]
> >>> dir(hyper)[/color][/color]
> ['Dir', 'Neighbors', 'PeriodicGrid', 'PeriodicPos', '__doc__',
> '__file__', '__name__', 'refcount'][color=green][color=darkred]
> >>> hyper.PeriodicG rid[/color][/color]
> <type 'hyper.Periodic Grid'>[/color]
^^^^^

I think that's your error. The extension type is declared to be
hyper.PeriodicG rid, where it actually is model.hyper.Per iodicGrid
(because hyper is in the model package).

Pickle stores g.__class__.__m odule__ (which is "hyper") and
g.__class__.__n ame__ (="PeriodicGrid ") to find the class object for
reimporting, and on unpickling, tries to do __import__("hyp er"), which
fails.

The tp_name slot of your extension type should be "model.hyper.Pe riodicGrid".

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca

pickling extension class

pickling extension class

Comment

Comment

Comment

Comment