Why can't you pickle instancemethods?

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • Chris
    #1

    Why can't you pickle instancemethods?

    Why can pickle serialize references to functions, but not methods?

    Pickling a function serializes the function name, but pickling a
    staticmethod, classmethod, or instancemethod generates an error. In
    these cases, pickle knows the instance or class, and the method, so
    what's the problem? Pickle doesn't serialize code objects, so why can't
    it serialize the name as it does for functions? Is this one of those
    features that's feasible, but not useful, so no one's ever gotten
    around to implementing it?

    Regards,
    Chris
    >>import pickle
    >>>
    >>def somefunc():
    .... return 1
    ....
    >>class Functions(objec t):
    .... @staticmethod
    .... def somefunc():
    .... return 1
    ....
    >>class Foo(object):
    .... pass
    ....
    >>f = Foo()
    >>f.value = somefunc
    >>print pickle.dumps(f)
    ccopy_reg
    _reconstructor
    p0
    (c__main__
    Foo
    p1
    c__builtin__
    object
    p2
    Ntp3
    Rp4
    (dp5
    S'value'
    p6
    c__main__
    somefunc
    p7
    sb.
    >>>
    >>f.value = Functions.somef unc
    >>print pickle.dumps(f)
    Traceback (most recent call last):
    File "<stdin>", line 1, in ?
    File "C:\Program Files\Python24\ lib\pickle.py", line 1386, in dumps
    Pickler(file, protocol, bin).dump(obj)
    File "C:\Program Files\Python24\ lib\pickle.py", line 231, in dump
    self.save(obj)
    File "C:\Program Files\Python24\ lib\pickle.py", line 338, in save
    self.save_reduc e(obj=obj, *rv)
    File "C:\Program Files\Python24\ lib\pickle.py", line 433, in
    save_reduce
    save(state)
    File "C:\Program Files\Python24\ lib\pickle.py", line 293, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Program Files\Python24\ lib\pickle.py", line 663, in
    save_dict
    self._batch_set items(obj.iteri tems())
    File "C:\Program Files\Python24\ lib\pickle.py", line 677, in
    _batch_setitems
    save(v)
    File "C:\Program Files\Python24\ lib\pickle.py", line 293, in save
    f(self, obj) # Call unbound method with explicit self
    File "C:\Program Files\Python24\ lib\pickle.py", line 765, in
    save_global
    raise PicklingError(
    pickle.Pickling Error: Can't pickle <function somefunc at 0x009EC5F0>:
    it's not the same object as __
    main__.somefunc

    Tags: None
    • mdsteele@gmail.com
      #2
      Re: Why can't you pickle instancemethods ?

      Chris wrote:
      Why can pickle serialize references to functions, but not methods?
      >
      Pickling a function serializes the function name, but pickling a
      staticmethod, classmethod, or instancemethod generates an error. In
      these cases, pickle knows the instance or class, and the method, so
      what's the problem? Pickle doesn't serialize code objects, so why can't
      it serialize the name as it does for functions? Is this one of those
      features that's feasible, but not useful, so no one's ever gotten
      around to implementing it?
      I have often wondered this myself. I'm convinced that it would in fact
      be useful -- more than once I've written a program that has lots of
      objects with function pointers, and where it was inconvenient that the
      method pointers could not be pickled. One compromise that I have used
      before is to write a class such as:

      class InstanceMethodS et(object):
      def __init__(self,m ethods):
      self.methods = set(methods)
      def __getstate__(se lf):
      return [(method.im_self , method.im_func. func_name)
      for method in self.method]
      def __setstate__(se lf,state):
      self.methods = set(getattr(obj ,name) for obj,name in state)

      Obviously, this particular example is crude and not terribly robust,
      but it seems to do the job -- it effectively lets you pickle a set of
      instance method pointers. I don't know of any reason why instance
      methods (or class or static methods) couldn't be pickled directly,
      unless perhaps there exists some kind of pathological corner case that
      would create Badness?

      -Matt

        Comment

        • Gabriel Genellina
          #3
          Re: Why can't you pickle instancemethods ?

          At Friday 20/10/2006 18:33, Chris wrote:
          >Why can pickle serialize references to functions, but not methods?
          Because all references must be globally accessible.
          >Pickling a function serializes the function name, but pickling a
          >staticmethod , classmethod, or instancemethod generates an error. In
          >these cases, pickle knows the instance or class, and the method, so
          >what's the problem? Pickle doesn't serialize code objects, so why can't
          >it serialize the name as it does for functions? Is this one of those
          >features that's feasible, but not useful, so no one's ever gotten
          >around to implementing it?
          Well, it's not so common to take a method from one class and glue it
          into an *instance* of another class...
          (Note that if you copy&paste a method from one *class* into another
          *class*, pickle works OK, as long as in the unpickling environment
          you rebuild your classes the same way)

          For static methods there is no way you can retrieve a globally usable
          name (like 'Functions.some func' in your example) - at least I don't
          know how to do it, they appear to have lost any reference to the
          defining class.
          For class methods you can build such reference using im_self and
          im_func.func_na me
          For instance methods use im_class and im_func.func_na me
          Then define your own __getstate__ and __setstate__.


          --
          Gabriel Genellina
          Softlab SRL

          _______________ _______________ _______________ _____
          Correo Yahoo!
          Espacio para todos tus mensajes, antivirus y antispam ¡gratis!
          ¡Abrí tu cuenta ya! - http://correo.yahoo.com.ar

            Comment

            • Chris
              #4
              Re: Why can't you pickle instancemethods ?

              mdsteele@gmail. com wrote:
              Chris wrote:
              Why can pickle serialize references to functions, but not methods?

              Pickling a function serializes the function name, but pickling a
              staticmethod, classmethod, or instancemethod generates an error. In
              these cases, pickle knows the instance or class, and the method, so
              what's the problem? Pickle doesn't serialize code objects, so why can't
              it serialize the name as it does for functions? Is this one of those
              features that's feasible, but not useful, so no one's ever gotten
              around to implementing it?
              >
              I have often wondered this myself. I'm convinced that it would in fact
              be useful -- more than once I've written a program that has lots of
              objects with function pointers, and where it was inconvenient that the
              method pointers could not be pickled. One compromise that I have used
              before is to write a class such as:
              >
              class InstanceMethodS et(object):
              def __init__(self,m ethods):
              self.methods = set(methods)
              def __getstate__(se lf):
              return [(method.im_self , method.im_func. func_name)
              for method in self.method]
              def __setstate__(se lf,state):
              self.methods = set(getattr(obj ,name) for obj,name in state)
              >
              Obviously, this particular example is crude and not terribly robust,
              but it seems to do the job -- it effectively lets you pickle a set of
              instance method pointers. I don't know of any reason why instance
              methods (or class or static methods) couldn't be pickled directly,
              unless perhaps there exists some kind of pathological corner case that
              would create Badness?
              >
              -Matt
              Thanks, that's quite clever. Although I think you'd still have to
              explicitly specify im_self for staticmethods. I could imagine a similar
              callable proxy that simply removes the direct method reference from the
              equation:

              class MethodProxy(obj ect):
              def __init__(self, obj, method):
              self.obj = obj
              if isinstance(meth od, basestring):
              self.methodName = method
              else:
              assert callable(method )
              self.methodName = method.func_nam e
              def __call__(self, *args, **kwargs):
              return getattr(self.ob j, self.methodName )(*args, **kwargs)

              picklableMethod = MethodProxy(som eObj, someObj.method)

                Comment

                • Steven Bethard
                  #5
                  Re: Why can't you pickle instancemethods ?

                  Chris wrote:
                  Why can pickle serialize references to functions, but not methods?
                  Here's the recipe I use::

                  def _pickle_method( method):
                  func_name = method.im_func. __name__
                  obj = method.im_self
                  cls = method.im_class
                  return _unpickle_metho d, (func_name, obj, cls)

                  def _unpickle_metho d(func_name, obj, cls):
                  for cls in cls.mro():
                  try:
                  func = cls.__dict__[func_name]
                  except KeyError:
                  pass
                  else:
                  break
                  return func.__get__(ob j, cls)

                  import copy_reg
                  import types
                  copy_reg.pickle (types.MethodTy pe, _pickle_method, _unpickle_metho d)

                  There may be some special cases where this fails, but I haven't run into
                  them yet.

                  STeVe

                    Comment

                    • mdsteele@gmail.com
                      #6
                      Re: Why can't you pickle instancemethods ?

                      Steven Bethard wrote:
                      Here's the recipe I use::
                      >
                      [...]
                      >
                      There may be some special cases where this fails, but I haven't run into
                      them yet.
                      Wow, that's a really nice recipe; I didn't even know about the copy_reg
                      module. I'll have to start using that.

                      I did notice one failure mode, however--it doesn't work with methods
                      named __foo because im_func.__name_ _ contains the *unmangled* version
                      of the function name, so when you try to unpickle the method, the try
                      statement never succeeds and you get an UnboundLocalErr or on func.

                      The good news is that I think it can be fixed by mangling the name
                      manually in _pickle_method( ), like so:

                      def _pickle_method( method):
                      func_name = method.im_func. __name__
                      obj = method.im_self
                      cls = method.im_class
                      if func_name.start swith('__') and not func_name.endsw ith('__'):
                      cls_name = cls.__name__.ls trip('_')
                      if cls_name: func_name = '_' + cls_name + func_name
                      return _unpickle_metho d, (func_name, obj, cls)

                        Comment

                        Previous template Next
                        Working...