problem doing unpickle in an exec statement

[Danny Shevitz]

Howdy,

In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.

I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.p y) to create the pickle.

create_pickle.p y: (run this first)

############### ############### ###############
import cPickle

# the pickle file name
file_name = 'd:\\temp\\test 1.pickle'

# define a class
class Tree(object):
pass


def main():
# instantiate
t = Tree()

# create the sweet pickle
fp = open(file_name, 'wb')
cPickle.dump(t, fp)
fp.close()

# try to unpickle directly
fp = open(file_name, 'rb')
result = cPickle.load(fp )
fp.close()
print "unpickling directly works just fine, result = ", result

if __name__=='__ma in__':
main()
############### ############### ###############

run this second:

exec_pickle.py
############### ############### ###############
# this file shows a problem with sweet pickle in an exec statement

# the pickle file name
file_name = 'd:\\temp\\test 1.pickle'

# code to be turned into a function
code_text = '''
def include():
print "this works!"
'''

# a function for creating functions
def create_fun(code _text):
clean_dict = {}
exec code_text in clean_dict
return clean_dict['include']

# include_fun is a bona fide function
include_fun = create_fun(code _text)

# this works
include_fun()


# now try to load the pickle in an exec statement
code_text = '''
def include(file_na me):
print "processing file_name: ", file_name
import cPickle
fp = open(file_name, "rb")
result = cPickle.load(fp )
fp.close()
print "result = ", result
'''

# create the new include_fun
include_fun = create_fun(code _text)

# run it
include_fun(fil e_name)

############### ############### ###############

Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

thanks,
Danny

[castironpi]

Re: problem doing unpickle in an exec statement

On Jul 23, 6:01 pm, Danny Shevitz <shev...@lanl.g ovwrote:

Howdy,
>
In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.
>
I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.p y) to create the pickle.
>
create_pickle.p y: (run this first)
>
############### ############### ###############
import cPickle
>
# the pickle file name
file_name = 'd:\\temp\\test 1.pickle'
>
# define a class
class Tree(object):
        pass
>
def main():
        # instantiate  
        t = Tree()
>
        # create the sweet pickle
        fp = open(file_name, 'wb')
        cPickle.dump(t, fp)
        fp.close()
>
        # try to unpickle directly
        fp = open(file_name, 'rb')
        result = cPickle.load(fp )
        fp.close()
        print "unpickling directly works just fine, result = ",result
>
if __name__=='__ma in__':
        main()
############### ############### ###############
>
run this second:
>
exec_pickle.py
############### ############### ###############
# this file shows a problem with sweet pickle in an exec statement
>
# the pickle file name
file_name = 'd:\\temp\\test 1.pickle'
>
# code to be turned into a function
code_text = '''
def include():
  print "this works!"
'''
>
# a function for creating functions
def create_fun(code _text):
        clean_dict = {}
        exec code_text in clean_dict
        return clean_dict['include']
>
# include_fun is a bona fide function
include_fun = create_fun(code _text)
>
# this works
include_fun()
>
# now try to load the pickle in an exec statement
code_text = '''
def include(file_na me):
  print "processing file_name: ", file_name
  import cPickle
  fp = open(file_name, "rb")
  result = cPickle.load(fp )
  fp.close()
  print "result = ", result
'''
>
# create the new include_fun
include_fun = create_fun(code _text)
>
# run it
include_fun(fil e_name)
>
############### ############### ###############
>
Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?
>
thanks,
Danny

Hi,

It works if you paste

# define a class
class Tree(object):
pass

into exec_pickle.py. There are inherent dilemmas in pickling
instances of non-primitive types, such as definition location, and
change in definition across versions. How does 'class Tree' fit in to
your design module?

[Peter Otten]

Re: problem doing unpickle in an exec statement

Danny Shevitz wrote:

Howdy,
>
In my app I need to exec user text that defines a function. I want this
function to unpickle an object. Pickle breaks because it is looking for
the object definition that isn't in the calling namespace.
>
I have mocked up a simple example that shows the problem. Run this
first code (from create_pickle.p y) to create the pickle.
>
create_pickle.p y: (run this first)
>
############### ############### ###############
import cPickle
>
# the pickle file name
file_name = 'd:\\temp\\test 1.pickle'
>
# define a class
class Tree(object):
pass
>
>
def main():
# instantiate
t = Tree()
>
# create the sweet pickle
fp = open(file_name, 'wb')
cPickle.dump(t, fp)
fp.close()
>
# try to unpickle directly
fp = open(file_name, 'rb')
result = cPickle.load(fp )
fp.close()
print "unpickling directly works just fine, result = ", result
>
if __name__=='__ma in__':
main()
############### ############### ###############
>
run this second:
>
exec_pickle.py
############### ############### ###############
# this file shows a problem with sweet pickle in an exec statement
>
# the pickle file name
file_name = 'd:\\temp\\test 1.pickle'
>
# code to be turned into a function
code_text = '''
def include():
print "this works!"
'''
>
# a function for creating functions
def create_fun(code _text):
clean_dict = {}
exec code_text in clean_dict
return clean_dict['include']
>
# include_fun is a bona fide function
include_fun = create_fun(code _text)
>
# this works
include_fun()
>
>
# now try to load the pickle in an exec statement
code_text = '''
def include(file_na me):
print "processing file_name: ", file_name
import cPickle
fp = open(file_name, "rb")
result = cPickle.load(fp )
fp.close()
print "result = ", result
'''
>
# create the new include_fun
include_fun = create_fun(code _text)
>
# run it
include_fun(fil e_name)
>
############### ############### ###############
>
Can anyone enlighten me what I need to do to exec_pickle.py
to get this to work?

Pickling saves the name of the module and the class (and of course the
instance data). Because you put the class in your main script the module
name is __main__, and when you unpickle later pickle imports __main__ which
unfortunately is now a different script that doesn't contain the/a Tree
class. To solve that problem put the Tree class in a separate module, say
tree.py, that can be imported by both create_pickle.p y and exec_pickle.py.
You only need an explicit

from tree import Tree

in create_pickle.

Peter