pickling multiple dictionaries

[manstey]

Hi,

I am running a script that produces about 450,000 dictionaries. I tried
putting them into a tuple and then pickling the tuple, but the tuple
gets too big. Can I pickle dictionaries one after another into the same
file and then read them out again?

Cheers,
Matthew

[Miki]

Re: pickling multiple dictionaries

Hello Matthew,

You can try either http://docs.python.org/lib/module-shelve.html or any
other database bindings with blobs.

HTH,
Miki

http://pythonwise.blogspot.com

[Hans Georg Krauthaeuser]

Re: pickling multiple dictionaries

manstey wrote:[color=blue]
> Hi,
>
> I am running a script that produces about 450,000 dictionaries. I tried
> putting them into a tuple and then pickling the tuple, but the tuple
> gets too big. Can I pickle dictionaries one after another into the same
> file and then read them out again?
>
> Cheers,
> Matthew
>[/color]
If you don't know, just try it:

In [1]:import pickle
In [2]:d1={'a':1}
In [3]:d2={'b':2}
In [4]:pfile=file('te st.p','wb')
In [5]:pickle.dump(d1 ,pfile)
In [6]:pickle.dump(d2 ,pfile)
In [7]:pfile.close()
In [8]:del d1
In [9]:del d2
In [10]:pfile=file('te st.p','rb')
In [11]:d1=pickle.load (pfile)
In [12]:d1
Out[12]:{'a': 1}
In [13]:d2=pickle.load (pfile)
In [14]:d2
Out[14]:{'b': 2}

If your data is *really* large, have a look to PyTables
(http://www.pytables.org/moin).

Regards,
Hans Georg

[manstey]

Re: pickling multiple dictionaries

Thanks very much. How large is *really* large for making pytables
worthwhile. Our python script generates an xml file of about 450Mb. Is
pytables worth using then?