Hi,
I encounter a problem with pickle.
I download a html from:
and parse it with BeautifulSoup.
This page is very huge.
When I use pickle to dump it, a RuntimeError: maximum recursion depth
exceeded occur.
I think it is cause by this problem at first :
But and then I do not think so, because I log recursion call of pickle
in file
I found that the recursion limit is exceeded in mid-way to expand
whole the BeautifulSoup object.
Not repeat to call some methods.
This is the code for test.
from BeautifulSoup import *
import pickle as pickle
import urllib
doc = urllib.urlopen( 'http://www.amazon.com/Magellan-Maestro-4040-
Widescreen-Navigator/dp/B000NMKHW6/ref=sr_1_2?
ie=UTF8&s=elect ronics&qid=1202 541889&sr=1-2')
import sys
sys.setrecursio nlimit(40000)
soup = BeautifulSoup(d oc)
print pickle.dumps(so up)
-------------------
What I want to ask is: Is this cause by the limit of recursion limit
and stack size?
I had tired cPickle at first, and then I try pickle, cPickle just stop
running program without any message.
I think it is also implement with recursion way, and it also over flow
stack when dumping soup.
Are there any version of pickle that implement with no-recursion way?
Thanks.
Victor Lin.
I encounter a problem with pickle.
I download a html from:
and parse it with BeautifulSoup.
This page is very huge.
When I use pickle to dump it, a RuntimeError: maximum recursion depth
exceeded occur.
I think it is cause by this problem at first :
But and then I do not think so, because I log recursion call of pickle
in file
I found that the recursion limit is exceeded in mid-way to expand
whole the BeautifulSoup object.
Not repeat to call some methods.
This is the code for test.
from BeautifulSoup import *
import pickle as pickle
import urllib
doc = urllib.urlopen( 'http://www.amazon.com/Magellan-Maestro-4040-
Widescreen-Navigator/dp/B000NMKHW6/ref=sr_1_2?
ie=UTF8&s=elect ronics&qid=1202 541889&sr=1-2')
import sys
sys.setrecursio nlimit(40000)
soup = BeautifulSoup(d oc)
print pickle.dumps(so up)
-------------------
What I want to ask is: Is this cause by the limit of recursion limit
and stack size?
I had tired cPickle at first, and then I try pickle, cPickle just stop
running program without any message.
I think it is also implement with recursion way, and it also over flow
stack when dumping soup.
Are there any version of pickle that implement with no-recursion way?
Thanks.
Victor Lin.
Comment