Is my web crawler being blocked?

[mh121]

I am trying to write a web crawler (for academic research purposes) that grabs the number of links different websites/domain names have from other websites, as listed on Google (for example, to get the number of websites linking to YouTube, you could type into Google 'Link:YouTube.c om' and get 11,100). I have a list of websites in a spreadsheet and would like to be able to output the number of links for each website in the sheet.

When I run the below script, it seems to run fast and accurately for a minute or so, or for several hundred websites. Then the crawler slows down, outputting that 0 links were found for all websites.

Am I getting this result because Google is detecting my crawler and blocking it? If so, is there anything that I can do, such as telling my crawler to sleep, and try again later (I've tried to sketch something out in the lower batch of code below)?

Thank you.
[CODE=python]
for i in range(len(lines )-1):
searchTerm = "link:"+ searchTerms[i]
br = MakeBrowser()
br.open('http://www.google.com' )
br.select_form( name='f')
br['q'] = searchTerm
resp = br.submit().rea dlines()
# Get the line number that starts with '<table'
htmlSplitter = re.compile('<.* ?>')
names=[]
y=htmlSplitter. split(resp[0])
value=''
for j in range(0,len(y)-1):
if y[j]==' linking to ':
value=string.re place(y[j-1],",","")

#...


br = MakeBrowser()
haveResp = False
while not haveResp:
try:
br.open('http://inventory.overt ure.com/d/' + \
'searchinventor y/suggestion/')
br.select_form( name='stst')
br['term'] = searchTerm
resp = br.submit().rea dlines()
haveResp = True
except urllib2.URLErro r:
time.sleep(10)[/CODE]

[oler1s]

Don't be surprised that your program is being blocked. You are, after all, hammering their servers with inefficient requests, at an unsustainable rate. Keep this up and your IP will be banned.

Obviously, the rate at which you issue requests is a problem. Using the web interface is inherently inefficient. Even worse, I doubt you coded your program to behave like a good client, like allowing for gzip compression of the webpage, caching, and so on. I haven't looked into it myself, but see if Google has a more efficient way of issuing these requests, through some API or something.

At least, don't keep requesting the Google search page over and over. You can construct a search term through the URL directly.

[dazzler]

if some www-site is completely blocking your app, your web crawler can also pretend to be someone else:

[code=python]
class AppURLopener(ur llib.FancyURLop ener):
version = "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8) Gecko/20051107 Firefox/1.5"
#so this is your application's version name (these version names I got from wikipedia, don't remember the article or if it exist anymore)

....

urllib._urlopen er = AppURLopener()
[/code]

of course this isn't so recommended way ;D

and if I remember correctly web site should tell in html if it want allow access to crawlers, so your app "should" first try to detect that code

[mh121]

Don't be surprised that your program is being blocked. You are, after all, hammering their servers with inefficient requests, at an unsustainable rate. Keep this up and your IP will be banned.

Obviously, the rate at which you issue requests is a problem. Using the web interface is inherently inefficient. Even worse, I doubt you coded your program to behave like a good client, like allowing for gzip compression of the webpage, caching, and so on. I haven't looked into it myself, but see if Google has a more efficient way of issuing these requests, through some API or something.

At least, don't keep requesting the Google search page over and over. You can construct a search term through the URL directly.

Thank you very much for your response. I have tried using urlopen through the URL directly with Google, but am finding that Google blocks this as well:

z=urlopen("http ://www.google.com/search?q=link%3 A" +www-site+ "S&rls=com.micr osoft:*&ie=UTF-8&oe=UTF-8&startIndex=1& startPage=1")
q=z.readlines()

I've found an API called PyGoogle that may work, though it seems to be used for more complicated tasks like getting a webpage's search ranking.

To accomplish more realistic searching/ good client behavior, are there any simple ways to allow for gzip compression or caching?

Thanks.