Getting the size of sourcecode

[mh121]

Hello,

I am trying to input a spreadsheet of possible domain names and output the length of the sourcecode of the webpage (if it exists). In doing this, I have three small questions (I am a newbie and apologize if the questions are simple):

1. How do I convert the length of the page to a string? I have looked around the web for Python 'tostring' and found several individually created functions, but I tried a few and had problems.

2. What is the best way to handle errors when a domain phrase doesn't lead to a good website? This will happen (I think) with the line z=br.open('http ://www.'+domainTer m)
for which the domainTerm might not lead to an active website.

3. Instead of getting the total number of characters on the sourcepage (which I get by looking at len(page) ), is there any way to get the number of lines?

Thank you,
Mitch

from mechanize import Browser
import re, time, urllib2

def MakeBrowser():
b = Browser()
headerString = 'mozilla/5.0 (x11; u; linux i686; en-us; rv:1.7.12) ' + \
'gecko/20050922 firefox/1.0.7 (debian package 1.0.7-1)'
h = [('User-agent', headerString)]
b.addheaders = h
b.set_handle_ro bots(False)
return(b)

f = open('bizornot1 .csv','r')
lines = f.readlines()
f.close()
f2 = open('bizornot1 _new.csv','w')
f2.write(lines[0].rstrip()+',Pag eSize'+"\n")
print(lines[0].rstrip()+",Pag eSize")

for i in range(len(lines )-1):
domainTerm = domainTerms[i]
br = MakeBrowser()
z=br.open('http ://www.'+domainTer m)
page=z.read()
f2.write(lines[i+1].rstrip()+','+l en(page)+"\n")
print(lines[i+1].rstrip()+','+l en(page))

f2.close()

[bvdet]

Hello,

I am trying to input a spreadsheet of possible domain names and output the length of the sourcecode of the webpage (if it exists). In doing this, I have three small questions (I am a newbie and apologize if the questions are simple):

1. How do I convert the length of the page to a string? I have looked around the web for Python 'tostring' and found several individually created functions, but I tried a few and had problems.

2. What is the best way to handle errors when a domain phrase doesn't lead to a good website? This will happen (I think) with the line z=br.open('http ://www.'+domainTer m)
for which the domainTerm might not lead to an active website.

3. Instead of getting the total number of characters on the sourcepage (which I get by looking at len(page) ), is there any way to get the number of lines?

Thank you,
Mitch

from mechanize import Browser
import re, time, urllib2

def MakeBrowser():
b = Browser()
headerString = 'mozilla/5.0 (x11; u; linux i686; en-us; rv:1.7.12) ' + \
'gecko/20050922 firefox/1.0.7 (debian package 1.0.7-1)'
h = [('User-agent', headerString)]
b.addheaders = h
b.set_handle_ro bots(False)
return(b)

f = open('bizornot1 .csv','r')
lines = f.readlines()
f.close()
f2 = open('bizornot1 _new.csv','w')
f2.write(lines[0].rstrip()+',Pag eSize'+"\n")
print(lines[0].rstrip()+",Pag eSize")

for i in range(len(lines )-1):
domainTerm = domainTerms[i]
br = MakeBrowser()
z=br.open('http ://www.'+domainTer m)
page=z.read()
f2.write(lines[i+1].rstrip()+','+l en(page)+"\n")
print(lines[i+1].rstrip()+','+l en(page))

f2.close()

I am not familiar with the 'mechanize' module. You can do the following with the 'urllib' module:[code=Python]from urllib import urlopen

h = urlopen('http://www.somewebsite .com/')

# source = h.read() # read page into a string
lineList = h.readlines() # read page into a list of strings
info = h.info()
trueURL = h.geturl()

print ('The number of lines is %d' % len(lineList))

print 'The number of words is %d' % sum([len(line.strip( ).split()) for line in lineList])

h.close()

try:
h = urlopen('http://www.invalidURL. com/')
except IOError, e:
print e[/code]
[code=Python]
>>> The number of lines is 110
The number of words is 807
[Errno socket error] (7, 'getaddrinfo failed')

>>> info
<httplib.HTTPMe ssage instance at 0x00E7A3C8>
>>> print info
Date: Thu, 09 Aug 2007 02:25:43 GMT
Server: Apache
Last-Modified: Tue, 07 Aug 2007 02:41:23 GMT
ETag: "744141-2b29-46b7dbd3"
Accept-Ranges: bytes
Content-Length: 11049
Connection: close
Content-Type: text/html

>>> trueURL
'http://www.bvdetailing .com/'
>>>
'''[/code]

[robin746]

1. How do I convert the length of the page to a string? I have looked around the web for Python 'tostring' and found several individually created functions, but I tried a few and had problems.

Please clarify. If you have read the page into a variable, it is (likely) already a string. The str() function converts to a string but I do not think that is what you want.

2. What is the best way to handle errors when a domain phrase doesn't lead to a good website?

What error does your module spit back? Wrap the code you have in a try/except block specifying this error, and then do what you want when it happens. For example, if the error is BadSillyError, do this:

try:
    # code to run
except BadSillyError:
    # what to do on failure
else:
    # continue with rest of cdoe if error does not happen

3. Instead of getting the total number of characters on the sourcepage (which I get by looking at len(page) ), is there any way to get the number of lines?

If you know lines are separated by carriage returns you can do something like:

lines = page.split('\n')
number_of_lines = len(lines)

But then you might want to eliminate blank lines. Or comments.

[robin746]

I have just published a full Line Of Code Counter that you can adapt to your purpose.

[mh121]

Thank you very much for your comments. What I meant for the destring function was actually just the str() function you provided. I tried out many of your suggestions today and, after trying out more tomorrow, if I continue to have questions, I will repost.