Output of HTML parsing

[Jackie]

Hi, all,

I want to get the information of the professors (name,title) from the
following link:

"http://www.economics.u toronto.ca/index.php/index/person/faculty/"

Ideally, I'd like to have a output file where each line is one Prof,
including his name and title. In practice, I use the CSV module.

The following is my program:


--------------- Program
----------------------------------------------------

import urllib,re,csv

url = "http://www.economics.u toronto.ca/index.php/index/person/
faculty/"

sock = urllib.urlopen( url)
htmlSource = sock.read()
sock.close()

namePattern = re.compile(r'cl ass="name">(.*) </a>')
titlePattern = re.compile(r'</a>,&nbsp;(.*)\s *</td>')

name = namePattern.fin dall(htmlSource )
title_temp = titlePattern.fi ndall(htmlSourc e)
title =[]
for item in title_temp:
item_new=" ".join(item.spl it()) #Suppress the
spaces between 'title' and </td>
title.extend([item_new])


output =[]
for i in range(len(name) ):
output.insert(i ,[name[i],title[i]]) #Generate a list of
[name, title]

writer = csv.writer(open ("professor.csv ", "wb"))
writer.writerow s(output) #output CSV file

-------------- End of Program
----------------------------------------------

My questions are:

1.The code above assume that each Prof has a tilte. If any one of them
does not, the name and title will be mismatched. How to program to
allow that title can be empty?

2.Is there any easier way to get the data I want other than using
list?

3.Should I close the opened csv file("professor .csv")? How to close
it?

Thanks!

Jackie

[Sebastian Wiesner]

Re: Output of HTML parsing

[ Jackie <jackie.BPUG@gm ail.com]

1.The code above assume that each Prof has a tilte. If any one of them
does not, the name and title will be mismatched. How to program to
allow that title can be empty?
>
2.Is there any easier way to get the data I want other than using
list?

Use BeautifulSoup.

3.Should I close the opened csv file("professor .csv")? How to close
it?

Assign the file object to a separate name (e.g. stream) and then invoke its
close method after writing all csv data to it.

--
Freedom is always the freedom of dissenters.
(Rosa Luxemburg)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (GNU/Linux)

iD8DBQBGcp64n3I EGILecb4RAlblAK CmypoYjyPSciI0N aC7A9dcPIa3owCg kn3G
owa3lSPAMdTDhze jhuF8ztg=
=FP0v
-----END PGP SIGNATURE-----

[Stefan Behnel]

Re: Output of HTML parsing

Jackie wrote:

I want to get the information of the professors (name,title) from the
following link:
>
"http://www.economics.u toronto.ca/index.php/index/person/faculty/"

That's even XHTML, no need to go through BeautifulSoup. Use lxml instead.

http://codespeak.net/lxml

Ideally, I'd like to have a output file where each line is one Prof,
including his name and title. In practice, I use the CSV module.
----------------------------------------------------
>
import urllib,re,csv
>
url = "http://www.economics.u toronto.ca/index.php/index/person/
faculty/"
>
sock = urllib.urlopen( url)
htmlSource = sock.read()
sock.close()

import lxml.etree as et
url = "http://www.economics.u toronto.ca/index.php/index/person/faculty/"
tree = et.parse(url)

namePattern = re.compile(r'cl ass="name">(.*) </a>')
titlePattern = re.compile(r'</a>,&nbsp;(.*)\s *</td>')
>
name = namePattern.fin dall(htmlSource )
title_temp = titlePattern.fi ndall(htmlSourc e)
title =[]
for item in title_temp:
item_new=" ".join(item.spl it()) #Suppress the
spaces between 'title' and </td>
title.extend([item_new])
>
>
output =[]
for i in range(len(name) ):
output.insert(i ,[name[i],title[i]]) #Generate a list of
[name, title]

# untested
get_name_text = et.XPath('norma lize-space(td[a/@class="name"]')
name_list = []
for name_row in tree.xpath('//tr[td/a/@class = "name"]'):
name_list.appen d(
tuple(get_name_ text(name_row). split(",", 3) + ["","",""])[:3] )

writer = csv.writer(open ("professor.csv ", "wb"))
writer.writerow s(output) #output CSV file

writer = csv.writer(open ("professor.csv ", "wb"))
writer.writerow s(name_list) #output CSV file

-------------- End of Program
----------------------------------------------
>
3.Should I close the opened csv file("professor .csv")? How to close
it?

I guess it has a "close()" function?

Stefan

[Jackie]

Re: Output of HTML parsing

On 6 15 , 2 01 , Stefan Behnel <stefan.behne l-n05...@web.dewr ote:

Jackie wrote:

import lxml.etree as et
url = "http://www.economics.u toronto.ca/index.php/index/person/faculty/"
tree = et.parse(url)
>

Stefan- -
>
- -

Thank you. But when I tried to run the above part, the following
message showed up:

Traceback (most recent call last):
File "D:\TS\Python\w orkspace\eco_de partment\lxml_v er.py", line 3, in
<module>
tree = et.parse(url)
File "etree.pyx" , line 1845, in etree.parse
File "parser.pxi ", line 928, in etree._parseDoc ument
File "parser.pxi ", line 932, in etree._parseDoc umentFromURL
File "parser.pxi ", line 849, in etree._parseDoc FromFile
File "parser.pxi ", line 557, in etree._BasePars er._parseDocFro mFile
File "parser.pxi ", line 631, in etree._handlePa rseResult
File "parser.pxi ", line 602, in etree._raisePar seError
etree.XMLSyntax Error: line 2845: Premature end of data in tag html
line 8

Could you please tell me where went wrong?

Thank you

Jackie

[Stefan Behnel]

Re: Output of HTML parsing

Jackie schrieb:

On 6 15 , 2 01 , Stefan Behnel <stefan.behne l-n05...@web.dewr ote:

>Jackie wrote:

>

>import lxml.etree as et
>url = "http://www.economics.u toronto.ca/index.php/index/person/faculty/"
>tree = et.parse(url)
>>

>

>Stefan- -
>>
>- -

>
Thank you. But when I tried to run the above part, the following
message showed up:
>
Traceback (most recent call last):
File "D:\TS\Python\w orkspace\eco_de partment\lxml_v er.py", line 3, in
<module>
tree = et.parse(url)
File "etree.pyx" , line 1845, in etree.parse
File "parser.pxi ", line 928, in etree._parseDoc ument
File "parser.pxi ", line 932, in etree._parseDoc umentFromURL
File "parser.pxi ", line 849, in etree._parseDoc FromFile
File "parser.pxi ", line 557, in etree._BasePars er._parseDocFro mFile
File "parser.pxi ", line 631, in etree._handlePa rseResult
File "parser.pxi ", line 602, in etree._raisePar seError
etree.XMLSyntax Error: line 2845: Premature end of data in tag html
line 8
>
Could you please tell me where went wrong?

Ah, ok, then the page is not actually XHTML, but broken HTML. Use this idiom
instead:

parser = et.HTMLParser()
tree = et.parse(url, parser)

Stefan