HTML Parsing

[mtuller]

Alright. I have tried everything I can find, but am not getting
anywhere. I have a web page that has data like this:

<tr >
<td headers="col1_1 " style="width:21 %" >
<span class="hpPageTe xt" >LETTER</span></td>
<td headers="col2_1 " style="width:13 %; text-align:right" >
<span class="hpPageTe xt" >33,699</span></td>
<td headers="col3_1 " style="width:13 %; text-align:right" >
<span class="hpPageTe xt" >1.0</span></td>
<td headers="col4_1 " style="width:13 %; text-align:right" >
</tr>

What is show is only a small section.

I want to extract the 33,699 (which is dynamic) and set the value to a
variable so that I can insert it into a database. I have tried parsing
the html with pyparsing, and the examples will get it to print all
instances with span, of which there are a hundred or so when I use:

for srvrtokens in printCount.sear chString(printe rListHTML):
print srvrtokens

If I set the last line to srvtokens[3] I get the values, but I don't
know grab a single line and then set that as a variable.

I have also tried Beautiful Soup, but had trouble understanding the
documentation, and HTMLParser doesn't seem to do what I want. Can
someone point me to a tutorial or give me some pointers on how to
parse html where there are multiple lines with the same tags and then
be able to go to a certain line and grab a value and set a variable's
value to that?

Thanks,

Mike

[Gabriel Genellina]

Re: HTML Parsing

En Sat, 10 Feb 2007 20:07:43 -0300, mtuller <mituller@gmail .comescribió:

<tr >
<td headers="col1_1 " style="width:21 %" >
<span class="hpPageTe xt" >LETTER</span></td>
<td headers="col2_1 " style="width:13 %; text-align:right" >
<span class="hpPageTe xt" >33,699</span></td>
<td headers="col3_1 " style="width:13 %; text-align:right" >
<span class="hpPageTe xt" >1.0</span></td>
<td headers="col4_1 " style="width:13 %; text-align:right" >
</tr>
>
I want to extract the 33,699 (which is dynamic) and set the value to a
variable so that I can insert it into a database. I have tried parsing
[...]
I have also tried Beautiful Soup, but had trouble understanding the
documentation, and HTMLParser doesn't seem to do what I want. Can[...]

Just try harder with BeautifulSoup, should work OK for your use case.
Unfortunately I can't give you an example right now.

--
Gabriel Genellina

[Ayaz Ahmed Khan]

Re: HTML Parsing

"mtuller" typed:

I have also tried Beautiful Soup, but had trouble understanding the
documentation

As Gabriel has suggested, spend a little more time going through the
documentation of BeautifulSoup. It is pretty easy to grasp.

I'll give you an example: I want to extract the text between the
following span tags in a large HTML source file.

<span class="title">L inux Kernel Bluetooth CAPI Packet Remote Buffer Overflow Vulnerability</span>

>>import re
>>from BeautifulSoup import BeautifulSoup
>>from urllib2 import urlopen
>>soup = BeautifulSoup(u rlopen('http://www.someurl.tld/'))
>>title = soup.find(name= 'span', attrs={'class': 'title'}, text=re.compile (r'^Linux \w+'))
>>title

u'Linux Kernel Bluetooth CAPI Packet Remote Buffer Overflow Vulnerability'

--
Ayaz Ahmed Khan

A witty saying proves nothing, but saying something pointless gets
people's attention.

[John Machin]

Re: HTML Parsing

On Feb 11, 6:05 pm, Ayaz Ahmed Khan <a...@dev.slash .nullwrote:

"mtuller" typed:
>

I have also tried Beautiful Soup, but had trouble understanding the
documentation

>
As Gabriel has suggested, spend a little more time going through the
documentation of BeautifulSoup. It is pretty easy to grasp.
>
I'll give you an example: I want to extract the text between the
following span tags in a large HTML source file.
>
<span class="title">L inux Kernel Bluetooth CAPI Packet Remote Buffer Overflow Vulnerability</span>
>

>import re
>from BeautifulSoup import BeautifulSoup
>from urllib2 import urlopen
>soup = BeautifulSoup(u rlopen('http://www.someurl.tld/'))
>title = soup.find(name= 'span', attrs={'class': 'title'}, text=re.compile (r'^Linux \w+'))
>title

>
u'Linux Kernel Bluetooth CAPI Packet Remote Buffer Overflow Vulnerability'
>

One can even use ElementTree, if the HTML is well-formed. See below.
However if it is as ill-formed as the sample (4th "td" element not
closed; I've omitted it below), then the OP would be better off
sticking with Beautiful Soup :-)

C:\junk>type element_soup.py
from xml.etree import cElementTree as ET
import cStringIO

guff = """
<tr >
<td headers="col1_1 " style="width:21 %" >
<span class="hpPageTe xt" >LETTER</span></td>
<td headers="col2_1 " style="width:13 %; text-align:right" >
<span class="hpPageTe xt" >33,699</span></td>
<td headers="col3_1 " style="width:13 %; text-align:right" >
<span class="hpPageTe xt" >1.0</span></td>
</tr>
"""

tree = ET.parse(cStrin gIO.StringIO(gu ff))
for elem in tree.getiterato r('td'):
key = elem.get('heade rs')
assert elem[0].tag == 'span'
value = elem[0].text
print repr(key), repr(value)

C:\junk>\python 25\python element_soup.py
'col1_1' 'LETTER'
'col2_1' '33,699'
'col3_1' '1.0'

HTH,
John

[Fredrik Lundh]

Re: HTML Parsing

John Machin wrote:

One can even use ElementTree, if the HTML is well-formed. See below.
However if it is as ill-formed as the sample (4th "td" element not
closed; I've omitted it below), then the OP would be better off
sticking with Beautiful Soup :-)

or get the best of both worlds:

http://effbot.org/zone/element-soup.htm

</F>

[Stefan Behnel]

Re: HTML Parsing

John Machin wrote:

One can even use ElementTree, if the HTML is well-formed. See below.
However if it is as ill-formed as the sample (4th "td" element not
closed; I've omitted it below), then the OP would be better off
sticking with Beautiful Soup :-)

Or (as we were talking about the best of both worlds already) use lxml's HTML
parser, which is also capable of parsing pretty disgusting HTML-like tag soup.

Stefan