HTML to dictionary

**Tina I** · Feb 27 '07, 10:25 AM

Re: HTML to dictionary

Tina I wrote:

Hi everyone,
>
I have a small, probably trivial even, problem. I have the following HTML:

>
> METAR:
>
>ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004 NOSIG
> 
>
> short-TAF:
>
>ENBR 270800Z 270918 VRB05KT 9999 FEW020 SCT040
> 
>
> long-TAF:
>
>ENBR 271212 VRB05KT 9999 FEW020 BKN030 TEMPO 2012 2000 SNRA VV010
>BECMG 2124 15012KT
>

>
I need to make this into a dictionary like this:
>
dictionary = {"METAR:" : "ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004
NOSIG" , "short-TAF:" : "ENBR 270800Z 270918 VRB05KT 9999 FEW020 SCT040"
, "long-Taf:" : "ENBR 271212 VRB05KT 9999 FEW020 BKN030 TEMPO 2012 2000
SNRA VV010 BECMG 2124 15012KT"}
>
I have played around with BeautifulSoup but I'm stuck at stripping off
the tags and chop it up to what I need to put in the dict. If someone
can offer some hints or example to get me going I would greatly
appreciate it.
>
Thanks!
Tina

Forgot to mention that the "METAR:", "short-TAF", and "long-TAF" is
always named as such wheras the line of data ("ENBR 271212 VRB05KT 9999
FEW020 BKN030 TEMPO 2012 2000 SNRA VV010 ") is dynamic and can be
anything...

Tina

**bearophileHUGS@lycos.com** · Feb 27 '07, 11:25 AM

Re: HTML to dictionary

Tina I:

I have a small, probably trivial even, problem. I have the following HTML:

This is a little data munging problem.
If it's a one-shot problem, then you can just load it with a browser,
copy and paste it as text, and then process the lines of the text in a
simple way (splitting lines according to ":", and using the stripped
pairs to feed a dict).

If there are more Html files, or you want to automate things more, you
can use html2text:

html2text: THE ASCIINATOR (aka html2txt)

http://www.aaronsw.com/2002/html2text/

A little script like this may help you:

from html2text import html2text
txt = html2text(the_h tml_data)
lines = str(txt).replac e("**", "").strip().spl itlines()
fields = [[field.strip() for field in line.split(":")] for line in
lines]
print dict(fields)

Note that splitlines() is tricky, if you find some problems, then you
may want a smarter splitter.

Bye,
bearophile

**WEINHANDL Herbert** · Feb 27 '07, 11:25 AM

Re: HTML to dictionary

Tina I schrieb:

Hi everyone,
>
I have a small, probably trivial even, problem. I have the following HTML:

>
> METAR:
>
>ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004 NOSIG
>

....

BeautifulSoup is really fun to work with ;-)

I have played around with BeautifulSoup but I'm stuck at stripping off
the tags and chop it up to what I need to put in the dict. If someone
can offer some hints or example to get me going I would greatly
appreciate it.
>
Thanks!
Tina

#!/usr/bin/python
# -*- coding: utf-8 -*-

from BeautifulSoup import BeautifulSoup, Tag, NavigableString

html = """<html<head>< title>Title</title</head>
<body>
<bMETAR: </bENBR 270920Z 00000KT 9999 ... 
<bshort-TAF:</bENBR 270800Z 270918 VRB05KT ... 
<blong-TAF: </bENBR 271212 VRB05KT 9999 ... 
</body>
</html>
"""

soup = BeautifulSoup( html, convertEntities ='html' )
bolds = soup.findAll( 'b' )

dict = {}

for b in bolds :
key = b.next.strip()
val = b.next.next.str ip()
print 'key=', key
print 'val=', val, '\n'
dict[key] = val

print dict

#---- end ----

happy pythoning

Herbert

**Paul Boddie** · Feb 27 '07, 11:45 AM

Re: HTML to dictionary

On 27 Feb, 11:08, Tina I <tina...@bestem selv.comwrote:

>
I have a small, probably trivial even, problem. I have the following HTML:

>
> METAR:
>
>ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004 NOSIG
> 
>
> short-TAF:
>
>ENBR 270800Z 270918 VRB05KT 9999 FEW020 SCT040
> 
>
> long-TAF:
>
>ENBR 271212 VRB05KT 9999 FEW020 BKN030 TEMPO 2012 2000 SNRA VV010
>BECMG 2124 15012KT
>

This looks almost like XHTML which means that you might be able to use
a normal XML parser.

I need to make this into a dictionary like this:
>
dictionary = {"METAR:" : "ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004
NOSIG" , "short-TAF:" : "ENBR 270800Z 270918 VRB05KT 9999 FEW020 SCT040"
, "long-Taf:" : "ENBR 271212 VRB05KT 9999 FEW020 BKN030 TEMPO 2012 2000
SNRA VV010 BECMG 2124 15012KT"}

So what you want to do is to find each "b" element, extract the
contents to produce a dictionary key, and then find all following text
nodes up to the "br" element, extracting the contents of those nodes
to produce the corresponding dictionary value.

Now, with a DOM/XPath library, the first part is quite
straightforward . Let's first parse the document, though:

import libxml2dom # my favourite ;-)
d = libxml2dom.pars e(the_file) # add html=1 if it's HTML

Now, let's get the "b" elements providing the keys:

key_elements = d.xpath("//b")

The above will find all "b" elements throughout the document. If
that's too broad a search, you can specify something more narrow. For
example:

key_elements = d.xpath("/html/body/b")

At this point, key_elements should contain a list of nodes, each
corresponding to a "b" element, and you can get the contents of each
element by asking for all the text nodes inside it and joining them
together, stripping the whitespace off each end to make the dictionary
key itself:

def get_key(key_ele ment):
texts = []
# Get all text child nodes, collecting the contents.
for n in key_element.xpa th("text()"):
texts.append(n. nodeValue)
# Join them together, removing leading/trailing space.
return "".join(texts). strip()

(Currently, libxml2dom lets you ask an element for its nodeValue,
erroneously returning text inside that element, but I don't want to
promote this as a solution since I may change it at some point.)

The process of getting the dictionary values is a bit more difficult.
What we need to do is to ask for the following siblings of the "b"
element, then to loop over them until we find a "br" element. The
dictionary value is then obtained from the discovered text fragments
by joining them together and stripping whitespace from the ends:

def get_value(key_e lement):
texts = []
# Loop over nodes following the element...
for n in key_element.xpa th("following-sibling::node() "):
# Stop looping if we find a "br" element.
if n.nodeType == n.ELEMENT_NODE and n.localName == "br":
break
# Otherwise get the (assumed) text content.
texts.append(n. nodeValue)
# Join the texts and remove leading/trailing space.
return "".join(texts). strip()

So, putting this together, you should get something like this:

dictionary = {}
for key_element in key_elements:
dictionary[get_key(key_ele ment)] = get_value(key_e lement)

As always with HTML processing, your mileage may vary with such an
approach, but I hope this is helpful. You should also be able to use
something like 4Suite or PyXML with the above code, albeit possibly
slightly modified.

Paul

P.S. Hopefully, Google Groups won't wrap the code badly. Whatever
happened to the preview option, Google?

**Nikita the Spider** · Feb 27 '07, 05:55 PM

Re: HTML to dictionary

In article <brednaJRF_OSnn nYRVnzvA@teleno r.com>,
Tina I <tinaweb@bestem selv.comwrote:

Hi everyone,
>
I have a small, probably trivial even, problem. I have the following HTML:

METAR:

ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004 NOSIG
 

short-TAF:

ENBR 270800Z 270918 VRB05KT 9999 FEW020 SCT040
 

long-TAF:

ENBR 271212 VRB05KT 9999 FEW020 BKN030 TEMPO 2012 2000 SNRA VV010 BECMG
2124 15012KT

>
I need to make this into a dictionary like this:
>
dictionary = {"METAR:" : "ENBR 270920Z 00000KT 9999 FEW018 02/M01 Q1004
NOSIG" , "short-TAF:" : "ENBR 270800Z 270918 VRB05KT 9999 FEW020 SCT040"
, "long-Taf:" : "ENBR 271212 VRB05KT 9999 FEW020 BKN030 TEMPO 2012 2000
SNRA VV010 BECMG 2124 15012KT"}

Tina,
In addition to Beautiful Soup which others have mentioned, Connelly
Barnes' HTMLData module will take (X)HTML and convert it into a
dictionary for you:

Oregon State University

http://oregonstate.edu/~barnesc/htmldata/

Oregon State University delivers exceptional, accessible education and problem-solving innovation as Oregon's largest and statewide public research university.

THe dictionary won't have the exact format you want, but I think it
would be fairly easy for you to convert to what you're looking for.

I use HTMLData a lot. Beautiful Soup is great for parsing iteratively,
but if I just want to throw some HTML at a function and get data back,
HTMLData is my tool of choice.

Good luck with whatever you choose

--
Philip

Nikita the Spider

http://NikitaTheSpider.com/

The ultimate life protection

Whole-site HTML validation, link checking and more

**Tina I** · Feb 27 '07, 06:15 PM

Re: HTML to dictionary

Thanks people, I learned a lot!! :)

I went for Herbert's solution in my application but I explored, and
learned from, all of them.

Tina

HTML to dictionary

HTML to dictionary

Comment

Comment

Comment

Comment

Comment

Comment