html 2 plain text

[robin]

hi,
i remember seeing this simple python function which would take raw html
and output the content (body?) of the page as plain text (no <..> tags
etc)
i have been looking at htmllib and htmlparser but this all seems to
complicated for what i'm looking for. i just need the main text in the
body of some arbitrary webbpage to then do some natural-language
processing with it...
thanks for pointing me to some helpful resources!

robin

[Faber]

Re: html 2 plain text

robin wrote:
[color=blue]
> i remember seeing this simple python function which would take raw html
> and output the content (body?) of the page as plain text (no <..> tags
> etc)
> i have been looking at htmllib and htmlparser but this all seems to
> complicated for what i'm looking for. i just need the main text in the
> body of some arbitrary webbpage to then do some natural-language
> processing with it...
> thanks for pointing me to some helpful resources![/color]

Have a look at the Beautiful Soup library:

http://www.crummy.com/software/BeautifulSoup/

Regards

--
Faber

http://faberbox.com/

http://smarking.com/

A teacher must always teach to doubt his teaching. -- José Ortega y Gasset

[robin]

Re: html 2 plain text

lucks yummy. merci beaucoup.

robin

[Ravi Teja]

Re: html 2 plain text

> i remember seeing this simple python function which would take raw html[color=blue]
> and output the content (body?) of the page as plain text (no <..> tags
> etc)[/color]

http://www.aaronsw.com/2002/html2text/

[garabik-news-2005-05@kassiopeia.juls.savba.sk]

Re: html 2 plain text

robin <robin.meier@gm ail.com> wrote:[color=blue]
> hi,
> i remember seeing this simple python function which would take raw html
> and output the content (body?) of the page as plain text (no <..> tags
> etc)
> i have been looking at htmllib and htmlparser but this all seems to
> complicated for what i'm looking for. i just need the main text in the
> body of some arbitrary webbpage to then do some natural-language
> processing with it...
> thanks for pointing me to some helpful resources![/color]

text=re.sub(r'( ?s)\<.+?\>', '', html_text)
(this will keep html entities, though)

--
-----------------------------------------------------------
| Radovan GarabÃk http://kassiopeia.juls.savba.sk/~garabik/ |
| __..--^^^--..__ garabik @ kassiopeia.juls .savba.sk |
-----------------------------------------------------------
Antivirus alert: file .signature infected by signature virus.
Hi! I'm a signature virus! Copy me into your signature file to help me spread!

[Fredrik Lundh]

Re: html 2 plain text

garabik-news-2005-05@kassiopeia.j uls.savba.sk wrote:
[color=blue]
> text=re.sub(r'( ?s)\<.+?\>', '', html_text)
> (this will keep html entities, though)[/color]

here's a variation that handles that too:

http://effbot.org/zone/re-sub.htm#strip-html

</F>