trying to parse non valid html documents with HTMLParser

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • florent
    #1

    trying to parse non valid html documents with HTMLParser

    I'm trying to parse html documents from the web, using the HTMLParser
    class of the HTMLParser module (python 2.3), but some web documents are
    not fully valids. When the parser finds an invalid tag, he raises an
    exception. Then it seems impossible to resume the parsing just after
    where the exception was raised. I'd like to continue parsing an html
    document even if an invalid tag was found. Is it possible to do this ?

    Here is a little non valid html document.
    ----------
    <html>
    <body>
    <a href="""">bogus link</a>
    </body>
    </html>
    ----------
    Tags: None
    • Benjamin Niemann
      #2
      Re: trying to parse non valid html documents with HTMLParser

      florent wrote:
      [color=blue]
      > I'm trying to parse html documents from the web, using the HTMLParser
      > class of the HTMLParser module (python 2.3), but some web documents are
      > not fully valids.[/color]

      Some?? Most of them :(
      [color=blue]
      > When the parser finds an invalid tag, he raises an
      > exception. Then it seems impossible to resume the parsing just after
      > where the exception was raised. I'd like to continue parsing an html
      > document even if an invalid tag was found. Is it possible to do this ?[/color]

      AFAIK not with HTMLParser or htmllib. You might try (if you haven't done
      yet) htmllib and see, which parser is more forgiving.

      You might pipe the document through an external tool like HTML Tidy
      <http://www.w3.org/People/Raggett/tidy/> before you feed it into
      HTMLParser.


      --
      Benjamin Niemann
      Email: pink at odahoda dot de
      WWW: http://www.odahoda.de/

        Comment

        • Benji York
          #3
          Re: trying to parse non valid html documents with HTMLParser

          florent wrote:[color=blue]
          > I'm trying to parse html documents from the web, using the HTMLParser
          > class of the HTMLParser module (python 2.3), but some web documents are
          > not fully valids.[/color]

          From http://www.crummy.com/software/BeautifulSoup/:

          You didn't write that awful page. You're just trying to get
          some data out of it. Right now, you don't really care what
          HTML is supposed to look like.

          Neither does this parser.
          --
          Benji York

            Comment

            • florent
              #4
              Re: trying to parse non valid html documents with HTMLParser

              > AFAIK not with HTMLParser or htmllib. You might try (if you haven't done[color=blue]
              > yet) htmllib and see, which parser is more forgiving.[/color]

              Thanks, I'll try htmllib.
              In other case, I found a solution. Feeding data to the HTMLParser by
              chunks extracted from the string using string.split("< "), will allow me
              to loose only one tag at a time when an exception is raised !

                Comment

                • florent
                  #5
                  Re: trying to parse non valid html documents with HTMLParser

                  > From http://www.crummy.com/software/BeautifulSoup/:[color=blue]
                  >
                  > You didn't write that awful page. You're just trying to get
                  > some data out of it. Right now, you don't really care what
                  > HTML is supposed to look like.
                  >
                  > Neither does this parser.[/color]

                  True, I just want to extract some data from html documents. But the
                  problem is the same. The parser looses the position he was in the string
                  when he encounters a bad tag.

                    Comment

                    • Benji York
                      #6
                      Re: trying to parse non valid html documents with HTMLParser

                      florent wrote:[color=blue]
                      > True, I just want to extract some data from html documents. But the
                      > problem is the same. The parser looses the position he was in the string
                      > when he encounters a bad tag.[/color]

                      Are you saying that Beautiful Soup can't parse the HTML? If so, I'm
                      sure the author would like an example so he can "fix" it.
                      --
                      Benji York


                        Comment

                        • florent
                          #7
                          Re: trying to parse non valid html documents with HTMLParser

                          > AFAIK not with HTMLParser or htmllib. You might try (if you haven't done[color=blue]
                          > yet) htmllib and see, which parser is more forgiving.[/color]

                          You were right, the HTMLParser of htmllib is more permissive. He just
                          ignores the bad tags !

                            Comment

                            • florent
                              #8
                              Re: trying to parse non valid html documents with HTMLParser

                              > Are you saying that Beautiful Soup can't parse the HTML? If so, I'm[color=blue]
                              > sure the author would like an example so he can "fix" it.[/color]

                              I finally use the htmllib module wich is more permissive than the
                              HTMLParser module when parsing bad html documents.
                              Anyway, where can I find the author's contact informations ?

                                Comment

                                • Steve M
                                  #9
                                  Re: trying to parse non valid html documents with HTMLParser

                                  >You were right, the HTMLParser of htmllib is more permissive. He just
                                  ignores the bad tags !

                                  The HTMLParser on my distribution is a she. But then again, I am using
                                  ActivePython on Windows...

                                    Comment

                                    • Benjamin Niemann
                                      #10
                                      Re: trying to parse non valid html documents with HTMLParser

                                      Steve M wrote:
                                      [color=blue][color=green]
                                      >>You were right, the HTMLParser of htmllib is more permissive. He just[/color]
                                      > ignores the bad tags !
                                      >
                                      > The HTMLParser on my distribution is a she. But then again, I am using
                                      > ActivePython on Windows...[/color]

                                      Although building parsers is for some strange reason one of my favourite
                                      programming adventures, I do not have such a personal relationship with my
                                      classes ;)

                                      --
                                      Benjamin Niemann
                                      Email: pink at odahoda dot de
                                      WWW: http://www.odahoda.de/

                                        Comment

                                        Previous template Next
                                        Working...