Getting URL's

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • defcon8
    #1

    Getting URL's

    How do I get all the URL's in a page?

    Tags: None
    • Ju Hui
      #2
      Re: Getting URL's

      use
      htmlparser or regular expression

        Comment

        • defcon8
          #3
          Re: Getting URL's

          Thanks

            Comment

            • Paul McGuire
              #4
              Re: Getting URL's

              "defcon8" <defcon8@gmail. com> wrote in message
              news:1148019975 .277742.253400@ y43g2000cwc.goo glegroups.com.. .[color=blue]
              > How do I get all the URL's in a page?
              >[/color]

              pyparsing comes with a simple example that does this, too.

              -- Paul
              Download pyparsing at http://sourceforge.net/projects/pyparsing


                Comment

                • softwindow
                  #5
                  Re: Getting URL's

                  it is difficult to get all URL's in a page
                  you can use sgmllib module to parse html files
                  can get the standard href .

                    Comment

                    • Paul McGuire
                      #6
                      Re: Getting URL's

                      "softwindow " <softwindow@gma il.com> wrote in message
                      news:1148026352 .141338.89290@j 73g2000cwa.goog legroups.com...
                      [color=blue]
                      > it is difficult to get all URL's in a page[/color]
                      <snip>

                      Is this really so hard?:

                      -----------------
                      from pyparsing import Literal,Suppres s,CharsNotIn,Ca selessLiteral,\
                      Word,dblQuotedS tring,alphanums ,SkipTo,makeHTM LTags
                      import urllib

                      # extract all <a> anchor tags - makeHTMLTags defines a
                      # fairly robust pair of match patterns, not just "<tag>","</tag>"
                      linkOpenTag,lin kCloseTag = makeHTMLTags("a ")
                      link = linkOpenTag + \
                      SkipTo(linkClos eTag).setResult sName("body") + \
                      linkCloseTag.su ppress()

                      # read the HTML source from some random URL
                      serverListPage = urllib.urlopen( "http://www.google.com" )
                      htmlText = serverListPage. read()
                      serverListPage. close()

                      # use the link grammar to scan the HTML source
                      for toks,strt,end in link.scanString (htmlText):
                      print toks.startA.hre f,"->",toks.body

                      -----------------
                      Prints:
                      /url?sa=p&pref=i g&pval=2&q=http ://www.google.com/ig%3Fhl%3Den ->
                      Personalized Home
                      https://www.google.com/accounts/Logi...gle.com/&hl=en ->
                      Sign in
                      /imghp?hl=en&tab =wi&ie=UTF-8 -> Images
                      http://groups.google.com/grphp?hl=en&tab=wg&ie=UTF-8 -> Groups
                      http://news.google.com/nwshp?hl=en&tab=wn&ie=UTF-8 -> News
                      http://froogle.google.com/frghp?hl=en&tab=wf&ie=UTF-8 -> Froogle
                      /maphp?hl=en&tab =wl&ie=UTF-8 -> Maps
                      /intl/en/options/ -> more&nbsp;&raqu o;
                      /advanced_search ?hl=en -> Advanced Search
                      /preferences?hl= en -> Preferences
                      /language_tools? hl=en -> Language Tools
                      /intl/en/ads/ -> Advertising&nbs p;Programs
                      /services/ -> Business Solutions
                      /intl/en/about.html -> About Google


                      -- Paul


                        Comment

                        Previous template Next
                        Working...