Urllib vs. FireFox

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Gilles Ganault

    #1

    Urllib vs. FireFox

    Hello

    After scratching my head as to why I failed finding data from a web
    using the "re" module, I discovered that a web page as downloaded by
    urllib doesn't match what is displayed when viewing the source page in
    FireFox.

    For instance, when searching Amazon for "Wargames":

    URLLIB:
    <a
    href="http://www.amazon.fr/Wargames-Matthew-Broderick/dp/B00004RJ7H"><sp an
    class="srTitle" >Wargames</span></a>

    ~ Matthew Broderick, Dabney Coleman, John Wood, et Ally Sheedy
    <span class="bindingB lock">(<span class="binding" >Cassette
    vidéo</span- 2000)</span></td></tr>

    FIREFOX:
    <div class="productT itle"><a
    href="http://www.amazon.fr/Wargames-Matthew-Broderick/dp/B00004RJ7H/ref=sr_1_1?ie=U TF8&s=dvd&qid=1 224872998&sr=8-1">
    Wargames</a<span class="binding" ~ Matthew Broderick, Dabney
    Coleman, John Wood, et Ally Sheedy</span><span class="binding" >
    (<span class="format"> Cassette vidéo</span- 2000)</span></div>

    Why do they differ?

    Thank you.
  • Stefan Behnel

    #2
    Re: Urllib vs. FireFox

    Gilles Ganault wrote:
    After scratching my head as to why I failed finding data from a web
    using the "re" module, I discovered that a web page as downloaded by
    urllib doesn't match what is displayed when viewing the source page in
    FireFox.
    >
    For instance, when searching Amazon for "Wargames":
    >
    URLLIB:
    <a
    href="http://www.amazon.fr/Wargames-Matthew-Broderick/dp/B00004RJ7H"><sp an
    class="srTitle" >Wargames</span></a>
    >
    ~ Matthew Broderick, Dabney Coleman, John Wood, et Ally Sheedy
    <span class="bindingB lock">(<span class="binding" >Cassette
    vidéo</span- 2000)</span></td></tr>
    >
    FIREFOX:
    <div class="productT itle"><a
    href="http://www.amazon.fr/Wargames-Matthew-Broderick/dp/B00004RJ7H/ref=sr_1_1?ie=U TF8&s=dvd&qid=1 224872998&sr=8-1">
    Wargames</a<span class="binding" ~ Matthew Broderick, Dabney
    Coleman, John Wood, et Ally Sheedy</span><span class="binding" >
    (<span class="format"> Cassette vidéo</span- 2000)</span></div>
    >
    Why do they differ?
    The browser sends a different client identifier than urllib, and the server
    sends back different page content depending on what client is asking.

    Stefan

    Comment

    • Rex

      #3
      Re: Urllib vs. FireFox

      Right. If you want to get the same results with your Python script
      that you did with Firefox, you can modify the browser headers in your
      code.

      Here's an example with urllib2:


      By the way, if you're doing non-trivial web scraping, the mechanize
      module might make your work much easier. You can install it with
      easy_install.


      Comment

      • Mike Driscoll

        #4
        Re: Urllib vs. FireFox

        On Oct 24, 2:53 pm, Rex <rex.eastbou... @gmail.comwrote :
        Right. If you want to get the same results with your Python script
        that you did with Firefox, you can modify the browser headers in your
        code.
        >
        Here's an example with urllib2:http://vsbabu.org/mt/archives/2003/0...g_http_headers...
        >
        By the way, if you're doing non-trivial web scraping, the mechanize
        module might make your work much easier. You can install it with
        easy_install.http://wwwsearch.sourceforge.net/mechanize/
        Or if you just need to query stuff on Amazon, then you might find this
        module helpful:

        http://pypi.python.org/pypi/Python-Amazon/

        -------------------
        Mike Driscoll

        Blog: http://blog.pythonlibrary.org
        Python Extension Building Network: http://www.pythonlibrary.org

        Comment

        • Lie Ryan

          #5
          Re: Urllib vs. FireFox

          On Fri, 24 Oct 2008 20:38:37 +0200, Gilles Ganault wrote:
          Hello
          >
          After scratching my head as to why I failed finding data from a web
          using the "re" module, I discovered that a web page as downloaded by
          urllib doesn't match what is displayed when viewing the source page in
          FireFox.
          >
          Cookies?

          Comment

          • Tim Roberts

            #6
            Re: Urllib vs. FireFox

            Lie Ryan <lie.1296@gmail .comwrote:
            >
            >Cookies?
            Yes, please. I'll take two. Chocolate chip. With milk.
            --
            Tim Roberts, timr@probo.com
            Providenza & Boekelheide, Inc.

            Comment

            • Gilles Ganault

              #7
              Re: Urllib vs. FireFox

              On Fri, 24 Oct 2008 13:15:49 -0700 (PDT), Mike Driscoll
              <kyosohma@gmail .comwrote:
              >On Oct 24, 2:53 pm, Rex <rex.eastbou... @gmail.comwrote :
              >By the way, if you're doing non-trivial web scraping, the mechanize
              >module might make your work much easier. You can install it with
              >easy_install .http://wwwsearch.sourceforge.net/mechanize/
              >
              >Or if you just need to query stuff on Amazon, then you might find this
              >module helpful:
              >
              >http://pypi.python.org/pypi/Python-Amazon/
              Thanks a bunch. I didn't know about the AWS service.

              Comment

              Working...