Python Screen Scraper

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • James Stroud
    #1

    Python Screen Scraper

    Hello,

    Does anyone know of an example, however modest, of a screenscraper
    authored in python? I am using Firefox.

    Basically, I am answering problems via my browser and being scored for
    each problem. I have a tendency to go past my peak for training
    efficiency, so I would like to scrape the result page for each problem I
    answer, compile statistics, and have a program alert me when I should
    stop (based on score and accuracy--assuming training value is related to
    changes in these metrics).

    I have no idea how to go about writing such a beast and I am hoping that
    I could get some pointers or an example that could get me going in the
    right direction.

    Parsing, etc, is not a problem, but I'm not exactly sure how I might
    interface python with Firefox, forwarding scraped pages to my browser
    (or forwarding from the browser to the scraper).

    Thanks in advance for any help or advice.

    James
    Tags: None
    • Michael Bentley
      #2
      Re: Python Screen Scraper


      On Apr 24, 2007, at 11:50 AM, James Stroud wrote:
      Hello,
      >
      Does anyone know of an example, however modest, of a screenscraper
      authored in python? I am using Firefox.
      >
      Basically, I am answering problems via my browser and being scored for
      each problem. I have a tendency to go past my peak for training
      efficiency, so I would like to scrape the result page for each
      problem I
      answer, compile statistics, and have a program alert me when I should
      stop (based on score and accuracy--assuming training value is
      related to
      changes in these metrics).
      >
      I have no idea how to go about writing such a beast and I am hoping
      that
      I could get some pointers or an example that could get me going in the
      right direction.
      >
      Parsing, etc, is not a problem, but I'm not exactly sure how I might
      interface python with Firefox, forwarding scraped pages to my browser
      (or forwarding from the browser to the scraper).
      >
      Thanks in advance for any help or advice.
      Possibly the easiest thing will be to read from firefox' cache.
      Otherwise I think your only real options are to either build a proxy
      or sniff the wire...


        Comment

        • Jeremy Sanders
          #3
          Re: Python Screen Scraper

          Michael Bentley wrote:
          Possibly the easiest thing will be to read from firefox' cache.
          Otherwise I think your only real options are to either build a proxy
          or sniff the wire...
          Maybe another way would be to write a firefox addon/plugin. I believe python
          is now supported...

          --
          Jeremy Sanders
          Personal webspace of Jeremy Sanders
          http://www.jeremysanders.net/

            Comment

            • Cameron Laird
              #4
              Re: Python Screen Scraper

              In article <mailman.6889.1 177417037.32031 .python-list@python.org >,
              Michael Bentley <michael@jedimi ndworks.comwrot e:
              >
              >On Apr 24, 2007, at 11:50 AM, James Stroud wrote:
              >
              >Hello,
              >>
              >Does anyone know of an example, however modest, of a screenscraper
              >authored in python? I am using Firefox.
              >>
              >Basically, I am answering problems via my browser and being scored for
              >each problem. I have a tendency to go past my peak for training
              >efficiency, so I would like to scrape the result page for each
              >problem I
              >answer, compile statistics, and have a program alert me when I should
              >stop (based on score and accuracy--assuming training value is
              >related to
              >changes in these metrics).
              >>
              >I have no idea how to go about writing such a beast and I am hoping
              >that
              >I could get some pointers or an example that could get me going in the
              >right direction.
              >>
              >Parsing, etc, is not a problem, but I'm not exactly sure how I might
              >interface python with Firefox, forwarding scraped pages to my browser
              >(or forwarding from the browser to the scraper).
              >>
              >Thanks in advance for any help or advice.
              >
              >Possibly the easiest thing will be to read from firefox' cache.
              >Otherwise I think your only real options are to either build a proxy
              >or sniff the wire...
              >
              >
              Oh, there are *lots* of ways to go about this. How crucial is the
              role of Firefox in all this? <URL: http://www.greasespot.net/ >
              might be the right answer. Are you open to use of languages other
              than Python?

                Comment

                • kyosohma@gmail.com
                  #5
                  Re: Python Screen Scraper

                  On Apr 24, 7:17 am, Michael Bentley <mich...@jedimi ndworks.comwrot e:
                  On Apr 24, 2007, at 11:50 AM, James Stroud wrote:
                  >
                  >
                  >
                  Hello,
                  >
                  Does anyone know of an example, however modest, of a screenscraper
                  authored in python? I am using Firefox.
                  >
                  Basically, I am answering problems via my browser and being scored for
                  each problem. I have a tendency to go past my peak for training
                  efficiency, so I would like to scrape the result page for each
                  problem I
                  answer, compile statistics, and have a program alert me when I should
                  stop (based on score and accuracy--assuming training value is
                  related to
                  changes in these metrics).
                  >
                  I have no idea how to go about writing such a beast and I am hoping
                  that
                  I could get some pointers or an example that could get me going in the
                  right direction.
                  >
                  Parsing, etc, is not a problem, but I'm not exactly sure how I might
                  interface python with Firefox, forwarding scraped pages to my browser
                  (or forwarding from the browser to the scraper).
                  >
                  Thanks in advance for any help or advice.
                  >
                  Possibly the easiest thing will be to read from firefox' cache.
                  Otherwise I think your only real options are to either build a proxy
                  or sniff the wire...
                  You should be able to parse the html too. There are quite a few tools
                  out there for that purpose, "Beautiful Soup" being a good example.

                  Mike

                    Comment

                    • James Stroud
                      #6
                      Re: Python Screen Scraper

                      James Stroud wrote:
                      Hello,
                      >
                      Does anyone know of an example, however modest, of a screenscraper
                      authored in python? I am using Firefox.
                      >
                      Basically, I am answering problems via my browser and being scored for
                      each problem. I have a tendency to go past my peak for training
                      efficiency, so I would like to scrape the result page for each problem I
                      answer, compile statistics, and have a program alert me when I should
                      stop (based on score and accuracy--assuming training value is related to
                      changes in these metrics).
                      >
                      I have no idea how to go about writing such a beast and I am hoping that
                      I could get some pointers or an example that could get me going in the
                      right direction.
                      >
                      Parsing, etc, is not a problem, but I'm not exactly sure how I might
                      interface python with Firefox, forwarding scraped pages to my browser
                      (or forwarding from the browser to the scraper).
                      >
                      Thanks in advance for any help or advice.
                      >
                      James
                      Thanks to everyone for your suggestions.

                      James

                        Comment

                        • tstrokes
                          #7
                          Re: Python Screen Scraper

                          On Apr 24, 6:17 am, Michael Bentley <mich...@jedimi ndworks.comwrot e:
                          On Apr 24, 2007, at 11:50 AM, James Stroud wrote:
                          >
                          >
                          >
                          Hello,
                          >
                          Does anyone know of an example, however modest, of a screenscraper
                          authored in python? I am using Firefox.
                          >
                          Basically, I am answering problems via my browser and being scored for
                          each problem. I have a tendency to go past my peak for training
                          efficiency, so I would like to scrape the result page for each
                          problem I
                          answer, compile statistics, and have a program alert me when I should
                          stop (based on score and accuracy--assuming training value is
                          related to
                          changes in these metrics).
                          >
                          I have no idea how to go about writing such a beast and I am hoping
                          that
                          I could get some pointers or an example that could get me going in the
                          right direction.
                          >
                          Parsing, etc, is not a problem, but I'm not exactly sure how I might
                          interface python with Firefox, forwarding scraped pages to my browser
                          (or forwarding from the browser to the scraper).
                          >
                          Thanks in advance for any help or advice.
                          >
                          Possibly the easiest thing will be to read from firefox' cache.
                          Otherwise I think your only real options are to either build a proxy
                          or sniff the wire...
                          You could try twill
                          http://twill.idyll.org/python-api.html


                            Comment

                            • skotjs
                              #8
                              Re: Python Screen Scraper

                              On Apr 24, 5:50 am, James Stroud <jstr...@mbi.uc la.eduwrote:
                              Hello,
                              >
                              Does anyone know of an example, however modest, of a screenscraper
                              authored in python? I am using Firefox.
                              >
                              Basically, I am answering problems via my browser and being scored for
                              each problem. I have a tendency to go past my peak for training
                              efficiency, so I would like to scrape the result page for each problem I
                              answer, compile statistics, and have a program alert me when I should
                              stop (based on score and accuracy--assuming training value is related to
                              changes in these metrics).
                              >
                              I have no idea how to go about writing such a beast and I am hoping that
                              I could get some pointers or an example that could get me going in the
                              right direction.
                              >
                              Parsing, etc, is not a problem, but I'm not exactly sure how I might
                              interface python with Firefox, forwarding scraped pages to my browser
                              (or forwarding from the browser to the scraper).
                              >
                              Thanks in advance for any help or advice.
                              >
                              James
                              Hey James,

                              This might be of interest to you: http://www.screen-scraper.com/suppor..._in_python.php

                              Scott

                                Comment

                                Previous template Next
                                Working...