ClientCookie bug

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • John J. Lee

    #16
    Re: ClientCookie bug (followup)

    Gary Feldman <gafStopSpamDat a@ziplink.stopa llspam.net> writes:
    [color=blue]
    > On 18 Aug 2003 14:43:06 +0100, jjl@pobox.com (John J. Lee) wrote:
    >[color=green]
    > >Hmm, good idea, but I really don't want to split the documentation up
    > >-- one page is simpler -- and the example is instructive for people[/color]
    >
    > Then definitely blockquote it (or indent it some other way), and consider
    > putting it into a smaller font, or using a grey background, or something
    > else to indicate that it's a digression.[/color]

    Again, that would be a good idea if it *were* a digression, but it's
    necessary for understanding what the module gets up to. Without that
    understanding in the reader's mind, it's hard to explain the code that
    one uses in practice if it's any more complicated than urlopen. And
    the very top of the page says:

    | import ClientCookie
    | response = ClientCookie.ur lopen("http://foo.bar.com/")
    |
    |This function behaves identically to urllib2.urlopen , except that it
    |deals with cookies automatically. That's probably all you need to
    |know.

    So you don't even have to read further than that for most purposes. I
    can't see how to improve on that, but I'm happy to learn how!

    [color=blue]
    > Orange would draw attention to
    > it; you want the opposite.[/color]

    Only the comment is in emacs-orange (well, my copy of python-mode uses
    that kind of rust-orange for Python comments), so it doesn't
    particularly draw attention to that block of code more than the rest.
    And I *do* want to draw attention to the comment, so people read the
    comment before the code.

    Admittedly, it doesn't seem to work ;-) (on a sample of one
    misinterpreter, Mark, so far -- I only just added that comment
    recently, though there are several other warnings that cover the same
    ground elsewhere).


    John

    Comment

    • Anand Pillai

      #17
      Re: ClientCookie bug (followup)

      I am working on a Cookie module which works *with* urllib2 rather
      than on top of it like the existing ClientCookie module. It uses
      the Cookie module which comes with python standard library.

      This module is written as an extension of my Harvestman webcrawler.
      The alpha code is ready. We are doing testing right now.

      Details will be posted to my website at
      http://members.lycos.co.uk/anandpillai within say 2 weeks or so.

      -Anand


      jjl@pobox.com (John J. Lee) wrote in message news:<87k79a25p i.fsf@pobox.com >...[color=blue]
      > cartermark46@uk mail.com (Mark Carter) writes:
      >[color=green][color=darkred]
      > > > I'll investigate further.[/color]
      > >
      > > Here are the results from running tests in ClientCookie 0.4.4.a:[/color]
      > [...][color=green]
      > > The upshot of this is that load_cookie_dat a() now works in win 98 and
      > > xp.
      > > load_from_regis try() works from win xp; it works from win 98 if and
      > > only if you set the USERNAME environment variable.[/color]
      >
      > You missed the username argument.
      >
      > cookiejar.load_ from_registry(u sername="mark")
      >
      > (should only be required for win9x family)
      >
      >[color=green]
      > > I appreciate that all the stuff about request2 and response2 may not
      > > be to your liking - but at the moment I'm just trying to figure out
      > > what works, and what doesn't. We can also worry about the delayload
      > > business later.[/color]
      >
      > No really, I wasn't joking: you *never* need to use add_cookie_head er
      > / extract_cookies if you're using urllib2 (at least, I can't think of
      > any possible reason to do so). It can only break things.
      >
      >[color=green]
      > > What do you think about the idea of actually setting up an Aapache web
      > > page to test these things 'for real'?[/color]
      >
      > I've done limited testing on Windows with 'fake' cookies from a local
      > Apache server, and on wine on linux. As I said, though, I don't have
      > a networked Windows OS, so it's inconvenient to test these things in a
      > 'real' situation. And my machine currently doesn't boot into Windows
      > without physically switching cables around (security & obscure
      > hardware issues, not software ones), which means I currently can't be
      > bothered to test it on Windows :-P. So, your feedback is appreciated.
      >
      >
      > John[/color]

      Comment

      • Mark Carter

        #18
        Re: ClientCookie bug (followup)

        > No really, I wasn't joking: you *never* need to use add_cookie_head er[color=blue]
        > / extract_cookies if you're using urllib2 (at least, I can't think of
        > any possible reason to do so). It can only break things.[/color]

        I must admit that I don't really know what I am doing. How would you
        simplify the following code:

        def go8():
        import ClientCookie
        c = ClientCookie.MS IECookieJar(del ayload=1)
        c.load_from_reg istry(username= 'mcarter') #only need username for
        win9x

        import urllib2
        url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
        request = urllib2.Request (url)
        response = urllib2.urlopen (request)
        request2 = urllib2.Request (url)
        c.add_cookie_he ader(request2)
        response2 = urllib2.urlopen (request2)

        print response2.getur l()
        print response2.info( ) # headers
        for line in response2.readl ines(): # body
        print line

        Comment

        • Anand Pillai

          #19
          Re: ClientCookie bug (followup)

          Hi John

          I wanted to add cookies support to harvestman, your module
          looked ideal for it.

          'We', nothing royal about it. It is just me and my friend
          & co-developer Nirmal Chidambaram. Apparently he has found
          a way around some of the bugs in Clientcookie. He has written
          a new module using the existing Cookie module of python &
          urllib2. One of the problems 'we' had with Clientcookie is that
          it uses its own 'urlopen' methods which does not fit our
          applications needs, so 'we' had to find a way around it.

          Once the code is ready, I will post it on my webpage, and
          of course it is not a module in itself, so I think an
          announcement to c.l.py is out of place.

          Regards

          -Anand


          jjl@pobox.com (John J. Lee) wrote in message news:<87ada5de3 j.fsf@pobox.com >...[color=blue]
          > pythonguy@Hotpo p.com (Anand Pillai) writes:
          >[color=green]
          > > I am working on a Cookie module which works *with* urllib2 rather
          > > than on top of it like the existing ClientCookie module. It uses
          > > the Cookie module which comes with python standard library.[/color]
          >
          > Interesting, though I don't know quite what you mean.
          >
          > First, if there's a way to work more closely with urllib2 than I've
          > figured out (which is quite possible), this patch needs to know about
          > it, so please post a comment:
          >
          > http://www.python.org/sf/759792
          >
          > If I understand what you mean, ClientCookie only works 'on top of'
          > rather than 'with' urllib2 to the extent that it currently has to
          > cut-n-paste code to add cookie handling to urllib2. That patch is
          > designed to remove the need to cut-n-paste, which would mean you'd do
          > urllib2.urlopen (after building an OpenerDirector that has an
          > HTTPCookieProce ssor from ClientCookie) instead of ClientCookie.ur lopen
          > as is required at present.
          >
          > Seco[/color]
          nd: is your module intended to do what ClientCookie does[color=blue]
          > (ie. figure out what cookies should be set and returned, and do so),
          > or is it just a more OO way of getting and returning Cookie headers?
          > I guess the latter?
          >
          >[color=green]
          > > This module is written as an extension of my Harvestman webcrawler.
          > > The alpha code is ready. We are doing testing right now.[/color]
          >
          > Is this the Royal We? ;-)
          >
          >[color=green]
          > > Details will be posted to my website at
          > > http://members.lycos.co.uk/anandpillai within say 2 weeks or so.[/color]
          > [...]
          >
          > Please do post an announcement to c.l.py.announce , or I'll forget.
          >
          >
          > John[/color]

          Comment

          • John J. Lee

            #20
            Re: ClientCookie bug (followup)

            pythonguy@Hotpo p.com (Anand Pillai) writes:
            [...][color=blue]
            > 'We', nothing royal about it. It is just me and my friend
            > & co-developer Nirmal Chidambaram. Apparently he has found
            > a way around some of the bugs in Clientcookie. He has written[/color]

            It'd be great if you made me aware what those bugs are!

            (BTW, no intent to offend with my comment about your plurality, or
            lack thereof -- it's just that the convention of using 'we' in source
            code comments is common enough that I've sometimes found myself using
            it even when writing code alone, which is funny.)

            [color=blue]
            > a new module using the existing Cookie module of python &
            > urllib2. One of the problems 'we' had with Clientcookie is that
            > it uses its own 'urlopen' methods which does not fit our
            > applications needs, so 'we' had to find a way around it.[/color]

            As I said before, if you know how to do that, please comment on the
            RFE I referenced in my last post. Jeremy Hylton is planning to look
            at the patch associated with that RFE in detail sometime, and you
            could save him some time if you know a way to do this without patching
            urllib2. And I'd like to know how to do it, too :-)

            [color=blue]
            > Once the code is ready, I will post it on my webpage, and
            > of course it is not a module in itself, so I think an
            > announcement to c.l.py is out of place.[/color]
            [...]

            Would you mind sending me an email?

            Thanks


            John

            Comment

            • Anand Pillai

              #21
              Re: ClientCookie bug (followup)

              In fact, I am not aware of how Nirmal has done it.
              He is wrapping up the coding soon, when I will get to
              see how he has done it. He is doing the persistency part
              of the cookies, and I guess he is using anydbm for it. Details
              later.

              I will send a mail to you, once I get the code from him
              and review it. Meanwhile you can visit his homepage here

              บาคาร่า ออนไลน์ FZ888 เว็บตรง ฝากถอนออโต้ ไม่มีขั้นต่ำ สมัครบาคาร่า 24 ชม คาสิโน รวมค่ายชั้นนำ แทงบาคาร่า เว็บบาคาร่า 2025


              There might be some details on the work there.

              Regards

              -Anand

              jjl@pobox.com (John J. Lee) wrote in message news:<87smnwh3l p.fsf@pobox.com >...[color=blue]
              > pythonguy@Hotpo p.com (Anand Pillai) writes:
              > [...][color=green]
              > > 'We', nothing royal about it. It is just me and my friend
              > > & co-developer Nirmal Chidambaram. Apparently he has found
              > > a way around some of the bugs in Clientcookie. He has written[/color]
              >
              > It'd be great if you made me aware what those bugs are!
              >
              > (BTW, no intent to offend with my comment about your plurality, or
              > lack thereof -- it's just that the convention of using 'we' in source
              > code comments is common enough that I've sometimes found myself using
              > it even when writing code alone, which is funny.)
              >
              >[color=green]
              > > a new module using the existing Cookie module of python &
              > > urllib2. One of the problems 'we' had with Clientcookie is that
              > > it uses its own 'urlopen' methods which does not fit our
              > > applications needs, so 'we' had to find a way around it.[/color]
              >
              > As I said before, if you know how to do that, please comment on the
              > RFE I referenced in my last post. Jeremy Hylton is planning to look
              > at the patch associated with that RFE in detail sometime, and you
              > could save him some time if you know a way to do this without patching
              > urllib2. And I'd like to know how to do it, too :-)
              >
              >[color=green]
              > > Once the code is ready, I will post it on my webpage, and
              > > of course it is not a module in itself, so I think an
              > > announcement to c.l.py is out of place.[/color]
              > [...]
              >
              > Would you mind sending me an email?
              >
              > Thanks
              >
              >
              > John[/color]

              Comment

              • John J. Lee

                #22
                Re: ClientCookie bug (followup)

                cartermark46@uk mail.com (Mark Carter) writes:
                [color=blue][color=green]
                > > No really, I wasn't joking: you *never* need to use add_cookie_head er
                > > / extract_cookies if you're using urllib2 (at least, I can't think of
                > > any possible reason to do so). It can only break things.[/color]
                >
                > I must admit that I don't really know what I am doing. How would you
                > simplify the following code:[/color]
                [...]

                It doesn't tend to encourage people to help you if you don't even
                *try* to apply their advice, and just expect them to do all your work
                for you. Read that paragraph you quoted (and the rest I posted in
                that message and previously), try doing what it suggests, then come
                back here if you need to.


                John

                Comment

                • Robin Munn

                  #23
                  Re: ClientCookie bug (followup)

                  John J. Lee <jjl@pobox.co m> wrote:[color=blue]
                  > pythonguy@Hotpo p.com (Anand Pillai) writes:
                  > [...][color=green]
                  >> 'We', nothing royal about it. It is just me and my friend
                  >> & co-developer Nirmal Chidambaram. Apparently he has found
                  >> a way around some of the bugs in Clientcookie. He has written[/color]
                  >
                  > It'd be great if you made me aware what those bugs are!
                  >
                  > (BTW, no intent to offend with my comment about your plurality, or
                  > lack thereof -- it's just that the convention of using 'we' in source
                  > code comments is common enough that I've sometimes found myself using
                  > it even when writing code alone, which is funny.)[/color]

                  I do that all the time; "we" is the default in my code comments and I
                  have rarely, I think never, used "I". I usually think of it as myself
                  reading through the code with another coder, who is tasked with taking
                  over code maintenance from me. I'm explaining to him what the code is
                  intended to do: "So here, we want to search through the list for..."

                  --
                  Robin Munn <rmunn@pobox.co m> | http://www.rmunn.com/ | PGP key 0x6AFB6838
                  -----------------------------+-----------------------+----------------------
                  "Remember, when it comes to commercial TV, the program is not the product.
                  YOU are the product, and the advertiser is the customer." - Mark W. Schumann

                  Comment

                  • John J. Lee

                    #24
                    Re: ClientCookie bug (followup)

                    Robin Munn <rmunn@pobox.co m> writes:
                    [color=blue]
                    > John J. Lee <jjl@pobox.co m> wrote:[/color]
                    [...][color=blue][color=green]
                    > > lack thereof -- it's just that the convention of using 'we' in source
                    > > code comments is common enough that I've sometimes found myself using
                    > > it even when writing code alone, which is funny.)[/color]
                    >
                    > I do that all the time; "we" is the default in my code comments and I
                    > have rarely, I think never, used "I". I usually think of it as myself
                    > reading through the code with another coder, who is tasked with taking
                    > over code maintenance from me. I'm explaining to him what the code is
                    > intended to do: "So here, we want to search through the list for..."[/color]

                    That's a good excuse. :-)


                    John

                    Comment

                    • Mark Carter

                      #25
                      Re: ClientCookie bug (followup)

                      jjl@pobox.com (John J. Lee) wrote in message news:<87isos0zn g.fsf@pobox.com >...[color=blue]
                      > cartermark46@uk mail.com (Mark Carter) writes:
                      >[color=green][color=darkred]
                      > > > No really, I wasn't joking: you *never* need to use add_cookie_head er
                      > > > / extract_cookies if you're using urllib2 (at least, I can't think of
                      > > > any possible reason to do so). It can only break things.[/color]
                      > >
                      > > I must admit that I don't really know what I am doing. How would you
                      > > simplify the following code:[/color]
                      > [...]
                      >
                      > It doesn't tend to encourage people to help you if you don't even
                      > *try* to apply their advice, and just expect them to do all your work
                      > for you. Read that paragraph you quoted (and the rest I posted in
                      > that message and previously), try doing what it suggests, then come
                      > back here if you need to.[/color]

                      I've looked again, and add_cookie_head er() definitely(!) is required
                      for the following code to work:

                      def go10():
                      #works with win xp
                      import ClientCookie
                      c = ClientCookie.MS IECookieJar(del ayload=1)
                      c.load_from_reg istry(username= 'mcarter') #only need username for
                      win9x

                      import urllib2
                      url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
                      request = urllib2.Request (url)
                      response = urllib2.urlopen (request)
                      request2 = urllib2.Request (url)
                      c.add_cookie_he ader(request2)
                      response2 = urllib2.urlopen (request2)

                      print response2.getur l()
                      print response2.info( ) # headers
                      for line in response2.readl ines(): # body
                      print line

                      If I take out add_cookie_head er(), then the wrong page (i.e. the
                      registration page) is returned.

                      I will investigate further and let you know what I find.

                      Comment

                      • John J. Lee

                        #26
                        Re: ClientCookie bug (followup)

                        cartermark46@uk mail.com (Mark Carter) writes:
                        [color=blue]
                        > jjl@pobox.com (John J. Lee) wrote in message news:<87isos0zn g.fsf@pobox.com >...[color=green]
                        > > cartermark46@uk mail.com (Mark Carter) writes:[/color][/color]
                        [...][color=blue]
                        > I've looked again, and add_cookie_head er() definitely(!) is required
                        > for the following code to work:[/color]

                        Oh, sorry, I missed something: you're using urllib.urlopen. You need
                        to call ClientCookie.ur lopen instead (this is still using urllib2 --
                        ClientCookie uses urllib2 to do most of the work).

                        You seem to be opening the same URL twice for no obvious reason, so I
                        removed that below:


                        def go10():
                        #works with win xp
                        import ClientCookie
                        c = ClientCookie.MS IECookieJar(del ayload=1)
                        c.load_from_reg istry(username= 'mcarter') #only need username for win9x

                        url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
                        response = ClientCookie.ur lopen(url)

                        print response.info() # headers
                        print response.read() # body


                        John

                        Comment

                        • Mark Carter

                          #27
                          Re: ClientCookie bug (followup)

                          > Oh, sorry, I missed something: you're using urllib.urlopen. You need[color=blue]
                          > to call ClientCookie.ur lopen instead (this is still using urllib2 --
                          > ClientCookie uses urllib2 to do most of the work).
                          >
                          > You seem to be opening the same URL twice for no obvious reason, so I
                          > removed that below:
                          >
                          >
                          > def go10():
                          > #works with win xp
                          > import ClientCookie
                          > c = ClientCookie.MS IECookieJar(del ayload=1)
                          > c.load_from_reg istry(username= 'mcarter') #only need username for win9x
                          >
                          > url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
                          > response = ClientCookie.ur lopen(url)
                          >
                          > print response.info() # headers
                          > print response.read() # body[/color]


                          At the risk of sounding thick ... this doesn't work either! I couldn't
                          get load_cookie_dat a() to work using the above template, either. What
                          seems odd for starters is the line
                          response = ClientCookie.ur lopen(url)

                          What makes it odd is that to an offhand observer, there is nothing
                          visibly connecting the urlopen() command to c. Sure, some fancy stuff
                          may be going on under the hood; but like I say, ClientCookie.ur lopen()
                          doesn't seem connected with the MSIECookieJar instance just by looking
                          at it. It looks like I really want to be saying something like
                          response = c.urlopen(url)
                          but that just gives me
                          AttributeError: MSIECookieJar instance has no attribute 'urlopen'

                          Comment

                          • John J. Lee

                            #28
                            Re: ClientCookie bug (followup)

                            cartermark46@uk mail.com (Mark Carter) writes:
                            [...][color=blue]
                            > At the risk of sounding thick ... this doesn't work either! I couldn't
                            > get load_cookie_dat a() to work using the above template, either. What
                            > seems odd for starters is the line
                            > response = ClientCookie.ur lopen(url)
                            >
                            > What makes it odd is that to an offhand observer, there is nothing
                            > visibly connecting the urlopen() command to c. Sure, some fancy stuff[/color]
                            [...]

                            D'Oh!

                            def go10():
                            #works with win xp
                            import ClientCookie
                            c = ClientCookie.MS IECookieJar(del ayload=1)
                            c.load_from_reg istry(username= 'mcarter') #only need username for win9x
                            opener = ClientCookie.bu ild_opener(Clie ntCookie.HTTPCo okieProcessor(c ))

                            url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
                            response = opener.open(url )

                            print response.info() # headers
                            print response.read() # body


                            All of this is explained in the web page, though. [attempts to looks
                            at web page] Well, it would be if the link weren't broken or I could
                            log in to my SF account to fix it... Sigh.

                            BTW, if you want to use ClientCookie.ur lopen rather than opener.open,
                            you can call ClientCookie.in stall_opener(op ener) to install a global
                            opener, just as you would with urllib2.


                            John

                            Comment

                            • John J. Lee

                              #29
                              Re: ClientCookie bug (followup)

                              cartermark46@uk mail.com (Mark Carter) writes:

                              [...][color=blue][color=green]
                              > > All of this is explained in the web page, though.[/color]
                              >
                              > You are right! All the technical information is there. But can I
                              > suggest a change in the docs that might spell it out a bit more
                              > obviously to someone coming to the module from fresh? Something
                              > like:[/color]

                              Certainly!


                              [...][color=blue]
                              > h2: Using Internet Explorer or Netscape/Mozilla cookies
                              >
                              > c = ClientCookie.MS IECookieJar(del ayload=1) # see note 1
                              > c.load_from_reg istry(username= 'joe bloggs') # see note 2
                              > opener = ClientCookie.bu ild_opener(Clie ntCookie.HTTPCo okieProcessor(c ))
                              >
                              > url = 'http://foo.bar.com/'
                              > response = opener.open(url )
                              >
                              > print response.info() # headers
                              > print response.read() # body[/color]

                              Possibly, but there's a balance to be struck between convenience and
                              documentation bloat. Since "Cooperatin g with Netscape/Mozilla and
                              Internet Explorer" explains the CookieJar part, and the very next
                              section is "Using your own CookieJar instance", it seems to flow quite
                              well to me. The first sentence of that section is kind of misleading
                              though, so I'll change that.

                              [...][color=blue]
                              > 2. Usually only win98 and 95 users need supply the username. As an
                              > alternative
                              > to loading from the registry, consider using the line
                              > c.load_cookie_d ata('cookie-file.txt')
                              > instead.[/color]

                              Thanks -- worthwhile addition.

                              [...][color=blue]
                              > should be at the top, where people will see it first, and before
                              > those more complicated examples.[/color]

                              It is at the top -- after an example of the simplest usages and an
                              explanation of what CookieJar is actually for, which I think have to
                              come first.


                              John

                              Comment

                              Working...