ClientCookie bug

[Mark Carter]

I am using Windows 98, python 2.3, ClientCookie 0.4.3a.

When I do:
import ClientCookie
import os
c = ClientCookie.MS IECookieJar(del ayload=1)
c.load_from_reg istry()

I get the response:
Traceback (most recent call last):
File "C:\My Documents\mycoo kies.py", line 33, in ?
c.load_from_reg istry()
File "C:\PYTHON23\Li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
line 230, in load_from_regis try
self.load(filen ame, ignore_discard, ignore_expires)
File "C:\PYTHON23\Li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
line 245, in load
self._really_lo ad(index, filename, ignore_discard, ignore_expires)
File "C:\PYTHON23\Li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
line 258, in _really_load
user_name = string.lower(os .environ['USERNAME'])
File "C:\PYTHON23\Li b\os.py", line 417, in __getitem__
return self.data[key.upper()]
KeyError: 'USERNAME'

Basically, it doesn't like USERNAME as an environment variable.

[Syver Enstad]

Re: ClientCookie bug

cartermark46@uk mail.com (Mark Carter) writes:
[color=blue]
> I am using Windows 98, python 2.3, ClientCookie 0.4.3a.
>
> When I do:
> import ClientCookie
> import os
> c = ClientCookie.MS IECookieJar(del ayload=1)
> c.load_from_reg istry()
>
> I get the response:
> Traceback (most recent call last):
> File "C:\My Documents\mycoo kies.py", line 33, in ?
> c.load_from_reg istry()
> File "C:\PYTHON23\Li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
> line 230, in load_from_regis try
> self.load(filen ame, ignore_discard, ignore_expires)
> File "C:\PYTHON23\Li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
> line 245, in load
> self._really_lo ad(index, filename, ignore_discard, ignore_expires)
> File "C:\PYTHON23\Li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
> line 258, in _really_load
> user_name = string.lower(os .environ['USERNAME'])
> File "C:\PYTHON23\Li b\os.py", line 417, in __getitem__
> return self.data[key.upper()]
> KeyError: 'USERNAME'
>
> Basically, it doesn't like USERNAME as an environment variable.[/color]

I suspect this only works on "real" win32 (the NT family) operating
systems, which define USERNAME, and not hybrid dos/win32 systems like
the win9x family.

[John J. Lee]

Re: ClientCookie bug

Syver Enstad <syver@inout.no > writes:
[color=blue]
> cartermark46@uk mail.com (Mark Carter) writes:
>[color=green]
> > I am using Windows 98, python 2.3, ClientCookie 0.4.3a.
> >
> > When I do:
> > import ClientCookie
> > import os
> > c = ClientCookie.MS IECookieJar(del ayload=1)
> > c.load_from_reg istry()
> >
> > I get the response:[/color][/color]
[...][color=blue][color=green]
> > KeyError: 'USERNAME'
> >
> > Basically, it doesn't like USERNAME as an environment variable.[/color]
>
> I suspect this only works on "real" win32 (the NT family) operating
> systems, which define USERNAME, and not hybrid dos/win32 systems like
> the win9x family.[/color]

Anybody know the best way to get the username without win32all
installed?

Mark: I don't have a win9x box, but try replacing that
os.environ['USERNAME'] with getpass.getuser (). You'll need to stick
an import getpass in there too, of course. Let me know if that works.

The rest of the code should work OK on win9x.


John

[Mark Carter]

Re: ClientCookie bug (fix offered)

> > I am using Windows 98, python 2.3, ClientCookie 0.4.3a.[color=blue][color=green]
> >
> > When I do:
> > c.load_from_reg istry()
> >
> > I get the response:
> > KeyError: 'USERNAME'[/color][/color]
[color=blue]
> I suspect this only works on "real" win32 (the NT family) operating[/color]

I have a fix/workaround for this - and would like it included in ClientCookie.
It my problems out real nice! Interested?

BTW, I can't seem to locate the project on Sourceforge.

[Mark Carter]

Re: ClientCookie bug

jjl@pobox.com (John J. Lee) wrote in message news:<87fzk4sav 2.fsf@pobox.com >...[color=blue]
> Syver Enstad <syver@inout.no > writes:
>[color=green]
> > cartermark46@uk mail.com (Mark Carter) writes:
> >[color=darkred]
> > > I am using Windows 98, python 2.3, ClientCookie 0.4.3a.
> > >
> > > When I do:
> > > import ClientCookie
> > > import os
> > > c = ClientCookie.MS IECookieJar(del ayload=1)
> > > c.load_from_reg istry()
> > >
> > > I get the response:[/color][/color]
> [...][color=green][color=darkred]
> > > KeyError: 'USERNAME'
> > >
> > > Basically, it doesn't like USERNAME as an environment variable.[/color]
> >
> > I suspect this only works on "real" win32 (the NT family) operating
> > systems, which define USERNAME, and not hybrid dos/win32 systems like
> > the win9x family.[/color]
>
> Anybody know the best way to get the username without win32all
> installed?
>
> Mark: I don't have a win9x box, but try replacing that
> os.environ['USERNAME'] with getpass.getuser (). You'll need to stick
> an import getpass in there too, of course. Let me know if that works.
>
> The rest of the code should work OK on win9x.
>
>
> John[/color]

My apologies - the bug exists in my own brain, not in the code. I had in mind
that a fix would be required in ClientCookie - but now I know better.

To get it to work, I simply called:
os.environ['USERNAME'] = 'mark carter' # or whatever your name is
before calling
c.load_from_reg istry() # c is a MSIECookieJar

It's an embarassingly simple solution to the problem - but it took me a while
to figure it out. Although there is nothing wrong with code, perhaps it would
help new users of the library to mention it. Maybe the solution is somewhat
obvious in retrospect - but it took me a while before the penny dropped.

Another approach - which I actually prefer - is to copy the cookie to the local
directory and load it using load_cookie_dat a(). At first, I was having no end
of problems with it - until I discovered that you should use the (binary)
cookie, not an ASCII cookie that you obtain from performing a cookie export
from MSIE. ... and don't forget to call ClientCookie.MS IECookieJar() WITHOUT
the delayload argument.

So, sorry for the false alarm - but hope my investigations will prove useful to
others.

[John J. Lee]

Re: ClientCookie bug

cartermark46@uk mail.com (Mark Carter) writes:
[color=blue]
> jjl@pobox.com (John J. Lee) wrote in message news:<87fzk4sav 2.fsf@pobox.com >...[/color]
[...][color=blue]
> My apologies - the bug exists in my own brain, not in the code. I had in mind[/color]

Your brain is fine, Mark. It's a real bug.

[color=blue]
> that a fix would be required in ClientCookie - but now I know better.
>
> To get it to work, I simply called:
> os.environ['USERNAME'] = 'mark carter' # or whatever your name is
> before calling
> c.load_from_reg istry() # c is a MSIECookieJar[/color]

That's a workaround, but it shouldn't be necessary. Would you mind
trying the getpass fix I suggested? I don't know if getpass.getuser
works on w9x, but it's worth a shot.


[...][color=blue]
> Another approach - which I actually prefer - is to copy the cookie to the local
> directory and load it using load_cookie_dat a(). At first, I was having no end[/color]

Should be no need for that.

[color=blue]
> of problems with it - until I discovered that you should use the (binary)
> cookie, not an ASCII cookie that you obtain from performing a cookie export
> from MSIE. ... and don't forget to call ClientCookie.MS IECookieJar() WITHOUT
> the delayload argument.[/color]
[...]

Why without the delayload argument? There are features that don't yet
work with that, but that's documented. Is something else not working?


John

[JanC]

Re: ClientCookie bug (fix offered)

cartermark46@uk mail.com (Mark Carter) schreef:
[color=blue]
> BTW, I can't seem to locate the project on Sourceforge.[/color]

<http://wwwsearch.sourc eforge.net/>

--
JanC

"Be strict when sending and tolerant when receiving."
RFC 1958 - Architectural Principles of the Internet - section 3.9

[Mark Carter]

Re: ClientCookie bug (followup)

> I'll try it on a win 98 machine, and report back the results asap.

---
USERNAME

I promised that I would try getuser() on a win 98 machine. Alas,
typing:

import getpass
print getpass.getuser ()

produces:

ImportError: No module named pwd

So, it's the same problem as occurs on win95 (unsuprisingly)

---
DELAYLOAD

John: Why without the delayload argument?

My response:

The following code:

import ClientCookie
c = ClientCookie.MS IECookieJar(del ayload=1)
c.load_cookie_d ata("hemscott-cookie.bin")
import urllib2
url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
request = urllib2.Request (url)
response = urllib2.urlopen (request)
request2 = urllib2.Request (url)
c.add_cookie_he ader(request2)
response2 = urllib2.urlopen (request2)
print response2.getur l()
print response2.info( ) # headers
for line in response2.readl ines(): # body
print line


produces the error:

Traceback (most recent call last):
File "C:\My Documents\markc \ClientCookie\c lp04.py", line 9, in ?
c.add_cookie_he ader(request2)
File "C:\PYTHON23\Li b\site-packages\Client Cookie\_ClientC ookie.py",
line 1170, in add_cookie_head er
cookies.extend( self._get_cooki es_for_domain(
File "C:\PYTHON23\Li b\site-packages\Client Cookie\_ClientC ookie.py",
line 1050, in _get_cookies_fo r_domain
return self._cookies_f or_domain(domai n, request, unverifiable)
File "C:\PYTHON23\li b\site-packages\Client Cookie\_MSIECoo kieJar.py",
line 112, in _cookies_for_do main
if self.delayload and cookies["//+delayload"] is not None:
KeyError: '//+delayload'

whereas running it without delayload=1 causes it to run successfully.

[John J. Lee]

Re: ClientCookie bug (followup)

cartermark46@uk mail.com (Mark Carter) writes:
[...][color=blue]
> John: Why without the delayload argument?
>
> My response:
>
> The following code:
>
> import ClientCookie
> c = ClientCookie.MS IECookieJar(del ayload=1)
> c.load_cookie_d ata("hemscott-cookie.bin")
> import urllib2
> url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
> request = urllib2.Request (url)
> response = urllib2.urlopen (request)
> request2 = urllib2.Request (url)
> c.add_cookie_he ader(request2)
> response2 = urllib2.urlopen (request2)
> print response2.getur l()
> print response2.info( ) # headers
> for line in response2.readl ines(): # body
> print line[/color]

Take note of this comment from the web page:

| # Don't copy this blindly! You probably want to follow the examples
| # above, not this one.

No matter how many times and how many places I say this, everybody
seems to be driven to make the same mistakes. I guess people don't
believe it's as simple as calling urlopen, though I explictly say it
*is* that simple in most cases in the second sentence of the
documentation.

DON'T USE *BOTH* add_cookie_head er/extract_cookies *AND* urlopen.
urlopen is all you need, unless you're not using urllib2. Probably
won't do any actual harm, but it's completely pointless and
obfuscatory. You're also failing to use ClientCookie.ur lopen (which
unlike urllib2.urlopen , knows about cookies), because you blindly
copied the third example from the web page. You're also using Request
objects for no apparent reason, and you're attempting to fetch the
same url twice, again because you're blindly copying. Sigh -- if
people have to blindly copy, why not copy from the code I actually
*tell* people to copy from, instead of the code I explictly tell
people *not* to copy from? Ngghhh! :-)

You want something like this:

import ClientCookie
c = ClientCookie.MS IECookieJar(del ayload=1)
c.load_cookie_d ata("hemscott-cookie.bin")
url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
response = ClientCookie.ur lopen(url)

print response.read()
response.close( )

[color=blue]
> produces the error:
>
> Traceback (most recent call last):[/color]
[...][color=blue]
> whereas running it without delayload=1 causes it to run successfully.[/color]

Anyway, you have found another bug, so all is forgiven.

(the MSIE delayload feature is still a bit of a mess -- luckily, I can
pin the blame on the original author of that code, since it's pretty
directly ported from Perl ;)


John

[Mark Carter]

Re: ClientCookie bug (followup)

> *tell* people to copy from, instead of the code I explictly tell[color=blue]
> people *not* to copy from? Ngghhh! :-)[/color]

[color=blue]
> Anyway, you have found another bug, so all is forgiven.[/color]
[color=blue]
> (the MSIE delayload feature is still a bit of a mess -- luckily, I can
> pin the blame on the original author of that code, since it's pretty
> directly ported from Perl ;)[/color]


If we shadows have offended,
Think but this, and all is mended-
That you have but slumb'red here
While these visions did appear.
And this weak and idle theme,
No more yielding but a dream,
Gentles, do not reprehend.
If you pardon, we will mend.
And, as I am an honest Puck,
If we have unearned luck
Now to scape the serpent's tongue,
We will make amends ere long;
Else the Puck a liar call.
So, good night unto you all.
Give me your hands, if we be friends,
And Robin shall restore amends.


But seriously ... please standby and I will inform you of the outcome
of your changes.

[Mark Carter]

Re: ClientCookie bug (followup)

> You want something like this:[color=blue]
>
> import ClientCookie
> c = ClientCookie.MS IECookieJar(del ayload=1)
> c.load_cookie_d ata("hemscott-cookie.bin")
> url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
> response = ClientCookie.ur lopen(url)
>
> print response.read()
> response.close( )[/color]


It doesn't work on my XP machine at least. This is proboably why
people have been doing it all wrong, allegedly.

I'll investigate further. Apologies for the time lag.

[John J. Lee]

Re: ClientCookie bug (followup)

cartermark46@uk mail.com (Mark Carter) writes:
[color=blue][color=green]
> > You want something like this:
> >
> > import ClientCookie
> > c = ClientCookie.MS IECookieJar(del ayload=1)
> > c.load_cookie_d ata("hemscott-cookie.bin")
> > url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
> > response = ClientCookie.ur lopen(url)
> >
> > print response.read()
> > response.close( )[/color]
>
>
> It doesn't work on my XP machine at least. This is proboably why
> people have been doing it all wrong, allegedly.[/color]

I don't see how -- the examples in question (the one everybody seems
driven to copy from, and the ones people should copy from) does not
include any reference to any CookieJar.

[color=blue]
> I'll investigate further. Apologies for the time lag.[/color]

Thanks


John

[John J. Lee]

Re: ClientCookie bug (followup)

Gary Feldman <gafStopSpamDat a@ziplink.stopa llspam.net> writes:
[color=blue]
> On 16 Aug 2003 11:33:13 +0100, jjl@pobox.com (John J. Lee) wrote:
>[color=green]
> >
> >Take note of this comment from the web page:
> >
> >| # Don't copy this blindly! You probably want to follow the examples
> >| # above, not this one.[/color]
>
> Since the purpose of that example seems to be to show how things work under
> the hood, may I suggest putting it on a separate page, and replacing it
> here with something like "Here is a <a href...>lower level example</a> that
> shows how this works, though you would rarely want to implement things at
> this low level."[/color]

Hmm, good idea, but I really don't want to split the documentation up
-- one page is simpler -- and the example is instructive for people
who actually want to understand what the module does. And if people
manage to miss that comment (highlighted in orange, for heavens sake),
well...


John

[Mark Carter]

Re: ClientCookie bug (followup)

> I'll investigate further.

Here are the results from running tests in ClientCookie 0.4.4.a:

def go7():
#ClientCookie 0.4.4.a:
#works from win xp and win 98

# I prefer this method as a better way than using
load_from_regis try()
#use this method!!

import ClientCookie
c = ClientCookie.MS IECookieJar() # do NOT set delayload
#c.user_name = "mark carter"
c.load_cookie_d ata("hemscott-cookie.bin")
#c.load_from_re gistry()
print c

import urllib2
url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
request = urllib2.Request (url)
response = urllib2.urlopen (request)
c.extract_cooki es(response, request)
#let's say this next request requires a cookie that was set in
response
request2 = urllib2.Request (url)
c.add_cookie_he ader(request2)
response2 = urllib2.urlopen (request2)

print response2.getur l()
print response2.info( ) # headers
print response2.read( )
response2.close ()



def go8():
#ClientCookie 0.4.4.a:
#contains bug in Win98 when environ variable is commented out,
#but works in win98 when environ variable is set
#Works in win xp , regardless of the USERNAME line

#this works - I can now import into Hemscott
import ClientCookie
c = ClientCookie.MS IECookieJar(del ayload=1)
#c.user_name = "mark carter"
#os.environ['USERNAME'] = 'mcarter' #needed by
load_from_regis try()
c.load_from_reg istry()

import urllib2
url = 'http://businessplus.he mscott.net/corp/crp03733.htm'
request = urllib2.Request (url)
response = urllib2.urlopen (request)
#c.extract_cook ies(response, request)
# let's say this next request requires a cookie that was set in
response
request2 = urllib2.Request (url)
c.add_cookie_he ader(request2)
response2 = urllib2.urlopen (request2)

print response2.getur l()
print response2.info( ) # headers
for line in response2.readl ines(): # body
print line


The upshot of this is that load_cookie_dat a() now works in win 98 and
xp.
load_from_regis try() works from win xp; it works from win 98 if and
only if you set the USERNAME environment variable.

I appreciate that all the stuff about request2 and response2 may not
be to your liking - but at the moment I'm just trying to figure out
what works, and what doesn't. We can also worry about the delayload
business later.

What do you think about the idea of actually setting up an Aapache web
page to test these things 'for real'?

[Gary Feldman]

Re: ClientCookie bug (followup)

On 18 Aug 2003 14:43:06 +0100, jjl@pobox.com (John J. Lee) wrote:
[color=blue]
>Hmm, good idea, but I really don't want to split the documentation up
>-- one page is simpler -- and the example is instructive for people[/color]

Then definitely blockquote it (or indent it some other way), and consider
putting it into a smaller font, or using a grey background, or something
else to indicate that it's a digression. Orange would draw attention to
it; you want the opposite.

Gary