not able to HTTPS page from python

[muttu2244@yahoo.com]

Hi all,

Am trying to read a email ids which will be in the form of links ( on
which if we click, they will redirect to outlook with their respective
email ids).

And these links are in the HTTPS page, a secured http page.

The point is that am able to read some links with HTTP page, but am not
able to read the same when I try with HTTPS.



Using the following code from sgmllib am able to read the links,





class MyParser(sgmlli b.SGMLParser):

def __init__(self):

sgmllib.SGMLPar ser.__init__(se lf)

self.inside_a = False

self.address = ''

def start_a(self,at trs):

if DEBUG:

print "start_a"

print attrs

for attr,value in attrs:

if attr == 'href' and value.startswit h('mailto:'):

self.address = value[7:]

self.inside_a = True

def end_a(self):

if DEBUG:

print "end_a"

if self.address:

print '"%s" <%s>' % (self.nickname, self.address)

mailIdList.appe nd(self.address )



self.inside_a = False

self.address = self.nickname = ''

def handle_data(sel f,data):

if self.inside_a:

self.nickname = data





And for the proxy authentication and the https handler am using the
following lines of code





authinfo = urllib2.HTTPBas icAuthHandler()



proxy_support = urllib2.ProxyHa ndler ({"http" :
"http://user:password@p roxyname:port"} )



opener = urllib2.build_o pener(proxy_sup port, authinfo,
urllib2.HTTPSHa ndler)



urllib2.install _opener(opener)





Then am trying to call the parser for the links in a particular https
page which will be given as a command line argument. Which will read me
all the links in that page.



p = MyParser()

for ln in urllib2.urlopen ( sys.argv[1] ):

p.feed(ln)

p.close()





NOTE : I have installed python with _ssl support also.





So with this code am able to read the links with HTTP page but not for
the HTTPS page.

AM NOT GETTING ANY ERRORS EITHER BUT ITS NOT READING THE LINKS, THAT
ARE PRESENT IN THE GIVEN HTTPS PAGE



Could you please tell me am I doing some thing wrong in the above code
for any of the handlers.





I have got struck here from so many days, please give me the solution
for this.



Thanks and regards

YOGI

[Fredrik Lundh]

Re: not able to HTTPS page from python

<muttu2244@yaho o.com> wrote:
[color=blue]
> AM NOT GETTING ANY ERRORS EITHER BUT ITS NOT READING THE LINKS, THAT
> ARE PRESENT IN THE GIVEN HTTPS PAGE[/color]

HAVE YOU TRIED ADDING A PRINT STATEMENT TO THE FEED LOOP SO
YOU CAN SEE WHAT YOU'RE GETTING BACK FROM THE SERVER ?

</f>

[Steve Holden]

Re: not able to HTTPS page from python

Fredrik Lundh wrote:[color=blue]
> <muttu2244@yaho o.com> wrote:
>
>[color=green]
>>AM NOT GETTING ANY ERRORS EITHER BUT ITS NOT READING THE LINKS, THAT
>>ARE PRESENT IN THE GIVEN HTTPS PAGE[/color]
>
>
> HAVE YOU TRIED ADDING A PRINT STATEMENT TO THE FEED LOOP SO
> YOU CAN SEE WHAT YOU'RE GETTING BACK FROM THE SERVER ?
>[/color]
COULD YOU GUYS BE QUIET, PLEASE, I'M TRYING TO WORK HERE!

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Larry Bates]

Re: not able to HTTPS page from python

It is possible that the links have been obscured (something
I do on my own web pages) by inserting Javascript that creates
the links on the fly using document.write( ). That way web
spiders can't go through the web pages and easily pick up email
addresses to send spam to all my employees. Just a thought
since you have spent days on this.

-Larry Bates


muttu2244@yahoo .com wrote:[color=blue]
> Hi all,
>
> Am trying to read a email ids which will be in the form of links ( on
> which if we click, they will redirect to outlook with their respective
> email ids).
>
> And these links are in the HTTPS page, a secured http page.
>
> The point is that am able to read some links with HTTP page, but am not
> able to read the same when I try with HTTPS.
>
>
>
> Using the following code from sgmllib am able to read the links,
>
>
>
>
>
> class MyParser(sgmlli b.SGMLParser):
>
> def __init__(self):
>
> sgmllib.SGMLPar ser.__init__(se lf)
>
> self.inside_a = False
>
> self.address = ''
>
> def start_a(self,at trs):
>
> if DEBUG:
>
> print "start_a"
>
> print attrs
>
> for attr,value in attrs:
>
> if attr == 'href' and value.startswit h('mailto:'):
>
> self.address = value[7:]
>
> self.inside_a = True
>
> def end_a(self):
>
> if DEBUG:
>
> print "end_a"
>
> if self.address:
>
> print '"%s" <%s>' % (self.nickname, self.address)
>
> mailIdList.appe nd(self.address )
>
>
>
> self.inside_a = False
>
> self.address = self.nickname = ''
>
> def handle_data(sel f,data):
>
> if self.inside_a:
>
> self.nickname = data
>
>
>
>
>
> And for the proxy authentication and the https handler am using the
> following lines of code
>
>
>
>
>
> authinfo = urllib2.HTTPBas icAuthHandler()
>
>
>
> proxy_support = urllib2.ProxyHa ndler ({"http" :
> "http://user:password@p roxyname:port"} )
>
>
>
> opener = urllib2.build_o pener(proxy_sup port, authinfo,
> urllib2.HTTPSHa ndler)
>
>
>
> urllib2.install _opener(opener)
>
>
>
>
>
> Then am trying to call the parser for the links in a particular https
> page which will be given as a command line argument. Which will read me
> all the links in that page.
>
>
>
> p = MyParser()
>
> for ln in urllib2.urlopen ( sys.argv[1] ):
>
> p.feed(ln)
>
> p.close()
>
>
>
>
>
> NOTE : I have installed python with _ssl support also.
>
>
>
>
>
> So with this code am able to read the links with HTTP page but not for
> the HTTPS page.
>
> AM NOT GETTING ANY ERRORS EITHER BUT ITS NOT READING THE LINKS, THAT
> ARE PRESENT IN THE GIVEN HTTPS PAGE
>
>
>
> Could you please tell me am I doing some thing wrong in the above code
> for any of the handlers.
>
>
>
>
>
> I have got struck here from so many days, please give me the solution
> for this.
>
>
>
> Thanks and regards
>
> YOGI
>[/color]