Submitting forms over HTTPS with mechanize

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • Rex
    #1

    Submitting forms over HTTPS with mechanize

    Hello,

    I am working on an academic research project where I need to log in to
    a website (www.lexis.com) over HTTPS and execute a bunch of queries to
    gather a data set. I just discovered the mechanize module, which seems
    great because it's a high-level tool. However, I can't find any decent
    documentation for mechanize apart from the docstrings, which are
    pretty thin. So I just followed some other examples I found online, to
    produce the following:

    baseurl = 'http://www.lexis.com/'
    br = mechanize.Brows er()
    br.set_handle_r obots(False)
    br.addheaders = [('User-Agent', 'Firefox')]
    br.open(baseurl )
    br.select_form( name="formauth" )
    br["USER_ID"]="my_user_id "
    br["PASSWORD"]="my_passwor d"
    result = br.submit()

    This code hangs at br.submit(), and I can't tell what I'm doing wrong.
    Typically I would inspect the HTTP data with an HTTP debugging proxy
    (Fiddler), but I guess since this is HTTPS I can't do that. Any
    glaring errors in my code?

    By the way, does anyone have suggestions for Python modules that I
    should use instead of mechanize (and that are sufficiently easy)? If
    mechanize fails, I might try modifying some similar Perl code a friend
    sent me that logs into lexis.com.

    Thanks so much,

    Rex
    Tags: None
    • Larry Bates
      #2
      Re: Submitting forms over HTTPS with mechanize

      Rex wrote:
      Hello,
      >
      I am working on an academic research project where I need to log in to
      a website (www.lexis.com) over HTTPS and execute a bunch of queries to
      gather a data set. I just discovered the mechanize module, which seems
      great because it's a high-level tool. However, I can't find any decent
      documentation for mechanize apart from the docstrings, which are
      pretty thin. So I just followed some other examples I found online, to
      produce the following:
      >
      baseurl = 'http://www.lexis.com/'
      br = mechanize.Brows er()
      br.set_handle_r obots(False)
      br.addheaders = [('User-Agent', 'Firefox')]
      br.open(baseurl )
      br.select_form( name="formauth" )
      br["USER_ID"]="my_user_id "
      br["PASSWORD"]="my_passwor d"
      result = br.submit()
      >
      This code hangs at br.submit(), and I can't tell what I'm doing wrong.
      Typically I would inspect the HTTP data with an HTTP debugging proxy
      (Fiddler), but I guess since this is HTTPS I can't do that. Any
      glaring errors in my code?
      >
      By the way, does anyone have suggestions for Python modules that I
      should use instead of mechanize (and that are sufficiently easy)? If
      mechanize fails, I might try modifying some similar Perl code a friend
      sent me that logs into lexis.com.
      >
      Thanks so much,
      >
      Rex
      I've used mechanize quite successfully but others have suggested Twill
      http://twill.idyll.org/. It seems to be at least documented.

      -Larry

        Comment

        • Rex
          #3
          Re: Submitting forms over HTTPS with mechanize

          On Sep 3, 10:17 pm, Larry Bates <larry.ba...@vi talEsafe.comwro te:
          Rex wrote:
          Hello,
          >
          I am working on an academic research project where I need to log in to
          a website (www.lexis.com) over HTTPS and execute a bunch of queries to
          gather a data set. I just discovered the mechanize module, which seems
          great because it's a high-level tool. However, I can't find any decent
          documentation for mechanize apart from the docstrings, which are
          pretty thin. So I just followed some other examples I found online, to
          produce the following:
          >
          baseurl = 'http://www.lexis.com/'
          br = mechanize.Brows er()
          br.set_handle_r obots(False)
          br.addheaders = [('User-Agent', 'Firefox')]
          br.open(baseurl )
          br.select_form( name="formauth" )
          br["USER_ID"]="my_user_id "
          br["PASSWORD"]="my_passwor d"
          result = br.submit()
          >
          This code hangs at br.submit(), and I can't tell what I'm doing wrong.
          Typically I would inspect the HTTP data with an HTTP debugging proxy
          (Fiddler), but I guess since this is HTTPS I can't do that. Any
          glaring errors in my code?
          >
          By the way, does anyone have suggestions for Python modules that I
          should use instead of mechanize (and that are sufficiently easy)? If
          mechanize fails, I might try modifying some similar Perl code a friend
          sent me that logs into lexis.com.
          >
          Thanks so much,
          >
          Rex
          >
          I've used mechanize quite successfully but others have suggested Twillhttp://twill.idyll.org/.  It seems to be at least documented.
          >
          -Larry
          Thanks for the reply, Larry. I ran my code again and it worked; there
          was probably some temporary issue with either my computer or the
          server that caused it to hang.

            Comment

            Previous template Next
            Working...