Help to automatically traverse a login session

**Colin McKinnon** · Aug 10 '06, 09:45 PM

Re: Help to automatically traverse a login session

joe t. wrote:

The subject may sound a little cryptic, so i'll try my best to explain.
Details are unavailable, as i am under a nondisclosure agreement, but
i'm looking for general principles and tips, not necessarily fixes for
existing code.
>

<snip long winded explanation>

So you want to copy someone else's data and you've only got an HTTP
interface intended for humans to the remote system.

There's plenty of companies doing this already - no need to be shy.

How simple it is depends on how well their site is written - assuming it is
well written you should be able to parse the pages with the XML parser. How
to get the pages? That's rather up to you - you could use a site ripper
like pavuk or write your own spider, e.g. using snoopy.

HTH

C.

**Andy Hassall** · Aug 10 '06, 09:55 PM

Re: Help to automatically traverse a login session

On 10 Aug 2006 14:25:33 -0700, "joe t." <thookerov@gmai l.comwrote:

>There is a website that requires me to log in using a web-form.
>Obviously, POST vars are sent and verified and on success i'm given a
>Session and/or Cookie. Within this logged-in area, there are links
>leading to data query result pages. "Click here for your recent
>transactions " kind of thing.
>
>Those results pages are what i want to get to, but through some kind of
>script that parses the results that get served out, not by user
>interaction. i want to send a request for a link within that logged in
>area and have the results served to my script, then parse out specific
>data from those results and in turn serve them to a user in my own
>page.
>
>i know that sounds shady, but the login is legitimate, the data access
>is legitimate, and the credentials are also valid. The problem is, i
>can't request a direct database link to the server hosting the actual
>data because of this nondisclosure agreement. It would require
>divulging the reasons for the need for such access, which my employer
>is not willing to reveal at this time.
>
>If there's anyone who can offer ideas or help, and wishes to keep
>possible answers off the public board, please email me. i realize this
>is a long shot, and i doubt that even if there IS a way, that anyone
>would be willing/able. But i gotta try.

Whilst this sort of situation is never the best way of doing things, sometimes
it's the only way. If you really do have to go down this route then there is a
particularly nice Perl module called WWW::Mechanize.

Obviously it's not PHP, but you can call Perl from PHP.

Client Challenge

http://search.cpan.org/search?query=www%3A%3Amechanize&mode=all

Whilst you're in Perl, then it also has various HTML parsing modules, the most
obvious being HTML::Parser, which can deal with HTML even if it's of dubious
quality.

Client Challenge

http://search.cpan.org/~gaas/HTML-Parser-3.55/Parser.pm

So combined you can have a Perl script that does all the hard stuff and then
returns its results in an easily machine-readable form to PHP.

--
Andy Hassall :: andy@andyh.co.u k :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

**axlq** · Aug 11 '06, 12:35 AM

Re: Help to automatically traverse a login session

In article <1155245133.537 125.73740@74g20 00cwt.googlegro ups.com>,
joe t. <thookerov@gmai l.comwrote:

>There is a website that requires me to log in using a web-form.
>...
>Those results pages are what i want to get to, but through some kind of
>script that parses the results that get served out, not by user
>interaction.

I once did this to gather a huge amount of historical data from a
horse-racing web site. I had to write the application in Java. It
would log in with my userID and password, submit queries to forms,
save the HTML result pages sent back, then parse the tabular data in
those pages into comma-delimited text data.

It was a much bigger project than I anticipated. I suspect there
are some macro automation tools out there that will let you do it
more easily.

-Alex

**joe t.** · Aug 11 '06, 12:05 PM

Re: Help to automatically traverse a login session

axlq wrote:

In article <1155245133.537 125.73740@74g20 00cwt.googlegro ups.com>,
joe t. <thookerov@gmai l.comwrote:

There is a website that requires me to log in using a web-form.
...
Those results pages are what i want to get to, but through some kind of
script that parses the results that get served out, not by user
interaction.

>
I once did this to gather a huge amount of historical data from a
horse-racing web site. I had to write the application in Java. It
would log in with my userID and password, submit queries to forms,
save the HTML result pages sent back, then parse the tabular data in
those pages into comma-delimited text data.
>
It was a much bigger project than I anticipated. I suspect there
are some macro automation tools out there that will let you do it
more easily.
>
-Alex

Thanks all of you for the suggestions. i will investigate these options
and try to report back on success.
-joe

**R. Rajesh Jeba Anbiah** · Aug 13 '06, 09:05 AM

Re: Help to automatically traverse a login session

joe t. wrote:
<snip>

There is a website that requires me to log in using a web-form.
Obviously, POST vars are sent and verified and on success i'm given a
Session and/or Cookie. Within this logged-in area, there are links
leading to data query result pages. "Click here for your recent
transactions" kind of thing.
>
Those results pages are what i want to get to, but through some kind of
script that parses the results that get served out, not by user
interaction. i want to send a request for a link within that logged in
area and have the results served to my script, then parse out specific
data from those results and in turn serve them to a user in my own
page.

<snip>

Such "web scraping" can be done with cURL <http://in.php.net/curl>
(need to set cookie support). Not all sites would allow web scraping
and will try to block automation with "CAPTCHA" (google it). Some sites
will even use Ajax based rendering which will then make the cURL
process a big tough (though I heard that cURL can work with Mozilla
JavaScript engine). In that case, it will be better to go for Delphi or
VB 6 as we can use WebBrowser component and can automate clicks, etc
with DOM object.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

**R. Rajesh Jeba Anbiah** · Aug 13 '06, 09:05 AM

Re: Help to automatically traverse a login session

joe t. wrote:
<snip>

There is a website that requires me to log in using a web-form.
Obviously, POST vars are sent and verified and on success i'm given a
Session and/or Cookie. Within this logged-in area, there are links
leading to data query result pages. "Click here for your recent
transactions" kind of thing.
>
Those results pages are what i want to get to, but through some kind of
script that parses the results that get served out, not by user
interaction. i want to send a request for a link within that logged in
area and have the results served to my script, then parse out specific
data from those results and in turn serve them to a user in my own
page.

<snip>

Such "web scraping" can be done with cURL <http://in.php.net/curl>
(need to set cookie support). Not all sites would allow web scraping
and will try to block automation with "CAPTCHA" (google it). Some sites
will even use Ajax based rendering which will then make the cURL
process a big tough (though I heard that cURL can work with Mozilla
JavaScript engine). In that case, it will be better to go for Delphi or
VB 6 as we can use WebBrowser component and can automate clicks, etc
with DOM object.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

Help to automatically traverse a login session

Help to automatically traverse a login session

Comment

Comment

Comment

Comment

Comment

Comment