Automate access to password protected sites

[Darren]

Hello,

we're fully paid up and signed up to a password protected site, so this
isn't hacking. In this site we can download a price list in CSV
format.

What I'd like to do is have a PHP script log into the site and download
the CSV file so we can then display it on our website.

Is this posible?

Darren

[Gordon Burditt]

Re: Automate access to password protected sites

>we're fully paid up and signed up to a password protected site, so this[color=blue]
>isn't hacking. In this site we can download a price list in CSV
>format.
>
>What I'd like to do is have a PHP script log into the site and download
>the CSV file so we can then display it on our website.[/color]

Are you sure this is a permitted use of the price list?
Being allowed to access the list and publishing it are quite
different from the point of view of copyright law.
[color=blue]
>Is this posible?[/color]

With cURL you can submit the login page with your info filled in,
get back a session cookie (or whatever types the site uses), then
use that cookie to go to other places on the site. It may require
some research as to what variables need to be sent, (including
HTTP_REFERER, maybe) and this can break if the site changes them.

Gordon L. Burditt

[d]

Re: Automate access to password protected sites

"Darren" <groups@darrenl udlam.plus.com> wrote in message
news:1138122708 .477954.222130@ g47g2000cwa.goo glegroups.com.. .[color=blue]
> Hello,
>
> we're fully paid up and signed up to a password protected site, so this
> isn't hacking. In this site we can download a price list in CSV
> format.
>
> What I'd like to do is have a PHP script log into the site and download
> the CSV file so we can then display it on our website.
>
> Is this posible?[/color]

It's very possible. As other posts have said - make sure it's not against
the site's rules to do this. Just because you can manually view the list
doesn't mean to say you can do it automatically.

The best way to do it is to use cURL. I say that because unless you feel
particularly comfortable with writing and interpreting HTTP requests
(they're not hard, but you do need to do some reading), cURL will do most of
the work for you. I would do it like this:

Step 1. Read the page where you would log in, using cURL. Get the HTML and
find the form. Get the variable names from the form (for username and
password) - this is done in case they change the variable names, your script
will still work.

Step 2. Create your post variables containing the variable names from step
1 and your own username and password. Submit these variables to the action
of the form you read in step 1.

Step 3. Now, cURL should be remembering your session ID/cookie (you might
have to set the cookie jar/file settings for cURL - this is covered
adequately on the PHP site, http://www.php.net/curl).

Step 4. Using cURL, access the CSV's url - cURL should tell the server what
it needs to hear to accept that you're logged on, and the file should be
available.

I hope that helps.

dave
[color=blue]
> Darren
>[/color]

[Alvaro G. Vicario]

Re: Automate access to password protected sites

*** Darren escribió/wrote (24 Jan 2006 09:11:48 -0800):[color=blue]
> we're fully paid up and signed up to a password protected site, so this
> isn't hacking. In this site we can download a price list in CSV
> format.
>
> What I'd like to do is have a PHP script log into the site and download
> the CSV file so we can then display it on our website.[/color]

In this case I'd normally do a system() call and use wget (if available).
It takes care itself of all the dirty details.

wget --help

[...]

HTTP options:
--http-user=USER set http user to USER.
--http-passwd=PASS set http password to PASS.

If the site is not using HTTP authentication, though, you'd need to send
the login form data and probably create a cookie. The right tool is curl.

--
-+ Álvaro G. Vicario - Burgos, Spain
++ http://bits.demogracia.com es mi sitio para programadores web
+- http://www.demogracia.com es mi web de humor libre de cloro
--

[Carl Vondrick]

Re: Automate access to password protected sites

Darren wrote:[color=blue]
> Hello,
>
> we're fully paid up and signed up to a password protected site, so this
> isn't hacking. In this site we can download a price list in CSV
> format.
>
> What I'd like to do is have a PHP script log into the site and download
> the CSV file so we can then display it on our website.
>
> Is this posible?
>
> Darren
>[/color]
How is the site protected? WWW-Auth or regular form? Cookies?

In any case, you can use fsockopen
[http://us3.php.net/manual/en/function.fsockopen.php] to simulate a
connection to the server. You just need to pass the right headers.

I prefer this way because it stays within PHP, without branching to *NIX
command line. This should work just fine on Windows or *NIX.

[d]

Re: Automate access to password protected sites

"Carl Vondrick" <usenet@carlsof t.net> wrote in message
news:UjXBf.1592 8$Jd.14345@news svr25.news.prod igy.net...[color=blue]
> Darren wrote:[color=green]
>> Hello,
>>
>> we're fully paid up and signed up to a password protected site, so this
>> isn't hacking. In this site we can download a price list in CSV
>> format.
>>
>> What I'd like to do is have a PHP script log into the site and download
>> the CSV file so we can then display it on our website.
>>
>> Is this posible?
>>
>> Darren
>>[/color]
> How is the site protected? WWW-Auth or regular form? Cookies?
>
> In any case, you can use fsockopen
> [http://us3.php.net/manual/en/function.fsockopen.php] to simulate a
> connection to the server. You just need to pass the right headers.
>
> I prefer this way because it stays within PHP, without branching to *NIX
> command line. This should work just fine on Windows or *NIX.[/color]

It also doesn't need any extensions, like cURL. It does, though, take a bit
more knowledge about HTTP, but I totally agree with you - I use fsockopen
for all my HTTP needs ;)

dave