PHP behaving like a user using a browser

[David Rasmussen]

Some sites seem to be session driven in the sense that if I visit the
homepage and do a few clicks I can navigate anywhere I want, but if I
paste the current location into a new browser window after having
navigated to some page, it doesn't work. It just returns to the start
page or says "timeout" etc.

This means that I can't read these pages from PHP with

$string = file_get_conten ts('http://some.url/blah/deep/link');

or whatever.

I guess the way to do then is to make PHP appear as a user-driven
browser as far as the page is concerned. And then start at the start
page and navigate down to the page in question "pressing" buttons and
"choosing" menu items from drop down items etc. from within PHP.

But how do I do this?

/David

[R. Rajesh Jeba Anbiah]

Re: PHP behaving like a user using a browser

David Rasmussen wrote:[color=blue]
> Some sites seem to be session driven in the sense that if I visit the[/color]
[color=blue]
> homepage and do a few clicks I can navigate anywhere I want, but if I[/color]
[color=blue]
> paste the current location into a new browser window after having
> navigated to some page, it doesn't work. It just returns to the start[/color]
[color=blue]
> page or says "timeout" etc.[/color]
<snip>

http://groups.google.com/groups?selm=HL-dne9pJr1ESb7fRVn-sQ%40comcast.com

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[Chung Leong]

Re: [FAQ] PHP behaving like a user using a browser


"R. Rajesh Jeba Anbiah" <ng4rrjanbiah@r ediffmail.com> wrote in message
news:1114360559 .452324.288110@ l41g2000cwc.goo glegroups.com.. .[color=blue]
> David Rasmussen wrote:[color=green]
> > Some sites seem to be session driven in the sense that if I visit the[/color]
>[color=green]
> > homepage and do a few clicks I can navigate anywhere I want, but if I[/color]
>[color=green]
> > paste the current location into a new browser window after having
> > navigated to some page, it doesn't work. It just returns to the start[/color]
>[color=green]
> > page or says "timeout" etc.[/color]
> <snip>
>
> http://groups.google.com/groups?selm...%40comcast.com
>
> --
> <?php echo 'Just another PHP saint'; ?>
> Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/
>[/color]

I forgot to mention how to obtain the cookie.

------------------------
Q. My PHP application retrieves a page from a web site that uses cookies.
How do I get the cookie?
A. Use fopen() to open a connection to the server, then call
stream_get_meta _data() to obtain information about the connection. The
function returns an associative array. The 'wrapper_data' element holds an
array containing the HTTP response headers. Loop through it and parse the
string that begins with "Set-Cookie."

----------------------

[R. Rajesh Jeba Anbiah]

Re: [FAQ] How do I retrieve a page from a web site? (Was: PHP behaving like a user using a browser)

Q: How do I retrieve a page from a web site?
A: Pass a URL to file() or file_get_conten ts(). The former returns the
contents as an array of lines. The latter returns the same as string.

Example:

$html = file_get_conten ts('http://www.example.com/');

Q: How do I retrieve a page from a web site that does browser
detection?
A: Use ini_set() to change the configuration option "user_agent ." This
sets the User-Agent header sent by PHP.

Example:

ini_set('user_a gent', 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT
5.1)');
$html = file_get_conten ts('http://www.example.com/');

Q: How do I retrieve a page from a web site that requires a cookie?
A: Use stream_context_ create() to create a HTTP context with Cookie as
one of the headers. Then, if you are coding in PHP 5, pass the context
to file() or file_get_conten ts() as the third parameter. In PHP 4
either function accepts a context, so you need to open the URL with
fopen() and retrieve the data a chunk at a time with fread().

Example:

$opts = array(
'http'=>array(
'method'=> 'GET',
'header'=>
"Accept-language: en\r\n" .
"Cookie: foo=bar\r\n"
)
);

$context = stream_context_ create($opts);
$f = fopen($url, "rb", false, $context);
while($data = fread($f, 1024)) {
echo $data;

}

stream_context_ create() is available in PHP 4.3.0 and above. If you are
using an older version, you would need the cURL functions or use
fsockopen() to open the connection and send the cookie header with
fputs().

Example 1:

$ch = curl_init();
curl_setopt($ch , CURLOPT_URL, $url);
curl_setopt($ch , CURLOPT_HTTPHEA DER,
array('Cookie: foo=bar'));
curl_exec($ch);
curl_close($ch) ;

Example 2:

$fp = fsockopen($host , $port);
fputs($fp, "GET / HTTP/1.0\r\n");
fputs($fp, "Host: $host\r\n");
fputs($fp, "Cookie: foo=bar\r\n\r\n ");

while ($data = fgets($fp, 1024)) {
echo $data;
}

Refer:

http://www.php.net/curl

http://curl.haxx.se/libcurl/php/examples/?ex=cookiejar.php

Q. My PHP application retrieves a page from a web site that uses
cookies. How do I get the cookie?
A. Use fopen() to open a connection to the server, then call
stream_get_meta _data() to obtain information about the connection. The
function returns an associative array. The 'wrapper_data' element holds
an array containing the HTTP response headers. Loop through it and
parse the string that begins with "Set-Cookie."

Refer:

http://www.phpclasses.org/httpclient

http://pear.php.net/package/HTTP_Client

++++++
@revision 2 Janwillem Borleffs added other examples
@revision 3 URLs changed to www.example.com. Added reference links. Few
double quotes quickly converted to single quotes
@todo Cleanup. Trim

[Micha³ Wo¼niak]

Re: [FAQ] How do I retrieve a page from a web site? (Was: PHP behaving like a user using a browser)

Great thing, but it misses an issue:
Suppose I have a file, let's call it incl.php, that I include in my
scripts, but I don't want to be accessible directly - when somebody
tries http://my.server/incl.php, though the file is actually there, I
want to display the webserver's 404 Error default message. I don't want
it to be hardcoded, though (like
$err404='<HTML> <BODY>...</BODY></HTML>';), in other words I'd like to do
something like:

$err404=file_ge _contents('http ://localhost/foobar.htm'); # foobar.htm
does not exist
$err404=str_rep lace('http://localhost/foobar.htm', __FILE__, $err404);
echp $err404;
die();

But - file_get_conten ts returns noting and an error:
Warning: file_get_conten ts(http://localhost/no_such_file.php): failed to
open stream: HTTP request failed! HTTP/1.1 404 Not Found in blahblah.php
on line 666.

what shuld I do to get the err404 page? Besides CURL, there must be
another way. :)

TIA
Mike

[R. Rajesh Jeba Anbiah]

Re: How do I retrieve a page from a web site? (Was: PHP behaving like a user using a browser)

Michal Wozniak wrote:
<snip>[color=blue]
> what shuld I do to get the err404 page? Besides CURL, there must be
> another way. :)[/color]

http://www.php.net/fsockopen

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/