CURL and PHP Redirect

[niftyhawk@gmail.com]

Hi All,

I have a small curl function which spits out a bunch of headers.

The CURL Code is:

$url="http://blahblah.com?Lo gon";
$ch = curl_init();
curl_setopt($ch , CURLOPT_URL, $url);
curl_setopt($ch , CURLOPT_FOLLOWL OCATION, 1);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
curl_setopt($ch , CURLOPT_USERAGE NT, $_SERVER["HTTP_USER_AGEN T"]);
curl_setopt($ch , CURLOPT_REFERER ,
"http://blahblah.com/default.asp?v=" );
curl_setopt($ch , CURLOPT_VERBOSE , 0);
curl_setopt($ch , CURLOPT_HEADER, 1);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIE LDS,
"FormId=0&Confi g=0&RedirectURL =$RedirectURL&I dentification=$ UserID&Verifica tion=$Password& DB=$DB&Remember Logon=1&UseEnha ncedContent=1") ;
curl_setopt ($ch, CURLOPT_COOKIEJ AR, 'Ocookie.txt');
$content = curl_exec ($ch); # This returns HTML
curl_close ($ch);
print $content;


The HTML sspit out by the above code is:

HTTP/1.1 100 Continue Server: Microsoft-IIS/5.1 Date: Fri, 17 Mar 2006
00:14:25 GMT X-Powered-By: ASP.NET HTTP/1.1 200 OK Server:
Microsoft-IIS/5.1 Date: Fri, 17 Mar 2006 00:14:25 GMT X-Powered-By:
ASP.NET Connection: close Content-Type: text/html Refresh: 0;
URL=http://blahblah.com/default.asp?v= Set-Cookie: osaa=guest;
expires=Monday, 16-Mar-2009 00:14:25 GMT; path=/ Set-Cookie:
osab=guest; expires=Monday, 16-Mar-2009 00:14:25 GMT; path=/

All this is printed along with the existing HTML on which the above
curl code is written. How do I redirect the existing page to the one
setup in the headers by making sure of passing the whole header
information which is spit out by the curl code?

Thanks

[R. Rajesh Jeba Anbiah]

Re: CURL and PHP Redirect


niftyhawk@gmail .com wrote:[color=blue]
> Hi All,
>
> I have a small curl function which spits out a bunch of headers.
>
> The CURL Code is:
>
> $url="http://blahblah.com?Lo gon";
> $ch = curl_init();
> curl_setopt($ch , CURLOPT_URL, $url);
> curl_setopt($ch , CURLOPT_FOLLOWL OCATION, 1);
> curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
> curl_setopt($ch , CURLOPT_USERAGE NT, $_SERVER["HTTP_USER_AGEN T"]);
> curl_setopt($ch , CURLOPT_REFERER ,
> "http://blahblah.com/default.asp?v=" );
> curl_setopt($ch , CURLOPT_VERBOSE , 0);
> curl_setopt($ch , CURLOPT_HEADER, 1);
> curl_setopt ($ch, CURLOPT_POST, 1);
> curl_setopt ($ch, CURLOPT_POSTFIE LDS,
> "FormId=0&Confi g=0&RedirectURL =$RedirectURL&I dentification=$ UserID&Verifica tion=$Password& DB=$DB&Remember Logon=1&UseEnha ncedContent=1") ;
> curl_setopt ($ch, CURLOPT_COOKIEJ AR, 'Ocookie.txt');
> $content = curl_exec ($ch); # This returns HTML
> curl_close ($ch);
> print $content;
>
>
> The HTML sspit out by the above code is:
>
> HTTP/1.1 100 Continue Server: Microsoft-IIS/5.1 Date: Fri, 17 Mar 2006
> 00:14:25 GMT X-Powered-By: ASP.NET HTTP/1.1 200 OK Server:
> Microsoft-IIS/5.1 Date: Fri, 17 Mar 2006 00:14:25 GMT X-Powered-By:
> ASP.NET Connection: close Content-Type: text/html Refresh: 0;
> URL=http://blahblah.com/default.asp?v= Set-Cookie: osaa=guest;
> expires=Monday, 16-Mar-2009 00:14:25 GMT; path=/ Set-Cookie:
> osab=guest; expires=Monday, 16-Mar-2009 00:14:25 GMT; path=/
>
> All this is printed along with the existing HTML on which the above
> curl code is written.[/color]

IIRC, this is happening as you have used curl_setopt($ch ,
CURLOPT_HEADER, 1);. Set it to 0.
[color=blue]
> How do I redirect the existing page to the one
> setup in the headers by making sure of passing the whole header
> information which is spit out by the curl code?[/color]

CURLOPT_FOLLOWL OCATION should handle that case, IMHO. But, also you
have wide range of options to store and set HTTP headers in libcurl.
If I'm right, you're missing CURLOPT_COOKIEF ILE (should be set to the
same file of cookie jar) as it is needed for reading/setting the cookie
headers when sending requests. Make sure the cookie file is readable
and writable.

FWIW, for testing purpose, you may enable verbose log:

$fp_err = fopen('verbose_ file.txt', 'ab+');
fwrite($fp_err, date('Y-m-d H:i:s')."\n\n") ; //add timestamp to the
verbose log
curl_setopt($ch , CURLOPT_VERBOSE , 1);
curl_setopt($ch , CURLOPT_FAILONE RROR, true);
curl_setopt($ch , CURLOPT_STDERR, $fp_err);

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[niftyhawk@gmail.com]

Re: CURL and PHP Redirect

Hi,

CURLOPT_FOLLOWL OCATION doesnt seem to handle the case of redirection.
Any other ideas? Verbose log was useful to know the header information.

Thanks

[niftyhawk@gmail.com]

Re: CURL and PHP Redirect

Also, CURLOPT_COOKIEF ILE works great but seems to make the page load
slower.. Is there a faster alternative to it? :)

[R. Rajesh Jeba Anbiah]

Re: CURL and PHP Redirect

niftyhawk@gmail .com wrote:[color=blue]
> Also, CURLOPT_COOKIEF ILE works great but seems to make the page load
> slower.. Is there a faster alternative to it? :)[/color]

Among the above two questions which is your concern now? I guess,
both of them are now sorted out.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[niftyhawk@gmail.com]

Re: CURL and PHP Redirect

Hi,

I was able to use CURLOPT_COOKIEF ILE to write cookies and do my
redirections fine. But my only concern now is it takes a lot of time
for CURLOPT_COOKIEF ILE to update cookies. I just wanted to know how we
can speed up this process?

Thanks


R. Rajesh Jeba Anbiah wrote:[color=blue]
> niftyhawk@gmail .com wrote:[color=green]
> > Also, CURLOPT_COOKIEF ILE works great but seems to make the page load
> > slower.. Is there a faster alternative to it? :)[/color]
>
> Among the above two questions which is your concern now? I guess,
> both of them are now sorted out.
>
> --
> <?php echo 'Just another PHP saint'; ?>
> Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/[/color]

[R. Rajesh Jeba Anbiah]

Re: CURL and PHP Redirect

niftyhawk@gmail .com wrote:[color=blue]
> Hi,
>
> I was able to use CURLOPT_COOKIEF ILE to write cookies and do my
> redirections fine. But my only concern now is it takes a lot of time
> for CURLOPT_COOKIEF ILE to update cookies. I just wanted to know how we
> can speed up this process?[/color]
<snip>

I guess, GG ate my yesterday reply. I mentioned that you should
check verbose log and also check if there is any locking issues with
the file. Also, try to place the cookie file in the current path.

Also, CURLOPT_COOKIEF ILE is for setting http headers; for setting
http headers, it will fetch the cookie from that file.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[niftyhawk@gmail.com]

Re: CURL and PHP Redirect

Hi,

Following is the verbose log for my curl transfer with progress meter
on. I am unable to figure out why IIS is taking 4 seconds to spit out
the response..any ideas?

2006-03-22 15:09:00

* About to connect() to www2.lib.ncsu.e du port 80
* Trying 152.1.79.69... * connected
* Connected to www2.lib.ncsu.e du (152.1.79.69) port 80[color=blue]
> GET /catalog/?view=&Nty=1&N= 0&Ntk=Keyword&N tt=cats&No= HTTP/1.1[/color]
Host: www2.lib.ncsu.e du
Accept: */*
Cookie: JSESSIONID=39F9 A1F5288148C802B D3B92C2978CFF

< HTTP/1.1 200 OK
< Date: Wed, 22 Mar 2006 21:10:48 GMT
< Server: Apache/2.0.52 (Unix) JRun/4.0 mod_jk/1.2.8
< Transfer-Encoding: chunked
< Content-Type: text/html;charset=UT F-8
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 1187 0 1187 0 0 5623 0 --:--:-- --:--:--
--:--:-- 5623
100 25210 0 25210 0 0 89473 0 --:--:-- --:--:--
--:--:-- 335k* Connection #0 to host www2.lib.ncsu.e du left intact

* Closing connection #0
2006-03-22 15:09:00

* About to connect() to 172.19.10.249 port 80
* Trying 172.19.10.249.. . * connected
* Connected to 172.19.10.249 (172.19.10.249) port 80[color=blue]
> GET /abc/xyztest.asp?v= HTTP/1.1[/color]
Host: 172.19.10.249
Accept: */*
Cookie: ASPSESSIONIDASS TQTRR=MNOLKGNAN DGKFMMNCAGGNPHD ;
osah=ALBANY+PAR K; osag=Chicago+Pu blic+Library; osaf=1; osae=1;
osad=2644; osac=doe; osab=john; osaa=JOHN

% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01
--:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:02
--:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:03
--:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- 0:00:04
--:--:-- 0< HTTP/1.1 200 OK
< Server: Microsoft-IIS/5.1
< Date: Wed, 22 Mar 2006 21:08:57 GMT
< X-Powered-By: ASP.NET
< Content-Length: 29727
< Content-Type: text/html
< Cache-control: private

100 29727 100 29727 0 0 7270 0 0:00:04 0:00:04
--:--:-- 9670* Connection #0 to host 172.19.10.249 left intact

* Closing connection #0


Thanks

[R. Rajesh Jeba Anbiah]

Re: CURL and PHP Redirect

niftyhawk@gmail .com wrote:[color=blue]
> Hi,
>
> Following is the verbose log for my curl transfer with progress meter
> on. I am unable to figure out why IIS is taking 4 seconds to spit out
> the response..any ideas?[/color]
<snip>

I haven't tried turning on progress meter option yet and so I'm not
sure about it. Not sure, it affects the speed.

Could you try manually with browser and cross check the http headers
with the one set via curl; and tweak the options so that it sends
exactly the same http headers sent by browser. And, what's the time
taken for browser?

Could you post the result of print_r(curl_ge tinfo($ch)); I guess,
it's taking too many redirections.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[niftyhawk@gmail.com]

Re: CURL and PHP Redirect

Hi,

The result of print_r(curl_ge tinfo($ch)); is as follows:

Array ( [url] => http://localhost/abc/default.asp?xyz = [http_code] => 0
[header_size] => 0 [request_size] => 0 [filetime] => 0
[ssl_verify_resu lt] => 0 [redirect_count] => 0 [total_time] => 0
[namelookup_time] => 0 [connect_time] => 0 [pretransfer_tim e] => 0
[size_upload] => 0 [size_download] => 0 [speed_download] => 0
[speed_upload] => 0 [download_conten t_length] => 0
[upload_content_ length] => 0 [starttransfer_t ime] => 0 [redirect_time]
=> 0 )

I will cross check the http headers and try to tweak curl options..

Thanks

[R. Rajesh Jeba Anbiah]

Re: CURL and PHP Redirect

niftyhawk@gmail .com wrote:[color=blue]
> Hi,
>
> The result of print_r(curl_ge tinfo($ch)); is as follows:
>
> Array ( [url] => http://localhost/abc/default.asp?xyz = [http_code] => 0
> [header_size] => 0 [request_size] => 0 [filetime] => 0
> [ssl_verify_resu lt] => 0 [redirect_count] => 0 [total_time] => 0
> [namelookup_time] => 0 [connect_time] => 0 [pretransfer_tim e] => 0
> [size_upload] => 0 [size_download] => 0 [speed_download] => 0
> [speed_upload] => 0 [download_conten t_length] => 0
> [upload_content_ length] => 0 [starttransfer_t ime] => 0 [redirect_time]
> => 0 )[/color]

Obviously, you're using print_r(curl_ge tinfo($ch)); before
curl_exec(). Use it immediately after curl_exec()

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.bl ogspot.com/

[niftyhawk@gmail.com]

Re: CURL and PHP Redirect

Sorry, My Bad..

Here's the complete output:

The Local System Performace signifies that my curl code is pulling
stuff out a website running on local system.

The Remote System Performace signifies that my curl code is pulling
stuff out a website running on a remote system.

The remote system performance is faster.. eventhough i tweaked a lot of
curl, i am still unable to figure out why it is so slow on the local
machine.. :(

Local SYSTEM Performance
=============== =====

Array ( [url] => http://abc.com/share/xyz.dll?Logon [content_type] =>
text/html [http_code] => 200 [header_size] => 952 [request_size] => 305
[filetime] => -1 [ssl_verify_resu lt] => 0 [redirect_count] => 0
[total_time] => 0.052765 [namelookup_time] => 0.000145 [connect_time]
=> 0.000797 [pretransfer_tim e] => 0.000897 [size_upload] => 0
[size_download] => 84 [speed_download] => 1591 [speed_upload] => 0
[download_conten t_length] => 0 [upload_content_ length] => 0
[starttransfer_t ime] => 0.052367 [redirect_time] => 0 )

Array ( [url] => http://abc.com/share/default.asp?v= [content_type] =>
text/html [http_code] => 200 [header_size] => 357 [request_size] => 425
[filetime] => -1 [ssl_verify_resu lt] => 0 [redirect_count] => 0
[total_time] => 2.757411 [namelookup_time] => 4E-05 [connect_time] =>
0.000555 [pretransfer_tim e] => 0.000664 [size_upload] => 0
[size_download] => 29654 [speed_download] => 10754 [speed_upload] => 0
[download_conten t_length] => 29654 [upload_content_ length] => 0
[starttransfer_t ime] => 2.754259 [redirect_time] => 0 )


Remote System PERFORMANCE (This is very fast for some reason)
=============== =============== =============== =============== =
Array ( [url] => http://xyz.com/share/xyz.dll?Logon [content_type] =>
text/html [http_code] => 200 [header_size] => 863 [request_size] => 307
[filetime] => -1 [ssl_verify_resu lt] => 0 [redirect_count] => 0
[total_time] => 0.116149 [namelookup_time] => 5.8E-05 [connect_time] =>
0.034908 [pretransfer_tim e] => 0.034969 [size_upload] => 0
[size_download] => 213 [speed_download] => 1833 [speed_upload] => 0
[download_conten t_length] => 0 [upload_content_ length] => 0
[starttransfer_t ime] => 0.115845 [redirect_time] => 0 )

Array ( [url] => http://xyz.com/share/default.asp?v= [content_type] =>
text/html [http_code] => 200 [header_size] => 311 [request_size] => 445
[filetime] => -1 [ssl_verify_resu lt] => 0 [redirect_count] => 0
[total_time] => 0.870313 [namelookup_time] => 4.8E-05 [connect_time] =>
0.034595 [pretransfer_tim e] => 0.034689 [size_upload] => 0
[size_download] => 30134 [speed_download] => 34624 [speed_upload] => 0
[download_conten t_length] => 30134 [upload_content_ length] => 0
[starttransfer_t ime] => 0.756889 [redirect_time] => 0 )

[R. Rajesh Jeba Anbiah]

Re: CURL and PHP Redirect

niftyhawk@gmail .com wrote:[color=blue]
> Sorry, My Bad..
>
> Here's the complete output:
>
> The Local System Performace signifies that my curl code is pulling
> stuff out a website running on local system.
>
> The Remote System Performace signifies that my curl code is pulling
> stuff out a website running on a remote system.
>
> The remote system performance is faster.. eventhough i tweaked a lot of
> curl, i am still unable to figure out why it is so slow on the local
> machine.. :([/color]
<snip>

IIRC, libcurl on Windows machine will be little slower as compared
to Linux; not sure though.

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/