Having Trouble With CURL Function, Please Help

[devranger]

I am using the below CURL Function and can not figure out why it is not
retruning the results from the post. Can anyone take a look and tell
me what I may be doing wrong? I am just not seeing it.

As you can see at the bottom of the script if (strstr($Line, 'HOUSTON
YAMAHA MOTORSPORTS')) then it is reading the correct page, otherwise if
(strstr($Line, 'Yamaha Dealer Locator')) then it is reading the
incorrect page of the initial URL called
http://www.yamaha-motor.com/sport/de....aspx?ls=sport.


So when I run it what appears to be happening is the it is finding the

http://www.yamaha-motor.com/sport/dealers/dealerhome/home.aspx?ls=sport

but not he returned page from the post.

It is a fairly simple script. Your help is much appreciated on how I
can resolve this issue.

Thank you.

<?php

$data="MasterTe mplate:_PageTem plate:cphCenter Content:CenterC ontent:fcDropDo wnList=ATVs&Mas terTemplate:_Pa geTemplate:cphC enterContent:Ce nterContent:zip TextBox=77090";

curlgrabber("ht tp://www.yamaha-motor.com/sport/dealers/dealerhome/home.aspx?ls=sp ort",
$data); // Calls The CURL Function

unset($data);


function curlgrabber ($url, $data)
{


// create a new curl resource
$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch , CURLOPT_URL, $url);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, true);
curl_setopt($ch , CURLOPT_FOLLOWL OCATION, true);
curl_setopt($ch , CURLOPT_CONNECT TIMEOUT, 600);
curl_setopt($ch , CURLOPT_TIMEOUT , 600);
curl_setopt($ch , CURLOPT_POST, true);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);

// grab URL, and return output

$Contents = curl_exec($ch);

// close curl resource, and free up system resources


if (curl_errno($ch )) {
print curl_error($ch) ;
} else {
curl_close($ch) ;
unset($ch);
}


$Lines = preg_split("/\n/",$Contents );

$counter=0;
$strTag="False" ;

foreach($Lines as $Line)
{

if (strstr($Line, 'HOUSTON YAMAHA MOTORSPORTS')) // EXAMPLE ONLY!
{
echo "Post Worked And Is Reading Correct Page";
}


if (strstr($Line, 'Yamaha Dealer Locator')) // EXAMPLE ONLY!
{
echo "Post did not work. This is reading the initial screen at
http://www.yamaha-motor.com/sport/dealers/dealerhome/home.aspx?ls=sp ort";
}


}// End For Each
}// End Function

?>

[Andy Hassall]

Re: Having Trouble With CURL Function, Please Help

On 13 Dec 2006 10:44:06 -0800, "devranger" <knee-dragger@hotmail .comwrote:

>$data="MasterT emplate:_PageTe mplate:cphCente rContent:Center Content:fcDropD ownList=ATVs&Ma sterTemplate:_P ageTemplate:cph CenterContent:C enterContent:zi pTextBox=77090" ;

[snip]

>curl_setopt($c h, CURLOPT_POSTFIE LDS, $data);

What's with the colons in the data? That doesn't look like the sort of data
that you'd normally send in a POST.

And if you capture the POST from visiting that page with a browser, there's a
colossal amount of junk posted along with the form, nearly 2k of the stuff.
Which probably has some meaning to the page. Try it out with LiveHTTPHeaders or
similar.

When pasting all this back in exactly as the $data field it works, but it's so
cryptic I have no idea if it'd continue to work. (And depending on what you're
doing with the data you're getting out of the page the owners of the page may
not be happy about it either - have you just asked them for the information
directly?)

--
Andy Hassall :: andy@andyh.co.u k :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

[Rik]

Re: Having Trouble With CURL Function, Please Help

Andy Hassall wrote:

On 13 Dec 2006 10:44:06 -0800, "devranger" <knee-dragger@hotmail .com>
wrote:
>

>>

$data="MasterTe mplate:_PageTem plate:cphCenter Content:CenterC ontent:fcDropDo
wnList=ATVs&Mas terTemplate:_Pa geTemplate:cphC enterContent:Ce nterContent:zip
TextBox=77090";

>[snip] curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);

>
What's with the colons in the data? That doesn't look like the sort
of data
that you'd normally send in a POST.

Yup, AFAIK it stil should e ampersands.
Let cURL handle it though:
$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $topost );

--
Rik Wasmus

[devranger]

Re: Having Trouble With CURL Function, Please Help

Rik wrote:

Andy Hassall wrote:

On 13 Dec 2006 10:44:06 -0800, "devranger" <knee-dragger@hotmail .com>
wrote:

>

$data="MasterTe mplate:_PageTem plate:cphCenter Content:CenterC ontent:fcDropDo
wnList=ATVs&Mas terTemplate:_Pa geTemplate:cphC enterContent:Ce nterContent:zip
TextBox=77090";

[snip] curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);

What's with the colons in the data? That doesn't look like the sort
of data
that you'd normally send in a POST.

>
Yup, AFAIK it stil should e ampersands.
Let cURL handle it though:
$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $topost );
>
--
Rik Wasmus

Rik -

So you are suggesting I use an array but what do you mean by e
ampersads? Does that mean the ":" should be and &?

The reason I placed the colons is that they are just part of the field
input name I thought was necessary for the post. For instance, one
field on the form is called
MasterTemplate: _PageTemplate:c phCenterContent :CenterContent: zipTextBox.
Looks real weird to me as well but I do not know any other way to post
to the field than to call it as specified in the input name of the form
i.e. <input
name="MasterTem plate:_PageTemp late:cphCenterC ontent:CenterCo ntent:zipTextBo x"
type="text" maxlength="5"
id="MasterTempl ate__PageTempla te_cphCenterCon tent_CenterCont ent_zipTextBox"
tabindex="1" class="dealerte xt" style="width:10 0px;" size="20" />

Is this incorrect?

Thank you.

[Jerry Stuckle]

Re: Having Trouble With CURL Function, Please Help

devranger wrote:

Rik wrote:
>

>>Andy Hassall wrote:
>>

>>>On 13 Dec 2006 10:44:06 -0800, "devranger" <knee-dragger@hotmail .com>
>>>wrote:
>>>
>>>

>>$data="Master Template:_PageT emplate:cphCent erContent:Cente rContent:fcDrop Do
>>wnList=ATVs&M asterTemplate:_ PageTemplate:cp hCenterContent: CenterContent:z ip
>>TextBox=77090 ";
>>

>>>>[snip] curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);
>>>
>>What's with the colons in the data? That doesn't look like the sort
>>>of data
>>>that you'd normally send in a POST.

>>
>>Yup, AFAIK it stil should e ampersands.
>>Let cURL handle it though:
>>$topost = array(
> 'key1' ='value',
> 'key2' ='value2'
> //etc...
>>);
>>curl_setopt($ ch, CURLOPT_POSTFIE LDS, $topost );
>>
>>--
>>Rik Wasmus

>
>
Rik -
>
So you are suggesting I use an array but what do you mean by e
ampersads? Does that mean the ":" should be and &?
>
The reason I placed the colons is that they are just part of the field
input name I thought was necessary for the post. For instance, one
field on the form is called
MasterTemplate: _PageTemplate:c phCenterContent :CenterContent: zipTextBox.
Looks real weird to me as well but I do not know any other way to post
to the field than to call it as specified in the input name of the form
i.e. <input
name="MasterTem plate:_PageTemp late:cphCenterC ontent:CenterCo ntent:zipTextBo x"
type="text" maxlength="5"
id="MasterTempl ate__PageTempla te_cphCenterCon tent_CenterCont ent_zipTextBox"
tabindex="1" class="dealerte xt" style="width:10 0px;" size="20" />
>
Is this incorrect?
>
Thank you.
>

It looks like your data is correct, but there could be all kinds of
other reasons. For instance, they might be checking the HTTP_REFERER to
ensure it comes from their own page. Or they could have started a
session on the first page and check for it on the second. Or any of
several other things.

And I agree with Andy. It looks like maybe you're trying to scrape
something from their page. They may not like it, but perhaps if you ask
nicely they'll give it to you.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[devranger]

Re: Having Trouble With CURL Function, Please Help


Jerry Stuckle wrote:

devranger wrote:

Rik wrote:

>Andy Hassall wrote:
>
>>On 13 Dec 2006 10:44:06 -0800, "devranger" <knee-dragger@hotmail .com>
>>wrote:
>>
>>
>$data="MasterT emplate:_PageTe mplate:cphCente rContent:Center Content:fcDropD o
>wnList=ATVs&Ma sterTemplate:_P ageTemplate:cph CenterContent:C enterContent:zi p
>TextBox=77090" ;
>
>>>[snip] curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);
>>
>What's with the colons in the data? That doesn't look like the sort
>>of data
>>that you'd normally send in a POST.
>
>Yup, AFAIK it stil should e ampersands.
>Let cURL handle it though:
>$topost = array(
'key1' ='value',
'key2' ='value2'
//etc...
>);
>curl_setopt($c h, CURLOPT_POSTFIE LDS, $topost );
>
>--
>Rik Wasmus

Rik -

So you are suggesting I use an array but what do you mean by e
ampersads? Does that mean the ":" should be and &?

The reason I placed the colons is that they are just part of the field
input name I thought was necessary for the post. For instance, one
field on the form is called
MasterTemplate: _PageTemplate:c phCenterContent :CenterContent: zipTextBox.
Looks real weird to me as well but I do not know any other way to post
to the field than to call it as specified in the input name of the form
i.e. <input
name="MasterTem plate:_PageTemp late:cphCenterC ontent:CenterCo ntent:zipTextBo x"
type="text" maxlength="5"
id="MasterTempl ate__PageTempla te_cphCenterCon tent_CenterCont ent_zipTextBox"
tabindex="1" class="dealerte xt" style="width:10 0px;" size="20" />

Is this incorrect?

Thank you.

>
It looks like your data is correct, but there could be all kinds of
other reasons. For instance, they might be checking the HTTP_REFERER to
ensure it comes from their own page. Or they could have started a
session on the first page and check for it on the second. Or any of
several other things.
>
And I agree with Andy. It looks like maybe you're trying to scrape
something from their page. They may not like it, but perhaps if you ask
nicely they'll give it to you.
>
--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

Jerry / Andy

I am just doing this CURL as a test example. See I am new to
programming and I am learning php programming through a book called php
web development. I am catching on and in the book I was reading about
all the cool things you can do with CURL, so I started to do some
testing becasue this is how I learn. I have been sucessful with
posting to a few pages on different sites, but this one gave me trouble
and is kind of driving me crazy on why. I really have no use for the
data just getting it to work or figuring out why it may not work.

Now I see and have learned a couple things. You mentioned HTTP_REFERER
or they could have started a session. See I have learned through you
response becasue I had no idea these were limitation to CURL. That is
the book never mention it. Thank you.

Just for my reference is there any other limitation of CURL I should be
aware of?

Thank you.

[Rik]

Re: Having Trouble With CURL Function, Please Help

devranger wrote:

I am just doing this CURL as a test example. See I am new to
programming and I am learning php programming through a book called
php
web development. I am catching on and in the book I was reading about
all the cool things you can do with CURL, so I started to do some
testing becasue this is how I learn. I have been sucessful with
posting to a few pages on different sites, but this one gave me
trouble
and is kind of driving me crazy on why. I really have no use for the
data just getting it to work or figuring out why it may not work.
>
Now I see and have learned a couple things. You mentioned
HTTP_REFERER
or they could have started a session. See I have learned through you
response becasue I had no idea these were limitation to CURL. That is
the book never mention it. Thank you.
>
Just for my reference is there any other limitation of CURL I should
be
aware of?

It's not a limitation of cURL, it's a check they might have on their own
website. You're better of reading up on the normal HTTP protocol, and how
some major sites check their visitors. cURL can set a HTTP_REFERER for you
without a problem, or open the 'previous page' so get a session started/a
cookie. It's just a pain to figure out what they're checking.

Try to go to the page 'manually', and check all headers sent to & from the
site. If you use Mozilla, check uit LiveHTTPHeaders , for MSIE there is
Fiddler. THey will show you what is sent, try to emulate that with cURL.
--
Rik Wasmus

[devranger]

Re: Having Trouble With CURL Function, Please Help

I have been reading up on what you suggested Rik and testing different
options but I am still not successful in getting the results returned.
I have set the curl_setopt($ch , CURLOPT_REFERER , $url); and also set up
to curl sessions, one to capture the cookie and the other to post with
the cookie (see below): I really feel it may have something to do
with the long input names and : in the input name of my $data variable,
but Jerry said the data vaiable was correct.

Does anyone have any other suggestion on how I could get this working?
Your help is much appreciated, I feel I have learned so much so far.

Thank you!!!!



$data="MasterTe mplate:_PageTem plate:cphCenter Content:CenterC ontent:fcDropDo wnList=ATVs&Mas terTemplate:_Pa geTemplate:cphC enterContent:Ce nterContent:zip TextBox=77090";

$ch = curl_init();
curl_setopt($ch , CURLOPT_URL,$ur l);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
curl_setopt($ch , CURLOPT_FOLLOWL OCATION, 1);
curl_setopt($ch , CURL_ERROR, 1);
curl_setopt($ch , CURLOPT_COOKIEJ AR, "cookie.txt ");
curl_setopt($ch , CURLOPT_COOKIEF ILE, "cookie.txt ");
$result = curl_exec ($ch);
curl_close ($ch);

// create a new curl resource

$ch = curl_init();

// set URL and other appropriate options
curl_setopt($ch , CURLOPT_URL, $url);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, true);
curl_setopt($ch , CURLOPT_POST, true);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);
curl_setopt($ch , CURLOPT_FOLLOWL OCATION, true);
curl_setopt($ch , CURL_ERROR, 1);
curl_setopt($ch , CURLOPT_REFERER , $url);
curl_setopt($ch , CURLOPT_COOKIEJ AR, "cookie.txt ");
curl_setopt($ch , CURLOPT_COOKIEF ILE, "cookie.txt ");
curl_setopt($ch , CURLOPT_CONNECT TIMEOUT, 600);
curl_setopt($ch , CURLOPT_TIMEOUT , 600);

// grab URL, and return output

$Contents = curl_exec($ch);

[Jerry Stuckle]

Re: Having Trouble With CURL Function, Please Help

devranger wrote:

I have been reading up on what you suggested Rik and testing different
options but I am still not successful in getting the results returned.
I have set the curl_setopt($ch , CURLOPT_REFERER , $url); and also set up
to curl sessions, one to capture the cookie and the other to post with
the cookie (see below): I really feel it may have something to do
with the long input names and : in the input name of my $data variable,
but Jerry said the data vaiable was correct.
>
Does anyone have any other suggestion on how I could get this working?
Your help is much appreciated, I feel I have learned so much so far.
>
Thank you!!!!
>
>
>
$data="MasterTe mplate:_PageTem plate:cphCenter Content:CenterC ontent:fcDropDo wnList=ATVs&Mas terTemplate:_Pa geTemplate:cphC enterContent:Ce nterContent:zip TextBox=77090";
>
$ch = curl_init();
curl_setopt($ch , CURLOPT_URL,$ur l);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, 1);
curl_setopt($ch , CURLOPT_FOLLOWL OCATION, 1);
curl_setopt($ch , CURL_ERROR, 1);
curl_setopt($ch , CURLOPT_COOKIEJ AR, "cookie.txt ");
curl_setopt($ch , CURLOPT_COOKIEF ILE, "cookie.txt ");
$result = curl_exec ($ch);
curl_close ($ch);
>
// create a new curl resource
>
$ch = curl_init();
>
// set URL and other appropriate options
curl_setopt($ch , CURLOPT_URL, $url);
curl_setopt($ch , CURLOPT_RETURNT RANSFER, true);
curl_setopt($ch , CURLOPT_POST, true);
curl_setopt($ch , CURLOPT_POSTFIE LDS, $data);
curl_setopt($ch , CURLOPT_FOLLOWL OCATION, true);
curl_setopt($ch , CURL_ERROR, 1);
curl_setopt($ch , CURLOPT_REFERER , $url);
curl_setopt($ch , CURLOPT_COOKIEJ AR, "cookie.txt ");
curl_setopt($ch , CURLOPT_COOKIEF ILE, "cookie.txt ");
curl_setopt($ch , CURLOPT_CONNECT TIMEOUT, 600);
curl_setopt($ch , CURLOPT_TIMEOUT , 600);
>
// grab URL, and return output
>
$Contents = curl_exec($ch);
>

You're going to have to ask the owner of the data you wish to mine.
There are all kinds of ways of protecting data from mining, and only he
knows which one(s) he's using.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Rik]

Re: Having Trouble With CURL Function, Please Help

devranger wrote:

I have been reading up on what you suggested Rik and testing different
options but I am still not successful in getting the results returned.
I have set the curl_setopt($ch , CURLOPT_REFERER , $url); and also set
up
to curl sessions, one to capture the cookie and the other to post with
the cookie (see below): I really feel it may have something to do
with the long input names and : in the input name of my $data
variable,
but Jerry said the data vaiable was correct.
>
Does anyone have any other suggestion on how I could get this working?
Your help is much appreciated, I feel I have learned so much so far.
>
Thank you!!!!
>
>
>
>

$data="MasterTe mplate:_PageTem plate:cphCenter Content:CenterC ontent:fcDropDo
wnList=ATVs&Mas terTemplate:_Pa geTemplate:cphC enterContent:Ce nterContent:zip
TextBox=77090";

Probably:
$data=array(
'MasterTemplate :_P.......ist' ='ATV',
'MasterTemplate :_P...ox' ='77090');

Grtz,
--
Rik Wasmus