Automated web browing

[mr_marcin]

Hi

Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.

The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

Thanks for help
Marcin

[Manuel Lemos]

Re: Automated web browing

on 01/17/2008 07:52 PM mr_marcin said the following:

Hi
>
Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.
>
I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.
>
The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

You may want to try this HTTP client class. Basically it acts like a
browser accessing pages, submitting forms, collecting cookies, handling
redirection, etc. which seems what you need to retrieve the pages with
the prices you want to grab.

http://www.phpclasses.org/httpclient


--

Regards,
Manuel Lemos

PHP professionals looking for PHP jobs
http://www.phpclasses.org/professionals/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Jerry Stuckle]

Re: Automated web browing

mr_marcin wrote:

Hi
>
Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.
>
I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.
>
The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.
>
Thanks for help
Marcin
>

cURL will allow you to get or post to pages, and will return the data.
I much prefer it over the HTTPClient class. It's more flexible.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Marlin Forbes]

Re: Automated web browing

mr_marcin wrote:

Hi
>
Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.

Hi there,

SimpleTest has a class included called SimpleBrowser, which does what
you want, with a very intuitive API. It's not too fast tho...

SimpleTest: http://www.lastcraft.com/simple_test.php

Or, you can interactively setup browsing sessions with the Selenium IDE
and then use the PHP client for the Selenium Remote Control to run them...

Selenium IDE: http://www.openqa.org/selenium-ide/
Selenium RC: http://www.openqa.org/selenium-rc/
PHP Client for Selenium: http://pear.php.net/package/Testing_Selenium

Misc:

http://blog.thinkphp.de/archives/133-Practical-Testing-PHP-Applications-with-Selenium.html

Regards,
Marlin Forbes
Freelance Developer
Data Shaman
datashaman.com
+27 (0)82 501-6647

[mr_marcin]

Re: Automated web browing

Or, you can interactively setup browsing sessions with the Selenium IDE

and then use the PHP client for the Selenium Remote Control to run them...
>
Selenium IDE:http://www.openqa.org/selenium-ide/
Selenium RC:http://www.openqa.org/selenium-rc/
PHP Client for Selenium:http://pear.php.net/package/Testing_Selenium

This sounds like a quite easy to use package, but will this be
efficient enough? I will check all options next week.

[mr_marcin]

Re: Automated web browing

cURL will allow you to get or post to pages, and will return the data.

I much prefer it over the HTTPClient class. It's more flexible.
>

I guess this approach requires some manual job, but you are right -
thats the most flexible and probably most effective way.

[R. Rajesh Jeba Anbiah]

Re: Automated web browing

On Jan 18, 2:52 am, mr_marcin <mar...@cme.plw rote:

Hi
>
Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.
>
I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.
>
The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.

1. If you're looking for client tools http://www.iopus.com/imacros/firefox/
2. Web scraping with cURL or HTTPClient class
3. Look for the Web services (SOAP, XML, etc)

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[Manuel Lemos]

Re: Automated web browing

Hello,

on 01/17/2008 10:15 PM Jerry Stuckle said the following:

>Does anybody have some idea how to input some text into inputbox on
>one page, than press some button on that page, that will load another
>page, and finally read the responde? Suppose I want to write a price
>comparision engine, where I would like to parse shops website for
>price each time user wants.
>>
>I have found similar feature in Symfony framework, called sfBrowser
>(or sfTestBrowser). These are made for automated functional testing,
>but should provide the functinality I am requesting.
>>
>The question is: will this be efficient enough? Maybe there are other
>ways to achieve this? Of course I can always try to make it more
>manually - look for some pattern in url (search is usually done via
>GET), and parse output html.
>>
>Thanks for help
>Marcin
>>

>
cURL will allow you to get or post to pages, and will return the data. I
much prefer it over the HTTPClient class. It's more flexible.

I wonder which HTTP client you are talking about. The HTTP client I
mentioned wraps around Curl or socket functions depending on which is
more convinient to use in each PHP setup. This is the HTTP client class
I meant:

http://www.phpclasses.org/httpclient

As for Curl being flexible, I wonder what you are talking about.

Personally I find it very odd that you cannot read retrieved pages with
Curl in small chunks at a time without having to use callbacks. This is
bad because it makes very difficult to retrieve and process large pages
without using external files nor exceeding the PHP memory limits.

--

Regards,
Manuel Lemos

PHP professionals looking for PHP jobs
http://www.phpclasses.org/professionals/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Jerry Stuckle]

Re: Automated web browing

Manuel Lemos wrote:

Hello,
>
on 01/17/2008 10:15 PM Jerry Stuckle said the following:

>>Does anybody have some idea how to input some text into inputbox on
>>one page, than press some button on that page, that will load another
>>page, and finally read the responde? Suppose I want to write a price
>>comparision engine, where I would like to parse shops website for
>>price each time user wants.
>>>
>>I have found similar feature in Symfony framework, called sfBrowser
>>(or sfTestBrowser). These are made for automated functional testing,
>>but should provide the functinality I am requesting.
>>>
>>The question is: will this be efficient enough? Maybe there are other
>>ways to achieve this? Of course I can always try to make it more
>>manually - look for some pattern in url (search is usually done via
>>GET), and parse output html.
>>>
>>Thanks for help
>>Marcin
>>>

>cURL will allow you to get or post to pages, and will return the data. I
>much prefer it over the HTTPClient class. It's more flexible.

>
I wonder which HTTP client you are talking about. The HTTP client I
mentioned wraps around Curl or socket functions depending on which is
more convinient to use in each PHP setup. This is the HTTP client class
I meant:
>
http://www.phpclasses.org/httpclient
>

The same one.

As for Curl being flexible, I wonder what you are talking about.
>

I can do virtually anything with it that I can do with a browser, with
the exception of client side scripting. Also much less overhead than
the httpclient class.

Personally I find it very odd that you cannot read retrieved pages with
Curl in small chunks at a time without having to use callbacks. This is
bad because it makes very difficult to retrieve and process large pages
without using external files nor exceeding the PHP memory limits.
>

So? I never needed to. First of all, I have no need to retrieve huge
pages. The larges I've ever downloaded (a table with lots of info) was
a little over 3MB and Curl and PHP handled it just fine.

But if the text were split, you need to do additional processing to
handle splits at inconvenient locations. Much easier to add everything
to a temporary file and read it back in the way I need to so it.

But that's one of the advantages of cURL - it gives me the option of
doing the callbacks or not.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Jerry Stuckle]

Re: Automated web browing

Manuel Lemos wrote:

<snip junk>

Manuel,

I'm not going to argue with you about whether the HTTPClass is easier to
use or whatever.

My single point was that cURL is more flexible. You can do anything
with cURL that you can with the HTTPClient class and more. That is
pretty obvious - because the HTTPClient class is built on cURL - so if
cURL can't do it, neither can the HTTPClient class.

But being built on cURL, the HTTPClient class restricts what you can do.
So it is less flexible.

You can sit there and argue all you want as to the other merits of your
class. I won't bite. Because that was not my point.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Manuel Lemos]

Re: Automated web browing

Hello,

on 01/19/2008 06:52 PM Jerry Stuckle said the following:

I'm not going to argue with you about whether the HTTPClass is easier to
use or whatever.
>
My single point was that cURL is more flexible. You can do anything
with cURL that you can with the HTTPClient class and more. That is
pretty obvious - because the HTTPClient class is built on cURL - so if
cURL can't do it, neither can the HTTPClient class.
>
But being built on cURL, the HTTPClient class restricts what you can do.
So it is less flexible.

No, that is not the way it works. I already explained that to you.

The HTTP client class uses Curl when fsockopen calls cannot be used
under the current PHP setup. Curl is used as a better than nothing solution.

For instance before PHP 4.3.0 you can only make SSL request with curl.
The class used curl for SSL requests, but of course, with curl it cannot
not send or receive streamed data in small chunks that never exceed the
PHP memory limits.

If you want that flexibility you need to use PHP 4.3.0 or newer. Then
the class will use fsockopen for SSL requests.

In any case, the HTTP client class abstracts that for you. You do not
need to adapt your application code depending on the PHP version, as the
class does it for you.

I developed the HTTP client class not just as a mere curl wrapper, but
to actually add some benefits on top of curl/fsockopen. So, it was meant
to add flexibility, not to remove it.

That is why I questioned you about you flexibility statement. Maybe you
tried an old version of the HTTP client class and you found some
limitations that no longer exist. But if you still find it less
flexible, I want to understand what you are talking about.



--

Regards,
Manuel Lemos

PHP professionals looking for PHP jobs
http://www.phpclasses.org/professionals/

PHP Classes - Free ready to use OOP components written in PHP
http://www.phpclasses.org/

[Jerry Stuckle]

Re: Automated web browing

Manuel Lemos wrote:

Hello,
>
on 01/19/2008 06:52 PM Jerry Stuckle said the following:

>I'm not going to argue with you about whether the HTTPClass is easier to
>use or whatever.
>>
>My single point was that cURL is more flexible. You can do anything
>with cURL that you can with the HTTPClient class and more. That is
>pretty obvious - because the HTTPClient class is built on cURL - so if
>cURL can't do it, neither can the HTTPClient class.
>>
>But being built on cURL, the HTTPClient class restricts what you can do.
> So it is less flexible.

>
No, that is not the way it works. I already explained that to you.
>
The HTTP client class uses Curl when fsockopen calls cannot be used
under the current PHP setup. Curl is used as a better than nothing solution.
>
For instance before PHP 4.3.0 you can only make SSL request with curl.
The class used curl for SSL requests, but of course, with curl it cannot
not send or receive streamed data in small chunks that never exceed the
PHP memory limits.
>
If you want that flexibility you need to use PHP 4.3.0 or newer. Then
the class will use fsockopen for SSL requests.
>
In any case, the HTTP client class abstracts that for you. You do not
need to adapt your application code depending on the PHP version, as the
class does it for you.
>
I developed the HTTP client class not just as a mere curl wrapper, but
to actually add some benefits on top of curl/fsockopen. So, it was meant
to add flexibility, not to remove it.
>
That is why I questioned you about you flexibility statement. Maybe you
tried an old version of the HTTP client class and you found some
limitations that no longer exist. But if you still find it less
flexible, I want to understand what you are talking about.
>
>
>

Manuel,

You are obviously not able to step back and take an objective look at
your classes. I have tried to discuss much of this with you previously,
but you have consistently argued about unrelated things.

I really don't feel like continuing this argument. Please let me know
when you can look at it objectively, and I will be happy to *discuss* it
with you.

I will continue to recommend cURL for the reasons I have outlined. The
difference is I have no relationship with cURL, other than as a user of
the library.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Paul Lautman]

Re: Automated web browing

mr_marcin wrote:

Hi
>
Does anybody have some idea how to input some text into inputbox on
one page, than press some button on that page, that will load another
page, and finally read the responde? Suppose I want to write a price
comparision engine, where I would like to parse shops website for
price each time user wants.
>
I have found similar feature in Symfony framework, called sfBrowser
(or sfTestBrowser). These are made for automated functional testing,
but should provide the functinality I am requesting.
>
The question is: will this be efficient enough? Maybe there are other
ways to achieve this? Of course I can always try to make it more
manually - look for some pattern in url (search is usually done via
GET), and parse output html.
>
Thanks for help
Marcin

Take a look at Snoopy
http://sourceforge.net/project/showf...?group_id=2091