Grabbing data in parallel connections?

[Joe]

I need to connect to 10 web sites to grab content from them. I would
like to connect to each site simultaneously so that I can obtain the
data as fast as possible.

I am familar doing this with perl by using parallel sockets or the
module LWP-Parallel. So what would be the best method to do this in
php? Sockets, forks?

Also, if possible, could someone provide me with a good reference site
that will help me accomplish this?


Thanks

[steve]

Re: Grabbing data in parallel connections?

"Joe114" wrote:[color=blue]
> I need to connect to 10 web sites to grab content from them. I[/color]
would[color=blue]
> like to connect to each site simultaneously so that I can obtain[/color]
the[color=blue]
> data as fast as possible.
>
> I am familar doing this with perl by using parallel sockets or the
> module LWP-Parallel. So what would be the best method to do this[/color]
in[color=blue]
> php? Sockets, forks?
>
> Also, if possible, could someone provide me with a good reference[/color]
site[color=blue]
> that will help me accomplish this?
>
>
> Thanks[/color]

I don’t know if it is possible in PHP. What you can do is fork to unix
and use lynx or wget, and have them write the data to different files.
Hope others have better solutions.

--
http://www.dbForumz.com/ This article was posted by author's request
Articles individually checked for conformance to usenet standards
Topic URL: http://www.dbForumz.com/PHP-Grabbing...ict142156.html
Visit Topic URL to contact author (reg. req'd). Report abuse: http://www.dbForumz.com/eform.php?p=475432

[Chung Leong]

Re: Grabbing data in parallel connections?


"Joe" <jkindness@roge rs.com> wrote in message
news:5667e6ba.0 408221723.11a6d c7d@posting.goo gle.com...[color=blue]
> I need to connect to 10 web sites to grab content from them. I would
> like to connect to each site simultaneously so that I can obtain the
> data as fast as possible.
>
> I am familar doing this with perl by using parallel sockets or the
> module LWP-Parallel. So what would be the best method to do this in
> php? Sockets, forks?
>
> Also, if possible, could someone provide me with a good reference site
> that will help me accomplish this?
>
>
> Thanks[/color]

Just use good'o fopen().

See http://www.php.net/stream_set_blocking/ and
http://www.php.net/stream_select/.

[Frank]

Re: Grabbing data in parallel connections?

Hi Joe,

I've done something similar a few years ago, using nonblocking sockets.
You could create your 10 connections and then read from them in a loop.
Have a look at the PHP socket documentation. But using "raw" sockets, you
will have to implement (part of) the HTTP yourself. But it might be that
today there are new functions in PHP -like fopen- that support HTTP and can
be used in nonblocking mode. I am not really up-to-date at the moment.
If the docs are of no help, let me know and I look up the code and send it
to you.


Cheers
Frank

P.S.
I just remembered that some time ago someone else posted nearly the same
questions and I send nearly the same reply:)
Or is it just a deja vu?


Joe wrote:
[color=blue]
> I need to connect to 10 web sites to grab content from them. I would
> like to connect to each site simultaneously so that I can obtain the
> data as fast as possible.
>
> I am familar doing this with perl by using parallel sockets or the
> module LWP-Parallel. So what would be the best method to do this in
> php? Sockets, forks?
>
> Also, if possible, could someone provide me with a good reference site
> that will help me accomplish this?
>
>
> Thanks[/color]

[R. Rajesh Jeba Anbiah]

Re: Grabbing data in parallel connections?

Frank <f.passek@web.d e> wrote in message news:<2p37aoFg7 0r7U1@uni-berlin.de>...[color=blue]
> Hi Joe,
>
> I've done something similar a few years ago, using nonblocking sockets.
> You could create your 10 connections and then read from them in a loop.
> Have a look at the PHP socket documentation. But using "raw" sockets, you
> will have to implement (part of) the HTTP yourself. But it might be that
> today there are new functions in PHP -like fopen- that support HTTP and can
> be used in nonblocking mode. I am not really up-to-date at the moment.
> If the docs are of no help, let me know and I look up the code and send it
> to you.[/color]

If you don't mind to share your thoughts, you can post a piece/part
of your code so that it helps others.

--
| Just another PHP saint |
Email: rrjanbiah-at-Y!com

[Frank]

Re: Grabbing data in parallel connections?

R. Rajesh Jeba Anbiah wrote:[color=blue]
>
> If you don't mind to share your thoughts, you can post a piece/part
> of your code so that it helps others.
>[/color]

Hi,

sure, I found it between some old pizza boxes;)
I have not tested it again, so I don't know if it will work with a more
recent PHP version. Thus it might not work, it might have bugs, but it
provides a starting point.

Have fun
Frank



When I recall correctly, you can use it as follows:

$this->Connection=n ew HTTPConnection( new URL("http:://www.whatever.co m"),
"", 30, NONBLOCKING);
if (!$this->Connection->makeGETRequest ())
die($this->Connection->getErrorMessag e());

while($this->Connection->readBlock())
{}

print $this->Connection->getContents( );



file http_class.inc. php

<?php

include_once "class.url.inc. php";

//define constants for use with choice of blocking/nonblocking mode
/** @global string For use with HTTPConnection constructor, sets connection
blocking */
define ("BLOCKING", 1);

/** @global string For use with HTTPConnection constructor, sets connection
nonblocking */
define ("NONBLOCKIN G", 0);

/** Objects of this class provide a way to access http-resources.
* Currently only the GET-Method is supported
*/
class HTTPConnection
{
//-----------------
//-- public methods

/** Constructor creates an unconnected object and initializes
member-variables
* @acces public
* @param &$url a reference to an url-object
* @param $sendhd string containing a valid http-header to send along,
default value is ""
* @param $timeout numeric timeout in seconds for the connect call of
fsockopen, default is 1s
* @param $blocking boolean flag, if set to true (default) the socket
will be opened in blocking mode
*/
function HTTPConnection( &$url, $sendhd="", $timeout=1,
$blockmode=BLOC KING)
{
$this->Status=false ;
$this->URL = $url;
$this->IP = false;
$this->Socket=false ;
$this->Timeout=$timeo ut;
$this->Buffer=false ;
$this->Header=false ;
$this->Contents=false ;
$this->SendHeader=$se ndhd;
$this->Blocking=$bloc kmode;
}


/** This method opens a socket to a http-resource and sends a GET-Request
* If the connection is opened in blocking mode, the respond of the
resource
* is fetched, parsed and stored in the internal buffers
* When using an unblocking connection, the client has to call readBlock
to fetch
* the respond
*
* @access public
* @returns true if the request was successfull, false otherwise
* @see readBlock()
*/
function makeGETRequest( )
{

//make dns-lookup and connect socket
if ($this->findIP())
if($this->openSocket() )
{
$path=$this->URL->getPath();
if (strlen($qs=$th is->URL->getQueryString ()))
$path.="?$qs";
$str="GET $path HTTP/1.0\r\nHost:".$ this->URL->getHost()."\r\ n"
$this->SendHeader."\r \n\r\n";

//write http-request
if (fputs($this->Socket, $str) != strlen($str))
{
$this->ErrorMessage=" Unknown error writing request to socket";
return false;
}
//clear buffer
$this->Buffer="";

//when using blocking mode fetch, parse and store respond
if ($this->Blocking)
{
while (!feof($this->Socket))
$this->Buffer.=fread( $this->Socket, 1024);

return $this->parseBuffer( );
}

return true;
}
return false;
}

/** Returns the header of the repond from the last http-request as an
associative array
*
* @access public
* @returns array
*/
function getHeader()
{
return $this->Header;
}

/** Returns the body of the repond from the last http-request
*
* @access public
* @returns string
*/
function getContents()
{
return $this->Contents;
}

/** Returns the textual description of the last error, may be an empty
string, if no error occured so far
*
* @access public
* @returns string
*/
function getErrorMessage ()
{
return $this->ErrorMessage ;
}

/** Reads $blocksize bytes from socket and appends them to internal buffer
*
* @access public
* @returns false if an error occured, false otherwise
*/
function readBlock($bloc ksize=1024)
{
$this->Buffer.=fread( $this->Socket, $blocksize);
if (!feof($this->Socket))
return true;

$this->parseBuffer( );
return false;
}

/** Closes the socket
*
* @access public
* @returns true if the socket could be closed, false otherwise
*/
function close()
{
return fclose($this->Socket);
}

//-----------------
//-- private methods

/** Look up IP of host and store it
*
* @access private
* @returns IP when dns-lookup was succesfull, false otherwise
*/
function findIP()
{
$h=$this->URL->getHost();
if (ereg("[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}", $h))
return $this->IP=$h;

if ( ($ip=gethostbyn ame($h)) == $h )
{
$this->ErrorMessage=" Could not find DNS-Entry for host: $h";
return false;
}

return $this->IP=$ip;
}



/** Opens a socket
*
* @access private
* @returns true if the socket could be opened, false otherwise
*/
function openSocket()
{
if ($this->Socket=fsockop en($this->IP, $this->URL->getPort(), &$errno,
&$errstr, $this->Timeout))
{
socket_set_bloc king($this->Socket, $this->Blocking);
return true;
}

if (!strlen($errst r))
$this->ErrorMessage=" Unknown error opening Socket";
else
$this->ErrorMessage=" Error $errno opening Socket: $errstr";

return false;
}

/** Parse http-respond as stored in buffer into header and body
*
* @access private
* @returns true
*/
function parseBuffer()
{
$this->Header="";
if ( ($p=strpos($thi s->Buffer, "\r\n\r\n"))!== false )
$this->parseHeader(su bstr($this->Buffer, 0, $p));
else
$p=-4;
$this->Contents=subst r($this->Buffer, $p+4);
return true;
}

/** Parse http-header into associative array
* The array keys are the names of the headerfields in uppercase letters
*
* @access priavate
* @param $hd string holding the http-header
* @returns void
*/
function parseHeader($hd )
{
$lines=explode( "\n", $hd);
$n=count($lines );

for ($i=0; $i<$n; ++$i)
if (strlen(trim($l ines[$i])))
{
$parts=split("^[a-zA-Z0-9_-]+:", $lines[$i]);
if (count($parts)> 1)
{
$key=substr($li nes[$i], 0, strpos($lines[$i], ":"));
$this->Header[strtoupper(trim ($key))]=trim($parts[1]);
}
}
}
};

?>



file class.url.inc.p hp

<?php
function URL($urlstr)
{
$u=trim($urlstr );
if (strlen($u))
$this->parseURL($u) ;
}

function parseURL($urlst r)
{
$this->IsValid=fals e;
//--- ist ein Protokol angegeben?
$parts=explode( ":/", $urlstr);
if (count($parts)< 2)
$urlstr="http://$urlstr";

$parts=@parse_u rl($urlstr);

$this->Protocol = strtolower($par ts["scheme"]);
$this->Host = strtolower($par ts["host"]);
$this->Path = $parts["path"];
$this->Port = $parts["port"];
$this->Query = $parts["query"];
$this->Fragment = $parts["fragment"];
$this->User = $parts["user"];
$this->Password = $parts["pass"];


if (!strlen($this->Protocol))
$this->Protocol="http ";

if (!strlen($this->Port))
$this->Port="80";

if (!strlen($this->Path))
$this->Path="/";

if ($this->Host)
$this->IsValid=true ;
else return false;

$this->buildCanonical ();
return true;
}


function getHost() { return $this->Host; }

function getPath(){ return $this->Path; }


function getPort() { return $this->Port; }


function getQueryString( ){ return $this->Query;}


function getAsString()
{
return $this->Canonical;
}

function buildCanonical( )
{
$this->Canonical= strtolower($thi s->Protocol).
"://".
strtolower($thi s->Host);

if ($this->Port!="80")
$this->Canonical.=":" .$this->Port;

$this->Canonical.=$th is->Path;

if (strlen($this->Query))
$this->Canonical.="?" .$this->Query;
if (strlen($this->Fragment))
$this->Canonical.="#" .$this->Fragment;
return true;
}

function isValid()
{
return $this->IsValid;
}

var $Canonical=fals e;
var $IsValid=false;
var $Host="";
var $Port=80;
var $Protocol="http ";
var $Path="/";
}
?>