Quickest way to Request a Page?

[samthemist]

For example: POST, GET www.thescripts. com etc

What's the fastest way of doing this? As the program i'm making relys on speed, so the fastest way would help alot.

I have tried an equivalent in VB6 (don't ask why), and it's doing around 500ms on a 8mbps connection.

Is there one that can go at that speed or faster?

[miller]

http://search.cpan.org/search?query=lw ptut

- Miller

[KevinADC]

no script will speed up your connection speed.

[miller]

The only way to speed things up is to code for a "304" response (meaning page content hasn't changed). You do this by working with the Last-Modified and ETag headers. There's plenty of documentation available for this, so just start reading up on lwp.

- Miller

[samthemist]

no script will speed up your connection speed.

I understand that, though there will always be a script that is faster then another at requesting pages.

Is LWP definitely the fastest method?

[miller]

Is LWP definitely the fastest method?

No. All implementations will be pretty much the same. The only real speed improvements are made outside of the actual fetching process. Is the process pre-loaded? Do you check for a 304 response? What are you using for parsing (easy time crunch)?

All fetching methods will use the exact same amount of time. LWP is just the standard and most comprehensive utility.

- Miller

[samthemist]

Well i used LWP, and tested it and another method i found with GetTickCount. LWP took roughly 1750-2000ms and the other method 1000-1250ms

sub Request {
  ($host, $file) = ($_[1] =~ m#([^/]*)(/.*)?$#i);
  Request: {
    socket(sock, AF_INET, SOCK_STREAM, getprotobyname('tcp'));
    connect(sock, pack('S n a4 x8', AF_INET, 80, (gethostbyname($host))[4]));
    select(sock); $|=1; select(STDOUT); 
    if ($_[3] != '') { $referer = "Referer: $_[3]\015\012" } else { $referer = ''; }
    if (length($_[2]) == 0) {
      print sock "$_[0] $file HTTP/1.1\015\012",
                "Accept: */*\015\012",
                "$referer",
                "Accept-Language: en-ca\015\012",
                "UA-CPU: x86\015\012",
                "Accept-Encoding: gzip, deflate\015\012",
                "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506)\015\012",
                "Host: $host\015\012",
                "Connection: Keep-Alive\015\012",
                "$fcookie\015\012";
    } elsif ($_[0] == 'POST') {
      print sock "POST $file HTTP/1.1\015\012",
                "Accept: */*\015\012",
                "$referer",
                "Accept-Language: en-ca\015\012",
                "Content-Type: application/x-www-form-urlencoded\015\012",
                "UA-CPU: x86\015\012",
                "Accept-Encoding: gzip, deflate\015\012",
                "User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; SLCC1; .NET CLR 2.0.50727; .NET CLR 3.0.04506)\015\012",
                "Host: $host\015\012",
                "Content-Length: ", length($_[2]), "\015\012",
                "Connection: Keep-Alive\015\012",
                "Cache-Control: no-cache\015\012",
                "$fcookie\015\012$_[2]";
    }
    vec($rin = '', fileno(sock), 1) = 1;
    select($rin, undef, undef, 60) || die "ERROR: $!";
    $numread = 0;
    while (($numread < 5) && ($thisread = read(sock, $status_line, 5 - $numread, $numread))) {
      $numread+= $thisread; }
    defined($thisread) || die "ERROR: $!";
    
    $content = '';
    if ($status_line !~ m#^HTTP/#) {
      $content.= $_ while read(sock, $_, 16384);
    } else {
      $status_line.= <sock>;
      ($status_code) = ($status_line =~ m#^HTTP/\d+\.\d+\s+(\d+)#);
      $headers = '';
      while (<sock>) { last if /^\015?\012/; $headers.= $_; }
      $headers =~ s/\015?\012[ \t]+/ /g ;
      
      @headers = split(/\n/, $headers);
      foreach $header (@headers) {
        if ($header =~ /^set-cookie:/i) {
          @cookies = split(/:.(.+?)=/, $header);
          @value = split(/(.+?);/, @cookies[2]);
          if ($cookie !~ /@cookies[1]=/) {
            $cookie.= @cookies[1].'=';
            $cookie.= @value[1].'; '; }
          elsif ($cookie =~ /@cookies[1]=/) {
            $cookie =~ substr($cookie, index($cookie, 
              @cookies[1]) + length(@cookies[1]) + 1, 
              index($cookie, ";", index($cookie, @cookies[1])) - 
              index($cookie, @cookies[1]) - length(@cookies[1]) - 1, @value[1]);
          }
        }
      }
      if (substr($cookie, length($cookie) - 4, length($cookie)) =~ /;./) {
        $fcookie = "Cookie: ".substr($cookie, 0, length($cookie) - 2)."\015\012"; }

      if ($status_code =~ /^(301|302|303)$/) {
        unless (($file) = ($headers =~ /^location:[ \t]*(\S*)/im)) {
          die "ERROR: No location in redirect"; }
        ($numredirects++ > 5) && die "ERROR: Too many location redirects";
        close(sock); $file.= 'phtml'; redo Request;
      }
      ($status_code == 200) || die "ERROR: HTTP Response not OK";
      $content.= $_ while read(sock, $_, 16384);
    }
    close(sock);
  }
  if ($headers =~ /content-encoding:.?gzip/i) {
    $content = Compress::Zlib::memGunzip($content); }
  return $content;
}

$result = Request("GET", "thescripts.com", "thescripts.com")

Is there anyway to make this even faster?

[miller]

Well i used LWP, and tested it and another method i found with GetTickCount. LWP took roughly 1750-2000ms and the other method 1000-1250ms

....

Is there anyway to make this even faster?

Are you certain that the time differential is from the actual processing time? Or is that the total time, therefore including the compile time of the script?

There are definitely modules out there with a smaller footprint than LWP. But if you discount the compile time, most likely all of them will be about the same with regard to processing time. Again, this is just a logical deduction rather than one tested intensely in practice. However, I would guess that if you use Time::HiRes to get the time differentials for the actual processing, they would be roughly equivalent.

That is why I mentioned the script being cached, instead of loading each time for processing. Most of the speed improvements will be made irregardless of the implementation method for the fetching. This is my hypothesis.

- Miller

[KevinADC]

maybe if you run your scripts with fastcgi or mod_perl you can boost the overall speed. But since perl is interpreted it has the extra compile time to account for. The operating system might be your best bet unless it has to cross platform compatible.

[samthemist]

I basically used Win32::GetTickC ount before and after a post.

Then took them away to get the difference, thus the time to post.

[miller]

Add the timing within the script.

[CODE=perl]
use File::Slurp qw(append_file) ;
use Time::HiRes qw(gettimeofday );

my $start = gettimeofday():

# Do all your processing here.
# ....

my $end = gettimeofday();
append_file("$0 .timelog", scalar(localtim e) . " took " . ($end - $start) . "\n");
[/CODE]

- Miller

[samthemist]

use File::Slurp

i don't think i have that, where can i get it?

[miller]

cpan File::Slurp
cpanfaq How do I install perl modules?

- Miller

[samthemist]

cpan File::Slurp
cpanfaq How do I install perl modules?

- Miller

Wed Jul 11 20:20:51 2007 took 1.390625