pagecrawling websites with Python

[writeson]

Hi all,

We've got an application we wrote in Python called pagecrawler that
generates a list of URL's based on sql queries. It then runs through
this list of URL's 'browsing' one of our staging servers for all those
URL's. We do this to build the site dynamically, but each page
generated by the URL is saved as a static HTML file. Anyway, the
pagecrawler program uses Python threads to try and build the pages as
fast as it can. The list of URL's is stored in a queue and the thread
objects get URL's from the queue and run them till the queue is empty.
This works okay but it still seems to take a long time to build the
site this way, even though the actual pages only take milliseconds to
run (the pages are generated with PHP on separate server). Does anyone
have any insight if this is a reasonable approach to build web pages,
or if we should look at another design?

Thanks in advance,
Doug

[Swaroop C H]

Re: pagecrawling websites with Python

On 1 Apr 2005 11:58:11 -0800, writeson <writeson@chart er.net> wrote:[color=blue]
> We've got an application we wrote in Python called pagecrawler that[/color]
<snip />[color=blue]
> Does anyone have any insight if this is a reasonable approach to build web pages,
> or if we should look at another design?[/color]

I don't have an answer to your particular question, but maybe you can
have a look at how the HarvestMan works:

http://freshmeat.net/projects/harvestman

Regards,
--
Swaroop C H
Blog: http://www.swaroopch.info
Book: http://www.byteofpython.info

[writeson]

Re: pagecrawling websites with Python

Swaroop,

Thanks for the reply, I'll take a look at HarvestMan and see if we can
use it directly, or get some ideas from the source code. :)

Doug