Is Python good for web crawlers?

[Tempo]

I was wondering if python is a good language to build a web crawler
with? For example, to construct a program that will routinely search x
amount of sites to check the availability of a product. Or to search
for news articles containing the word 'XYZ'. These are just random
ideas to try to explain my question a bit further. Well if you have an
opinion about this please let me know becasue I am very interested to
hear what you have to say. Thanks.

[Andrew Gwozdziewycz]

Re: Is Python good for web crawlers?

On 7 Feb 2006 08:33:28 -0800, Tempo <bradfordh@gmai l.com> wrote:[color=blue]
> I was wondering if python is a good language to build a web crawler
> with? For example, to construct a program that will routinely search x
> amount of sites to check the availability of a product. Or to search
> for news articles containing the word 'XYZ'. These are just random
> ideas to try to explain my question a bit further. Well if you have an
> opinion about this please let me know becasue I am very interested to
> hear what you have to say. Thanks.
>[/color]

Google supplies a basic webcrawler as a google desktop plugin called
Kongulo (http://sourceforge.net/projects/goog-kongulo/) which is
written in python. I would think python would be perfect for this sort
of application. Your bottleneck is always going to be downloading the
page.

--
Andrew Gwozdziewycz <apgwoz@gmail.c om>

http://ihadagreatview.org

http://plasticandroid.org

[Tempo]

Re: Is Python good for web crawlers?

Why do you say that the bottleneck of the crawler will always be
downloading the page? Is it becasue there isn't already a modual to do
this and I will have to start from scratch? Or a bandwidth issue?

[Diez B. Roggisch]

Re: Is Python good for web crawlers?

Tempo wrote:
[color=blue]
> Why do you say that the bottleneck of the crawler will always be
> downloading the page? Is it becasue there isn't already a modual to do
> this and I will have to start from scratch? Or a bandwidth issue?[/color]

Because of bandwidth - not necessarily yours directly, but the maximum flow
between your uplink and the site in question. It will always take at least
a fractioin of a second up to several seconds until the data is there - in
that time, lots of python code can run.

Diez

[Tempo]

Re: Is Python good for web crawlers?

Does a web crawler have to download an entire page if it only needs to
check if the product is in stock on a page? Or if it just needs to
search for one match of a certain word on a page?

[Tim Parkin]

Re: Is Python good for web crawlers?

Tempo wrote:
[color=blue]
>Does a web crawler have to download an entire page if it only needs to
>check if the product is in stock on a page? Or if it just needs to
>search for one match of a certain word on a page?
>
>
>[/color]
Typically you would download the whole html file and then perform any
analysis on this. It is possible to parse the stream of characters as
they come back from the server but this would statistically only reduce
the download time by a half (presuming the item you want is of a single
byte in length and can appear anywhere in the html). In reality, unless
the pages you are requesting are very large (200k+) or your bandwidth
very expensive (in time and/or capacity) then it is probably easier for
you to just download the whole file.

I would recommend that you use BeautifulSoup to parse badly formatted
html documents (which is most of the web). (google 'beautiful soup' and
you should find it easily).

Tim Parkin

[Tempo]

Re: Is Python good for web crawlers?

I took your advice and got a copy of BeautifulSoup, but I am having
trouble installing the module. Any advice? I noticed that I just can't
put it into the 'lib' directory of python to install it.

[Tim Parkin]

Re: Is Python good for web crawlers?

Tempo wrote:
[color=blue]
>I took your advice and got a copy of BeautifulSoup, but I am having
>trouble installing the module. Any advice? I noticed that I just can't
>put it into the 'lib' directory of python to install it.
>
>
>[/color]
Just save the file in the same directory as your project then you should
be able to use the sample code.

Tim Parkin

[Paul Rubin]

Re: Is Python good for web crawlers?

"Tempo" <bradfordh@gmai l.com> writes:[color=blue]
> I was wondering if python is a good language to build a web crawler
> with? For example, to construct a program that will routinely search x
> amount of sites to check the availability of a product. Or to search
> for news articles containing the word 'XYZ'. These are just random
> ideas to try to explain my question a bit further.[/color]

I've written a few of these in Python. The language itself is fine
for this. The built-in libraries do most of what you'd hope, though
they have room for improvement. Generally I use urllib.read() to get
the whole html page as a string, then process it from there. I just
look for the substrings I'm interested in, making no attempt to
actually parse the html into a DOM or anything like that.

[Xavier Morel]

Re: Is Python good for web crawlers?

Paul Rubin wrote:[color=blue]
> Generally I use urllib.read() to get
> the whole html page as a string, then process it from there. I just
> look for the substrings I'm interested in, making no attempt to
> actually parse the html into a DOM or anything like that.
>[/color]
BeautifulSoup works *really* well when you want to parse the source
(e.g. when you don't want to use string matching, or when the structures
you're looking for are a bit too complicated for simple string
matching/substring search)

The API of the package is extremely simple, straightforward and... obvious.

[Paul Rubin]

Re: Is Python good for web crawlers?

Xavier Morel <xavier.morel@m asklinn.net> writes:[color=blue]
> BeautifulSoup.. .
> The API of the package is extremely simple, straightforward and... obvious.[/color]

I did not find that. I spent a few minutes looking at the
documentation and it wasn't obvious at all how to use it. Maybe I
could have figured it out with more effort, but I got whatever the
immediate task was done without it instead. It does look like a nice
package but the docs need improvement.

[Tempo]

Re: Is Python good for web crawlers?

I agree. I think the way that I will learn to use most of it is by
going through the source code.

[Magnus Lycka]

Re: Is Python good for web crawlers?

Tempo wrote:[color=blue]
> I was wondering if python is a good language to build a web crawler
> with? For example, to construct a program that will routinely search x
> amount of sites to check the availability of a product. Or to search
> for news articles containing the word 'XYZ'. These are just random
> ideas to try to explain my question a bit further. Well if you have an
> opinion about this please let me know becasue I am very interested to
> hear what you have to say. Thanks.[/color]

I dunno, but there are these two guys, Sergey Brin and Lawrence Page,
who wrote a web crawler in Python. As far as I understood, they were
fairly successful with it. I think they called their system Koogle,
Bugle, or Gobble or something like that. Goo...can't remember.

See http://www-db.stanford.edu/~backrub/google.html

They've also employed some clever Python programmers, such as Greg
Stein, Alex Martelli (isn't he a bot?) and some obscure dutch
mathematician called Guido van something. It seems they still like
Python.

[Alex Martelli]

Re: Is Python good for web crawlers?

Magnus Lycka <lycka@carmen.s e> wrote:
...[color=blue]
> I dunno, but there are these two guys, Sergey Brin and Lawrence Page,
> who wrote a web crawler in Python. As far as I understood, they were
> fairly successful with it. I think they called their system Koogle,
> Bugle, or Gobble or something like that. Goo...can't remember.
>
> See http://www-db.stanford.edu/~backrub/google.html[/color]

Yeah, I've heard of them, too.

[color=blue]
> They've also employed some clever Python programmers, such as Greg
> Stein, Alex Martelli (isn't he a bot?) and some obscure dutch
> mathematician called Guido van something. It seems they still like
> Python.[/color]

Bot? me? did I fail a Turing test again without even noticing?!


Alex

[Simon Brunning]

Re: Is Python good for web crawlers?

On 2/8/06, Alex Martelli <aleaxit@yahoo. com> wrote:[color=blue]
>
> Bot? me? did I fail a Turing test again without even noticing?![/color]

If you'd noticed the test, you'd have passed.

;-)

--
Cheers,
Simon B,
simon@brunningo nline.net,

http://www.brunningonline.net/simon/blog/