Web crawler on python

[sonich]

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

[Mr.SpOOn]

Re: Web crawler on python

On Sun, Oct 26, 2008 at 9:54 PM, sonich <SoniaTerpugova @gmail.comwrote :

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

What about BeautifulSoup?

http://www.crummy.com/software/BeautifulSoup/

[James Mills]

Re: Web crawler on python

On Mon, Oct 27, 2008 at 6:54 AM, sonich <SoniaTerpugova @gmail.comwrote :

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/examples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"

[Support Desk]

RE: Web crawler on python



-----Original Message-----
From: James Mills [mailto:prologic @shortcircuit.n et.au]
Sent: Sunday, October 26, 2008 5:26 PM
To: sonich
Cc: python-list@python.org
Subject: Re: Web crawler on python

On Mon, Oct 27, 2008 at 6:54 AM, sonich <SoniaTerpugova @gmail.comwrote :

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.wsgi/projects/pymills/file/330d047ff663/e

xamples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"

[Alex]

Re: Web crawler on python

On Oct 26, 9:54 pm, sonich <SoniaTerpug... @gmail.comwrote :

I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?

You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
or you can have a look at my project on launchpad
https://code.launchpad.net/~esaurito...r/experimental.
It's a single site crawler but you can easily modified it.

Bye.

Alex