Python "robots.txt" parser broken since 2003

[John Nagle]

This bug, "[ 813986 ] robotparser interactively prompts for username and
password", has been open since 2003. It killed a big batch job of ours
last night.

Module "robotparse r" naively uses "urlopen" to read "robots.txt " URLs.
If the server asks for basic authentication on that file, "robotparse r"
prompts for the password on standard input. Which is rarely what you
want. You can demonstrate this with:

import robotparser
url = 'http://mueblesmoraleda .com' # this site is password-protected.
parser = robotparser.Rob otFileParser()
parser.set_url( url)
parser.read() # Prompts for password

That's the tandard, although silly, "urllib" behavior.

This was reported in 2003, and a patch was uploaded in 2005, but the patch
never made it into Python 2.4 or 2.5.

A temporary workaround is this:

import robotparser
def prompt_user_pas swd(self, host, realm):
return None, None
robotparser.URL opener.prompt_u ser_passwd = prompt_user_pas swd # temp patch


John Nagle

[Terry Reedy]

Re: Python "robots.tx t" parser broken since 2003


"John Nagle" <nagle@animats. comwrote in message
news:FvtWh.1182 4$YL5.6282@news svr29.news.prod igy.net...
| This was reported in 2003, and a patch was uploaded in 2005, but the
patch
| never made it into Python 2.4 or 2.5.

If the patch is still open, perhaps you could review it.

tjr

[John Nagle]

Re: Python &quot;robots.tx t&quot; parser broken since 2003

Terry Reedy wrote:

"John Nagle" <nagle@animats. comwrote in message
news:FvtWh.1182 4$YL5.6282@news svr29.news.prod igy.net...
| This was reported in 2003, and a patch was uploaded in 2005, but the
patch
| never made it into Python 2.4 or 2.5.
>
If the patch is still open, perhaps you could review it.
>

I tried it on Python 2.4 and it's in our production system now.
But someone who regularly does check-ins should do this.

John Nagle

[Steven Bethard]

Re: Python &quot;robots.tx t&quot; parser broken since 2003

John Nagle wrote:

Terry Reedy wrote:

>"John Nagle" <nagle@animats. comwrote in message
>news:FvtWh.118 24$YL5.6282@new ssvr29.news.pro digy.net...
>| This was reported in 2003, and a patch was uploaded in 2005, but the
>patch
>| never made it into Python 2.4 or 2.5.
>>
>If the patch is still open, perhaps you could review it.
>>

I tried it on Python 2.4 and it's in our production system now.
But someone who regularly does check-ins should do this.

If you post such a review (even just the short sentence above) to the
patch tracker, it often increases the chance of someone committing the
patch.

Steve

[Nikita the Spider]

Re: Python &quot;robots.tx t&quot; parser broken since 2003

In article <FvtWh.11824$YL 5.6282@newssvr2 9.news.prodigy. net>,
John Nagle <nagle@animats. comwrote:

This bug, "[ 813986 ] robotparser interactively prompts for username and
password", has been open since 2003. It killed a big batch job of ours
last night.
>
Module "robotparse r" naively uses "urlopen" to read "robots.txt " URLs.
If the server asks for basic authentication on that file, "robotparse r"
prompts for the password on standard input. Which is rarely what you
want. You can demonstrate this with:
>
import robotparser
url = 'http://mueblesmoraleda .com' # this site is password-protected.
parser = robotparser.Rob otFileParser()
parser.set_url( url)
parser.read() # Prompts for password
>
That's the tandard, although silly, "urllib" behavior.

John,
robotparser is (IMO) suboptimal in a few other ways, too.
- It doesn't handle non-ASCII characters. (They're infrequent but when
writing a spider which sees thousands of robots.txt files in a short
time, "infrequent " can become "daily").
- It doesn't account for BOMs in robots.txt (which are rare).
- It ignores any Expires header sent with the robots.txt
- It handles some ambiguous return codes (e.g. 503) that it ought to
pass up to the caller.

I wrote my own parser to address these problems. It probably suffers
from the same urllib hang that you've found (I have not encountered it
myself) and I appreciate you posting a fix. Here's the code &
documentation in case you're interested:

http://NikitaTheSpider.com/python/rerp/

Cheers

--
Philip

http://NikitaTheSpider.com/

Whole-site HTML validation, link checking and more

[John Nagle]

Re: Python &quot;robots.tx t&quot; parser broken since 2003

Steven Bethard wrote:

John Nagle wrote:
>

>Terry Reedy wrote:
>>

>>"John Nagle" <nagle@animats. comwrote in message
>>news:FvtWh.11 824$YL5.6282@ne wssvr29.news.pr odigy.net...
>>| This was reported in 2003, and a patch was uploaded in 2005, but
>>the patch
>>| never made it into Python 2.4 or 2.5.
>>>
>>If the patch is still open, perhaps you could review it.
>>>

> I tried it on Python 2.4 and it's in our production system now.
>But someone who regularly does check-ins should do this.

>
>
If you post such a review (even just the short sentence above) to the
patch tracker, it often increases the chance of someone committing the
patch.
>
Steve

OK, updated the tracker comments.

John Nagle