Lynx and wget - redirect problems for PHP scripts

[Martin]

I have a PHP script that should be run a few times a day. Currently I run
this script manually as I haven't been able to set up Lynx or wget to run it
as a cronjob. Or to be more specific: the script works, but both wget and
Lynx fail after 10 or 20 redirects, then it warns:

wget: 20 redirections exceeded.
Lynx: Alert!: Redirection limit of 10 URL's reached.

What the scripts does is download various HTML pages (allowed by the
owners) - and some of them have 20-100 pages. The script loops through the
pages and for each new does a redirect. After 10 or 20 it is stopped, due to
Lynx and wget.

Any idea how to fix this ? Can the 10 or 20 limits be modified ?

Martin.

[Jochen Daum]

Re: Lynx and wget - redirect problems for PHP scripts

Hi Martin!

On Wed, 10 Sep 2003 22:31:22 +0200, "Martin" <marjor@home.se > wrote:
[color=blue]
>I have a PHP script that should be run a few times a day. Currently I run
>this script manually as I haven't been able to set up Lynx or wget to run it
>as a cronjob. Or to be more specific: the script works, but both wget and
>Lynx fail after 10 or 20 redirects, then it warns:
>
>wget: 20 redirections exceeded.
>Lynx: Alert!: Redirection limit of 10 URL's reached.
>
>What the scripts does is download various HTML pages (allowed by the
>owners) - and some of them have 20-100 pages. The script loops through the
>pages and for each new does a redirect. After 10 or 20 it is stopped, due to
>Lynx and wget.
>
>Any idea how to fix this ? Can the 10 or 20 limits be modified ?[/color]

Maybe ask this question in comp.os.linux.a pps, but maybe you can
modify your script, so that it keeps a database table of pages to
download. Then call it every minute and stop it after 55 seconds. This
way, you don't need redirects and its also more reliable towards slow
connection from the sites you download from.

HTH, Jochen
--
PHP DB Edit Toolkit -- PHP scripts for building
database editing interfaces.

http://sourceforge.net/projects/phpdbedittk/

[Nikolai Chuvakhin]

Re: Lynx and wget - redirect problems for PHP scripts

"Martin" <marjor@home.se > wrote in message
news:<ySL7b.104 283$GK.571@news 2.bredband.com> ...[color=blue]
>
> wget: 20 redirections exceeded.
>
> Any idea how to fix this ? Can the 10 or 20 limits be modified ?[/color]

wget seems to have this limit hard-coded in retr.c:

/* Maximum number of allowed redirections. 20 was chosen as a
"reasonable " value, which is low enough to not cause havoc, yet
high enough to guarantee that normal retrievals will not be hurt by
the check. */

#define MAX_REDIRECTION S 20

Cheers,
NC

[Martin]

Re: Lynx and wget - redirect problems for PHP scripts

Thanks guys - appreciate your help.

Martin.

"Martin" <marjor@home.se > wrote in message
news:ySL7b.1042 83$GK.571@news2 .bredband.com.. .[color=blue]
> I have a PHP script that should be run a few times a day. Currently I run
> this script manually as I haven't been able to set up Lynx or wget to run[/color]
it[color=blue]
> as a cronjob. Or to be more specific: the script works, but both wget and
> Lynx fail after 10 or 20 redirects, then it warns:
>
> wget: 20 redirections exceeded.
> Lynx: Alert!: Redirection limit of 10 URL's reached.
>
> What the scripts does is download various HTML pages (allowed by the
> owners) - and some of them have 20-100 pages. The script loops through the
> pages and for each new does a redirect. After 10 or 20 it is stopped, due[/color]
to[color=blue]
> Lynx and wget.
>
> Any idea how to fix this ? Can the 10 or 20 limits be modified ?
>
> Martin.
>
>[/color]