Robots.txt ?

[Maverick]

How can I regulate an agent or bot to my site monthly ?
they are visiting to often
I'd like to allow them monthly only

any ideas ?

tia

[Martin Kissner]

Re: Robots.txt ?

Maverick wrote :[color=blue]
> How can I regulate an agent or bot to my site monthly ?
> they are visiting to often
> I'd like to allow them monthly only
>[/color]
Try:
<meta name="revisit-after" content="30 days">

HTH
Martin

--
Epur Si Muove (Gallileo Gallilei)

[Markku Uttula]

Re: Robots.txt ?

Maverick wrote:[color=blue]
> How can I regulate an agent or bot to my site monthly ?
> they are visiting to often
> I'd like to allow them monthly only
>
> any ideas ?[/color]

You could try generateing robots.txt dynamically. Based on the user
agent, just decide whether you wish to let it in today or not, and
based on that decision, output the user agent deny for it or don't.

It's good to know however that not all robots follow the robots.txt
(though the well behaving ones do). Another thing to note, is that
Google (for example) drops the pages it can't access from it's
database.

--
Markku Uttula

[Maverick]

Re: Robots.txt ?

Where does that go;
does that have to be in the head section of every page on the whole web
?

tia

[Martin Kissner]

Re: Robots.txt ?

Maverick wrote :[color=blue]
> Where does that go;
> does that have to be in the head section of every page on the whole web
> ?[/color]

you might want to read this:

http://www.robotstxt.org/wc/exclusion.html

Or just use Google with search terms like:
robot.txt
meta robot
and so on.

HTH
Martin

--
Epur Si Muove (Gallileo Gallilei)