Google Bot problems?

[Steve]

I have worked on a couple of sites which google's bot visits, partially lists
and then goes away again.

MSN and Yahoo are fine and working.

Can anyone please suggest what, if anything, is wrong with these sites? Frankly
I am a bit baffled at the moment! I am wondering if there is a problem with the
page headers and googlebot?

All suggestions appreciated.

Thanks in advance,




Steve

Sites & Google results are:-

http://tinyurl.com/6tr2b

http://tinyurl.com/5dj6w

[me]

Re: Google Bot problems?

"Steve" <pleasedonotspa mme@ireland.com > wrote in message
news:_mDXd.4915 3$Z14.37683@new s.indigo.ie...[color=blue]
> I have worked on a couple of sites which google's bot visits, partially[/color]
lists[color=blue]
> and then goes away again.
>
> MSN and Yahoo are fine and working.
>
> Can anyone please suggest what, if anything, is wrong with these sites?[/color]
Frankly[color=blue]
> I am a bit baffled at the moment! I am wondering if there is a problem[/color]
with the[color=blue]
> page headers and googlebot?
> All suggestions appreciated.
> Thanks in advance,
> Steve
>
> Sites & Google results are:-
>
> http://tinyurl.com/6tr2b
>
> http://tinyurl.com/5dj6w[/color]

Place this in the head of every page:

<meta name="robots" content="ALL">

IIRC bots may enter from any page so this may help.
Good Luck,
me

[Steve]

Re: Google Bot problems?

>[color=blue]
> Place this in the head of every page:
>
> <meta name="robots" content="ALL">
>
> IIRC bots may enter from any page so this may help.
> Good Luck,
> me
>
>[/color]

Thanks me...will give it a go!



Steve

[Big Bill]

Re: Google Bot problems?

On Wed, 09 Mar 2005 19:07:34 +0000, Steve
<pleasedonotspa mme@ireland.com > wrote:
[color=blue][color=green]
>>
>> Place this in the head of every page:
>>
>> <meta name="robots" content="ALL">
>>
>> IIRC bots may enter from any page so this may help.
>> Good Luck,
>> me
>>
>>[/color]
>
>Thanks me...will give it a go!
>
>
>
>Steve[/color]

Thoughtful of me to mentin it but it won't have any effect. Please may
we see the robots txt for each site.

BB
--
www.kruse.co.uk/ SEO@kruse.demon .co.uk
Affordable SEO!
--

[Big Bill]

Re: Google Bot problems?

On Wed, 9 Mar 2005 09:01:36 -0600, "me" <anonymous@_.co m> wrote:
[color=blue]
>"Steve" <pleasedonotspa mme@ireland.com > wrote in message
>news:_mDXd.491 53$Z14.37683@ne ws.indigo.ie...[color=green]
>> I have worked on a couple of sites which google's bot visits, partially[/color]
>lists[color=green]
>> and then goes away again.
>>
>> MSN and Yahoo are fine and working.
>>
>> Can anyone please suggest what, if anything, is wrong with these sites?[/color]
>Frankly[color=green]
>> I am a bit baffled at the moment! I am wondering if there is a problem[/color]
>with the[color=green]
>> page headers and googlebot?
>> All suggestions appreciated.
>> Thanks in advance,
>> Steve
>>
>> Sites & Google results are:-
>>
>> http://tinyurl.com/6tr2b
>>
>> http://tinyurl.com/5dj6w[/color]
>
>Place this in the head of every page:
>
><meta name="robots" content="ALL">
>
>IIRC bots may enter from any page so this may help.
>Good Luck,
>me[/color]

It won't do anything. Robots ignore meta tags like that as they index
what they can anyway by default. It would be useful if the relevant
robots.txts were displayed here.

BB.

--
www.kruse.co.uk/ SEO@kruse.demon .co.uk
Affordable SEO!
--

[Adrienne]

Re: Google Bot problems?

Gazing into my crystal ball I observed Big Bill <kruse@cityscap e.co.uk>
writing in news:18qv21doap 5jfnq8j7ekdnkam 38damfhb6@4ax.c om:
[color=blue]
> On Wed, 9 Mar 2005 09:01:36 -0600, "me" <anonymous@_.co m> wrote:
>[color=green]
>>"Steve" <pleasedonotspa mme@ireland.com > wrote in message
>>news:_mDXd.49 153$Z14.37683@n ews.indigo.ie.. .[color=darkred]
>>> I have worked on a couple of sites which google's bot visits, partially
>>> lists and then goes away again.
>>>
>>> MSN and Yahoo are fine and working.
>>>
>>> Can anyone please suggest what, if anything, is wrong with these sites?
>>> Frankly I am a bit baffled at the moment! I am wondering if there is a
>>> problem with the page headers and googlebot?
>>> All suggestions appreciated.
>>> Thanks in advance,
>>> Steve
>>>
>>> Sites & Google results are:-
>>>
>>> http://tinyurl.com/6tr2b
>>>
>>> http://tinyurl.com/5dj6w[/color]
>>
>>Place this in the head of every page:
>>
>><meta name="robots" content="ALL">
>>
>>IIRC bots may enter from any page so this may help.
>>Good Luck,
>>me[/color]
>
> It won't do anything. Robots ignore meta tags like that as they index
> what they can anyway by default. It would be useful if the relevant
> robots.txts were displayed here.
>
> BB.
>
> --
> www.kruse.co.uk/ SEO@kruse.demon .co.uk
> Affordable SEO!
> --
>[/color]

1. http://www.barrabooks.com/robots.txt

User-agent: *
Disallow: /picture_library/
Disallow: /Store/
Disallow: /CCS/
Disallow: /webstat/
Disallow: /plesk-stat/
Disallow: /php/

2. http://www.stevenhenson.com/robots.txt

User-agent: *
Disallow: /picture_library/
Disallow: /Store/
Disallow: /CCS/
Disallow: /webstat/
Disallow: /plesk-stat/
Disallow: /php/

I don't see anything unusual do you?

--
Adrienne Boswell

http://www.cavalcade-of-coding.info

Please respond to the group so others can share

[Steve]

Re: Google Bot problems?

> 1. http://www.barrabooks.com/robots.txt[color=blue]
>
> User-agent: *
> Disallow: /picture_library/
> Disallow: /Store/
> Disallow: /CCS/
> Disallow: /webstat/
> Disallow: /plesk-stat/
> Disallow: /php/
>
> 2. http://www.stevenhenson.com/robots.txt
>
> User-agent: *
> Disallow: /picture_library/
> Disallow: /Store/
> Disallow: /CCS/
> Disallow: /webstat/
> Disallow: /plesk-stat/
> Disallow: /php/
>
> I don't see anything unusual do you?
>[/color]

Thank you Adrienne - I see nothing nasty in the robot.txt files as well! It
still leaves me baffled as to why Google is behaving this way.




Steve

[phil_gg04@treefic.com]

Re: Google Bot problems?

> I have worked on a couple of sites which google's bot visits,
partially lists[color=blue]
> and then goes away again.
> MSN and Yahoo are fine and working.[/color]

How long have you waited? You do have to be patient. MSN have made
some claims about indexing more often than Google and I believe this
could be true. Google does find the front page for one of the sites.

(BB, the robots.txt files are at http://www.barrabooks.com/robots.txt
and http://www.stevenhenson.com/robots.txt.)

--Phil.

[me]

Re: Google Bot problems?

"Steve" <pleasedonotspa mme@ireland.com > wrote in message
news:_mDXd.4915 3$Z14.37683@new s.indigo.ie...[color=blue]
> I have worked on a couple of sites which google's bot visits, partially[/color]
lists[color=blue]
> and then goes away again.
>
> MSN and Yahoo are fine and working.
>
> Can anyone please suggest what, if anything, is wrong with these sites?[/color]
Frankly[color=blue]
> I am a bit baffled at the moment! I am wondering if there is a problem[/color]
with the[color=blue]
> page headers and googlebot?
> All suggestions appreciated.
> Thanks in advance,
>
> Steve
>
> Sites & Google results are:-
>
> http://tinyurl.com/6tr2b
>
> http://tinyurl.com/5dj6w[/color]

Don't assume that if you submit your site to one engine the others will also
pick it up (unless it's Google). Submit your site to Google directly:

http://www.google.com/addurl/

Incidentally the others search engines do eventually pick up any site that
Google lists, they know who the top search engine is even if MS *still*
doesn't. ;-)

This page still does not have <meta name="robots" content="ALL"> in the head
so I assume there may be others that don't. It may not be absolute necessity
to have this tag on every page but what can it hurt? I always plan for the
worst and hope for the best.

http://www.stevenhenson.com/spider_map.htm

There's also a trailing slash after the word "ALL" in the tag you're using.
I don't know if this will cause a problem but I would omit it just in case.

If you have submitted your site recently it may (surely?) take two weeks to
several months before it gets listed.
Good Luck,
me

[Adrienne]

Re: Google Bot problems?

Gazing into my crystal ball I observed Steve
<pleasedonotspa mme@ireland.com > writing in
news:FcXXd.4923 7$Z14.37695@new s.indigo.ie:
[color=blue][color=green]
>> 1. http://www.barrabooks.com/robots.txt
>>
>> User-agent: *
>> Disallow: /picture_library/
>> Disallow: /Store/
>> Disallow: /CCS/
>> Disallow: /webstat/
>> Disallow: /plesk-stat/
>> Disallow: /php/
>>
>> 2. http://www.stevenhenson.com/robots.txt
>>
>> User-agent: *
>> Disallow: /picture_library/
>> Disallow: /Store/
>> Disallow: /CCS/
>> Disallow: /webstat/
>> Disallow: /plesk-stat/
>> Disallow: /php/
>>
>> I don't see anything unusual do you?
>>[/color]
>
> Thank you Adrienne - I see nothing nasty in the robot.txt files as
> well! It still leaves me baffled as to why Google is behaving this way.
>
>[/color]

Not sure if this is what is causing it, but on www.stevenhenson.com there
are some markup errors that might be confusing Google. Google _might_
think it is going in an endless loop, therefore getting out before it
sticks its toe in the water.<http://validator.w3.org/check?uri=http%3A%2F%
2Fwww.stevenhen son.com%2F&char set=%28detect+a utomatically%29 &doctype=%
28detect+automa tically%29&ss=1 >

www.barrabooks.com does not have any markup errors, but it could be
upgraded to stylesheets instead of tables, and use semantic markup, eg:
<span class="style2 xbig"><strong>W elcome to Barra Books</strong></span>
should be
<h1>Welcome to Barra Books</h1>

--
Adrienne Boswell

http://www.cavalcade-of-coding.info

Please respond to the group so others can share

[Steve]

Re: Google Bot problems?

me wrote:[color=blue]
> "Steve" <pleasedonotspa mme@ireland.com > wrote in message
> news:_mDXd.4915 3$Z14.37683@new s.indigo.ie...
>[color=green]
>>I have worked on a couple of sites which google's bot visits, partially[/color]
>
> lists
>[color=green]
>>and then goes away again.
>>
>>MSN and Yahoo are fine and working.
>>
>>Can anyone please suggest what, if anything, is wrong with these sites?[/color]
>
> Frankly
>[color=green]
>>I am a bit baffled at the moment! I am wondering if there is a problem[/color]
>
> with the
>[color=green]
>>page headers and googlebot?
>>All suggestions appreciated.
>>Thanks in advance,
>>
>>Steve
>>
>>Sites & Google results are:-
>>
>>http://tinyurl.com/6tr2b
>>
>>http://tinyurl.com/5dj6w[/color]
>
>
> Don't assume that if you submit your site to one engine the others will also
> pick it up (unless it's Google). Submit your site to Google directly:
>
> http://www.google.com/addurl/
>
> Incidentally the others search engines do eventually pick up any site that
> Google lists, they know who the top search engine is even if MS *still*
> doesn't. ;-)
>
> This page still does not have <meta name="robots" content="ALL"> in the head
> so I assume there may be others that don't. It may not be absolute necessity
> to have this tag on every page but what can it hurt? I always plan for the
> worst and hope for the best.
>
> http://www.stevenhenson.com/spider_map.htm
>
> There's also a trailing slash after the word "ALL" in the tag you're using.
> I don't know if this will cause a problem but I would omit it just in case.
>
> If you have submitted your site recently it may (surely?) take two weeks to
> several months before it gets listed.
> Good Luck,
> me
>
>[/color]
Thank you!

I have added the robots ALL tag to the spider_map page and submitted that page
to Google. Not sure if this will work but the spider_map page is a page
deliberately designed to make life easy for the spiders...so hopefully we might
make some headway!

Google keeps visiting the sites, looks at one or two pages (according to the
logs) and then goes away again.

Every now and then Google shows the sites as having a couple of pages listed -
and then reduces then down to the minimum information (i.e. just one page). Its
rather like Google cannot make its mind up to include the site and keeps coming
back for a nibble!


Steve

[Big Bill]

Re: Google Bot problems?

On Thu, 10 Mar 2005 07:22:54 GMT, Adrienne <arbpen2003@sbc global.net>
wrote:
[color=blue]
>Gazing into my crystal ball I observed Big Bill <kruse@cityscap e.co.uk>
>writing in news:18qv21doap 5jfnq8j7ekdnkam 38damfhb6@4ax.c om:
>[color=green]
>> On Wed, 9 Mar 2005 09:01:36 -0600, "me" <anonymous@_.co m> wrote:
>>[color=darkred]
>>>"Steve" <pleasedonotspa mme@ireland.com > wrote in message
>>>news:_mDXd.4 9153$Z14.37683@ news.indigo.ie. ..
>>>> I have worked on a couple of sites which google's bot visits, partially
>>>> lists and then goes away again.
>>>>
>>>> MSN and Yahoo are fine and working.
>>>>
>>>> Can anyone please suggest what, if anything, is wrong with these sites?
>>>> Frankly I am a bit baffled at the moment! I am wondering if there is a
>>>> problem with the page headers and googlebot?
>>>> All suggestions appreciated.
>>>> Thanks in advance,
>>>> Steve
>>>>
>>>> Sites & Google results are:-
>>>>
>>>> http://tinyurl.com/6tr2b
>>>>
>>>> http://tinyurl.com/5dj6w
>>>
>>>Place this in the head of every page:
>>>
>>><meta name="robots" content="ALL">
>>>
>>>IIRC bots may enter from any page so this may help.
>>>Good Luck,
>>>me[/color]
>>
>> It won't do anything. Robots ignore meta tags like that as they index
>> what they can anyway by default. It would be useful if the relevant
>> robots.txts were displayed here.
>>
>> BB.
>>
>> --
>> www.kruse.co.uk/ SEO@kruse.demon .co.uk
>> Affordable SEO!
>> --
>>[/color]
>
>1. http://www.barrabooks.com/robots.txt
>
>User-agent: *
>Disallow: /picture_library/
>Disallow: /Store/
>Disallow: /CCS/
>Disallow: /webstat/
>Disallow: /plesk-stat/
>Disallow: /php/
>
>2. http://www.stevenhenson.com/robots.txt
>
>User-agent: *
>Disallow: /picture_library/
>Disallow: /Store/
>Disallow: /CCS/
>Disallow: /webstat/
>Disallow: /plesk-stat/
>Disallow: /php/
>
>I don't see anything unusual do you?[/color]

No but something's somewhere. I'll have to come back to this as I'm
trying to download a windows update and watch Zev fron Lexx in the
shower just at the mo.

BB
--
www.kruse.co.uk/ SEO@kruse.demon .co.uk
Affordable SEO!
--

[Big Bill]

Re: Google Bot problems?

On Thu, 10 Mar 2005 21:24:11 +0000, Steve
<pleasedonotspa mme@ireland.com > wrote:
[color=blue]
>me wrote:[color=green]
>> "Steve" <pleasedonotspa mme@ireland.com > wrote in message
>> news:_mDXd.4915 3$Z14.37683@new s.indigo.ie...
>>[color=darkred]
>>>I have worked on a couple of sites which google's bot visits, partially[/color]
>>
>> lists
>>[color=darkred]
>>>and then goes away again.
>>>
>>>MSN and Yahoo are fine and working.
>>>
>>>Can anyone please suggest what, if anything, is wrong with these sites?[/color]
>>
>> Frankly
>>[color=darkred]
>>>I am a bit baffled at the moment! I am wondering if there is a problem[/color]
>>
>> with the
>>[color=darkred]
>>>page headers and googlebot?
>>>All suggestions appreciated.
>>>Thanks in advance,
>>>
>>>Steve
>>>
>>>Sites & Google results are:-
>>>
>>>http://tinyurl.com/6tr2b
>>>
>>>http://tinyurl.com/5dj6w[/color]
>>
>>
>> Don't assume that if you submit your site to one engine the others will also
>> pick it up (unless it's Google). Submit your site to Google directly:
>>
>> http://www.google.com/addurl/
>>
>> Incidentally the others search engines do eventually pick up any site that
>> Google lists, they know who the top search engine is even if MS *still*
>> doesn't. ;-)
>>
>> This page still does not have <meta name="robots" content="ALL"> in the head
>> so I assume there may be others that don't. It may not be absolute necessity
>> to have this tag on every page but what can it hurt? I always plan for the
>> worst and hope for the best.
>>
>> http://www.stevenhenson.com/spider_map.htm
>>
>> There's also a trailing slash after the word "ALL" in the tag you're using.
>> I don't know if this will cause a problem but I would omit it just in case.
>>
>> If you have submitted your site recently it may (surely?) take two weeks to
>> several months before it gets listed.
>> Good Luck,
>> me
>>
>>[/color]
>Thank you!
>
>I have added the robots ALL tag to the spider_map page and submitted that page
>to Google. Not sure if this will work[/color]

It won't. It's pointless other than it takes up space.
[color=blue]
> but the spider_map page is a page
>deliberately designed to make life easy for the spiders...so hopefully we might
>make some headway![/color]

How many links on it, by the way?

BB
--
www.kruse.co.uk/ SEO@kruse.demon .co.uk
Affordable SEO!
--

[Big Bill]

Re: Google Bot problems?

On Thu, 10 Mar 2005 08:23:50 -0600, "me" <anonymous@_.co m> wrote:
[color=blue]
>"Steve" <pleasedonotspa mme@ireland.com > wrote in message
>news:_mDXd.491 53$Z14.37683@ne ws.indigo.ie...[color=green]
>> I have worked on a couple of sites which google's bot visits, partially[/color]
>lists[color=green]
>> and then goes away again.
>>
>> MSN and Yahoo are fine and working.
>>
>> Can anyone please suggest what, if anything, is wrong with these sites?[/color]
>Frankly[color=green]
>> I am a bit baffled at the moment! I am wondering if there is a problem[/color]
>with the[color=green]
>> page headers and googlebot?
>> All suggestions appreciated.
>> Thanks in advance,
>>
>> Steve
>>
>> Sites & Google results are:-
>>
>> http://tinyurl.com/6tr2b
>>
>> http://tinyurl.com/5dj6w[/color]
>
>Don't assume that if you submit your site to one engine the others will also
>pick it up (unless it's Google). Submit your site to Google directly:
>
>http://www.google.com/addurl/
>
>Incidentally the others search engines do eventually pick up any site that
>Google lists, they know who the top search engine is even if MS *still*
>doesn't. ;-)
>
>This page still does not have <meta name="robots" content="ALL"> in the head
>so I assume there may be others that don't. It may not be absolute necessity
>to have this tag on every page but what can it hurt? I always plan for the
>worst and hope for the best.
>
>http://www.stevenhenson.com/spider_map.htm
>
>There's also a trailing slash after the word "ALL" in the tag you're using.
>I don't know if this will cause a problem but I would omit it just in case.
>
>If you have submitted your site recently it may (surely?) take two weeks to
>several months before it gets listed.
>Good Luck,
>me
>[/color]

(?)

BB
--
www.kruse.co.uk/ SEO@kruse.demon .co.uk
Affordable SEO!
--

[Tim]

Re: Google Bot problems?

"me" <anonymous@_.co m> wrote:
[color=blue][color=green]
>> Place this in the head of every page:
>>
>> <meta name="robots" content="ALL">[/color][/color]

Big Bill <kruse@cityscap e.co.uk> posted:
[color=blue]
> It won't do anything. Robots ignore meta tags like that as they index
> what they can anyway by default.[/color]

Perhaps that particular example might be ignored, but some robots do pay
attention to robot meta statements in the HTML head. Mostly about what
they should ignore, in some way, rather than what they should look at.

I recommend having a look through <http://www.google.com/webmasters/> for
more information about the robots, as well as why Google might be ignoring
the site. For instance, your wad of meta keywords might be doing you more
harm than good.

--
If you insist on e-mailing me, use the reply-to address (it's real but
temporary). But please reply to the group, like you're supposed to.

This message was sent without a virus, please delete some files yourself.