How to detect whether human or robot visitor?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Baron Samedi

    #1

    How to detect whether human or robot visitor?

    I coded up a hit counter, then extended it to see who was reading my
    blog, by matching IP. The problem is that I am swamped by crawlers.

    How can I detect a human, or a crawler? If I can handle one, I can
    negate it for the other.

    Should I somehow user $_SERVER['USER_AGENT'] ? or something else?

    Thanks in advance.

  • Rik

    #2
    Re: How to detect whether human or robot visitor?

    Baron Samedi <Papa.Legba.666 @gmail.comwrote :
    I coded up a hit counter, then extended it to see who was reading my
    blog, by matching IP. The problem is that I am swamped by crawlers.
    >
    How can I detect a human, or a crawler? If I can handle one, I can
    negate it for the other.
    >
    Should I somehow user $_SERVER['USER_AGENT'] ? or something else?
    Nice crawlers: yes.
    Evil crawlers: no, they just copy a valid UA-string.

    <http://www.user-agents.org/index.shtml>

    --
    Rik Wasmus
    Posted on Usenet, not any forum you might see this in.
    Ask Smart Questions: http://tinyurl.com/anel

    Comment

    • Rik

      #3
      Re: How to detect whether human or robot visitor?

      Rik <luiheidsgoeroe @hotmail.comwro te:
      Evil crawlers: no, they just copy a valid UA-string.
      Euhm, as if a user-agent string can 'valid'. Well, only standard header
      rules apply :-).
      What I mean to say is: they will copy a string of a known 'normal' UA.
      --
      Rik Wasmus
      Posted on Usenet, not any forum you might see this in.
      Ask Smart Questions: http://tinyurl.com/anel

      Comment

      • Benjamin

        #4
        Re: How to detect whether human or robot visitor?

        On Mar 10, 7:18 pm, "Baron Samedi" <Papa.Legba.... @gmail.comwrote :
        I coded up a hit counter, then extended it to see who was reading my
        blog, by matching IP. The problem is that I am swamped by crawlers.
        >
        How can I detect a human, or a crawler? If I can handle one, I can
        negate it for the other.
        You can't really. Any device can send any user agent it wants. (A
        awful bot can send a Firefox user agent.)
        >
        Should I somehow user $_SERVER['USER_AGENT'] ? or something else?
        Try using the get_browser function. It returns all sorts of
        information about the USER_AGENT. http://www.php.net/get-browser
        >
        Thanks in advance.

        Comment

        • Gordon Burditt

          #5
          Re: How to detect whether human or robot visitor?

          >I coded up a hit counter, then extended it to see who was reading my
          >blog, by matching IP. The problem is that I am swamped by crawlers.
          Nice crawlers for search engines identify themselves in the user
          agent string. Also, nice crawlers obey robots.txt, so you can
          exclude portions of your site if you want. Of course, that part
          won't be indexed. Evil bots fake user agent strings of ordinary
          users.
          >How can I detect a human, or a crawler? If I can handle one, I can
          >negate it for the other.
          Unfortunately, evil bots can hire humans to work for them, if you
          had in mind such things as CAPTCHAs (decoding warped text in images).
          >Should I somehow user $_SERVER['USER_AGENT'] ? or something else?
          The user agent string is one thing you can use, mostly to detect nice
          crawlers.

          Comment

          • Baron Samedi

            #6
            Re: How to detect whether human or robot visitor?

            On 11 Mar, 08:36, Rik <luiheidsgoe... @hotmail.comwro te:
            Baron Samedi <Papa.Legba.... @gmail.comwrote :
            I coded up a hit counter, then extended it to see who was reading my
            blog, by matching IP. The problem is that I am swamped by crawlers.
            >
            How can I detect a human, or a crawler? If I can handle one, I can
            negate it for the other.
            >
            Should I somehow user $_SERVER['USER_AGENT'] ? or something else?
            >
            Nice crawlers: yes.
            Evil crawlers: no, they just copy a valid UA-string.
            Well, that's a start. Even if exclude the 'nice' crawlers, it will
            help me to sort out. I am drowning in hits and have to eyeball them
            anyway. What I am thinking of doing, since more and more people have
            static IP addresses nowadays (but not all, sigh), is to just look for
            IP known addresses of friends or registered users, since those are
            what interest me personally. I am not looking for perfection, just a
            feel for who is looking at what.


            >
            <http://www.user-agents.org/index.shtml>
            That is one long list. Hopefully, I can snarfle some code somewhere
            online.

            Thanks much for the feedback.

            >
            --
            Rik Wasmus
            Posted on Usenet, not any forum you might see this in.
            Ask Smart Questions:http://tinyurl.com/anel

            Comment

            • sally@mander.com

              #7
              Re: How to detect whether human or robot visitor?

              In article <12v70rmtpklsie e@corp.supernew s.com>,
              gordonb.qm9ec@b urditt.org says...
              I coded up a hit counter, then extended it to see who was reading my
              blog, by matching IP. The problem is that I am swamped by crawlers.
              >
              Nice crawlers for search engines identify themselves in the user
              agent string. Also, nice crawlers obey robots.txt, so you can
              exclude portions of your site if you want. Of course, that part
              won't be indexed. Evil bots fake user agent strings of ordinary
              users.
              >
              How can I detect a human, or a crawler? If I can handle one, I can
              negate it for the other.
              >
              Unfortunately, evil bots can hire humans to work for them, if you
              had in mind such things as CAPTCHAs (decoding warped text in images).
              >
              Should I somehow user $_SERVER['USER_AGENT'] ? or something else?
              >
              The user agent string is one thing you can use, mostly to detect nice
              crawlers.
              >
              >
              You might also want to set up what I call a "junk pot" as opposed to
              honey pot. (as far as I know I invented this one... ) The idea
              is to get them to waste their time somewhere away from your "real site"

              Take all the free space you have on your server and fill it with junk
              files. HTML zips jpgs anything at all. Just use as much and as many files
              as you can. If you can put up a dummy forum with guest write permissions
              thats cool too. Use lots of sub-directories with very long names too.
              Block access to it in your robots.txt so genuine crawlers dont waste
              their time. Put your main site under a single entry point 1 step below
              normal.

              sit back and watch the crawlers waste their time in your junk bin.

              Actually I redirect google into mine too.
              Its fun to watch their thieving archive fill up with junk instead of
              the copyright material they normally steal from everyone.
              (google just proves if youre big enough you can ignore any law)

              I'm sure the idea can be fine tuned by the good people here or
              perhaps somewhere more appropriate.

              Comment

              • Jerry Stuckle

                #8
                Re: How to detect whether human or robot visitor?

                sally@mander.co m wrote:
                In article <12v70rmtpklsie e@corp.supernew s.com>,
                gordonb.qm9ec@b urditt.org says...
                >>I coded up a hit counter, then extended it to see who was reading my
                >>blog, by matching IP. The problem is that I am swamped by crawlers.
                >Nice crawlers for search engines identify themselves in the user
                >agent string. Also, nice crawlers obey robots.txt, so you can
                >exclude portions of your site if you want. Of course, that part
                >won't be indexed. Evil bots fake user agent strings of ordinary
                >users.
                >>
                >>How can I detect a human, or a crawler? If I can handle one, I can
                >>negate it for the other.
                >Unfortunatel y, evil bots can hire humans to work for them, if you
                >had in mind such things as CAPTCHAs (decoding warped text in images).
                >>
                >>Should I somehow user $_SERVER['USER_AGENT'] ? or something else?
                >The user agent string is one thing you can use, mostly to detect nice
                >crawlers.
                >>
                >>
                >
                You might also want to set up what I call a "junk pot" as opposed to
                honey pot. (as far as I know I invented this one... ) The idea
                is to get them to waste their time somewhere away from your "real site"
                >
                Take all the free space you have on your server and fill it with junk
                files. HTML zips jpgs anything at all. Just use as much and as many files
                as you can. If you can put up a dummy forum with guest write permissions
                thats cool too. Use lots of sub-directories with very long names too.
                Block access to it in your robots.txt so genuine crawlers dont waste
                their time. Put your main site under a single entry point 1 step below
                normal.
                >
                sit back and watch the crawlers waste their time in your junk bin.
                >
                Actually I redirect google into mine too.
                Its fun to watch their thieving archive fill up with junk instead of
                the copyright material they normally steal from everyone.
                (google just proves if youre big enough you can ignore any law)
                >
                I'm sure the idea can be fine tuned by the good people here or
                perhaps somewhere more appropriate.
                >
                Let's see. Sit back and watch all those spiders use up all of your
                bandwidth. So your hosting company shuts you down until you pay for
                more bandwidth. Not a good idea, IMHO.

                And if you think google is stealing copyright material I suggest you
                talk to an attorney about 'fair use' rather than spouting unfounded
                claims. That could get YOU in trouble.

                --
                =============== ===
                Remove the "x" from my email address
                Jerry Stuckle
                JDS Computer Training Corp.
                jstucklex@attgl obal.net
                =============== ===

                Comment

                Working...