Any suggestion to deal with web crawler?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Alex Shi

    Any suggestion to deal with web crawler?

    Any good idea or a reference link to deal with search engine
    for php web pages? Thanks in advance!

    Alex

  • Jeffrey Silverman

    #2
    Re: Any suggestion to deal with web crawler?

    On Tue, 16 Mar 2004 13:49:43 -0500, Alex Shi wrote:
    [color=blue]
    > Any good idea or a reference link to deal with search engine
    > for php web pages? Thanks in advance!
    >
    > Alex[/color]

    Huh? That is not nearly enough information for me to know what you are
    trying to ask.

    --
    Jeffrey D. Silverman | jeffrey AT jhu DOT edu
    Website | http://www.wse.jhu.edu/newtnotes/

    Comment

    • somaBoy MX

      #3
      Re: Any suggestion to deal with web crawler?


      "Alex Shi" <chpshi@stonix. com> schreef in bericht
      news:rXH5c.9627 3$is7.10318@nnt p-post.primus.ca. ..[color=blue]
      > Any good idea or a reference link to deal with search engine
      > for php web pages? Thanks in advance!
      >
      > Alex
      >[/color]

      From my experience, most search engines handle PHP pretty well, even with a
      lot of parameters in the URLstring. The only thing you might want to do is
      not use the word "id" in your urlstring because apparently some robots
      presume that's a session id and don't index the page.


      ..soma


      Comment

      • Pedro Graca

        #4
        Re: Any suggestion to deal with web crawler?

        Alex Shi wrote:[color=blue]
        > Any good idea or a reference link to deal with search engine
        > for php web pages? Thanks in advance![/color]

        Put a file named "robots.txt " in the root directory of your site with
        the contents:

        User-agent: *
        Disallow: /


        See "The Web Robots Pages" for more info

        --
        USENET would be a better place if everybody read: : mail address :
        http://www.catb.org/~esr/faqs/smart-questions.html : is valid for :
        http://www.netmeister.org/news/learn2quote2.html : "text/plain" :
        http://www.expita.com/nomime.html : to 10K bytes :

        Comment

        • Alex Shi

          #5
          Re: Any suggestion to deal with web crawler?

          Sorry I failed to express myself clearly.

          I'm told that pure html page with carefully arranged keywords and
          page content can be very much search engine friendly. However
          the reason we go with PHP is that we need to generate intensive
          dynamic content with templates/frames. Usually my page script is
          layouted like following:

          <?
          // processing
          // processing
          .....
          // read html template file into a buffer/string
          // insert dynamic contents into the template buffer
          // send the buffer as a page by print or echo
          ?>

          To my understanding, this approach might be very much not
          friendly to search engine. I'd assume that some one may have
          expreriences do deal with such kind of issue and can share
          them with me. Thanks in advance.

          Alex
          [color=blue]
          >[color=green]
          > > Any good idea or a reference link to deal with search engine
          > > for php web pages? Thanks in advance!
          > >
          > > Alex[/color]
          >
          > Huh? That is not nearly enough information for me to know what you are
          > trying to ask.
          >
          > --
          > Jeffrey D. Silverman | jeffrey AT jhu DOT edu
          > Website | http://www.wse.jhu.edu/newtnotes/
          >[/color]

          Comment

          • Robert Peake

            #6
            Re: Any suggestion to deal with web crawler?

            Hi Alex,

            You don't need to worry about commenting your code. In fact, you don't need
            to worry about anything between <?PHP and ?> - search engines deal with the
            final output, in HTML, of your PHP code, not the code itself. "View Source"
            from any web browser on a page and you will see what the search engine cares
            about.

            Cheers,


            On 3/16/04 1:48 PM, in article PyK5c.96551$Wz2 .4601@nntp-post.primus.ca,
            "Alex Shi" <chpshi@stonix. com> wrote:
            [color=blue]
            > Sorry I failed to express myself clearly.
            >
            > I'm told that pure html page with carefully arranged keywords and
            > page content can be very much search engine friendly. However
            > the reason we go with PHP is that we need to generate intensive
            > dynamic content with templates/frames. Usually my page script is
            > layouted like following:
            >
            > <?
            > // processing
            > // processing
            > ....
            > // read html template file into a buffer/string
            > // insert dynamic contents into the template buffer
            > // send the buffer as a page by print or echo
            > ?>
            >
            > To my understanding, this approach might be very much not
            > friendly to search engine. I'd assume that some one may have
            > expreriences do deal with such kind of issue and can share
            > them with me. Thanks in advance.
            >
            > Alex
            >[color=green]
            >>[color=darkred]
            >>> Any good idea or a reference link to deal with search engine
            >>> for php web pages? Thanks in advance!
            >>>
            >>> Alex[/color]
            >>
            >> Huh? That is not nearly enough information for me to know what you are
            >> trying to ask.
            >>
            >> --
            >> Jeffrey D. Silverman | jeffrey AT jhu DOT edu
            >> Website | http://www.wse.jhu.edu/newtnotes/
            >>[/color][/color]

            --
            Robert Peake | Peake Professional Consulting
            Robert@PeakePro .com | http://www.peakepro.com/

            Comment

            • Alex Shi

              #7
              Re: Any suggestion to deal with web crawler?

              Thanks!
              That was really confusing me a lot :(
              I did guess it should be working as what you described but I thought
              I might be wrong :)
              You make me happy and relaxed :))

              Alex


              [color=blue]
              > Hi Alex,
              >
              > You don't need to worry about commenting your code. In fact, you don't[/color]
              need[color=blue]
              > to worry about anything between <?PHP and ?> - search engines deal with[/color]
              the[color=blue]
              > final output, in HTML, of your PHP code, not the code itself. "View[/color]
              Source"[color=blue]
              > from any web browser on a page and you will see what the search engine[/color]
              cares[color=blue]
              > about.
              >
              > Cheers,
              >
              >
              > On 3/16/04 1:48 PM, in article PyK5c.96551$Wz2 .4601@nntp-post.primus.ca,
              > "Alex Shi" <chpshi@stonix. com> wrote:
              >[color=green]
              > > Sorry I failed to express myself clearly.
              > >
              > > I'm told that pure html page with carefully arranged keywords and
              > > page content can be very much search engine friendly. However
              > > the reason we go with PHP is that we need to generate intensive
              > > dynamic content with templates/frames. Usually my page script is
              > > layouted like following:
              > >
              > > <?
              > > // processing
              > > // processing
              > > ....
              > > // read html template file into a buffer/string
              > > // insert dynamic contents into the template buffer
              > > // send the buffer as a page by print or echo
              > > ?>
              > >
              > > To my understanding, this approach might be very much not
              > > friendly to search engine. I'd assume that some one may have
              > > expreriences do deal with such kind of issue and can share
              > > them with me. Thanks in advance.
              > >
              > > Alex
              > >[color=darkred]
              > >>
              > >>> Any good idea or a reference link to deal with search engine
              > >>> for php web pages? Thanks in advance!
              > >>>
              > >>> Alex
              > >>
              > >> Huh? That is not nearly enough information for me to know what you are
              > >> trying to ask.
              > >>
              > >> --
              > >> Jeffrey D. Silverman | jeffrey AT jhu DOT edu
              > >> Website | http://www.wse.jhu.edu/newtnotes/
              > >>[/color][/color]
              >
              > --
              > Robert Peake | Peake Professional Consulting
              > Robert@PeakePro .com | http://www.peakepro.com/
              >[/color]

              Comment

              Working...