Best text search for many Gigabyte Database?

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • ngocviet
    #1

    Best text search for many Gigabyte Database?

    I have a so big DB, It's about 10-100GB of text.
    Now I want to create a searching function but It take too long when I
    use LIKE from MySQL
    Anyone know the best sollution?
    Can I use Google Desktop search or an others?
    Please help!

    Tags: None
    • Erwin Moller
      #2
      Re: Best text search for many Gigabyte Database?

      ngocviet wrote:
      I have a so big DB, It's about 10-100GB of text.
      Now I want to create a searching function but It take too long when I
      use LIKE from MySQL
      Anyone know the best sollution?
      Can I use Google Desktop search or an others?
      Please help!
      If you use mySQL, consider using FULLTEXT and MATCH AGAINST().
      Bottomline stays: If you have 100GB of data, some searchtime is needed, but
      using FULLTEXT can speed that up considerable.
      Check your mySQL documentation for details.

      Regards,
      Erwin Moller

        Comment

        • ngocviet
          #3
          Re: Best text search for many Gigabyte Database?

          If you use mySQL, consider using FULLTEXT and MATCH AGAINST().
          Bottomline stays: If you have 100GB of data, some searchtime is needed, but
          using FULLTEXT can speed that up considerable.
          Check your mySQL documentation for details.
          I've tried FULLTEXT but it take too long, about over 10 seconds with
          1.2 Gigabyte table.
          If need, I can convert database to other structure.

            Comment

            • Erwin Moller
              #4
              Re: Best text search for many Gigabyte Database?

              ngocviet wrote:
              >
              >If you use mySQL, consider using FULLTEXT and MATCH AGAINST().
              >Bottomline stays: If you have 100GB of data, some searchtime is needed,
              >but using FULLTEXT can speed that up considerable.
              >Check your mySQL documentation for details.
              >
              I've tried FULLTEXT but it take too long, about over 10 seconds with
              1.2 Gigabyte table.
              If need, I can convert database to other structure.
              Hi,

              If you use MySQL ISAM with FULLTEXT, and not LIKE, but MATCH AGAINST, you
              are using one of the fastest approaches a developer can set up (as far as I
              know).
              I expect the only way to increase searchspeed for a 100GB database filled
              with text, is throwing more/better hardware at it (more memory, faster disk
              IO, faster CPU, etc.).
              In general: When searching through a huge datastructure: Disk IO is the
              bottleneck. So faster disk IO will help the most.
              Dive into different RAID systems maybe.
              eg: If you have 2 HD delivering data at the same time (using some RAID),
              your query will maybe run twice as fast.

              Regards,
              Erwin Moller

                Comment

                • Kimmo Laine
                  #5
                  Re: Best text search for many Gigabyte Database?

                  "ngocviet" <ngocviet@gmail .comwrote in message
                  news:1170669914 .799189.11620@h 3g2000cwc.googl egroups.com...
                  >
                  >If you use mySQL, consider using FULLTEXT and MATCH AGAINST().
                  >Bottomline stays: If you have 100GB of data, some searchtime is needed,
                  >but
                  >using FULLTEXT can speed that up considerable.
                  >Check your mySQL documentation for details.
                  >
                  I've tried FULLTEXT but it take too long, about over 10 seconds with
                  1.2 Gigabyte table.
                  If need, I can convert database to other structure.

                  A search like that for such an amount of unindexed textual data simply has
                  to take some time. Live with it, or invest to hardware. Like the IT proverb
                  says: "If it doesn't work, just throw money at it".

                  --
                  "Ohjelmoija on organismi joka muuttaa kofeiinia koodiksi" - lpk
                  http://outolempi.net/ahdistus/ - Satunnaisesti päivittyvä nettisarjis
                  spam@outolempi. net | rot13(xvzzb@bhg byrzcv.arg)


                    Comment

                    • petersprc
                      #6
                      Re: Best text search for many Gigabyte Database?

                      In addition to MySQL FULLTEXT, you could also use Lucene
                      (Zend_Search_Lu cene in PHP or perhaps Solr running inside Tomcat).
                      Performance is quite good and Lucene is very flexible.

                      On Feb 5, 1:27 am, "ngocviet" <ngocv...@gmail .comwrote:
                      I have a so big DB, It's about 10-100GB of text.
                      Now I want to create a searching function but It take too long when I
                      use LIKE from MySQL
                      Anyone know the best sollution?
                      Can I use Google Desktop search or an others?
                      Please help!

                        Comment

                        Previous template Next
                        Working...