Python - what is the fastest database ?

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • martijn@gamecreators.nl

    Python - what is the fastest database ?

    H!,

    I'm testing things with Python with databases.
    But I have one big question.

    What is the 'fastest' database for the internet in combination with
    Python ?

    - with +/- 15 GB data.
    - fast internet SELECT query's.

    Python use bsddb but could he handle 15 GB fast ?

    A other question is:
    How is it possible that google (super big database) is super fast?
    What type database do they use / software ?

    Thanks Very Much,
    GC-Martijn

  • PA

    #2
    Re: Python - what is the fastest database ?


    On Feb 28, 2005, at 13:10, martijn@gamecre ators.nl wrote:
    [color=blue]
    > What type database do they use / software ?[/color]

    Hmmm... they don't use a "database" in the traditional sense of the
    term.



    Cheers

    --
    PA, Onnay Equitursay


    Comment

    • Peter Hansen

      #3
      Re: Python - what is the fastest database ?

      PA wrote:[color=blue]
      > On Feb 28, 2005, at 13:10, martijn@gamecre ators.nl wrote:[color=green]
      >> What type database do they use / software ?[/color]
      >
      > Hmmm... they don't use a "database" in the traditional sense of the term.
      >
      > http://www-db.stanford.edu/~backrub/google.html[/color]

      Section "4.3 Crawling the Web" in that discusses the role Python
      plays in Google, at least in the form it (Google) had when this
      paper was written.

      -Peter

      Comment

      • pyguy2@gmail.com

        #4
        Re: Python - what is the fastest database ?

        It depends on what you mean by database.

        If you want really fast I/O, try pytables.
        "PyTables is a hierarchical database package designed to efficiently
        manage very large amounts of data."



        some more comments from the webpage:

        # High performance I/O: On modern systems, and for large amounts of
        data, tables and array objects can be read and written at a speed only
        limited by the performance of the underlying I/O subsystem. Moreover,
        if your data is compressible, even faster than your I/O maximum
        throughput (!).

        # Support of files bigger than 2 GB: So that you won't be limited if
        you want to deal with very large datasets. In fact, PyTables support
        full 64-bit file addressing even on 32-bit platforms (provided that the
        underlying filesystem does so too, of course).

        # Architecture-independent: PyTables has been carefully coded (as HDF5
        itself) with little-endian/big-endian byte orderings issues in mind .
        So, you can write a file in a big-endian machine (like a Sparc or MIPS)
        and read it in other little-endian (like Intel or Alpha) without
        problems.

        # Portability: PyTables has been ported to many architectures, namely
        Linux, Windows, MacOSX, FreeBSD, Solaris, IRIX and probably works in
        many more. Moreover, it runs just fine also in 64 bit plaforms (like
        AMD64, Intel64, UltraSparc or MIPS RXX000 processors).

        Comment

        • Terry Reedy

          #5
          Re: Python - what is the fastest database ?


          <martijn@gamecr eators.nl> wrote in message
          news:1109592623 .116661.101880@ g14g2000cwa.goo glegroups.com.. .[color=blue][color=green]
          >> How is it possible that google (super big database) is super fast?[/color]
          > What type database do they use / software ?[/color]

          On the hardware side, Google's secret is massively parallel cluster
          computing, coupled with proprietary software for splitting tasks and
          joining results. They have perhaps 200,000 CPUs. A query might be given
          to hundreds of them for a fraction of a second.

          Terry J. Reedy



          Comment

          • km

            #6
            Re: Python - what is the fastest database ?

            Hi all,

            Google has specially designed file system 'Goolgle File System' too.

            KM
            -------------------------------------------------------------------------
            On Mon, Feb 28, 2005 at 01:43:53PM -0500, Terry Reedy wrote:[color=blue]
            >
            > <martijn@gamecr eators.nl> wrote in message
            > news:1109592623 .116661.101880@ g14g2000cwa.goo glegroups.com.. .[color=green][color=darkred]
            > >> How is it possible that google (super big database) is super fast?[/color]
            > > What type database do they use / software ?[/color]
            >
            > On the hardware side, Google's secret is massively parallel cluster
            > computing, coupled with proprietary software for splitting tasks and
            > joining results. They have perhaps 200,000 CPUs. A query might be given
            > to hundreds of them for a fraction of a second.
            >
            > Terry J. Reedy
            >
            >
            >
            > --[/color]

            Comment

            Working...