TPC H data

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • Shalu Gupta
    #1

    TPC H data

    Hello,

    We are trying to import the TPC-H data into postgresql using the COPY
    command and for the larger files we get an error due to insufficient
    memory space.

    We are using a linux system with Postgresql-7.3.4

    Is it that Postgresql cannot handle such large files or is there some
    other possible reason.

    Thanks
    Shalu Gupta
    NC State University.

    ---------------------------(end of broadcast)---------------------------
    TIP 5: Have you checked our extensive FAQ?

    PostgreSQL: Not Found
    http://www.postgresql.org/docs/faqs/FAQ.html


    Tags: None
    • Andrew Dunstan
      #2
      Re: [HACKERS] TPC H data

      Shalu Gupta wrote:
      [color=blue]
      >Hello,
      >
      >We are trying to import the TPC-H data into postgresql using the COPY
      >command and for the larger files we get an error due to insufficient
      >memory space.
      >
      >We are using a linux system with Postgresql-7.3.4
      >
      >Is it that Postgresql cannot handle such large files or is there some
      >other possible reason.
      >
      >Thanks
      >Shalu Gupta
      >NC State University.
      >
      >
      >[/color]

      Shalu,

      I loaded the largest TPC-H table (lineitem, roughly 6 million rows) the
      other day into a completely untuned 7.5devel PostgreSQL instance running
      on RH 9, and it didn't raise a sweat. I delayed creating the indexes
      until after the load. Data load took roughly 10 minutes, index creation
      took a further 35 minutes (there are 13 of them).

      HTH. (I'm just down the road from NCSU, would be happy to help out)

      cheers

      andrew



      ---------------------------(end of broadcast)---------------------------
      TIP 8: explain analyze is your friend

        Comment

        • Andrew Dunstan
          #3
          Re: [HACKERS] TPC H data

          Shalu Gupta wrote:
          [color=blue]
          >Hello,
          >
          >We are trying to import the TPC-H data into postgresql using the COPY
          >command and for the larger files we get an error due to insufficient
          >memory space.
          >
          >We are using a linux system with Postgresql-7.3.4
          >
          >Is it that Postgresql cannot handle such large files or is there some
          >other possible reason.
          >
          >Thanks
          >Shalu Gupta
          >NC State University.
          >
          >
          >[/color]

          Shalu,

          I loaded the largest TPC-H table (lineitem, roughly 6 million rows) the
          other day into a completely untuned 7.5devel PostgreSQL instance running
          on RH 9, and it didn't raise a sweat. I delayed creating the indexes
          until after the load. Data load took roughly 10 minutes, index creation
          took a further 35 minutes (there are 13 of them).

          HTH. (I'm just down the road from NCSU, would be happy to help out)

          cheers

          andrew



          ---------------------------(end of broadcast)---------------------------
          TIP 8: explain analyze is your friend

            Comment

            • scott.marlowe
              #4
              Re: [HACKERS] TPC H data

              On Wed, 21 Apr 2004, Shalu Gupta wrote:
              [color=blue]
              > Hello,
              >
              > We are trying to import the TPC-H data into postgresql using the COPY
              > command and for the larger files we get an error due to insufficient
              > memory space.
              >
              > We are using a linux system with Postgresql-7.3.4
              >
              > Is it that Postgresql cannot handle such large files or is there some
              > other possible reason.[/color]

              what method(s) are you using to load the data?


              ---------------------------(end of broadcast)---------------------------
              TIP 1: subscribe and unsubscribe commands go to majordomo@postg resql.org

                Comment

                • scott.marlowe
                  #5
                  Re: [HACKERS] TPC H data

                  On Wed, 21 Apr 2004, Shalu Gupta wrote:
                  [color=blue]
                  > Hello,
                  >
                  > We are trying to import the TPC-H data into postgresql using the COPY
                  > command and for the larger files we get an error due to insufficient
                  > memory space.
                  >
                  > We are using a linux system with Postgresql-7.3.4
                  >
                  > Is it that Postgresql cannot handle such large files or is there some
                  > other possible reason.[/color]

                  what method(s) are you using to load the data?


                  ---------------------------(end of broadcast)---------------------------
                  TIP 1: subscribe and unsubscribe commands go to majordomo@postg resql.org

                    Comment

                    • Mark Kirkwood
                      #6
                      Re: TPC H data

                      What scale factor TPC H are you importing?

                      additionally - might be worth giving the specs of the machine you are
                      doing this on.

                      (I seem to recall trying this with Pg 7.2 a while ago without this
                      issue, mind you - think I had ~1G of Ram and used the scale fact 1
                      dataset, i.e 1G)

                      regards

                      Mark


                      Shalu Gupta wrote:
                      [color=blue]
                      >Hello,
                      >
                      >We are trying to import the TPC-H data into postgresql using the COPY
                      >command and for the larger files we get an error due to insufficient
                      >memory space.
                      >
                      >We are using a linux system with Postgresql-7.3.4
                      >
                      >Is it that Postgresql cannot handle such large files or is there some
                      >other possible reason.
                      >
                      >Thanks
                      >Shalu Gupta
                      >NC State University.
                      >
                      >---------------------------(end of broadcast)---------------------------
                      >TIP 5: Have you checked our extensive FAQ?
                      >
                      > http://www.postgresql.org/docs/faqs/FAQ.html
                      >
                      >
                      >
                      >[/color]

                      ---------------------------(end of broadcast)---------------------------
                      TIP 9: the planner will ignore your desire to choose an index scan if your
                      joining column's datatypes do not match

                        Comment

                        • Mark Kirkwood
                          #7
                          Re: TPC H data

                          What scale factor TPC H are you importing?

                          additionally - might be worth giving the specs of the machine you are
                          doing this on.

                          (I seem to recall trying this with Pg 7.2 a while ago without this
                          issue, mind you - think I had ~1G of Ram and used the scale fact 1
                          dataset, i.e 1G)

                          regards

                          Mark


                          Shalu Gupta wrote:
                          [color=blue]
                          >Hello,
                          >
                          >We are trying to import the TPC-H data into postgresql using the COPY
                          >command and for the larger files we get an error due to insufficient
                          >memory space.
                          >
                          >We are using a linux system with Postgresql-7.3.4
                          >
                          >Is it that Postgresql cannot handle such large files or is there some
                          >other possible reason.
                          >
                          >Thanks
                          >Shalu Gupta
                          >NC State University.
                          >
                          >---------------------------(end of broadcast)---------------------------
                          >TIP 5: Have you checked our extensive FAQ?
                          >
                          > http://www.postgresql.org/docs/faqs/FAQ.html
                          >
                          >
                          >
                          >[/color]

                          ---------------------------(end of broadcast)---------------------------
                          TIP 9: the planner will ignore your desire to choose an index scan if your
                          joining column's datatypes do not match

                            Comment

                            Previous template Next
                            Working...