too much memory use

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • rohit
    #1

    too much memory use

    hi ,
    i am making a program for desktop search.
    in order to make it more effective im implementing the records in the
    form of a tree(files starting with 'a','b','c'.... have different
    trees ..so 26 trees in all) in memory and writing it down in file.
    the max size file has 16000 records...now to implement the tree in
    list i'm using line no as index ..and empty child nodes are
    represented as "\n"
    all this work is going on in the memory..
    problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
    cant think of an effective way to implement tree in memory(i can
    compact it on disk by writing just the index no..along with the record
    from which tree in memory can be reconstructed, but i have to
    implement tree as previous to implement random access)
    please help..
    rohit sethi
    delhi college of engineering

    Tags: None
    • Marc 'BlackJack' Rintsch
      #2
      Re: too much memory use

      In <1179860235.044 137.295080@a26g 2000pre.googleg roups.com>, rohit wrote:
      i am making a program for desktop search.
      in order to make it more effective im implementing the records in the
      form of a tree(files starting with 'a','b','c'.... have different
      trees ..so 26 trees in all) in memory and writing it down in file.
      the max size file has 16000 records...now to implement the tree in
      list i'm using line no as index ..and empty child nodes are
      represented as "\n"
      all this work is going on in the memory..
      problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
      cant think of an effective way to implement tree in memory(i can
      compact it on disk by writing just the index no..along with the record
      from which tree in memory can be reconstructed, but i have to
      implement tree as previous to implement random access)
      I'm not quite sure what exactly you have as in-memory data structures and
      how many "records" -- are you sure you don't keep references to objects
      you don't really need anymore? Or maybe you have object cycles and
      implemented the `__del__()` method on those objects?

      Anyway… If the data doesn't fit into memory anymore it's time to put
      them into a database. Either a `shelve`, an SQL database like SQLite or
      maybe an object database like zodb or Durus.

      Ciao,
      Marc 'BlackJack' Rintsch

        Comment

        • Tim Roberts
          #3
          Re: too much memory use

          rohit <rohitsethidce@ gmail.comwrote:
          >
          >i am making a program for desktop search.
          >in order to make it more effective im implementing the records in the
          >form of a tree(files starting with 'a','b','c'.... have different
          >trees ..so 26 trees in all) in memory and writing it down in file.
          >the max size file has 16000 records...now to implement the tree in
          >list i'm using line no as index ..and empty child nodes are
          >represented as "\n"
          >all this work is going on in the memory..
          >problem is the system eats up my 512 mb RAM +1gb virtual store n hangs
          >cant think of an effective way to implement tree in memory(i can
          >compact it on disk by writing just the index no..along with the record
          >from which tree in memory can be reconstructed, but i have to
          >implement tree as previous to implement random access)
          Why don't you just use a database? That's what they're designed for.
          --
          Tim Roberts, timr@probo.com
          Providenza & Boekelheide, Inc.

            Comment

            Previous template Next
            Working...