DBM scalability

[George Sakkis]

I'm trying to create a dbm database with around 4.5 million entries but the existing dbm modules
(dbhash, gdbm) don't seem to cut it. What happens is that the more entries are added, the more time
per new entry is required, so the complexity seems to be much worse than linear. Is this to be
expected and if so, should I expect better performance (i.e. linear or almost linear) from a real
database, e.g. sqlite ?

George

[Paul Rubin]

Re: DBM scalability

"George Sakkis" <gsakkis@rutger s.edu> writes:[color=blue]
> I'm trying to create a dbm database with around 4.5 million entries
> but the existing dbm modules (dbhash, gdbm) don't seem to cut
> it. What happens is that the more entries are added, the more time
> per new entry is required, so the complexity seems to be much worse
> than linear. Is this to be expected[/color]

No, not expected. See if you're using something like db.keys() which
tries to read all the keys from the db into memory, or anything like that.

[George Sakkis]

Re: DBM scalability

"Paul Rubin" <http://phr.cx@NOSPAM.i nvalid> wrote:
[color=blue]
> "George Sakkis" <gsakkis@rutger s.edu> writes:[color=green]
> > I'm trying to create a dbm database with around 4.5 million entries
> > but the existing dbm modules (dbhash, gdbm) don't seem to cut
> > it. What happens is that the more entries are added, the more time
> > per new entry is required, so the complexity seems to be much worse
> > than linear. Is this to be expected[/color]
>
> No, not expected. See if you're using something like db.keys() which
> tries to read all the keys from the db into memory, or anything like that.[/color]

It turns out it doesn't have to do with python or the dbm modules. The same program on a different
box and platform runs linearly, so I guess it has to do with the OS and/or the hard disk
configuration.

George