Database vs Data Structure?

[erikcw]

Hi,

I'm working on a web application where each user will be creating
several "projects" in there account, each with 1,000-50,000 objects.
Each object will consist of a unique name, an id, and some meta data.

The number of objects will grow and shrink as the user works with
their project.

I'm trying to decided whether to store the objects in the database
(each object gets it's own row) or to use some sort of data-structure
(maybe nested dictionaries or a custom class) and store the pickled
data-structure in a single row in the database (then unpickle the data
and query in memory).

A few requirements:
-Fast/scalable (web app)
-able to query objects based on name and id.
-will play nicely with versioning (undo/redo)

Any input on the best way to go?

Thanks!
Erik

[M.-A. Lemburg]

Re: Database vs Data Structure?

On 2008-04-18 05:37, erikcw wrote:

Hi,
>
I'm working on a web application where each user will be creating
several "projects" in there account, each with 1,000-50,000 objects.
Each object will consist of a unique name, an id, and some meta data.
>
The number of objects will grow and shrink as the user works with
their project.
>
I'm trying to decided whether to store the objects in the database
(each object gets it's own row) or to use some sort of data-structure
(maybe nested dictionaries or a custom class) and store the pickled
data-structure in a single row in the database (then unpickle the data
and query in memory).
>
A few requirements:
-Fast/scalable (web app)
-able to query objects based on name and id.
-will play nicely with versioning (undo/redo)
>
Any input on the best way to go?

Relational databases offer the best scalability and reliability,
so I'd go for those as backend.

If your data is mostly read-only and usually only touches
a small part of your database (e.g. the data for a day or two),
then you can increase query performance by keeping that part in
an in-memory database (e.g. use sqlite or Oracle TimesTen)
and only update the copy in case something changes in the
backend.

Regards,
--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Apr 18 2008)

>>Python/Zope Consulting and Support ... http://www.egenix.com/
>>mxODBC.Zope.D atabase.Adapter ... http://zope.egenix.com/
>>mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

_______________ _______________ _______________ _______________ ____________

:::: Try mxODBC.Zope.DA for Windows,Linux,S olaris,MacOSX for free ! ::::


eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611