ZODB: single database, multiple connections

[Petra Chong]

Hello all

I am using Python 2.3 and ZODB (without the rest of Zope) with the
following pattern:

* One process which writes stuff to a ZODB instance (call it test.db)
* Another process which reads stuff from the above ZODB instance
test.db

What I find is that when the first process writes, the second doesn't
pick up the changes. I am sure this must be because I am using ZODB
wrongly, but I can't find any examples that show how to use ZODB in
this way, and I can't find any API docs for FileStorage, Connection,
etc. Reading the source code (from C:\python23\lib \site-packages) has
not thrown up anything useful.

Here's my test code:

A simple database class:

class Database(object ):
PersistentObjec t = persistent.Pers istent
PersistentDict = BTrees.OOBTree. OOBTree

def __init__(self, filename, read_only = False):
self.storage = FileStorage.Fil eStorage(filena me, read_only =
read_only)
self.db = DB(self.storage )
self.connection = self.db.open()
self.dbroot = self.connection .root()


Write:


db = Database("test. db")
db.data = db.get_dictiona ry('data')

sz = len(db.data.key s())

class Datum(Persisten t):
def __init__(self, value):
self.value = value

if __name__ == '__main__':
# insert 10 things
for i in range(0, 10):
val = i + sz
d = Datum(val)
db.data[val] = d
transaction.com mit()

Read:

db = Database("test. db", read_only = True)

data = db.get_dictiona ry('data')

If I have a Python shell open and run the above two lines, if I run the
write process repeatedly, the above "data" object never contains any of
the newly added items. To pick them up I have to totally recreate the
"db" object.

I must be doing something wrongly, but I can't figure out what.

Any suggestions?

Thanks,

PC

[Tim Peters]

Re: ZODB: single database, multiple connections

[Petra Chong]

I am using Python 2.3 and ZODB (without the rest of Zope) with the
following pattern:
>
* One process which writes stuff to a ZODB instance (call it test.db)
* Another process which reads stuff from the above ZODB instance
test.db
>
What I find is that when the first process writes, the second doesn't
pick up the changes. I am sure this must be because I am using ZODB
wrongly, but I can't find any examples that show how to use ZODB in
this way, and I can't find any API docs for FileStorage, Connection,
etc. Reading the source code (from C:\python23\lib \site-packages) has
not thrown up anything useful.
>
Here's my test code:
>
A simple database class:
>
...
>
Write:
>
...
>
Read:
>
db = Database("test. db", read_only = True)
>
data = db.get_dictiona ry('data')
>
If I have a Python shell open and run the above two lines, if I run the
write process repeatedly, the above "data" object never contains any of
the newly added items. To pick them up I have to totally recreate the
"db" object.

You say that like it's hard to do ;-)

It's a decent way to proceed. ZODB is a database, and has
transactional semantics: you never see new object state on the read
side because you're /in/ a transaction, and a transaction guarantees
to give you a consistent view of the data. The view would be
inconsistent if it showed you state committed by different
transactions on the write side while you're still in the same
transaction on the read side.

I must be doing something wrongly, but I can't figure out what.

Seems to be a conceptual problem more than anything else.

Any suggestions?

You already know that tossing your connection and opening a new
connection will give you a newer view of the database, and it's
unclear why you don't think that's good enough. Other approaches
amount to telling ZODB (on the read side) that you're done with the
current transaction. For example, try doing

transaction.abo rt()

on the read side when you're ready to see newer object state.

BTW, a better place to ask about ZODB is the zodb-dev list:

http://mail.zope.org/mailman/listinfo/zodb-dev

It's not just for developers /of/ ZODB. Note that you need to
subscribe to it in order to post to it (that's a heavyweight anti-spam
gimmick common to all Zope lists).

[Petra Chong]

Re: ZODB: single database, multiple connections

If I have a Python shell open and run the above two lines, if I run the
write process repeatedly, the above "data" object never contains any of
the newly added items. To pick them up I have to totally recreate the
"db" object.

>
You say that like it's hard to do ;-)
>

It isn't, but this was the problem:

It took 15 seconds to open the database, so I thought that I shouldn't
be recreating the database and should be refreshing it. There is
nothing in the docs that says that the right behaviour is to recreate
it.

However, I then put some debug statements around it (I know, I know, I
could have used profiling) to find out exactly what was taking 15
seconds. Turned out there was some rubbish in the db that shouldn't
have been there. I got rid of it, and now it doesn't take 15 seconds to
recreate the database.

So- I am now recreating the db, and my problem is solved.

>
BTW, a better place to ask about ZODB is the zodb-dev list:
>

http://mail.zope.org/mailman/listinfo/zodb-dev

>
It's not just for developers /of/ ZODB. Note that you need to
subscribe to it in order to post to it (that's a heavyweight anti-spam
gimmick common to all Zope lists).

Thanks for that- it's been hard to track down information on ZODB.

Regards,

PC