serverless postgresql

[Jeff Bowden]

For ease of configuration and other reasons, I would like for my
single-user GUI app to be able to use postgresql in-process as a library
accessing a database created in the users home directory. I think I
could possibly get what I want by launching a captive copy of postmaster
with appropriate args but it seems conceptually cleaner to not have a
seperate process at all. Has anyone tried to do anything like this?

I've looked at sqlite and it might be workable for my project but I
would prefer the more standard and featureful SQL of postgres. In
particular sqlite lacks date arithmetic and has some funny type issues
(it claims to be typeless, except when it isn't, or something).





---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postg resql.org

[Enrico Weigelt]

Re: serverless postgresql

* Jeff Bowden <jlb@houseofdis traction.com> [2004-01-13 13:38:02 -0800]:
[color=blue]
> For ease of configuration and other reasons, I would like for my
> single-user GUI app to be able to use postgresql in-process as a library
> accessing a database created in the users home directory. I think I
> could possibly get what I want by launching a captive copy of postmaster
> with appropriate args but it seems conceptually cleaner to not have a
> seperate process at all. Has anyone tried to do anything like this?[/color]

hmm, i've also played around a little bit w/ postgresql standalone.
you could also do nice things like calling postmaster from inetd ;-)

but I dont think you really wanna have the postmaster in the same
process w/ your application, since it relies on forks, signals,
mmap() and some other things which directly affect the process control -
this probably infers with your application. On the application side
you cant really control, what it does, so it sounds quite difficult.
Having the postmaster in a separate process connected by a socket or
pty seems to be the better solution.


cu
--
---------------------------------------------------------------------
Enrico Weigelt == metux IT services

phone: +49 36207 519931 www: http://www.metux.de/
fax: +49 36207 519932 email: contact@metux.d e
cellphone: +49 174 7066481
---------------------------------------------------------------------
Diese Mail wurde mit UUCP versandt. http://www.metux.de/uucp/

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postg resql.org so that your
message can get through to the mailing list cleanly

[Rick Gigger]

Re: serverless postgresql

I have just about the same sort of needs now and concluded that postgres
just is not suited for embedding into apps like that. I am going with
sqlite and it is working fairly well. We just made user defined functions
in php for sqlite to match the date functions in postgres. Who knows what
other issues we may run into but so far sqlite is working fairly well.

I think that sqlite stores everything as a string but for searching,
sorting, ordering etc it uses the type info.

I to would absolutely love to just include a postgres dll and have postgres
in process and just store everything in a nice little file but from what I
have read hear that would involve major changes that the developers on not
interested in making. So far sqlite has done the job for me.

Also I am considering looking into firebird for an embedded database
solution since it can you pretty much what you are talking about here with
an in proccess db that uses a single file to store it's data. While I don't
want to switch my server stuff to it, it is probably more full featured than
sqlite (I'm sure it has date and arithmatic functions) and is made to work
well in embedded situations.

I would be intersted to see what you end up doing.

Rick

----- Original Message -----
From: "Jeff Bowden" <jlb@houseofdis traction.com>
To: "pgsql-general" <pgsql-general@postgre sql.org>
Sent: Tuesday, January 13, 2004 2:38 PM
Subject: [GENERAL] serverless postgresql

[color=blue]
> For ease of configuration and other reasons, I would like for my
> single-user GUI app to be able to use postgresql in-process as a library
> accessing a database created in the users home directory. I think I
> could possibly get what I want by launching a captive copy of postmaster
> with appropriate args but it seems conceptually cleaner to not have a
> seperate process at all. Has anyone tried to do anything like this?
>
> I've looked at sqlite and it might be workable for my project but I
> would prefer the more standard and featureful SQL of postgres. In
> particular sqlite lacks date arithmetic and has some funny type issues
> (it claims to be typeless, except when it isn't, or something).
>
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo@postg resql.org
>[/color]


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Tom Lane]

Re: serverless postgresql

"Rick Gigger" <rick@alpinenet working.com> writes:[color=blue]
> I to would absolutely love to just include a postgres dll and have postgres
> in process and just store everything in a nice little file but from what I
> have read hear that would involve major changes that the developers on not
> interested in making.[/color]

Not only are the developers uninterested in it, the developers actively
oppose it. We think an embedded database library cannot be reliable
enough to meet our notion of a "database", since it would be subject to
failures anytime the surrounding application has a bug. Keeping the
client code in a separate process is a far more robust design.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postg resql.org

[Jeff Bowden]

Re: serverless postgresql

Tom Lane wrote:
[color=blue]
>"Rick Gigger" <rick@alpinenet working.com> writes:
>
>[color=green]
>>I to would absolutely love to just include a postgres dll and have postgres
>>in process and just store everything in a nice little file but from what I
>>have read hear that would involve major changes that the developers on not
>>interested in making.
>>
>>[/color]
>
>Not only are the developers uninterested in it, the developers actively
>oppose it. We think an embedded database library cannot be reliable
>enough to meet our notion of a "database", since it would be subject to
>failures anytime the surrounding application has a bug. Keeping the
>client code in a separate process is a far more robust design.
>
>
>[/color]

That makes sense to me. I wonder if sqlite suffers for this problem
(e.g. app crashing and corrupting the database).

What about the notion of running postmaster on-demand as the user? Is
that something that anyone has experience with? It seems like it would
solve the complex configuration problems without compromising robustness
or requiring any special support other than sufficient command-line
parameters.

Oh yeah, that brings me to another question. I was looking at the
postmaster command-line switches and I couldn't find any that would
allow me to point it at an arbitrary config file but then I had a look
around and it seems that the .conf files are already used by the various
startup scripts. Does postmaster itself ever read the .conf files or is
it controlled strictly by switches?



---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[Dustin Sallings]

Re: serverless postgresql


On Jan 13, 2004, at 13:38, Jeff Bowden wrote:
[color=blue]
> but it seems conceptually cleaner to not have a seperate process at
> all.[/color]

That depends a lot on what you mean by cleaner. A few small programs
in their own process space dedicated to a specific set of functionality
with well-defined interfaces can make for a much more reliable
application (see postfix).

--
Dustin Sallings


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Tom Lane]

Re: serverless postgresql

Jeff Bowden <jlb@houseofdis traction.com> writes:[color=blue]
> That makes sense to me. I wonder if sqlite suffers for this problem
> (e.g. app crashing and corrupting the database).[/color]

Likely. I can tell you that Ann Harrison once told me she made a decent
amount of money as a consultant fixing broken Interbase/Firebird
database files. It would be hard to make a living in the same game for
Postgres. Now I don't think that Firebird is any buggier than Postgres.
But it comes in an embedded-library form; I'll bet lunch that most of
those data corruption problems were actually induced by crashes of
surrounding applications.
[color=blue]
> What about the notion of running postmaster on-demand as the user?[/color]

Possibly. You'd have to think carefully about what conditions the
postmaster should be shut down under, and especially what conditions
it should NOT be shut down under --- eg, a kill to the parent client
application shouldn't cause an ungraceful postmaster exit. It could
be tricky to get the signal handling right, especially under shells that
try to deliver signals to all children of a process being signaled.
On the whole I suspect it'd be easier just to leave the postmaster
running in the background...
[color=blue]
> Oh yeah, that brings me to another question. I was looking at the
> postmaster command-line switches and I couldn't find any that would
> allow me to point it at an arbitrary config file[/color]

The config files all live in $PGDATA and so are determined by the -D
switch. There was some talk of changing this, awhile back, but it
foundered on lack of consensus about exactly what to do instead.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Christopher Browne]

Re: serverless postgresql

jlb@houseofdist raction.com (Jeff Bowden) wrote:[color=blue]
> Tom Lane wrote:[color=green]
>>Not only are the developers uninterested in it, the developers actively
>>oppose it. We think an embedded database library cannot be reliable
>>enough to meet our notion of a "database", since it would be subject to
>>failures anytime the surrounding application has a bug. Keeping the
>>client code in a separate process is a far more robust design.[/color][/color]
[color=blue]
> That makes sense to me. I wonder if sqlite suffers for this problem
> (e.g. app crashing and corrupting the database).[/color]

I'm not sure its use has yet been so widespread that there is a good
feel for this. It is doubtless _possible_; "crash patterns" likely
_don't_ lead to enormous disasters when databases are small,
localized, and the apps probably _don't_ crash all the time.
[color=blue]
> What about the notion of running postmaster on-demand as the user?
> Is that something that anyone has experience with? It seems like it
> would solve the complex configuration problems without compromising
> robustness or requiring any special support other than sufficient
> command-line parameters.[/color]

I haven't had call for this being controlled "by the application," but
I can't see it being vastly troublesome. And I _don't_ see it
requiring lots of "command line" parameters; you just need to specify
the directory where the configuration is.
[color=blue]
> Oh yeah, that brings me to another question. I was looking at the
> postmaster command-line switches and I couldn't find any that would
> allow me to point it at an arbitrary config file but then I had a
> look around and it seems that the .conf files are already used by
> the various startup scripts. Does postmaster itself ever read the
> .conf files or is it controlled strictly by switches?[/color]

All of the .conf files are in one directory, and that directory is
controlled by either the value of environment variable PGDATA or the
"-D" command parameter.

It seems not-overly-valuable to have the .conf files be able to be
specified in random other locations.

Is there some particular reason you have in mind why you would want to
_ignore_ the configuration in $HOME/DBDIR and instead use
configuration in some other random location? I would think that
collecting the config into one directory, as is done right now, is a
_good_ thing.
--
let name="cbbrowne" and tld="ntlug.org" in name ^ "@" ^ tld;;

http://www3.sympatico.ca/cbbrowne/lsf.html

"Even in the area of anticompetitive conduct, Microsoft is mainly an
imitator." -- Ralph Nader (1998/11/11)

[Jeff Bowden]

Re: serverless postgresql

Tom Lane wrote:
[color=blue]
>Jeff Bowden <jlb@houseofdis traction.com> writes:
>
>[color=green]
>>What about the notion of running postmaster on-demand as the user?
>>
>>[/color]
>
>Possibly. You'd have to think carefully about what conditions the
>postmaster should be shut down under, and especially what conditions
>it should NOT be shut down under --- eg, a kill to the parent client
>application shouldn't cause an ungraceful postmaster exit. It could
>be tricky to get the signal handling right, especially under shells that
>try to deliver signals to all children of a process being signaled.
>On the whole I suspect it'd be easier just to leave the postmaster
>running in the background...
>
>[/color]

Details, details.... :-)
[color=blue][color=green]
>>Oh yeah, that brings me to another question. I was looking at the
>>postmaster command-line switches and I couldn't find any that would
>>allow me to point it at an arbitrary config file
>>
>>[/color]
>
>The config files all live in $PGDATA and so are determined by the -D
>switch. There was some talk of changing this, awhile back, but it
>foundered on lack of consensus about exactly what to do instead.
>
>[/color]

As long as it can be done.



---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[Christopher Browne]

Re: serverless postgresql

Quoth jlb@houseofdist raction.com (Jeff Bowden):[color=blue]
> For ease of configuration and other reasons, I would like for my
> single-user GUI app to be able to use postgresql in-process as a
> library accessing a database created in the users home directory. I
> think I could possibly get what I want by launching a captive copy of
> postmaster with appropriate args but it seems conceptually cleaner to
> not have a seperate process at all. Has anyone tried to do anything
> like this?[/color]

One of the properties of Unix is that spawning extra processes to "do
their thing" allows each application to be designed more 'cleanly.'

Curiously enough, the implementors of alternatives to the Sendmail MTA
have typically concluded that it was not only conceptually cleaner to
have a set of separate cooperating processes, but that they would also
reap benefits in terms of speed and improved security.

Thus, I think you'll find that people disagree with you on this. It
seems to me that it is conceptually cleaner to have the "applicatio n"
do "applicatio n stuff," and not muddle things up by mixing that with
the "database stuff."
--
If this was helpful, <http://svcs.affero.net/rm.php?r=cbbrow ne> rate me

http://cbbrowne.com/info/sgml.html

Rules of the Evil Overlord #87. "My vats of hazardous chemicals will
be covered when not in use. Also, I will not construct walkways above
them." <http://www.eviloverlor d.com/>

[Jim Wilson]

planner and join type mismatches

Tom Lane <tgl@sss.pgh.pa .us> said:
[color=blue]
> ---------------------------(end of broadcast)---------------------------
> TIP 9: the planner will ignore your desire to choose an index scan if your
> joining column's datatypes do not match
>[/color]

Re: tip 9 :-). In 7.3.5 is it possible that the planner is finding a type
mismatch when there isn't one, resulting in a seq scan being selected? I've
had a problem with joining tables on an int8 column (primary key column in one
of the tables). Tested left outer and where clause syntax.

TIA,

Jim Wilson


---------------------------(end of broadcast)---------------------------
TIP 5: Have you checked our extensive FAQ?

http://www.postgresql.org/docs/faqs/FAQ.html

[Tom Lane]

Re: planner and join type mismatches

"Jim Wilson" <jimw@kelcomain e.com> writes:[color=blue]
> Re: tip 9 :-). In 7.3.5 is it possible that the planner is finding a type
> mismatch when there isn't one, resulting in a seq scan being selected?[/color]

When you gave no details, how is anyone to know? For questions like
this, you need to post the table definitions, the exact query, and the
results of EXPLAIN ANALYZE on that query. BTW, pgsql-performance is a
more appropriate list than -general for such questions.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postg resql.org so that your
message can get through to the mailing list cleanly

[Rick Gigger]

Re: serverless postgresql

> Jeff Bowden <jlb@houseofdis traction.com> writes:[color=blue][color=green]
> > That makes sense to me. I wonder if sqlite suffers for this problem
> > (e.g. app crashing and corrupting the database).[/color]
>
> Likely. I can tell you that Ann Harrison once told me she made a decent
> amount of money as a consultant fixing broken Interbase/Firebird
> database files. It would be hard to make a living in the same game for
> Postgres. Now I don't think that Firebird is any buggier than Postgres.
> But it comes in an embedded-library form; I'll bet lunch that most of
> those data corruption problems were actually induced by crashes of
> surrounding applications.[/color]

I remember reading about how some the original commerical code from
Interbase came with some bugs that corrupted databases and that those bugs
were eventually fixed in Firebird. Don't know if that's related to this but
it could be.

Anyway since postgres uses WAL files to verify the integrity of the database
couldn't it more or less make the same guarantee's in an embedded version?
As long as the app uses the db libs unmodified and doesn't mess with the
files it creates how does simply making it embedded increase the change of
db errors resulting in database corruption?

In addition there are times when you want the ease of a relational database
but you don't need say the same level of reliability that you would on your
server. In these cases it would be nice to just have something that is easy
to drop into your app and have it just work. That being said I think that
in these cases the developer isn't usually going to care if it runs in
process or not. Maybe a good solution would be to make a postgres library
that could be included in process that would interface with the application.
All it would do is manager spawning the real postgres process. Then you
would have the reliability of a real server but the ease of use of an
embedded database. Couldn't that end up being pretty transparent to the
developer? If it was made to be easy postgres could become very popular as
an "embedded" db solution but still not have it's reputation diluted by
rampant database corruption from the application.


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Doug McNaught]

Re: serverless postgresql

"Rick Gigger" <rick@alpinenet working.com> writes:
[color=blue]
> Anyway since postgres uses WAL files to verify the integrity of the database
> couldn't it more or less make the same guarantee's in an embedded version?
> As long as the app uses the db libs unmodified and doesn't mess with the
> files it creates how does simply making it embedded increase the change of
> db errors resulting in database corruption?[/color]

If an app's internal data structures get corrupted and it goes nuts,
pretty much anything can happen to its address space.

-Doug

---------------------------(end of broadcast)---------------------------
TIP 3: if posting/reading through Usenet, please send an appropriate
subscribe-nomail command to majordomo@postg resql.org so that your
message can get through to the mailing list cleanly

[Jonathan Bartlett]

Re: serverless postgresql

> Anyway since postgres uses WAL files to verify the integrity of the database[color=blue]
> couldn't it more or less make the same guarantee's in an embedded version?
> As long as the app uses the db libs unmodified and doesn't mess with the
> files it creates how does simply making it embedded increase the change of
> db errors resulting in database corruption?[/color]

I have a different idea. I've been thinking about coding it, but haven't
had the time. This could be done with no changes to Postgres itself.

Basically, you would have a library which exported functions such as

pg_instance *pg_start(char *directory);
pg_get_connecti on(pg_instance *);
pg_stop(pg_inst ance *);
pg_initdb(char *directory)

pg_start would do the following:
1) check "directory" for an instance of the UNIX socket.
if it is there, make a Postgres connection
if it is not there, start the Postmaster server with "-k directory
-D directory/data" and then make a Postgres connection
2) Create a struct to contain the directory and any other data item we
need to connect to the database
3) Return this structure

pg_get_connecti on would just be a wrapper for pq_connect()

pg_stop would kill the database.

pg_initdb would simply run initdb

Does anyone see a reason why this wouldn't work?

Jon


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster