Where to log?

[Micha³ Wo¼niak]

Ok, I have a nice iface, I have a (hopefully) good user authentication, I
have a DB up-and-running, and I have PHP 4.3.4 (yeah, I know, I will
update...).

What I need is a logging utility. And the question is: where to keep the
logs?

We (me and my buddy) have written everything from scratch (including
sessions support) - in a quest to learn something useful. Now we need to
decide where to keep our logs?

We can either keep them in a DB (already running, so no
setup/configuration needed here), or in a file.
Obviously, the file has an advantage over the DB - no overhead, less
queries. But here's the bad part: there will be ~500 users using this
site. It most certainly *will* happen that a few will use it
simultaneously.

So, we can flock() the file. Great. I can see the queue getting dreadfuly
long, the execution time getting dangerously near the limit and the
users getting mad on the other side of the cable.

Well, we don't have to flock() it. Fantastic - I can just imagine what
will happen if ~10 instaces of the script will try to write to this file
simultaneously. ..

Ok, we can can even write a "utility" script, that will get invoked by
the other scripts (with fopen(), not include()) and will do the actual
logging. But - isn't that just a other way for flock()? :)

Well, why don't we use the DB? Oh, right - the overhead...

Any ideas? I'm run out of mine...

Cheers
Mike

[Jacob Atzen]

Re: Where to log?

On 2005-04-22, Micha³ Wo¼niak
<mikiwoz_remove _this@yahoo_rem ove_this.co.uk> wrote:[color=blue]
> So, we can flock() the file. Great. I can see the queue getting
> dreadfuly
> long, the execution time getting dangerously near the limit and the
> users getting mad on the other side of the cable.[/color]

If you only flock() it for the split second it actually is needed to
open and write to the log file you shouldn't see that kind of problem.
Even with 500 concurrent users (that is 500 people visiting your site at
the same time, not 500 concurrent requests) you should have plenty of
capacity to spare.

--
Cheers,
- Jacob Atzen

[Micha³ Wo¼niak]

Re: Where to log?

One quick glance of an experienced eye allowed to understand the blurred
and almost unreadable Jacob Atzen's handwriting:
[color=blue]
> On 2005-04-22, Micha? Wo?niak
> <mikiwoz_remove _this@yahoo_rem ove_this.co.uk> wrote:[color=green]
>> So, we can flock() the file. Great. I can see the queue getting
>> dreadfuly
>> long, the execution time getting dangerously near the limit and the
>> users getting mad on the other side of the cable.[/color]
>
> If you only flock() it for the split second it actually is needed to
> open and write to the log file you shouldn't see that kind of problem.
> Even with 500 concurrent users (that is 500 people visiting your site
> at the same time, not 500 concurrent requests) you should have plenty
> of capacity to spare.
>[/color]

Still not convinced, as there will be multiple write aactions to the file
per request...

Cheers
Mike

[Micha³ Wo¼niak]

Re: Where to log?

One more idea: syslog().
The only problem is: I don't want to have those logs in
the /var/log/syslog file, I'd rather having them in /some/other/file -
is it possible? Couldn't find anything on this on php.net :/

TIA
Mike

[Malcolm Dew-Jones]

Re: Where to log?

=?ISO-8859-2?Q?Micha=B3_Wo =BCniak?= (mikiwoz_remove _this@yahoo_rem ove_this.co.uk) wrote:
: Ok, I have a nice iface, I have a (hopefully) good user authentication, I
: have a DB up-and-running, and I have PHP 4.3.4 (yeah, I know, I will
: update...).

: What I need is a logging utility. And the question is: where to keep the
: logs?

: We (me and my buddy) have written everything from scratch (including
: sessions support) - in a quest to learn something useful. Now we need to
: decide where to keep our logs?

: We can either keep them in a DB (already running, so no
: setup/configuration needed here), or in a file.
: Obviously, the file has an advantage over the DB - no overhead, less
: queries. But here's the bad part: there will be ~500 users using this
: site. It most certainly *will* happen that a few will use it
: simultaneously.

: So, we can flock() the file. Great. I can see the queue getting dreadfuly
: long, the execution time getting dangerously near the limit and the
: users getting mad on the other side of the cable.

: Well, we don't have to flock() it. Fantastic - I can just imagine what
: will happen if ~10 instaces of the script will try to write to this file
: simultaneously. ..

Assuming you are on Linux or Unix.

Open the log file for append, and write entire lines at a time, no locking
is required (note _append_ mode, that's the key).

Up to a large size, (something like 64,000 bytes typical) each such line
is guaranteed to be written as a single atomic unit to the end of the
file, no locking required. This is a documented feature of append mode on
unix, and is done exactly for this kind of situation.

(In practise you rarely even have to write whole lines at a time, because
the program's file write will normally buffer your output and write it one
line at a time even if you don't do that part explicitly your self.)

This is true for linux/unix, but may not be true for windows.


--

This space not for rent.

[Jacob Atzen]

Re: Where to log?

On 2005-04-22, Malcolm Dew-Jones <yf110@vtn1.vic toria.tc.ca> wrote:[color=blue]
> Up to a large size, (something like 64,000 bytes typical) each such
> line is guaranteed to be written as a single atomic unit to the end of
> the file, no locking required. This is a documented feature of append
> mode on unix, and is done exactly for this kind of situation.[/color]

Where is this documented?

--
Cheers,
- Jacob Atzen

[Malcolm Dew-Jones]

Re: Where to log?

Jacob Atzen (jacob@aub.dk) wrote:
: On 2005-04-22, Malcolm Dew-Jones <yf110@vtn1.vic toria.tc.ca> wrote:
: > Up to a large size, (something like 64,000 bytes typical) each such
: > line is guaranteed to be written as a single atomic unit to the end of
: > the file, no locking required. This is a documented feature of append
: > mode on unix, and is done exactly for this kind of situation.

: Where is this documented?

I have no idea anymore where this is _directly_ stated, but here is one
example location in the linux man pages where this capability is _implied_
by mentioning an exception to it in NFS (from
http://www.die.net/doc/linux/man/man2/open.2.html, found by a quick
google)

open(2) - Linux man page
NAME
open, creat - open and possibly create a file or device
SYNOPSIS
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
(snip)
O_APPEND

The file is opened in append mode. Before each write, the file
pointer is positioned at the end of the file, as if with lseek.
O_APPEND may lead to corrupted files on NFS file systems if more
than one process appends data to a file at once. This is because
NFS does not support appending to a file, so the client kernel has
to simulate it, which can't be done without a race condition.

Note the caveat that multi processing appending doesn't work properly on
NFS. They only mention this because it is an exception to the normal
state of affairs - where multi processing appending does not lead to
corrupted files.

I'm sure a unix or linux group would have someone who knows the document
where the exact details are mentioned.

--

This space not for rent.

[Micha³ Wo¼niak]

Re: Where to log?

One quick glance of an experienced eye allowed to understand the blurred
and almost unreadable Malcolm Dew-Jones's handwriting:
[color=blue]
> Up to a large size, (something like 64,000 bytes typical) each such
> line is guaranteed to be written as a single atomic unit to the end
> of
> the file, no locking required. This is a documented feature of
> append mode on unix, and is done exactly for this kind of
> situation.[/color]

That would be great, if true. Doing a google search on this. :)

Thanks
Mike

[R. Rajesh Jeba Anbiah]

Re: Where to log?

Michal Wozniak wrote:[color=blue]
> Ok, I have a nice iface, I have a (hopefully) good user[/color]
authentication, I[color=blue]
> have a DB up-and-running, and I have PHP 4.3.4 (yeah, I know, I will
> update...).
>
> What I need is a logging utility. And the question is: where to keep[/color]
the[color=blue]
> logs?[/color]
<snip>

What to log? Access?? If so, why not apache's log?

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/