how does one trap for out-of-memory errors?

[lawrence]

Re: how does one trap for out-of-memory errors?

nc@iname.com (Nikolai Chuvakhin) wrote in message[color=blue][color=green]
> > I am not hell bent. I asked for alternatives.[/color]
>
> I think looking for an altermative in your situation was the right
> thing to do. Note also that you didn't have a compelling reason
> to do it "your" way in the first place; at least, my question to
> that extent:
>[color=green][color=darkred]
> > > But let me ask you: what will the application do
> > > with a PHP array that contains all entries in the weblog? Output
> > > them? Then the entries don't have to be in memory at all, they
> > > can be output straight from the DB buffer. Sort them? Again,
> > > they don't have to be in the memory for that, you should have
> > > taken care of it on the database level. Search through them?
> > > Same thing; too late to do it efficiently, had to be done on the
> > > database level.[/color][/color]
>
> went unanswered...[/color]

I'm not sure what you mean when you say I didn't answer your question.
I explained that I was trying to hide the database from the client
code. Putting the database return into a PHP array had been my answer
to that. You can, and have, argued that it is the wrong answer, but I
certainly answered your question about what it was that I was trying
to accomplish.





[color=blue][color=green]
> > Well, this code is being written first to work with MySql, but I'm
> > supposed to write it in such a way that would make it easy to switch
> > to PostGre.[/color]
>
> This is yet another reason why database abstraction layers are
> such a bad idea... PostgreSQL has a whole slew of features
> that are not available in MySQL: views, rules, triggers, stored
> procedures, etc. Database abstraction forces you to stick to
> the lowest common denominator, which robs you of opportunities
> to use the more advanced, higher-performance features...[/color]

Engineering of any kind is often about goals and tradeoffs.
Programmers would, in general, be delighted if, by magic, there was
some way to both take advantage of all the features of any database
and yet also easily port software from one database to another.
However, there is no easy way to achieve both these goals, so one
generally has to choose one. Either software can take advantage of the
special features of one piece of software, or it can be easily ported
from one database to another.

In our case, portability trumps advanced feature use. For another
project the opposite decision might be a wise one.

[lawrence]

Re: how does one trap for out-of-memory errors?

nc@iname.com (Nikolai Chuvakhin) wrote in message[color=blue][color=green]
> > The query object also needs to remember which entry it got last,
> > so it can get the next one.[/color]
>
> It already does, since it has a resource with an internal pointer
> to the next available entry...[/color]

It strikes me as breathtakingly complicated. If I want my code to do
one thing if the database has returned less than 50 rows, but
something else if the database has returned 50 or more rows, then a
request to count the number of rows returned needs to be sent down the
line, through the selectObject to the queryObject, and then returned
back up the line. And both objects need a getNextRow method, wrapping
(for MySQL) mysql_fetch_row . Or, if I want sometimes use
mysql_fetch_obj ect, or any of the other commands for getting rows,
then I'd have to write a separate method for each of those commands.
If I want the API of every queryObject to conform to the contract of a
standard interface, then I need to write these methods with that in
mind.

This may be the best way of doing things, I can't think of anything
better, but it strikes me as a large project, especially as I have
about 100 SQL statements for MySql alone.

[Zurab Davitiani]

Re: how does one trap for out-of-memory errors?

Nikolai Chuvakhin wrote on Friday 24 October 2003 15:08:
[color=blue]
> I am not saying you're wrong, but if you're right, I'd like to
> understand how it is possible for you to be right while I see
> what I see. No pun intended; I really would like to learn
> something here...[/color]

This is OT from the original discussion. Sorry for taking it in this
direction, but I'd like to clear up my original claim.

You are right. The issue doesn't affect primitive variables and arrays
thereof. It only affects objects. Objects don't seem to be destroyed (and
memory freed) until the script execution ends. e.g., if you have:

for($i=0; $i<10000; $i++) {
$myObj = new LogParser("Some BigLogFile.log" );
}

$myObj objects are not destroyed (to free the memory) even though the
variable with the same name is being reassigned. So, total memory
consumption goes up with each pass. And unset() doesn't help either in this
case.

But, if you have:

for($i=0; $i<10000; $i++) {
$contents = file_get_conten ts("SomeBigLogF ile.log");
}

the memory consumption will stay the same.

By going over some comments in previous posts on this topic, it seems like
PHP5 does have a better memory management with regard to objects. I will
test it when I have time, or if anyone has verified, feel free to post.

--
Business Web Solutions
ActiveLink, LLC

https://www.active-link.com/intranet/

[Alan Little]

Re: how does one trap for out-of-memory errors?

Carved in mystic runes upon the very living rock, the last words of
lawrence of comp.lang.php make plain:
[color=blue]
> I can't imagine how I can hide the database from the client code if I
> use database return resources, as you suggest above. But I now realize
> how limiting these arrays can be, in the presence of PHP's 8 meg
> limit. So I'm open to any graceful ideas for otherwise masking the
> database from the client code.[/color]

Unless I'm misunderstandin g, it seems like you're overlooking an obvious
solution. You have methods for transparently managing the queries, why
not have one for transparently retrieving the data from the resource
pointer? I have something like this in one of my applications, which
needs to be able to work with MySQL or Sybase. I simply wrote db_
functions to replace all the mysql_ and sybase_ functions, and made them
intelligent enough to handle the differences transparently.

--
Alan Little
Phorm PHP Form Processor

http://www.phorm.com/

[Nikolai Chuvakhin]

Re: how does one trap for out-of-memory errors?

Zurab Davitiani <agt@mindless.c om> wrote in message
news:<Lk6nb.131 9$X76.1047@news svr14.news.prod igy.com>...[color=blue]
> Nikolai Chuvakhin wrote on Friday 24 October 2003 15:08:
>[color=green]
> > I am not saying you're wrong, but if you're right, I'd like to
> > understand how it is possible for you to be right while I see
> > what I see. No pun intended; I really would like to learn
> > something here...[/color]
>
> This is OT from the original discussion.[/color]

I don't think so; the original discussion was about preventing
out-of-memory errors. The issue of how memory is allocated is,
I believe, relevant to that discussion.
[color=blue]
> You are right. The issue doesn't affect primitive variables and arrays
> thereof. It only affects objects. Objects don't seem to be destroyed (and
> memory freed) until the script execution ends.[/color]

Thank you, this indeed clears things up.

Cheers,
NC

[lawrence]

Re: how does one trap for out-of-memory errors?

Alan Little <alan@n-o-s-p-a-m-phorm.com> wrote in message news:<Xns942245 69416ACalanphor mcom@216.196.97 .132>...[color=blue]
> Carved in mystic runes upon the very living rock, the last words of
> lawrence of comp.lang.php make plain:
>[color=green]
> > I can't imagine how I can hide the database from the client code if I
> > use database return resources, as you suggest above. But I now realize
> > how limiting these arrays can be, in the presence of PHP's 8 meg
> > limit. So I'm open to any graceful ideas for otherwise masking the
> > database from the client code.[/color]
>
> Unless I'm misunderstandin g, it seems like you're overlooking an obvious
> solution. You have methods for transparently managing the queries, why
> not have one for transparently retrieving the data from the resource
> pointer? I have something like this in one of my applications, which
> needs to be able to work with MySQL or Sybase. I simply wrote db_
> functions to replace all the mysql_ and sybase_ functions, and made them
> intelligent enough to handle the differences transparently.[/color]

Yes, exactly. Sorry if I wasn't more clear, but that was what I did in
the end. I was a little dense, and did not at first see that as a
solution, but then in the end that is what I did.

[lawrence]

Re: how does one trap for out-of-memory errors?

Zurab Davitiani <agt@mindless.c om> wrote in message news:<Lk6nb.131 9$X76.1047@news svr14.news.prod igy.com>...[color=blue]
> Nikolai Chuvakhin wrote on Friday 24 October 2003 15:08:
>[color=green]
> > I am not saying you're wrong, but if you're right, I'd like to
> > understand how it is possible for you to be right while I see
> > what I see. No pun intended; I really would like to learn
> > something here...[/color]
>
> This is OT from the original discussion. Sorry for taking it in this
> direction, but I'd like to clear up my original claim.
>
> You are right. The issue doesn't affect primitive variables and arrays
> thereof. It only affects objects. Objects don't seem to be destroyed (and
> memory freed) until the script execution ends. e.g., if you have:
>
> for($i=0; $i<10000; $i++) {
> $myObj = new LogParser("Some BigLogFile.log" );
> }
>
> $myObj objects are not destroyed (to free the memory) even though the
> variable with the same name is being reassigned. So, total memory
> consumption goes up with each pass. And unset() doesn't help either in this
> case.
>
> But, if you have:
>
> for($i=0; $i<10000; $i++) {
> $contents = file_get_conten ts("SomeBigLogF ile.log");
> }
>
> the memory consumption will stay the same.
>
> By going over some comments in previous posts on this topic, it seems like
> PHP5 does have a better memory management with regard to objects. I will
> test it when I have time, or if anyone has verified, feel free to post.[/color]


I'm very glad you wrote this. I was just about to start a new thread,
with the title "Word of warning: garbage collection with objects not
efficient", but now that you've written this post I guess there is no
need. This is a lesson that I have learned empircally, through a
painful process. I never got out-of-memory messages until I started
using objects, but they've become more frequent as I've used more
objects. The most startling, and stark, example, came last week when I
took a function and moved it, unchanged, into an object. Suddenly,
many things in my code that had worked now got out-of-memory errors
with that class method. And mind you, I made no change to the function
except move into a class. As a function in procedural code, it brought
no memory errors, but as a class method it became a fatal bottle-neck
for the script.

So there is something to what you say. PHP's garbage collection is
much less automatic when it deals with objects. Anyone considering an
OO approach with PHP should be warned of this.

It also brings up the issue of how, exactly, to empty class variables
in PHP. Is it enough to do this:

$myVar = null;


??????

I have a class array which seems to make increasing demands on memory
with every use, even though I've started to do this at the end of each
use:

$dataTable["password"] = "";
$dataTable["username"] = "";
$dataTable["perPagePasswor d"] = "";

[lawrence]

Re: how does one trap for out-of-memory errors?

Zurab Davitiani <agt@mindless.c om> wrote in message:[color=blue]
> $myObj objects are not destroyed (to free the memory) even though the
>variable with the same name is being reassigned. So, total memory
>consumption goes up with each pass. And unset() doesn't help either[/color]
in this[color=blue]
>case.[/color]

This makes me nervous. If unset() does not work, what does? "null" I
hope?

I have empirically noticed what you describe. With object, in a for()
loop, memory consumption goes up with every pass. How is one to deal
with this?

[Zurab Davitiani]

Re: how does one trap for out-of-memory errors?

lawrence wrote on Monday 03 November 2003 09:28:
[color=blue]
> So there is something to what you say. PHP's garbage collection is
> much less automatic when it deals with objects. Anyone considering an
> OO approach with PHP should be warned of this.[/color]

I agree. Not only is it not automatic, but there's no way to even manually
free up the memory used by objects. Although I haven't had time to install
and test PHP5 in this regard, it'd be interesting to know if this is
implemented better.
[color=blue]
> It also brings up the issue of how, exactly, to empty class variables
> in PHP. Is it enough to do this:
>
> $myVar = null;
>
>
> ??????
>
> I have a class array which seems to make increasing demands on memory
> with every use, even though I've started to do this at the end of each
> use:
>
> $dataTable["password"] = "";
> $dataTable["username"] = "";
> $dataTable["perPagePasswor d"] = "";[/color]

I am not sure whether it will make any difference at all. i.e., I am not
sure the original memory used by object property will be freed. If you do
the testing on this, please post the results.

--
Business Web Solutions
ActiveLink, LLC

https://www.active-link.com/intranet/

[Zurab Davitiani]

Re: how does one trap for out-of-memory errors?

lawrence wrote on Monday 03 November 2003 11:56:
[color=blue]
> Zurab Davitiani <agt@mindless.c om> wrote in message:[color=green]
>> $myObj objects are not destroyed (to free the memory) even though the
>>variable with the same name is being reassigned. So, total memory
>>consumption goes up with each pass. And unset() doesn't help either[/color]
> in this[color=green]
>>case.[/color]
>
> This makes me nervous. If unset() does not work, what does? "null" I
> hope?
>
> I have empirically noticed what you describe. With object, in a for()
> loop, memory consumption goes up with every pass. How is one to deal
> with this?[/color]

Again, as I said before, there's no way to "deal" with this in PHP4 - no way
that I found anyway. Assigning to "", null, using unset(), etc. does not
free the memory.

I came accross this issue when I tried to run a long database update script
(from shell) with a lot of data using objects. I would get memory
allocation error at certain point. The way I resolved it (and still used
objects) was to launch a separate PHP process to query and update the data
for each record; so, all object references would get created inside that
process and die when that process was finished and memory would be freed
up. Now, this is not an option for most web apps, but it may work on longer
cron jobs when using objects is a necessity.

--
Business Web Solutions
ActiveLink, LLC

https://www.active-link.com/intranet/

[Nikolai Chuvakhin]

Re: how does one trap for out-of-memory errors?

lkrubner@geocit ies.com (lawrence) wrote in message
news:<da7e68e8. 0311031156.6555 c850@posting.go ogle.com>...[color=blue]
>
> With object, in a for() loop, memory consumption goes up
> with every pass. How is one to deal with this?[/color]

Just say no to OOP... PHP (or any other development tool, for that
matter) can't be all things to all people. My understanding is that
PHP was conceived as a lightweight high-performance highly portable
scripting environment. Hence, no dependence on system objects and
OOP added as an afterthought, an option for the developer to pay for
code reusability with loss of performance. What I personally find
very strange is the number of people willing to make the trade...
Sure, OOP sounds sophisticated, but the simple truth is, PHP was not
concieved sophisticated (in fact, I find it humorous that the PHP
team chose to replace Rasmus Lerdorf's initial meaning of PHP --
"Profession al Home Page" -- with their recursive "PHP: Hypertext
Preprocessor" routine). So I sincerely believe that if OOP is what
you like to do, your time is better spent working with ASP or JSP,
both of which require you to subscribe to the OOP methodology.

None of the above is meant to be a personal criticism of you; rather,
it's a general statement of (not necessarily unbiased) opinion on
application performance issues...

Cheers,
NC

[lawrence]

Re: how does one trap for out-of-memory errors?

Zurab Davitiani <agt@mindless.c om> wrote in message[color=blue][color=green]
> > I have empirically noticed what you describe. With object, in a for()
> > loop, memory consumption goes up with every pass. How is one to deal
> > with this?[/color]
>
> Again, as I said before, there's no way to "deal" with this in PHP4 - no way
> that I found anyway. Assigning to "", null, using unset(), etc. does not
> free the memory.
>
> I came accross this issue when I tried to run a long database update script
> (from shell) with a lot of data using objects. I would get memory
> allocation error at certain point. The way I resolved it (and still used
> objects) was to launch a separate PHP process to query and update the data
> for each record; so, all object references would get created inside that
> process and die when that process was finished and memory would be freed
> up. Now, this is not an option for most web apps, but it may work on longer
> cron jobs when using objects is a necessity.[/color]

Basically, OOP in PHP is broken. I am a bit stunned by this. If you
can't manually free the memory and the memory does not clear
automatically, then I think it is fair to descibe this feature as
"broken". For most applications, there is nothing the programmer can
do, except not do the application. There is no way for the programmer
to protect themself against this bug, no matter how talented they are,
and no matter how diligent.

I am very disappointed. As someone who has used PHP for most of their
work for the last 3 years, it is sad to learn that PHP has such a
major flaw. I thought it had matured and was now a robust,
sophisticated language for major web-development, but clearly, it is
still showing its roots as a scripting environment for "Personal Home
Pages."

People should be warned away from OOP in PHP. It should be advertised
as a strictly procedural language. Any large database return is likely
to give memory errors in OOP in PHP. People should be warned.

I'm surprised that I have not heard of this before. This should be a
fairly big deal.

[lawrence]

Re: how does one trap for out-of-memory errors?

Alan Little <alan@n-o-s-p-a-m-phorm.com> wrote in message[color=blue]
> Unless I'm misunderstandin g, it seems like you're overlooking an obvious
> solution. You have methods for transparently managing the queries, why
> not have one for transparently retrieving the data from the resource
> pointer? I have something like this in one of my applications, which
> needs to be able to work with MySQL or Sybase. I simply wrote db_
> functions to replace all the mysql_ and sybase_ functions, and made them
> intelligent enough to handle the differences transparently.[/color]

Now it seems that even this won't work. Getting one entry at a time or
getting the whole array at once uses the same memory, because the
memory garbage collection in objects in PHP seems to be broken.

[Nikolai Chuvakhin]

Re: how does one trap for out-of-memory errors?

lkrubner@geocit ies.com (lawrence) wrote in message
news:<da7e68e8. 0311040840.19c8 4f17@posting.go ogle.com>...[color=blue]
>
> Basically, OOP in PHP is broken. I am a bit stunned by this.[/color]

You shouldn't be. You have a language that has no typing to
speak of (Niklaus Wirth has long since given up screaming about
how little sense OOP makes without strong typing), which, unlike
VBscript, runs cross-platform, but, unlike JSP, does not require
a virtual machine. All interfaces to hardware and lower-level
software (such as databases) are implemented procedurally; in
other words, there are no system objects. Why do you expect OOP
to be anything but a pain in the neck in this framework?
[color=blue]
> I thought it had matured and was now a robust, sophisticated
> language for major web-development[/color]

Robust, yes. Sophisticated, no. Think of PHP as the assembly
language of Web development, and it will immediately become clear
to you that sophistication and robustness are conflicting notions.
Robust systems have a small footprint and thus are necessarily
feature-poor. Sophisticated systems are feature-rich and thus
must have a bigger footprint.
[color=blue]
> People should be warned away from OOP in PHP.[/color]

Amen! But then, people should be generally warned away from OOP...

Cheers,
NC

[Zurab Davitiani]

Re: how does one trap for out-of-memory errors?

Nikolai Chuvakhin wrote on Tuesday 04 November 2003 23:39:
[color=blue][color=green]
>> People should be warned away from OOP in PHP.[/color]
>
> Amen! But then, people should be generally warned away from OOP...[/color]

I respectfully disagree. There are times when you do need objects in your
apps. An existing example of this is DOM. Also, a lot of PHP OOP
problems/issues are being taken care of in PHP5, with major enhancements
and additions to OOP and related functionality. PHP is evolving, getting
new features, and fixing existing flaws, and resisting the use of the
available features will sometimes result in not taking the full advantage
of what they have to offer.

This does not mean that your PHP scripts have to look like Java code, but
use of objects in PHP are many times convenient and useful. As far as the
flaw pointed out in this (sub)thread is a PHP issue that will hopefully be
addressed in upcoming releases so that developers can take full advantage
of OO features that they expect from the environment to provide.

--
Business Web Solutions
ActiveLink, LLC

https://www.active-link.com/intranet/