PHP memory garbage collection

[Anze]

Hi!

I have a script that uses a lot of memory and an ISP that has memory-limit
set to 10 Mb. The script of course doesn't finish running. :(

The program is rather simple - it parses XML into an array and then starts
the loop that inserts data into the database. If I use a very large XML
file then the PHP crashes inside the loop (Fatal error: Allowed memory size
of 10485760 bytes exhausted) - indicating that the problem lies within loop
somewhere.

Can I trace memory consumption of a PHP script so I can determine where the
memory is wasted?

If PHP uses all the memory (and frees nothing) until the end of the script
then that could be the problem - is there any way I can force it to do
garbage collection?

Any comment on how PHP uses memory would be appreciated - I would really
hate to have to break the script into smaller pieces (or change ISP for
that matter :) ).

Thanks!

Anze

[Anze]

Re: PHP memory garbage collection


Anybody? Please? I really need to solve this...

Is there a way to measure current memory consumption?

Regards,

Anze

Anze wrote:[color=blue]
> Hi!
>
> I have a script that uses a lot of memory and an ISP that has memory-limit
> set to 10 Mb. The script of course doesn't finish running. :(
>
> The program is rather simple - it parses XML into an array and then starts
> the loop that inserts data into the database. If I use a very large XML
> file then the PHP crashes inside the loop (Fatal error: Allowed memory
> size of 10485760 bytes exhausted) - indicating that the problem lies
> within loop somewhere.
>
> Can I trace memory consumption of a PHP script so I can determine where
> the memory is wasted?
>
> If PHP uses all the memory (and frees nothing) until the end of the script
> then that could be the problem - is there any way I can force it to do
> garbage collection?
>
> Any comment on how PHP uses memory would be appreciated - I would really
> hate to have to break the script into smaller pieces (or change ISP for
> that matter :) ).
>
> Thanks!
>
> Anze[/color]

[Bert Melis]

Re: PHP memory garbage collection

Anze wrote:[color=blue]
> Anybody? Please? I really need to solve this...
>
> Is there a way to measure current memory consumption?
>
> Regards,
>
> Anze
>
> Anze wrote:
>[color=green]
>>Hi!
>>
>>I have a script that uses a lot of memory and an ISP that has memory-limit
>>set to 10 Mb. The script of course doesn't finish running. :(
>>
>>The program is rather simple - it parses XML into an array and then starts
>>the loop that inserts data into the database. If I use a very large XML
>>file then the PHP crashes inside the loop (Fatal error: Allowed memory
>>size of 10485760 bytes exhausted) - indicating that the problem lies
>>within loop somewhere.
>>
>>Can I trace memory consumption of a PHP script so I can determine where
>>the memory is wasted?
>>
>>If PHP uses all the memory (and frees nothing) until the end of the script
>>then that could be the problem - is there any way I can force it to do
>>garbage collection?
>>
>>Any comment on how PHP uses memory would be appreciated - I would really
>>hate to have to break the script into smaller pieces (or change ISP for
>>that matter :) ).
>>
>>Thanks!
>>
>>Anze[/color]
>
>[/color]
I'm already thinking of a solution, my fault...

Look into your script for where you have global variables which you
don't need anymore and then unset them (unset($var)).

If you can't solve it this way, you can contact your ISP to run the
script for you through SSH or something.

I don't think you would run the script often or do you? If you do, I
suggest thinking about the logic and effeciency about your piece of
programming.

[Anze]

Re: PHP memory garbage collection

Hi!
[color=blue]
> Look into your script for where you have global variables which you
> don't need anymore and then unset them (unset($var)).
>
> If you can't solve it this way, you can contact your ISP to run the
> script for you through SSH or something.
>
> I don't think you would run the script often or do you? If you do, I
> suggest thinking about the logic and effeciency about your piece of
> programming.[/color]

First of all, thank you for the answer!

The script will be run only from time to time. The problem is that I already
unset() all the variables that are allocated - at least global ones. All
other are local to functions and should be freed automatically (from what I
understand).

The script is pretty much straightforward - but it does use my CMS as a
backend. That is why I'm trying to see where the memory is being spent. I
am afraid this is only a tip of the iceberg and that other scripts could be
made more effective if I solved this problem.

Just a few minutes ago I found this:

http://www.zend.com/manual/function.memory-get-usage.php

It is not documented on the PHP site... :(

I will see if I can solve a problem by using this function.

Thank you again!

Regards,

Anze

[rush]

Re: PHP memory garbage collection

"Anze" <anzenews@volja .net> wrote in message
news:OXy4e.1109 6$F6.2184647@ne ws.siol.net...[color=blue]
> The script is pretty much straightforward - but it does use my CMS as a
> backend. That is why I'm trying to see where the memory is being spent. I
> am afraid this is only a tip of the iceberg and that other scripts could[/color]
be[color=blue]
> made more effective if I solved this problem.[/color]

for what is worth I _think_ php uses reference counting for it's memory
management. so maybe you could check if you produce some cyclic references
along the way. Also if you are using php less then version 5 then be aware
of default "copy by value" semantics, which could in some cases leave you
with a quite a few more objects then you expect. And lastly if it is a large
xml file mybe it is a xml parser and iteration over the DOM tree that is
taking over your memory.

If everything fails, maybe you could slice your work to smaller peaces, and
have more runs of the script each doing a smaller chunk of work.

As for finding host that would allow you to use more than 10 megs for php
script, I doubt that you can find one. If you do, he would prpobably have
severe problems with stability and availability of their servers ;)

rush
--

http://www.templatetamer.com/

http://www.folderscavenger.com/

[R. Rajesh Jeba Anbiah]

Re: PHP memory garbage collection

Anze wrote:[color=blue]
> Is there a way to measure current memory consumption?[/color]
<snip>

http://in.php.net/apd

--
<?php echo 'Just another PHP saint'; ?>
Email: rrjanbiah-at-Y!com Blog: http://rajeshanbiah.blogspot.com/

[Anze]

Re: PHP memory garbage collection

Hi!
[color=blue]
> for what is worth I _think_ php uses reference counting for it's memory
> management. so maybe you could check if you produce some cyclic references
> along the way. Also if you are using php less then version 5 then be aware[/color]

Yes, I do have them - I just never realized this could be a problem. Thanks
for the tip! But in this script I manually unset() all of the objects with
cyclic references so this shouldn't be a problem, right?
[color=blue]
> If everything fails, maybe you could slice your work to smaller peaces,
> and have more runs of the script each doing a smaller chunk of work.[/color]

I would hate to do that - it is the customer that uses this script to upload
some XML with article data and it would be very weird to ask of him to run
the script more than once. :)
[color=blue]
> As for finding host that would allow you to use more than 10 megs for php
> script, I doubt that you can find one. If you do, he would prpobably have
> severe problems with stability and availability of their servers ;)[/color]

Not really - they could set memory_limit to 10Mb and allow users to change
it via ini_set(). But this provider uses some other memory and time
limiting mechanism (not just memory_limit directive) so there is no way for
me to change it for some scripts.

But there is also an interesting thing I found while debugging. This code
apparently produces memory leaks:

<?php
class A
{
var $as=array();

function A()
{
for ($i=0;$i<9;$i++ )
{
$this->as[]='1';
};
}
};

for ($i=0;$i<10;$i+ +)
{
$a=&new A();
echo "mem: ".memory_get_us age()."<br />\n";
unset($a->as);
unset($a);
};
?>

The output is:
mem: 13984
mem: 14048
mem: 14112
mem: 14176
mem: 14240
mem: 14304
mem: 14368
mem: 14432
mem: 14496
mem: 14560

There is no memory leak if you change this line:
for ($i=0;$i<9;$i++ )
to:
for ($i=0;$i<8;$i++ )

Is this weird or is it just me? ;)
I'll check bug reports to see if someone else already stumbled across this -
it looks like a bug in PHP.

Regards,

Anze

[Anze]

Re: PHP memory garbage collection

> http://in.php.net/apd

Not exactly what I had in mind, but great tip anyway - thanks! It looks
promising. :)

Regards,

Anze

[Anze]

Re: PHP memory garbage collection

> I'll check bug reports to see if someone else already stumbled across this[color=blue]
> - it looks like a bug in PHP.[/color]

Someone else already reported this bug but developers seem to think this is
normal behaviour:

http://bugs.php.net/bug.php?id=31297

I don't agree with the developers, but then again - if it's only 64 bytes
per object it is not that much. Back to hunting other leaks... ;)

Regards,

Anze

[Tony Marston]

Re: PHP memory garbage collection


"Anze" <anzenews@volja .net> wrote in message
news:efw4e.1108 3$F6.2183915@ne ws.siol.net...[color=blue]
> Hi!
>
> I have a script that uses a lot of memory and an ISP that has memory-limit
> set to 10 Mb. The script of course doesn't finish running. :(
>
> The program is rather simple - it parses XML into an array and then starts
> the loop that inserts data into the database. If I use a very large XML
> file then the PHP crashes inside the loop (Fatal error: Allowed memory
> size
> of 10485760 bytes exhausted) - indicating that the problem lies within
> loop
> somewhere.[/color]

There'e your problem right there. You are reading the ENTIRE XML file into
memory before processing it. If you look at the documentation you will find
a method that requires reading the contents of the XML file in little chunks
at a time instead of one big mouthful.

--
Tony Marston

http://www.tonymarston.net

[color=blue]
> Can I trace memory consumption of a PHP script so I can determine where
> the
> memory is wasted?
>
> If PHP uses all the memory (and frees nothing) until the end of the script
> then that could be the problem - is there any way I can force it to do
> garbage collection?
>
> Any comment on how PHP uses memory would be appreciated - I would really
> hate to have to break the script into smaller pieces (or change ISP for
> that matter :) ).
>
> Thanks!
>
> Anze[/color]

[Anze]

Re: PHP memory garbage collection

> There'e your problem right there. You are reading the ENTIRE XML file into[color=blue]
> memory before processing it.[/color]

Not really, but thanks anyway. :)

The XML data is small enough and shouldn't pose a problem. It is the
processing that creates many objects and PHP for some reason does not do
garbage collection properly. It could be because of cyclic references - I
don't know. I'm learning how to use APD to test memory usage more
thoroughly. The XML data memory usage is negligible when compared to total
memory usage.

I am using my CMS objects to insert data in DB and this is the easiest way;
I would like to find the reason why PHP doesn't release unused memory
before I dig any deeper.

Thanks again!

Anze

[Ramius]

Re: PHP memory garbage collection

Anze wrote:[color=blue][color=green]
> > There'e your problem right there. You are reading the ENTIRE XML[/color][/color]
file into[color=blue][color=green]
> > memory before processing it.[/color]
>
> Not really, but thanks anyway. :)
>
> The XML data is small enough and shouldn't pose a problem.[/color]

I agree with Tony, you stated that you only run into this problem, "If
I use a very large XML file". So while the XML parsing may not be the
only problem, it is certainly a significant contributing factor.

If you aren't already using the "XML parser functions"
(http://us2.php.net/xml), then try them. You will find them to be
several times less memory intensive than DOM XML
(http://us2.php.net/domxml).