hitting the limits

[windandwaves]

Hi Folk

I am managing a site, www.friars.co.nz that seems to be hitting the limits
and I keep getting the 500 error. According to the people from webfarm it
is because my script are too demanding or not closed properly.

I dont believe you have to "close" PHP scripts or even database connections.

The site also accesses a 80 Megabyte database.

Do you know of a way I can find out where the problems are or improving the
performance of the site?

I uses ob_start and ob_end_flush, would that cause problems?

TIA
[color=blue]
> Nicolaas[/color]

[Jerry Stuckle]

Re: hitting the limits

windandwaves wrote:[color=blue]
> Hi Folk
>
> I am managing a site, www.friars.co.nz that seems to be hitting the limits
> and I keep getting the 500 error. According to the people from webfarm it
> is because my script are too demanding or not closed properly.
>
> I dont believe you have to "close" PHP scripts or even database connections.
>
> The site also accesses a 80 Megabyte database.
>
> Do you know of a way I can find out where the problems are or improving the
> performance of the site?
>
> I uses ob_start and ob_end_flush, would that cause problems?
>
> TIA
>
>[color=green]
>>Nicolaas[/color]
>
>
>[/color]

Well, ob_start and ob_end_flush require system resources; the bigger the page
the more resources it requires.

You might not be using many resources with it. You may be using a lot. It all
depends on the site and what you're using it for.


--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Rik]

Re: hitting the limits

windandwaves wrote:[color=blue]
> I am managing a site, www.friars.co.nz that seems to be hitting the
> limits and I keep getting the 500 error. According to the people
> from webfarm it is because my script are too demanding[/color]

I'm no server admin, but maybe you can ask them what the most demanding
requests are?
[color=blue]
> or not closed
> properly.
>
> I dont believe you have to "close" PHP scripts[/color]

Not to my knowledge, no.
[color=blue]
> or even database
> connections.[/color]

Normally no.
If your resources are stretched, it may be worth it to check wether your
script opens once (and only once) a connection to the database per request,
extracts all needed data, and then closes the connection, after which is
continues further processing. If your website is that popular, maybe there
is something to gain from using a persistent database connection, that
highly depends on your used scripts and connections.

I might be talking out of my ass here, I've unfortunately never had the
problem of being so popular :-).
[color=blue]
> The site also accesses a 80 Megabyte database.[/color]

Phah! That's huge compared to what I'm used to.
I assume you've normalized the database?
Created proper indexes for faster selecting?
[color=blue]
> Do you know of a way I can find out where the problems are or
> improving the performance of the site?[/color]

To check how much time (and probably resource) your script takes, loop over
it a couple of times on a local server (somewhere in the 100 or more), and
on specific places in your code add the following:

At the start:
$start = microtime(true) ;

At key locations in the script you want to know the time taken.
$end = microtime(true) ;
$time_taken[$some_specific_ name] += $end-$start;
$start = $end;

print_r($time_t aken) after a 100 or so loops will make it clear to you which
portions of your code take the most time, and check those specific portions
wether they can be done more effective.

If PHP < 5 use www.php.net's suggestion: create a function:
function microtime_float ()
{
list($usec, $sec) = explode(" ", microtime());
return ((float)$usec + (float)$sec);
}
and replace microtime(true) with microtime_float ().
[color=blue]
> I uses ob_start and ob_end_flush, would that cause problems?[/color]

It increases the use of resources. When stretched it might be a problem,
normally it won't. I wouldn't think it's main cause of the problem, but it
will add to it. Is there a specific reason you NEED ob functions? If not:
don't use them.


To decrease server-load you could think about a cache system: review the
dependancies of databasefields for each of your pages, create appropriate
timestamps with an index in the database if they don't exist already, and
cache the pages locally as html with a date/time. On a request, check wether
there is a timestamp in the database higher than of your cached html If not,
serve the cached file, if so, create new cachefile and serve that. Creating
of cachefiles should offcourse take place automatically on an update of a
certain database field, but that might be even more work.

I hope I've been of some help,
--
Rik Wasmus

[Shaun]

Re: hitting the limits

On Sat, 29 Apr 2006 13:40:42 +1200, "windandwav es"
<winandwaves@co ldmail.com> wrote:
[color=blue]
>Hi Folk
>
>I am managing a site, www.friars.co.nz that seems to be hitting the limits
>and I keep getting the 500 error. According to the people from webfarm it
>is because my script are too demanding or not closed properly.
>
>I dont believe you have to "close" PHP scripts or even database connections.[/color]

Ideally you shouldn't have to do either. PHP scripts automatically
close all resources upon termination, unless you've explicitly asked
them to do otherwise.

I've rarely encountered a 500 error using PHP, typically it crops up
with Perl scripts. In most situations, PHP will throw its own errors,
which are descriptive enough to trace the source of the problem.
(Granted, I just posted a few moments ago about an exception to this
case...)
[color=blue]
>The site also accesses a 80 Megabyte database.[/color]

Database as in...? MySQL? How much traffic are you getting? Are you
using persistent database connections, or normal ones?
[color=blue]
>Do you know of a way I can find out where the problems are or improving the
>performance of the site?[/color]

I'm not familiar with webfarm; with any luck, they give you a PHP
error log, or they send PHP's errors into Apache's error log. Whatever
error logs are available to you, check them religiously.
[color=blue]
>I uses ob_start and ob_end_flush, would that cause problems?[/color]

Jerry did that one justice in his response.

hth

-
Remove mypants to email.
<http://www.shaunc.com/>

[windandwaves]

Re: hitting the limits

Jerry Stuckle wrote:[color=blue]
> windandwaves wrote:[color=green]
>> Hi Folk
>>
>> I am managing a site, www.friars.co.nz that seems to be hitting the
>> limits and I keep getting the 500 error. According to the people
>> from webfarm it is because my script are too demanding or not closed
>> properly. I dont believe you have to "close" PHP scripts or even database
>> connections. The site also accesses a 80 Megabyte database.
>>
>> Do you know of a way I can find out where the problems are or
>> improving the performance of the site?
>>
>> I uses ob_start and ob_end_flush, would that cause problems?
>>
>> TIA
>>
>>[color=darkred]
>>> Nicolaas[/color]
>>
>>
>>[/color]
>
> Well, ob_start and ob_end_flush require system resources; the bigger
> the page the more resources it requires.
>
> You might not be using many resources with it. You may be using a
> lot. It all depends on the site and what you're using it for.[/color]

Hmm, yes, the good thin with ob_end is that I can resize the actual HMTL to
be very small, meaning that what we send is small - at least.

[windandwaves]

Re: hitting the limits

Rik wrote:
....[color=blue]
> To decrease server-load you could think about a cache system: review
> the dependancies of databasefields for each of your pages, create
> appropriate timestamps with an index in the database if they don't
> exist already, and cache the pages locally as html with a date/time.
> On a request, check wether there is a timestamp in the database
> higher than of your cached html If not, serve the cached file, if so,
> create new cachefile and serve that. Creating of cachefiles should
> offcourse take place automatically on an update of a certain database
> field, but that might be even more work.[/color]


Rik, that is a cool idea, would you just place the html in a database or is
there a special chache where pages are kept?

[windandwaves]

Re: hitting the limits

Shaun wrote:[color=blue]
> On Sat, 29 Apr 2006 13:40:42 +1200, "windandwav es"
> <winandwaves@co ldmail.com> wrote:
>[color=green]
>> Hi Folk
>>
>> I am managing a site, www.friars.co.nz that seems to be hitting the
>> limits and I keep getting the 500 error. According to the people
>> from webfarm it is because my script are too demanding or not closed
>> properly.[/color][/color]


Hmmm, yes, we have only about 10,000 unique visitors per month and a
relatively straight forward MySql database. However, we have shared
hosting, I think that is where the problem lies....

Thanks for your reply
[color=blue]
> Nicolaas[/color]

[Rik]

Re: hitting the limits

windandwaves wrote:[color=blue]
> Rik wrote:
> ...[color=green]
>> To decrease server-load you could think about a cache system: review
>> the dependancies of databasefields for each of your pages, create
>> appropriate timestamps with an index in the database if they don't
>> exist already, and cache the pages locally as html with a date/time.
>> On a request, check wether there is a timestamp in the database
>> higher than of your cached html If not, serve the cached file, if so,
>> create new cachefile and serve that. Creating of cachefiles should
>> offcourse take place automatically on an update of a certain database
>> field, but that might be even more work.[/color]
>
>
> Rik, that is a cool idea, would you just place the html in a database
> or is there a special chache where pages are kept?[/color]

An HTML page can perfectly be kept in the database, makes it even easier to
check wether a page has to be "rewritten" or not, possibly in 1 simple query
returning 0 or 1.

And if some items in a webpage aren't cachable, you could always try to
cache certain portions of code.

Grtz,
--
Rik Wasmus

[Jerry Stuckle]

Re: hitting the limits

windandwaves wrote:[color=blue]
> Jerry Stuckle wrote:
>[color=green]
>>windandwave s wrote:
>>[color=darkred]
>>>Hi Folk
>>>
>>>I am managing a site, www.friars.co.nz that seems to be hitting the
>>>limits and I keep getting the 500 error. According to the people
>>>from webfarm it is because my script are too demanding or not closed
>>>properly. I dont believe you have to "close" PHP scripts or even database
>>>connection s. The site also accesses a 80 Megabyte database.
>>>
>>>Do you know of a way I can find out where the problems are or
>>>improving the performance of the site?
>>>
>>>I uses ob_start and ob_end_flush, would that cause problems?
>>>
>>>TIA
>>>
>>>
>>>
>>>>Nicolaas
>>>
>>>
>>>[/color]
>>Well, ob_start and ob_end_flush require system resources; the bigger
>>the page the more resources it requires.
>>
>>You might not be using many resources with it. You may be using a
>>lot. It all depends on the site and what you're using it for.[/color]
>
>
> Hmm, yes, the good thin with ob_end is that I can resize the actual HMTL to
> be very small, meaning that what we send is small - at least.
>
>[/color]

Why even us it? I find very seldom do I need it. And it does add overhead.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Jerry Stuckle]

Re: hitting the limits

windandwaves wrote:[color=blue]
> Shaun wrote:
>[color=green]
>>On Sat, 29 Apr 2006 13:40:42 +1200, "windandwav es"
>><winandwaves@ coldmail.com> wrote:
>>
>>[color=darkred]
>>>Hi Folk
>>>
>>>I am managing a site, www.friars.co.nz that seems to be hitting the
>>>limits and I keep getting the 500 error. According to the people
>>>from webfarm it is because my script are too demanding or not closed
>>>properly.[/color][/color]
>
>
>
> Hmmm, yes, we have only about 10,000 unique visitors per month and a
> relatively straight forward MySql database. However, we have shared
> hosting, I think that is where the problem lies....
>
> Thanks for your reply
>
>[color=green]
>>Nicolaas[/color]
>
>
>[/color]

10K visitors a month and an 80Mb database are nothing. Any shared host should
be able to handle that, also.

If you think it's a problem, talk to your hosting company. Maybe the server
you're on is overloaded and they will move you to a less heavily loaded server.

I'm not familiar with webfarm - but they may be right, also. For instance, if
you don't close your mysql connection, it will be closed eventually. But the
connection will hang around until the garbage collector gets around to cleaning
things up.

And look at how efficient your code is. I've seen some pretty good code - but
I've seen some very inefficient code out there, also. And the inefficient code
will require a lot more resources.


--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[windandwaves]

Re: hitting the limits

Jerry Stuckle wrote:
.....[color=blue]
> Why even us it? I find very seldom do I need it. And it does add
> overhead.[/color]

I use it on all my sites because in that way, I can perfectly indent my code
but take out the 100s or even 1000s of spaces/tabs.

For all my sites, I make a login and when the person is logged in (i.e. me
the administrator), the page is produced with spaces so that I can analyse
the code.

I like writing xhtml strict with real simple xhtml that is compressed (i.e.
without spaces) so that the actual pages are super small. I do this to
protest against the dreamweavers out there, who create html that is often
over 100kb and just pollutes the superhighway with endless &nbsp; <td><img
src="spacer.gif ">etc......

HTMS
[color=blue]
> Nicolaas[/color]

[windandwaves]

Re: hitting the limits

Jerry Stuckle wrote:[color=blue]
> windandwaves wrote:[color=green]
>> Shaun wrote:
>>[color=darkred]
>>> On Sat, 29 Apr 2006 13:40:42 +1200, "windandwav es"
>>> <winandwaves@co ldmail.com> wrote:
>>>
>>>
>>>> Hi Folk
>>>>
>>>> I am managing a site, www.friars.co.nz that seems to be hitting the
>>>> limits and I keep getting the 500 error. According to the people
>>>> from webfarm it is because my script are too demanding or not
>>>> closed properly.[/color]
>>
>>
>>
>> Hmmm, yes, we have only about 10,000 unique visitors per month and a
>> relatively straight forward MySql database. However, we have shared
>> hosting, I think that is where the problem lies....
>>
>> Thanks for your reply
>>
>>[color=darkred]
>>> Nicolaas[/color]
>>
>>
>>[/color]
>
> 10K visitors a month and an 80Mb database are nothing. Any shared
> host should be able to handle that, also.
>
> If you think it's a problem, talk to your hosting company. Maybe the
> server you're on is overloaded and they will move you to a less
> heavily loaded server.
> I'm not familiar with webfarm - but they may be right, also. For
> instance, if you don't close your mysql connection, it will be closed
> eventually. But the connection will hang around until the garbage
> collector gets around to cleaning things up.[/color]

I will do a close database connection at the end of each script and see if
it makes a difference.
[color=blue]
> And look at how efficient your code is. I've seen some pretty good
> code - but I've seen some very inefficient code out there, also. And
> the inefficient code will require a lot more resources.[/color]

It is so hard for me to judge if the code is efficient. Really - who knows.
I know what efficient code is, but there are just so many variables. Even
making changes tot the php.ini could be huge.

Each page loads about 100Kb of libaries and functions... (probably 50 - 100
functions (small ones) in total)... Does that make a difference?
I have a couple of tables in the database with over 300,000 (small) rows)...
Does that make a difference?
I use a lot of session variables (i.e. I keep a log of each page visited for
each person), but all of them are stored in highly optimised Mysql tables
(e.g.

person ID, page ID, time
person ID, page ID, time
person ID, page ID, time
person ID, page ID, time

In the end, because it is such muddy water, it is probably cheaper to pay
for a dedicated server than to spend hours optimising your code. I can do
fast development and write code in such a way that it is easy to maintain.

Thanks for your reply. Much appreciated.
[color=blue]
> Nicolaas[/color]

[Jerry Stuckle]

Re: hitting the limits

windandwaves wrote:[color=blue]
> Jerry Stuckle wrote:
> ....
>[color=green]
>>Why even us it? I find very seldom do I need it. And it does add
>>overhead.[/color]
>
>
> I use it on all my sites because in that way, I can perfectly indent my code
> but take out the 100s or even 1000s of spaces/tabs.
>
> For all my sites, I make a login and when the person is logged in (i.e. me
> the administrator), the page is produced with spaces so that I can analyse
> the code.
>
> I like writing xhtml strict with real simple xhtml that is compressed (i.e.
> without spaces) so that the actual pages are super small. I do this to
> protest against the dreamweavers out there, who create html that is often
> over 100kb and just pollutes the superhighway with endless &nbsp; <td><img
> src="spacer.gif ">etc......
>
> HTMS
>
>[color=green]
>>Nicolaas[/color]
>
>
>[/color]

You can write perfectly good xml or html without obstart(). The two have
nothing to do with each other.

But compressing a page then expanding it just to make the code more readable
doesn't make sense. Just expand it in the file and serve it statically.
There's much less overhead.

I think the only time I've really needed obstart() is when I wanted to wrap
phpinfo() in another page. In that case I needed to get the output, parse it,
getting rid of the extra tags, then print it.


--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Rik]

Re: hitting the limits

windandwaves wrote:[color=blue]
> It is so hard for me to judge if the code is efficient. Really - who
> knows. I know what efficient code is, but there are just so many
> variables. Even making changes tot the php.ini could be huge.[/color]

Yup, but previous suggestion with microtime() would give you an indication
where the most time is taken.
[color=blue]
> Each page loads about 100Kb of libaries and functions... (probably 50
> - 100 functions (small ones) in total)... Does that make a difference?[/color]

Yes, probably not a lot, but why?
I'm having great difficulties imagining a a website where that much
libraries and functions are needed. Maybe it's time to include only files
that are actually needed?

It may be worth it to check the following url:

http://nl2.php.net/manual/en/language.oop5.autoload.php

[color=blue]
> I have a couple of tables in the database with over 300,000 (small)
> rows)... Does that make a difference?[/color]

Having a database doesn't make a differnce.
Exactly HOW you query the database does.

Normalize and create proper indexes.-

Grtz,
--
Rik Wasmus

[Jerry Stuckle]

Re: hitting the limits

windandwaves wrote:[color=blue]
> Jerry Stuckle wrote:
>[color=green]
>>windandwave s wrote:
>>[color=darkred]
>>>Shaun wrote:
>>>
>>>
>>>>On Sat, 29 Apr 2006 13:40:42 +1200, "windandwav es"
>>>><winandwave s@coldmail.com> wrote:
>>>>
>>>>
>>>>
>>>>>Hi Folk
>>>>>
>>>>>I am managing a site, www.friars.co.nz that seems to be hitting the
>>>>>limits and I keep getting the 500 error. According to the people
>>>>>from webfarm it is because my script are too demanding or not
>>>>>closed properly.
>>>
>>>
>>>
>>>Hmmm, yes, we have only about 10,000 unique visitors per month and a
>>>relatively straight forward MySql database. However, we have shared
>>>hosting, I think that is where the problem lies....
>>>
>>>Thanks for your reply
>>>
>>>
>>>
>>>>Nicolaas
>>>
>>>
>>>[/color]
>>10K visitors a month and an 80Mb database are nothing. Any shared
>>host should be able to handle that, also.
>>
>>If you think it's a problem, talk to your hosting company. Maybe the
>>server you're on is overloaded and they will move you to a less
>>heavily loaded server.
>>I'm not familiar with webfarm - but they may be right, also. For
>>instance, if you don't close your mysql connection, it will be closed
>>eventually. But the connection will hang around until the garbage
>>collector gets around to cleaning things up.[/color]
>
>
> I will do a close database connection at the end of each script and see if
> it makes a difference.
>
>[color=green]
>>And look at how efficient your code is. I've seen some pretty good
>>code - but I've seen some very inefficient code out there, also. And
>>the inefficient code will require a lot more resources.[/color]
>
>
> It is so hard for me to judge if the code is efficient. Really - who knows.
> I know what efficient code is, but there are just so many variables. Even
> making changes tot the php.ini could be huge.
>
> Each page loads about 100Kb of libaries and functions... (probably 50 - 100
> functions (small ones) in total)... Does that make a difference?
> I have a couple of tables in the database with over 300,000 (small) rows)...
> Does that make a difference?
> I use a lot of session variables (i.e. I keep a log of each page visited for
> each person), but all of them are stored in highly optimised Mysql tables
> (e.g.
>
> person ID, page ID, time
> person ID, page ID, time
> person ID, page ID, time
> person ID, page ID, time
>
> In the end, because it is such muddy water, it is probably cheaper to pay
> for a dedicated server than to spend hours optimising your code. I can do
> fast development and write code in such a way that it is easy to maintain.
>
> Thanks for your reply. Much appreciated.
>
>[color=green]
>>Nicolaas[/color]
>
>
>
>[/color]


Well, if every page is loading unnecessary functions, then yes, that will add
overhead. PHP has to parse all that code. I tend to break my functions in to
groups, and only load those which are necessary for the page.

The database itself isn't a problem. However, your access to the database could
be inefficient. For instance, you may be doing full table scans when the
appropriate index could allow an index scan. But that is all very dependent on
your the RDB you're using, database layout, the actual operations you're
performing on it and about a dozen other variables.

I wouldn't think session variables would have a huge effect on it, but it's also
possible. However, I would also look at what you're doing. Can you get
basically the same information from your Apache log?

Yes, it will take time to make it more efficient. But I think it would be worth
it. And a dedicated server has its own additional overhead for maintenance.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===