Large array stored in session variable causes server to choke

[MrBiggles]

Here's the sitch:

I read in a csv file with 60000 lines (20 fields per record), and store
the data to a local array. The file is read in and stored just fine and
pretty quick.
Now, if I try to assign that array to a session variable it chokes.
e.g. create array and load each element with a row from the file (btw,
each row is an array as well, using fgetcsv()). When local array is
loaded, I assign to session var as so:
$_SESSION['mydata'] = localArray;

What happens is the server will sit there and churn for a long time. CPU
goes to 99% use and memory use explodes on the apache thread. It will
keep churning a looong time, so long that I had to stop the server to
make it quit. This happens on my windows server as well as fedoracore4
server.

I've tried these variations all with the same results:
- store reference to array in session var (although since it's an array,
I think it makes no difference). e.g. $_SESSION['mydata'] =& myArray;

- read the file directly into a sesson var, instead of creating a local
array first. e.g. $_SESSION['mydata'][] = $row; (grabbing one row at a
time from file)

Same thing always happens.
Anybody have any insight into storing large arrays as session vars? Is
it bad practice? Can it be done? What's the scoop?

Any help appreciated!
Thanks
B

[Jerry Stuckle]

Re: Large array stored in session variable causes server to choke

MrBiggles wrote:[color=blue]
> Here's the sitch:
>
> I read in a csv file with 60000 lines (20 fields per record), and store
> the data to a local array. The file is read in and stored just fine and
> pretty quick.
> Now, if I try to assign that array to a session variable it chokes.
> e.g. create array and load each element with a row from the file (btw,
> each row is an array as well, using fgetcsv()). When local array is
> loaded, I assign to session var as so:
> $_SESSION['mydata'] = localArray;
>
> What happens is the server will sit there and churn for a long time. CPU
> goes to 99% use and memory use explodes on the apache thread. It will
> keep churning a looong time, so long that I had to stop the server to
> make it quit. This happens on my windows server as well as fedoracore4
> server.
>
> I've tried these variations all with the same results:
> - store reference to array in session var (although since it's an array,
> I think it makes no difference). e.g. $_SESSION['mydata'] =& myArray;
>
> - read the file directly into a sesson var, instead of creating a local
> array first. e.g. $_SESSION['mydata'][] = $row; (grabbing one row at a
> time from file)
>
> Same thing always happens.
> Anybody have any insight into storing large arrays as session vars? Is
> it bad practice? Can it be done? What's the scoop?
>
> Any help appreciated!
> Thanks
> B[/color]

Let's see... 60K records x 20 fields per record is 1.2M fields. That's
a lot of data. Then multiply by the average size of a field and you
have a lot of MB. I sure wouldn't try to store that much in a session!

Then PHP has to serialize the data to write it to the session file. And
unserialize when it reads the data in. A lot of work for that much data.

Doesn't look like a sound design to me... Maybe use a database instead?

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Jasen Betts]

Re: Large array stored in session variable causes server to choke

On 2006-02-11, MrBiggles <mrbiggles909@y ahoo.com> wrote:
[color=blue]
> I read in a csv file with 60000 lines (20 fields per record), and store
> the data to a local array. The file is read in and stored just fine and
> pretty quick.
> Now, if I try to assign that array to a session variable it chokes.
> e.g. create array and load each element with a row from the file (btw,
> each row is an array as well, using fgetcsv()). When local array is
> loaded, I assign to session var as so:
> $_SESSION['mydata'] = localArray;[/color]

Here I'm limited to something like 3.2K much more than that and the session
data is lost.
[color=blue]
> What happens is the server will sit there and churn for a long time. CPU
> goes to 99% use and memory use explodes on the apache thread. It will
> keep churning a looong time, so long that I had to stop the server to
> make it quit. This happens on my windows server as well as fedoracore4
> server.[/color]

maybe you need more ram.

Bye.
Jasen

[Jasen Betts]

RETRACTION - Large array stored in session variable causes server to choke

On 2006-02-12, Jasen Betts <jasen@free.net .nz> wrote:[color=blue]
> On 2006-02-11, MrBiggles <mrbiggles909@y ahoo.com> wrote:
>[color=green]
>> I read in a csv file with 60000 lines (20 fields per record), and store
>> the data to a local array. The file is read in and stored just fine and
>> pretty quick.
>> Now, if I try to assign that array to a session variable it chokes.
>> e.g. create array and load each element with a row from the file (btw,
>> each row is an array as well, using fgetcsv()). When local array is
>> loaded, I assign to session var as so:
>> $_SESSION['mydata'] = localArray;[/color]
>
> Here I'm limited to something like 3.2K much more than that and the session
> data is lost.[/color]

sorry, no that was a mozilla bug not displaying lines with that many
non-space characters... I've tested sessions upto 2M here - that's about the
limit for this old hardware...


hmm...

I hear (and see in my config file) that PHP gets (by default) 8M of ram
to play with.

60K x 20 is 1200K fields
there's only room for an about 8 bytes per field, four of which are
probably going to be a pointer of some sort there's probably another four
needed for memory allocation control, or a reference counte etc... looks
like you're doomed before you even start.

I did some testing:

$a=array();
for($c=200;$c<6 0000;$c+=200)
{
for($b=$c-200;$b<$c;$b++)
{
$a[$b]=array('1','2', '3','4','5','6' ,'7','8','9','0 ',
'a','b','c','d' ,'e','f','g','h ','i','j');
print('.');
}
print '<br>' .$c; flush();
};


I get about 5500 rows by 20 elements by 1 character each before I exceed
that limit

if you bump the limit up to say 400M you might have some success
(dependant on record sizes).

that means your server wants around 40G of ram to be able to handle 100
simultaneous requessts... (is 100 a reasonable figure??)

ISTM it might be time to re-evaluate the task and either do without the huge
array, or store and process it using some other language (like a databaee
using SQL or a custom app using C, or a combination oof the two)

Bye.
Jasen

[Jim Michaels]

Re: RETRACTION - Large array stored in session variable causes server to choke


"Jasen Betts" <jasen@free.net .nz> wrote in message
news:3f72.43ef0 7a3.18168@clunk er.homenet...[color=blue]
> On 2006-02-12, Jasen Betts <jasen@free.net .nz> wrote:[color=green]
>> On 2006-02-11, MrBiggles <mrbiggles909@y ahoo.com> wrote:
>>[color=darkred]
>>> I read in a csv file with 60000 lines (20 fields per record), and store
>>> the data to a local array. The file is read in and stored just fine and
>>> pretty quick.
>>> Now, if I try to assign that array to a session variable it chokes.
>>> e.g. create array and load each element with a row from the file (btw,
>>> each row is an array as well, using fgetcsv()). When local array is
>>> loaded, I assign to session var as so:
>>> $_SESSION['mydata'] = localArray;[/color]
>>
>> Here I'm limited to something like 3.2K much more than that and the
>> session
>> data is lost.[/color]
>
> sorry, no that was a mozilla bug not displaying lines with that many
> non-space characters... I've tested sessions upto 2M here - that's about
> the
> limit for this old hardware...
>
>
> hmm...
>
> I hear (and see in my config file) that PHP gets (by default) 8M of ram
> to play with.
>
> 60K x 20 is 1200K fields
> there's only room for an about 8 bytes per field, four of which are
> probably going to be a pointer of some sort there's probably another four
> needed for memory allocation control, or a reference counte etc... looks
> like you're doomed before you even start.[/color]

struct {
union {
long lval;
double dval;
struct {
char *val;
int len;
} str;
HashTable *ht;
zend_object_val ue obj;
} value;
zend_uint refcount;
zend_uchar type;
zend_uchar is_ref;
} zval;
so the sizeof(double)+ sizeof(unsigned int)+sizeof(uns igned
char)+sizeof(un signed char)
I forgot my C!
I think it's 8+4+1+1.14 bytes to start with, + the sizeof(the HashTable)
when that's all done.
not much detail there. the details about the internal structure of a
hashtable weren't given. sorry. But I suspect this struct is only necessary
once, not repeated every array element, due to the refcount. a refcount is
only needed by the parser/interpreter for certain purposes (like language
"references " & mainly keeping track of pointer counts under the hood).
[color=blue]
>
> I did some testing:
>
> $a=array();
> for($c=200;$c<6 0000;$c+=200)
> {
> for($b=$c-200;$b<$c;$b++)
> {
> $a[$b]=array('1','2', '3','4','5','6' ,'7','8','9','0 ',
> 'a','b','c','d' ,'e','f','g','h ','i','j');
> print('.');
> }
> print '<br>' .$c; flush();
> };
>
>
> I get about 5500 rows by 20 elements by 1 character each before I exceed
> that limit
>
> if you bump the limit up to say 400M you might have some success
> (dependant on record sizes).
>
> that means your server wants around 40G of ram to be able to handle 100
> simultaneous requessts... (is 100 a reasonable figure??)
>
> ISTM it might be time to re-evaluate the task and either do without the
> huge
> array, or store and process it using some other language (like a databaee
> using SQL or a custom app using C, or a combination oof the two)
>
> Bye.
> Jasen[/color]