memory pool?

[peter koch]

Re: memory pool?

On 11 Nov., 11:21, Uli Kunkel <genija...@yaho o.comwrote:

James Kanze wrote:

On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:

Juan Antonio Zaratiegui Vallecillo wrote:
I'm wondering if I could use reserve to allocate a few MB and
then if the size is not enough that the vector grows
automatically?  I read that vector resizing is very costly so
I presume that reserving just 1KB is not enough.

>

What you read was wrong.  Resizing isn't that costly for
primitive object types (like unsigned char).  There are other
reasons you might want to avoid it, but they probably don't
apply here.

>

--
James Kanze (GABI Software)             email:james.ka. ..@gmail.com
Conseils en informatique orientée objet/
                   Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

>
I tried 2 things:
>
1. reserve 3MB and then push_back every unsigned char (or byte) to the
vector.
2. Reserve 1KB on every callback

That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.

>
U didn't measure the time precisely, but it is approximatly the same.
It takes very long to complete, around 10 seconds.
This is to much.I don't know what the problem is.

That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.

/Peter

[Uli Kunkel]

Re: memory pool?

peter koch wrote:

On 11 Nov., 11:21, Uli Kunkel <genija...@yaho o.comwrote:

>James Kanze wrote:

>>On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:
>>>Juan Antonio Zaratiegui Vallecillo wrote:
>>>I'm wondering if I could use reserve to allocate a few MB and
>>>then if the size is not enough that the vector grows
>>>automaticall y? I read that vector resizing is very costly so
>>>I presume that reserving just 1KB is not enough.
>>What you read was wrong. Resizing isn't that costly for
>>primitive object types (like unsigned char). There are other
>>reasons you might want to avoid it, but they probably don't
>>apply here.
>>--
>>James Kanze (GABI Software) email:james.ka. ..@gmail.com
>>Conseils en informatique orientée objet/
>> Beratung in objektorientier ter Datenverarbeitu ng
>>9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

>I tried 2 things:
>>
>1. reserve 3MB and then push_back every unsigned char (or byte) to the
>vector.
>2. Reserve 1KB on every callback

>
That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.

>U didn't measure the time precisely, but it is approximatly the same.
>It takes very long to complete, around 10 seconds.
>This is to much.I don't know what the problem is.

>
That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.
>
/Peter

I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve (3*1024*1024);

//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
myPhoto.push_ba ck(pProgress->pbData[i]);
}
-------------------------------

What do you think?
Maybe I should use insert, instead of putting one char at the time.
I tried this but pbData is unsigned char* type and could't do it...

[Uli Kunkel]

Re: memory pool?

Uli Kunkel wrote:

peter koch wrote:

>On 11 Nov., 11:21, Uli Kunkel <genija...@yaho o.comwrote:

>>James Kanze wrote:
>>>On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:
>>>>Juan Antonio Zaratiegui Vallecillo wrote:
>>>>I'm wondering if I could use reserve to allocate a few MB and
>>>>then if the size is not enough that the vector grows
>>>>automatical ly? I read that vector resizing is very costly so
>>>>I presume that reserving just 1KB is not enough.
>>>What you read was wrong. Resizing isn't that costly for
>>>primitive object types (like unsigned char). There are other
>>>reasons you might want to avoid it, but they probably don't
>>>apply here.
>>>--
>>>James Kanze (GABI Software) email:james.ka. ..@gmail.com
>>>Conseils en informatique orientée objet/
>>> Beratung in objektorientier ter Datenverarbeitu ng
>>>9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
>>I tried 2 things:
>>>
>>1. reserve 3MB and then push_back every unsigned char (or byte) to the
>>vector.
>>2. Reserve 1KB on every callback

>>
>That is the wrong approach. You either reserve once at the beginning
>or you don't reserve at all. Reserving in small chunks will cause your
>algorithm to run much slower than without any reserves at all.

>>U didn't measure the time precisely, but it is approximatly the same.
>>It takes very long to complete, around 10 seconds.
>>This is to much.I don't know what the problem is.

>>
>That is far to much - something else must be wrong. You can isolate
>the vector code by simply appending "random data" to the vector
>without reading it from the source. If that is not a snap, you should
>post the codde here.
>>
>/Peter

>
I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve (3*1024*1024);
>
//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
myPhoto.push_ba ck(pProgress->pbData[i]);
}
-------------------------------
>
What do you think?
Maybe I should use insert, instead of putting one char at the time.
I tried this but pbData is unsigned char* type and could't do it...

I had one big oversight.
In the callback function I can define the length of each chunk.I thought
it is integer type.As I recall that is only 36635.
I check again and saw it is actually unsigned long.That was very stupid
of me.
Now I pass 1MB and it takes 2,3 seconds which is much better.

If I'm still doing something wrong I would appreciate any help.

[peter koch]

Re: memory pool?

On 11 Nov., 12:49, Uli Kunkel <genija...@yaho o.comwrote:

Uli Kunkel wrote:

peter koch wrote:

On 11 Nov., 11:21, Uli Kunkel <genija...@yaho o.comwrote:
>James Kanze wrote:
>>On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:
>>>Juan Antonio Zaratiegui Vallecillo wrote:
>>>I'm wondering if I could use reserve to allocate a few MB and
>>>then if the size is not enough that the vector grows
>>>automaticall y?  I read that vector resizing is very costly so
>>>I presume that reserving just 1KB is not enough.
>>What you read was wrong.  Resizing isn't that costly for
>>primitive object types (like unsigned char).  There are other
>>reasons you might want to avoid it, but they probably don't
>>apply here.
>>--
>>James Kanze (GABI Software)             email:james.ka. ...@gmail.com
>>Conseils en informatique orientée objet/
>>                   Beratung in objektorientier ter Datenverarbeitu ng
>>9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 0034
>I tried 2 things:

>

>1. reserve 3MB and then push_back every unsigned char (or byte) to the
>vector.
>2. Reserve 1KB on every callback

>

That is the wrong approach. You either reserve once at the beginning
or you don't reserve at all. Reserving in small chunks will cause your
algorithm to run much slower than without any reserves at all.
>U didn't measure the time precisely, but it is approximatly the same.
>It takes very long to complete, around 10 seconds.
>This is to much.I don't know what the problem is.

>

That is far to much - something else must be wrong. You can isolate
the vector code by simply appending "random data" to the vector
without reading it from the source. If that is not a snap, you should
post the codde here.

>

/Peter

>

I agree that it is better to allocate a couple of MB than to allocate
1KB so many times.
People on the group said that the latter isn't such a big deal.
Anyway the code is simple:
-----------------------------
//first I allocate 3MB in some initialization
myPhoto.reserve (3*1024*1024);

>

//then I assign one char at a time
for(unsigned long i=0; i < pProgress->lLength; i++)
{
    myPhoto.push_ba ck(pProgress->pbData[i]);

No need to push_back each item individually. Better would be to use
std::vector.ins ert or std::copy with a back_inserter. Look these up.

}
-------------------------------

>

What do you think?
Maybe I should use insert, instead of putting one char at the time.
I tried this but pbData is unsigned char* type and could't do it...

>
I had one big oversight.
In the callback function I can define the length of each chunk.I thought
it is integer type.As I recall that is only 36635.

It is implementation defined, but i doubt max(int) is 36635 - it might
be 32767.

I check again and saw it is actually unsigned long.That was very stupid
of me.
Now I pass 1MB and it takes 2,3 seconds which is much better.
>
If I'm still doing something wrong I would appreciate any help.

See above.

/Peter

[Uli Kunkel]

Re: memory pool?

peter koch wrote:

On 11 Nov., 12:49, Uli Kunkel <genija...@yaho o.comwrote:

>Uli Kunkel wrote:

>>peter koch wrote:
>>>On 11 Nov., 11:21, Uli Kunkel <genija...@yaho o.comwrote:
>>>>James Kanze wrote:
>>>>>On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:
>>>>>>Juan Antonio Zaratiegui Vallecillo wrote:
>>>>>>I'm wondering if I could use reserve to allocate a few MB and
>>>>>>then if the size is not enough that the vector grows
>>>>>>automatic ally? I read that vector resizing is very costly so
>>>>>>I presume that reserving just 1KB is not enough.
>>>>>What you read was wrong. Resizing isn't that costly for
>>>>>primitiv e object types (like unsigned char). There are other
>>>>>reasons you might want to avoid it, but they probably don't
>>>>>apply here.
>>>>>--
>>>>>James Kanze (GABI Software) email:james.ka. ..@gmail.com
>>>>>Conseils en informatique orientée objet/
>>>>> Beratung in objektorientier ter Datenverarbeitu ng
>>>>>9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
>>>>I tried 2 things:
>>>>1. reserve 3MB and then push_back every unsigned char (or byte) to the
>>>>vector.
>>>>2. Reserve 1KB on every callback
>>>That is the wrong approach. You either reserve once at the beginning
>>>or you don't reserve at all. Reserving in small chunks will cause your
>>>algorithm to run much slower than without any reserves at all.
>>>>U didn't measure the time precisely, but it is approximatly the same.
>>>>It takes very long to complete, around 10 seconds.
>>>>This is to much.I don't know what the problem is.
>>>That is far to much - something else must be wrong. You can isolate
>>>the vector code by simply appending "random data" to the vector
>>>without reading it from the source. If that is not a snap, you should
>>>post the codde here.
>>>/Peter
>>I agree that it is better to allocate a couple of MB than to allocate
>>1KB so many times.
>>People on the group said that the latter isn't such a big deal.
>>Anyway the code is simple:
>>-----------------------------
>>//first I allocate 3MB in some initialization
>>myPhoto.reser ve(3*1024*1024) ;
>>//then I assign one char at a time
>>for(unsigne d long i=0; i < pProgress->lLength; i++)
>>{
>> myPhoto.push_ba ck(pProgress->pbData[i]);

>
No need to push_back each item individually. Better would be to use
std::vector.ins ert or std::copy with a back_inserter. Look these up.
>

>>}
>>-------------------------------
>>What do you think?
>>Maybe I should use insert, instead of putting one char at the time.
>>I tried this but pbData is unsigned char* type and could't do it...

>I had one big oversight.
>In the callback function I can define the length of each chunk.I thought
>it is integer type.As I recall that is only 36635.

>
It is implementation defined, but i doubt max(int) is 36635 - it might
be 32767.
>

>I check again and saw it is actually unsigned long.That was very stupid
>of me.
>Now I pass 1MB and it takes 2,3 seconds which is much better.
>>
>If I'm still doing something wrong I would appreciate any help.

>
See above.
>
/Peter

I have a silly problem.
Insert takes const byte&. And variable pProgress->pbData that I pass is
unsigned char*.
Anyway it inserts an empty value, something is not right..

[Richard Herring]

Re: memory pool?

In message <gfc4ti$vfk$1@n ews.metronet.hr >, Uli Kunkel
<genijalac@yaho o.comwrites

>peter koch wrote:

>On 11 Nov., 12:49, Uli Kunkel <genija...@yaho o.comwrote:

>>Uli Kunkel wrote:

[...]

>>>I agree that it is better to allocate a couple of MB than to allocate
>>>1KB so many times.
>>>People on the group said that the latter isn't such a big deal.
>>>Anyway the code is simple:
>>>-----------------------------
>>>//first I allocate 3MB in some initialization
>>>myPhoto.rese rve(3*1024*1024 );
>>>//then I assign one char at a time
>>>for(unsign ed long i=0; i < pProgress->lLength; i++)
>>>{
>>> myPhoto.push_ba ck(pProgress->pbData[i]);

> No need to push_back each item individually. Better would be to use
>std::vector.in sert or std::copy with a back_inserter. Look these up.
>>

>>>}
>>>-------------------------------
>>>What do you think?
>>>Maybe I should use insert, instead of putting one char at the time.
>>>I tried this but pbData is unsigned char* type and could't do it...

[...]

>>If I'm still doing something wrong I would appreciate any help.

> See above.

>
>I have a silly problem.
>Insert takes const byte&. And variable pProgress->pbData that I pass is
>unsigned char*.

Use the two-iterator form of insert:

myPhoto.insert( myPhoto.end(), pProgress->pBData, pProgress->pBData+pProgre ss->lLength);

Pointers are iterators.

>Anyway it inserts an empty value, something is not right..

--
Richard Herring