memory pool?

[Uli Kunkel]

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that returns
1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Anyway thanks in advance for any suggestions.

[Pascal J. Bourguignon]

Re: memory pool?

Uli Kunkel <genijalac@yaho o.comwrites:

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that
returns 1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

Not on comp.lang.c++. For CString, you should ask on comp.lang.c.

Anyway thanks in advance for any suggestions.

typedef unsigned char byte;
const int KByte=1024;
const int MByte=1024*KByt e;
const int few=6;
std::vector<byt eimage(few*MByt e,0);

Then you can receive copy small buffers into it:


std::vector<byt e>::iterator pos=image.begin ();
std::vector<byt ebuffer(1*KByte );

while(receive(& buffer)){
pos=std::copy(b uffer.begin(),b uffer.end(),pos );
}


--
__Pascal Bourguignon__

[Juan Antonio Zaratiegui Vallecillo]

Re: memory pool?

Uli Kunkel escribió:

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that returns
1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?
>
Anyway thanks in advance for any suggestions.

Use a vector, taking advantage from its reserve() function.

#include <vector>

(...)

std::vector<uns igned charmyPhoto;
myPhoto.reserve (1024*1024);

for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1 024) >
}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]


Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise

best regards,

Zara

[Hendrik Schober]

Re: memory pool?

Juan Antonio Zaratiegui Vallecillo wrote:

Uli Kunkel escribió:

>I have a situation where I'm getting a picture from a camera in small
>chunks.
>So the photo is a few MBytes and I have a callback function that returns
>1KB until it finishes transfering the picture.
>Any suggestions or examples how should I put a picture in one variable?
>I heard something about using memory pools for this but I only need
>something simple for this occasion.
>Could I use CString?
>>
>Anyway thanks in advance for any suggestions.

>
>
Use a vector, taking advantage from its reserve() function.
>
#include <vector>
>
(...)
>
std::vector<uns igned charmyPhoto;
myPhoto.reserve (1024*1024);
>
for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1 024) >

This should read "append data to 'myPhoto'". Remember, reserved
memory is just that: reserved memory. Specifically, it's not
objects you could overwrite.

[...]
Zara

Schobi

[Uli Kunkel]

Re: memory pool?

Juan Antonio Zaratiegui Vallecillo wrote:

Uli Kunkel escribió:

>I have a situation where I'm getting a picture from a camera in small
>chunks.
>So the photo is a few MBytes and I have a callback function that
>returns 1KB until it finishes transfering the picture.
>Any suggestions or examples how should I put a picture in one variable?
>I heard something about using memory pools for this but I only need
>something simple for this occasion.
>Could I use CString?
>>
>Anyway thanks in advance for any suggestions.

>
>
Use a vector, taking advantage from its reserve() function.
>
#include <vector>
>
(...)
>
std::vector<uns igned charmyPhoto;
myPhoto.reserve (1024*1024);
>
for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1 024) >
}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]
>
>
Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise
>
best regards,
>
Zara

Thanks for the answers.
I'm wondering if I could use reserve to allocate a few MB and then if
the size is not enough that the vector grows automatically?
I read that vector resizing is very costly so I presume that reserving
just 1KB is not enough.

[peter koch]

Re: memory pool?

On 10 Nov., 12:49, Uli Kunkel <genija...@yaho o.comwrote:

Juan Antonio Zaratiegui Vallecillo wrote:
>
>
>
>
>

Uli Kunkel escribió:

I have a situation where I'm getting a picture from a camera in small
chunks.
So the photo is a few MBytes and I have a callback function that
returns 1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one variable?
I heard something about using memory pools for this but I only need
something simple for this occasion.
Could I use CString?

>

Anyway thanks in advance for any suggestions.

>

Use a vector, taking advantage from its reserve() function.

>

#include <vector>

>

(...)

>

    std::vector<uns igned charmyPhoto;
    myPhoto.reserve (1024*1024);

>

    for (size_t part=0; part !=1024;++part) {
       <read image data into &myPhoto(part*1 024) >
    }
(...)
    <you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]

>

Just customize such piece for your exact needs, substsituting all magic
constants with nice real constants.
This code skeleton takes as granted that char size is one byte, you may
need to correct the code it it is otherwise

>

best regards,

>

Zara

>
Thanks for the answers.
I'm wondering if I could use reserve to allocate a few MB and then if
the size is not enough that the vector grows automatically?
I read that vector resizing is very costly so I presume that reserving
just 1KB is not enough.

std::vector resizes autmatically. As an alternative, you could also
use std::deque which memorywise is more optimal with regard to
reallocation but overall is slightly slower (does more allocations)
than a std::vector.

/Peter

[Hendrik Schober]

Re: memory pool?

Uli Kunkel wrote:

[...]
I'm wondering if I could use reserve to allocate a few MB and then if
the size is not enough that the vector grows automatically?
I read that vector resizing is very costly so I presume that reserving
just 1KB is not enough.

There's a difference between a vector's size (member functions
'size()'/'resize()') and its capacity ('capacity()'/'reserve()').
The former is related to the number of actual objects in the
vector, the latter to the number of objects the vector is able
to store without having to reallocate.
When you tell a vector to reserve memory for an amount of
elements, it allocates the memory needed (or more), but does
not create the objects. When you tell it to resize to a
specific amount of objects, it will actually contain this
amount of constructed objects (while its capacity might be
higher). The capacity concept allows to write code adding
objects to a vector in the most obvious way (keep appending)
and, if performance or memory fragmentation etc. are issues,
reserve before-hand to reduce or eliminate reallocations. It's
only there for performance-reasons and (besides performance)
has no influence on your algorithm.

So when you know how many objects you will need before-hand,
reserve that amount and then keep appending incoming objects.
If not, strategies depend on your specific problem.

Schobi

[Uli Kunkel]

Re: memory pool?

Pascal J. Bourguignon wrote:

Uli Kunkel <genijalac@yaho o.comwrites:
>

>I have a situation where I'm getting a picture from a camera in small
>chunks.
>So the photo is a few MBytes and I have a callback function that
>returns 1KB until it finishes transfering the picture.
>Any suggestions or examples how should I put a picture in one variable?
>I heard something about using memory pools for this but I only need
>something simple for this occasion.
>Could I use CString?

>
Not on comp.lang.c++. For CString, you should ask on comp.lang.c.
>
>

>Anyway thanks in advance for any suggestions.

>
>
typedef unsigned char byte;
const int KByte=1024;
const int MByte=1024*KByt e;
const int few=6;
std::vector<byt eimage(few*MByt e,0);
>
Then you can receive copy small buffers into it:
>
>
std::vector<byt e>::iterator pos=image.begin ();
std::vector<byt ebuffer(1*KByte );
>
while(receive(& buffer)){
pos=std::copy(b uffer.begin(),b uffer.end(),pos );
}
>
>

I wrote some code and I have a problem that I can't open the picture.
Here is the simplified code:
----------------------------
//This part of the code pushes back every character to myPhoto vector

for(unsigned long i=0; i < Length; i++)
{
myPhoto.push_ba ck(srcData[i]);
}
if(complete) //Write myPhoto vector to a file
{
ofstream outf("C:\\test_ 0.jpg");
for ( pos=myPhoto.beg in()
; pos!=myPhoto.en d()
; ++pos)
{
outf << *pos;
}
outf.close();
}

//This part of the code append every chunk to the file
//It's for testing only
ofstream myFile ("C:\\test_1.jp g", ios::out | ios::binary | ios::app);
myFile.write ((char *)pProgress->pbData, pProgress->lLength);
myFile.close();
-----------------------------
What happens is that the first photo has 8 or 9 KB more than the second
one and it can't be opened.
The second picture is ok but I don't want to do it this way.
Also I opened them and they seem the samo at the begining but the first
one is larger..

I tried 2 different ways to writing them but with the same effect.

Any suggestions?

[Rolf Magnus]

Re: memory pool?

Uli Kunkel wrote:

I wrote some code and I have a problem that I can't open the picture.
Here is the simplified code:
----------------------------
//This part of the code pushes back every character to myPhoto vector
>
for(unsigned long i=0; i < Length; i++)
{
myPhoto.push_ba ck(srcData[i]);
}
if(complete) //Write myPhoto vector to a file
{
ofstream outf("C:\\test_ 0.jpg");
for ( pos=myPhoto.beg in()
; pos!=myPhoto.en d()
; ++pos)
{
outf << *pos;
}
outf.close();
}
>
//This part of the code append every chunk to the file
//It's for testing only
ofstream myFile ("C:\\test_1.jp g", ios::out | ios::binary | ios::app);
myFile.write ((char *)pProgress->pbData, pProgress->lLength);
myFile.close();
-----------------------------
What happens is that the first photo has 8 or 9 KB more than the second
one and it can't be opened.

The first file is opened in text mode, while the second is opened in binary
mode. Depending on the platform you're using, that means that the first file
might get altered in some way while being written.

[Uli Kunkel]

Re: memory pool?

Rolf Magnus wrote:

Uli Kunkel wrote:
>

>I wrote some code and I have a problem that I can't open the picture.
>Here is the simplified code:
>----------------------------
>//This part of the code pushes back every character to myPhoto vector
>>
>for(unsigned long i=0; i < Length; i++)
>{
>myPhoto.push_b ack(srcData[i]);
>}
>if(complete) //Write myPhoto vector to a file
>{
>ofstream outf("C:\\test_ 0.jpg");
>for ( pos=myPhoto.beg in()
>; pos!=myPhoto.en d()
>; ++pos)
>{
>outf << *pos;
>}
>outf.close() ;
> }
>>
>//This part of the code append every chunk to the file
>//It's for testing only
>ofstream myFile ("C:\\test_1.jp g", ios::out | ios::binary | ios::app);
>myFile.write ((char *)pProgress->pbData, pProgress->lLength);
>myFile.close() ;
>-----------------------------
>What happens is that the first photo has 8 or 9 KB more than the second
>one and it can't be opened.

>
The first file is opened in text mode, while the second is opened in binary
mode. Depending on the platform you're using, that means that the first file
might get altered in some way while being written.
>

That was the problem.
I didn't pay much attention on that part..
Thanks for the help.

[Juan Antonio Zaratiegui Vallecillo]

Re: memory pool?

Juan Antonio Zaratiegui Vallecillo escribió:

Uli Kunkel escribió:

>I have a situation where I'm getting a picture from a camera in small
>chunks.
>So the photo is a few MBytes and I have a callback function that
>returns 1KB until it finishes transfering the picture.
>Any suggestions or examples how should I put a picture in one variable?
>I heard something about using memory pools for this but I only need
>something simple for this occasion.
>Could I use CString?
>>
>Anyway thanks in advance for any suggestions.

>
>
Use a vector, taking advantage from its reserve() function.
>
#include <vector>
>
(...)
>
std::vector<uns igned charmyPhoto;
myPhoto.reserve (1024*1024);
>
for (size_t part=0; part !=1024;++part) {
<read image data into &myPhoto(part*1 024) >

ThankÅ› to all for the comments.

I really meant to say
<append imaged data into myPhoto.end()>
But it was too early in the morning

}
(...)
<you now have your photo in myPhoto.size() sequential bytes at
&myPhoto[0]
>
>

(...)

And I keep on favouring the use of reserve, as it will be surely faster
than letting the container resize as needed, because we already know the
final size.

Best regards,

Zara

PS: Any error in this new message,, it is because it is too late in the
afternoon ;-)

[Uli Kunkel]

Re: memory pool?

Juan Antonio Zaratiegui Vallecillo wrote:

Juan Antonio Zaratiegui Vallecillo escribió:

>Uli Kunkel escribió:

>>I have a situation where I'm getting a picture from a camera in small
>>chunks.
>>So the photo is a few MBytes and I have a callback function that
>>returns 1KB until it finishes transfering the picture.
>>Any suggestions or examples how should I put a picture in one variable?
>>I heard something about using memory pools for this but I only need
>>something simple for this occasion.
>>Could I use CString?
>>>
>>Anyway thanks in advance for any suggestions.

>>
>>
>Use a vector, taking advantage from its reserve() function.
>>
>#include <vector>
>>
>(...)
>>
> std::vector<uns igned charmyPhoto;
> myPhoto.reserve (1024*1024);
>>
> for (size_t part=0; part !=1024;++part) {
> <read image data into &myPhoto(part*1 024) >

ThankÅ› to all for the comments.
>
I really meant to say
<append imaged data into myPhoto.end()>
But it was too early in the morning
>

> }
>(...)
> <you now have your photo in myPhoto.size() sequential bytes at
>&myPhoto[0]
>>
>>

(...)
>
And I keep on favouring the use of reserve, as it will be surely faster
than letting the container resize as needed, because we already know the
final size.
>
Best regards,
>
Zara
>
PS: Any error in this new message,, it is because it is too late in the
afternoon ;-)

In the end I did something like this:
First I reserved 2 MB. Then I check the capacyty and reserve a 1KB more
if needed:

if(myPhoto.size () + pProgress->lLength < myPhoto.capacit y())
myPhoto.reserve (1024*1024);

Thank you all for your help.

[Default User]

Re: memory pool?

Pascal J. Bourguignon wrote:

Uli Kunkel <genijalac@yaho o.comwrites:
>

I have a situation where I'm getting a picture from a camera in
small chunks.
So the photo is a few MBytes and I have a callback function that
returns 1KB until it finishes transfering the picture.
Any suggestions or examples how should I put a picture in one
variable? I heard something about using memory pools for this but
I only need something simple for this occasion.
Could I use CString?

>
Not on comp.lang.c++. For CString, you should ask on comp.lang.c.

No, he should not. CString is not a C construct, but rather an MFC
class.




Brian

[James Kanze]

Re: memory pool?

On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:

Juan Antonio Zaratiegui Vallecillo wrote:
I'm wondering if I could use reserve to allocate a few MB and
then if the size is not enough that the vector grows
automatically? I read that vector resizing is very costly so
I presume that reserving just 1KB is not enough.

What you read was wrong. Resizing isn't that costly for
primitive object types (like unsigned char). There are other
reasons you might want to avoid it, but they probably don't
apply here.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[Uli Kunkel]

Re: memory pool?

James Kanze wrote:

On Nov 10, 12:49 pm, Uli Kunkel <genija...@yaho o.comwrote:

>Juan Antonio Zaratiegui Vallecillo wrote:
>I'm wondering if I could use reserve to allocate a few MB and
>then if the size is not enough that the vector grows
>automaticall y? I read that vector resizing is very costly so
>I presume that reserving just 1KB is not enough.

>
What you read was wrong. Resizing isn't that costly for
primitive object types (like unsigned char). There are other
reasons you might want to avoid it, but they probably don't
apply here.
>
--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

I tried 2 things:

1. reserve 3MB and then push_back every unsigned char (or byte) to the
vector.
2. Reserve 1KB on every callback

U didn't measure the time precisely, but it is approximatly the same.
It takes very long to complete, around 10 seconds.
This is to much.I don't know what the problem is.