canonical way for handling raw data

[Matthias Czapla]

Hi!

Whats the canonical way for handling raw data. I want to read a file without
making any assumption about its structure and store portions of it in memory
and compare ranges with constant byte sequences. _I_ would read it
into arrays of unsigned char and use C's memcmp(), but as you see Im a
novice C++ programmer and think that theres some better, typically used,
way.

Regards
lal

[Gianni Mariani]

Re: canonical way for handling raw data

Matthias Czapla wrote:[color=blue]
> Hi!
>
> Whats the canonical way for handling raw data. I want to read a file without
> making any assumption about its structure and store portions of it in memory
> and compare ranges with constant byte sequences. _I_ would read it
> into arrays of unsigned char and use C's memcmp(), but as you see Im a
> novice C++ programmer and think that theres some better, typically used,
> way.
>[/color]

I've seen all kinds of messes when handling raw data !

Before you go down writing memcmp everywhere, ask yourself, what do
these "chunks of raw data" do ?

Do you:
- concatenate them
- do you write to them
- do you convert them
- do you break them up into smaller chunks

..... write a list of operations you do with them.

Sometimes you'll benefit from using a regular vector<char> and sometimes
you need somthing a little fancier.

I tend to write code that avoids copying data and so I usually have a
"Buffer" class where I can create create chunks of raw data and
reference chunks within those chunks .... etc The idea is that data is
not copied.

[Matthias Czapla]

Re: canonical way for handling raw data

Gianni Mariani wrote:[color=blue]
> Matthias Czapla wrote:[color=green]
> > Hi!
> >
> > Whats the canonical way for handling raw data. I want to read a file without
> > making any assumption about its structure and store portions of it in memory
> > and compare ranges with constant byte sequences. _I_ would read it
> > into arrays of unsigned char and use C's memcmp(), but as you see Im a
> > novice C++ programmer and think that theres some better, typically used,
> > way.
> >[/color]
>
> I've seen all kinds of messes when handling raw data !
>
> Before you go down writing memcmp everywhere, ask yourself, what do
> these "chunks of raw data" do ?
>
> Do you:
> - concatenate them
> - do you write to them
> - do you convert them
> - do you break them up into smaller chunks
>
> .... write a list of operations you do with them.[/color]

Ok, I have an image file of some smartcard used in a digital camera which was
accidentally deleted/formatted. I want to search in this file for occurences
of one of several byte sequences which indicate the start of a JPEG picture.
So Im interested in the position of these sequences in the file.

I already wrote a pure C program which works seemingly well but Im currently
in the process of gronking C++ and want to reimplement the program the C++ way.

Regards
lal

[Thomas Matthews]

Re: canonical way for handling raw data

Matthias Czapla wrote:[color=blue]
> Hi!
>
> Whats the canonical way for handling raw data. I want to read a file without
> making any assumption about its structure and store portions of it in memory
> and compare ranges with constant byte sequences. _I_ would read it
> into arrays of unsigned char and use C's memcmp(), but as you see Im a
> novice C++ programmer and think that theres some better, typically used,
> way.
>
> Regards
> lal[/color]

The method for handling raw unstructured data is to read it into a
buffer, then parse the buffer.

One process that I use is to have classes for each datum type and have
the classes provide a "load from buffer" and "store to buffer"
methods. I then pass a pointer to the buffer and call the load
methods of the class. The load method would advance the buffer
pointer:
class MyClass
{
public:
void load_from_buffe r(unsigned char * & buffer_pointer) ;
};

void
MyClass ::
load_from_buffe r(unsigned char * & buffer_pointer)
{
my_item = *((/* type of my_item */ *) buffer_pointer) ;
buffer_pointer += sizeof /* type of my item */;
// ...
return;
}

also:
template <class AnyType>
AnyTtype load_from_buffe r(unsigned char * & buffer_pointer)
{
return *((AnyType *) buffer_pointer) ;
}



--
Thomas Matthews

C++ newsgroup welcome message:

http://www.slack.net/~shiva/welcome.txt

C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.l earn.c-c++ faq:

http://www.raos.demon.uk/acllc-c++/faq.html

Other sites:
http://www.josuttis.com -- C++ STL Library book

[Matthias Czapla]

Re: canonical way for handling raw data

Thomas Matthews wrote:[color=blue]
> The method for handling raw unstructured data is to read it into a
> buffer, then parse the buffer.
>
> One process that I use is to have classes for each datum type and have
> the classes provide a "load from buffer" and "store to buffer"
> methods. I then pass a pointer to the buffer and call the load
> methods of the class. The load method would advance the buffer
> pointer:
> class MyClass
> {
> public:
> void load_from_buffe r(unsigned char * & buffer_pointer) ;
> };
>
> void
> MyClass ::
> load_from_buffe r(unsigned char * & buffer_pointer)
> {
> my_item = *((/* type of my_item */ *) buffer_pointer) ;
> buffer_pointer += sizeof /* type of my item */;
> // ...
> return;
> }
>
> also:
> template <class AnyType>
> AnyTtype load_from_buffe r(unsigned char * & buffer_pointer)
> {
> return *((AnyType *) buffer_pointer) ;
> }[/color]

Tanks for your reply. I thought about using a separate class for I/O too.
The most important point for me in your explanation is the use of unsigned
char to hold the data. Mind you asking me whats the advantage of using
unsigned over signed char? Do you agree to using std::ifstream:: read() for
reading the data?

[Thomas Matthews]

Re: canonical way for handling raw data

Matthias Czapla wrote:[color=blue]
> Thomas Matthews wrote:
>
>
> Tanks for your reply. I thought about using a separate class for I/O too.
> The most important point for me in your explanation is the use of unsigned
> char to hold the data. Mind you asking me whats the advantage of using
> unsigned over signed char? Do you agree to using std::ifstream:: read() for
> reading the data?[/color]

Unsigned char allows usage of all the bits, without any worries about
overflow and signing. I just want a simple 'byte' or smallest
accessible unit. The 'signed' quantities have issues when it comes
to bitmanipulation (such as shifting).

I guess it's just my style. You can find good discussions about
signed and unsigned integral types in this newsgroup and
our neighbor news:comp.lang. c++.

You can use ifstream::read( ) as long as the file is opened in
binary mode. The binary mode tells the compiler/platform to
_NOT_ perform any translations on the data.

There are also claims that fread() is simpler and faster.
However, since developer time and quality is more important
than speed, go with ifstream::read( ).

In my Binary_Stream class, I have a pure virtual function:
unsigned long size_on_stream( ) const = 0;
All classes that use the Binary_Stream interface must provide
the size that they occupy on the stream. This allows one to
query an object about the size of data it requires in order
to allocate a buffer for reading:
unsigned long buffer_size = my_msg.size_on_ stream();
unsigned char * buffer = new unsigned char[buffer_size];
my_data_file.re ad(buffer, buffer_size);
unsigned char * buf_ptr(buffer) ;
my_msg.load_fro m_buffer(buf_pt r);
delete [] buffer;
One nice benefit is that objects can be written to and read
from a stream without knowing any details about the object!

--
Thomas Matthews

C++ newsgroup welcome message:

http://www.slack.net/~shiva/welcome.txt

C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.l earn.c-c++ faq:

http://www.raos.demon.uk/acllc-c++/faq.html

Other sites:
http://www.josuttis.com -- C++ STL Library book

[Thomas Matthews]

Re: canonical way for handling raw data

Thomas Matthews wrote:
[color=blue]
> Matthias Czapla wrote:
>[color=green]
>> Thomas Matthews wrote:[/color]
> I guess it's just my style. You can find good discussions about
> signed and unsigned integral types in this newsgroup and
> our neighbor news:comp.lang. c++.[/color]

That should be news:comp.lang. c.

--
Thomas Matthews

C++ newsgroup welcome message:

http://www.slack.net/~shiva/welcome.txt

C++ Faq: http://www.parashift.com/c++-faq-lite
C Faq: http://www.eskimo.com/~scs/c-faq/top.html
alt.comp.lang.l earn.c-c++ faq:

http://www.raos.demon.uk/acllc-c++/faq.html

Other sites:
http://www.josuttis.com -- C++ STL Library book

[Matthias Czapla]

Re: canonical way for handling raw data

Thomas Matthews wrote:[color=blue][color=green]
> > char to hold the data. Mind you asking me whats the advantage of using
> > unsigned over signed char? Do you agree to using std::ifstream:: read() for
> > reading the data?[/color]
>
> Unsigned char allows usage of all the bits, without any worries about
> overflow and signing. I just want a simple 'byte' or smallest
> accessible unit. The 'signed' quantities have issues when it comes
> to bitmanipulation (such as shifting).[/color]

I see.
[color=blue]
> I guess it's just my style. You can find good discussions about
> signed and unsigned integral types in this newsgroup and
> our neighbor news:comp.lang. c++.
>
> You can use ifstream::read( ) as long as the file is opened in
> binary mode. The binary mode tells the compiler/platform to
> _NOT_ perform any translations on the data.[/color]

Ill remember that.
[color=blue]
> There are also claims that fread() is simpler and faster.
> However, since developer time and quality is more important
> than speed, go with ifstream::read( ).[/color]

And as I stated elsewhere I want to do it the "C++ way".
[color=blue]
> In my Binary_Stream class, I have a pure virtual function:
> unsigned long size_on_stream( ) const = 0;
> All classes that use the Binary_Stream interface must provide
> the size that they occupy on the stream. This allows one to
> query an object about the size of data it requires in order
> to allocate a buffer for reading:
> unsigned long buffer_size = my_msg.size_on_ stream();
> unsigned char * buffer = new unsigned char[buffer_size];
> my_data_file.re ad(buffer, buffer_size);
> unsigned char * buf_ptr(buffer) ;
> my_msg.load_fro m_buffer(buf_pt r);
> delete [] buffer;
> One nice benefit is that objects can be written to and read
> from a stream without knowing any details about the object![/color]

Very nice. That has given me an idea about the topic. As it seems raw data
handling isnt too different from Cs and when I think about it this is
logical since this is very low level. Thank you for your help.

Regards
lal