single byte access in memblock

[JonLT]

Hi!

I'm trying to learn some network programming and right now im designing the package layout.
I think ill do it like this:
- first two bytes (an int): the size of the package
- third byte: the type of the package
- the rest: the data

The problem is that i dont know how to access a singe byte in a memblock created by malloc.
I guess i could use an unsiged char, but I'd like my app to be cross platform, and im not sure that a char is one byte on every arch.

Something like this:

unsigned int packageSize = 2 + 1 + dataSize;
unsiged char* package = (unsiged char*)malloc(packageSize);
package[0] = (unsigned char)packageSize;
package[2] = 2 // 2 being some predefined type

memcpy(package[3], data, dataSize);

Thanks!

[weaknessforcats]

So where is package[3] allocated?

You can't copy to memory you don't own.

Also, shouldn't your package be a struct?

[JonLT]

yeah a struct might be a good idear, but im not sure that it takes the same amount of mem on all archs. So if i send some bytes from the server and cast them to a package struct in the client, im not sure ill get the right thing.

What i really want is a type that uses the same amount of bytes on every arch. Do i need to make my own type, and if so - how would one do that?

I know the above code doesnt work, its more like a pointer towards what i would like to do

[Banfa]

When you say the package layout you mean the layout of the data block you are going to send across the network?

A lot depends on whether you intend this protocol to be transmitted over a TCP or UDP connection (or something else of course) but you may wish to consider also including

Package start indication: A TCP connection is a byte stream you may need to be able to pick out the start of a package in the stream.

Version Number: either message or protocol unless of course you only ever plan on having one version of the protocol that doesn't change once it is implemented to allow legacy and cutting edge devices to still effectively communicate.

A checksum of some sort: If you are using UDP then you have no guarantee of correct data delivery so a checksum might be useful in validating your packages.

When you define your package you will need to specify if the data carried is big endian or little endian (or uses some other byte ordering specification). That is if the package carries a 4 byte integer what order are the bytes of that integer transmitted in.

Are you sure that 1 byte (256 values) is enough for the package type? Remember no-one was ever going to need more than 640k of RAM in there PCs not so long ago.


sizeof(unsigned char) == sizeof(char) == 1

You are guaranteed this by the C or C++ (you haven't said which you are using) standard. Unfortunately what you are not guaranteed is the number of bits in a byte which is platform dependent.

[JonLT]

Thank you very much for that answer. It made me think :)
I didn't know anything about endianness, now i know a little bit. And i think a time-stamp in the header might be a good idear aswell.

Again: Thank you very much!

[Banfa]

Oh I forgot identifier of the sending unit. In the protocol I use/wrote we have

Start Bytes
Size
Protocol Version
Unit ID
Time Stamp (to 1/100 second)
Package Type
Payload
CRC16

Of course our packages have to be transmittable via a number of different transport mediums including TCP/IP and a modem link (and SMS is on the cards too but only conceptually at the moment).

The is no sequence number in the header because the majority of our packages are complete in themselves. In the few (1) case where the data to be transferred large (an image for example) it is carried by multiple packages in sequence. Then part of the payload is sequence number and max-sequence number to ensure the received packages can be verified as correct, complete and put back together in the right order.

If you are only using TCP this may not be required, on the other hand if are not and most of your packages carrier payloads of bits of a data block you wish to put sequence and max-sequence into the header.

[JonLT]

Thank you once again :)

I'm planing on using TCP for starters. The application im developing is a IRC like chat server and client. But I'd like it to be able to send more than just strings.
I'm using C++.
If TCP proves to slow i'll try UDP with some custom checking on top.

To get back to my initial question:
The following is what i'm planning to do:

The package:
- size (2 bytes)
- type (2 bytes)
- time stamp ( 'not sure yet' bytes)
- payload ( 'size' bytes)

When the client wants to send something, i reserve a memblock large enough for the payload and the additional information (the rest of the package). After that i just memcpy the data into the block at the predefined locations, and convert to big-endian (if it isnt already) and then send.
To manifest the package in the server i first receive the size of the package, and if needed convert its endianness. I then reserve enough mem for the rest of the package, and receive it (putting it in to the reserved block), again convert endianness if needed. To get the data from the package, first memcpy the type and the time stamp, and cast to some known type, and to get the payload of the package memcpy the data and cast to the type defined by the type field.

But the problem is this: will the result of the memcpy be correct on all archs (im not aiming for any embedding). I know this app will proberply never leave my home network, but i would like it to work elsewhere (just for kicks).
From what i understand an (un)signed int is minimum 2 bytes. Does that mean that important data can be stored outside the first two bytes, or will it be enough to send the first two bytes?
Is it even possible to send a float/double, whitout converting it to some costum format (saving the two parts (significand/exponent) in some clever way)?

Thanks!

[Banfa]

OK everything is bytes, it is up to you to define the size of the integers carried by your packages and it is up to the platforms to convert them to a form that is easy for them to handle.

Some protocols, MPEG2 for example, use the minimum number of bits required to carry the required range of values specified which leads to only sized fields 13 bits for example.

If you decide that for the purposes of your package an unsigned int is a 2 byte big endian value then that is how it is, it becomes the receiving software's problem to convert this to a appropriate data type for the platform in question.

The same with floating point numbers, you specify what standard you package will use and what order the bytes are in and it becomes the software on the platforms responsibility to read those bytes and convert them to a format the platform can use.

Remember what you are sending and receiving is not some integers and floating point values but an array of bytes that conform to a specification. It is the software's job to convert between that byte stream and useful data types.

You may have a structure that is the data for a given package type. If you want to produce portable, best practice code then what you can not do is use memcpy or casting to get the data from the array of bytes to the structure or vice-versa you must process the bytes individually to fill in your structure from the byte array.

That is you have to go the long way round using the bit operators (& | << >>).