Memory alignment

[Eric Sosman]

Re: Memory alignment

Why Tea wrote:

On Oct 3, 1:47 pm, Eric Sosman <Eric.Sos...@su n.comwrote:

>[...]
>I'd suggest not limiting yourself only to reading Question 2.6
>(for instance), but perusing some of the others as well.

>
Eric, thanks for the gentle nudge. I did google, but I didn't
know it was called "struct hack". It's hard to find the
right thing if you don't know the right term. [...]

.... which is why I suggest browsing the rest of the FAQ, too. No
need to memorize it, but get an idea of what's in it and what kind
of terminology describes it -- then next time you're puzzled you'll
have a better chance of finding an answer, or at least have the
vocabulary that lets you frame your question better.

--
Eric.Sosman@sun .com

[Keith Thompson]

Re: Memory alignment

Why Tea <ytlim1@gmail.c omwrites:

On Oct 3, 1:47 pm, Eric Sosman <Eric.Sos...@su n.comwrote:

[...]

>     You're full of questions, Why Tea, and that's a good thing.
>But has it occurred to you that other people have asked some of
>these same questions?  Has it occurred to you to look for a FAQ
>at some likely-sounding site like, oh, <http://www.c-faq.com/>?
>If you get lucky and find some useful material at such a site,
>I'd suggest not limiting yourself only to reading Question 2.6
>(for instance), but perusing some of the others as well.

>
Eric, thanks for the gentle nudge. I did google, but I didn't
know it was called "struct hack". It's hard to find the
right thing if you don't know the right term. Thanks to all
who took the time to reply.

Fair enough. I *know* it's called the "struct hack", and I've read
question 2.6 before, but I had trouble finding it myself, since the
answer to 2.6 doesn't use the phrase "struct hack". I happened to
remember that the URL includes the string "structhack ", so I was able
to use a Google advanced search to find it.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Keith Thompson]

Re: Memory alignment

Lowell Gilbert <lgusenet@be-well.ilk.orgwri tes:

Why Tea <ytlim1@gmail.c omwrites:

>On Oct 3, 12:48 pm, danmat...@gmail .com wrote:

[...]

>>Why would you want to declare a 1 char array to store 2 anyway?

>>
>Good question. This is found in some real embedded
>code to make more efficient of the memory. As I understood
>it, the last s[1] is just a placeholder as you can
>allocate more memory when needed. For example:
>>
>my_struct = malloc(sizeof(m y_struct_t) + MY_PAYLOAD_STRI NG_SIZE);

>
or even more likely, something more like
my_struct = malloc(sizeof(m y_struct_t) + strlen(my->struct->s));

+ 1. The length returned by strlen() doesn't include the terminating '\0'.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Keith Thompson]

Re: Memory alignment

jameskuyper@ver izon.net writes:
[...]

In C90, this was technically illegal, but as a practical matter it
worked on most (all?) implementations . It was sufficiently useful that
a modified version of the concept was added as an extension to many
complilers, but with the difference that the array was declared with a
size of 0, rather than 1. In C99, a modified version of the concept
was made finally standardized, under the name "flexible arrays". For
the C99 version, you should use:
>
typedef struct some_struct
{
int i;
short k,
int m;
char s[];
} some_struct_t;
>
With all three versions of this concept, it's required that the array
be declared at the end of the structure. The best way to handle the
allocation is as follows:
>
my_struct = malloc(offsetof (some_struct_t, s) +
MY_PAYLOAD_STRI NG_SIZE*sizeof my_struct.s[0]);
>
Using offsetof() rather than sizeof() makes a difference for the C90
version, because the size of the struct includes enough room for one
element of the array, whereas offsetof() does not.

[...]

I'm not convinced that's safe. If the string is very short, you might
allocate fewer than sizeof(some_str uct_t) bytes. That could, in some
circumstances, result in accessing memory that's within range of a
some_struct_t object, but outside the range of what was actually
allocated.

On the other hand, if you're doing this then you're not going to be
accessing the some_struct_t object as a whole. On the other other
hand, the compiler might be allowed to generate code that does so.
It's probably not going to cause any visible problems in practice,
especially since the memory allocated by malloc() will probably be big
enough to contain a my_struct_t object anyway, but it makes me
nervous. I'd rather risk allocating a byte or so too much than too
little, unless memory space is *really* critical and I'm willing to be
unwarrantedly chummy with the compiler (paraphrasing Dennis Ritchie).

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Why Tea]

Re: Memory alignment

On Oct 3, 2:10 pm, Keith Thompson <ks...@mib.orgw rote:

Why Tea <ytl...@gmail.c omwrites:

typedef struct some_struct
{
    int i;
    short k,
    int m;
    char s[1];
} some_struct_t;

>

Assuming 16 bit or 32-bit alignment, can I assume that
s always gets 4 or 8 bytes of allocation due to padding
in the following? (I.e. s is either 4 or 8 characters long)

>

some_struct_t *my_struct;
my_struct = malloc(sizeof(s ome_struct_t));

>

BTW, this is on a PowerPC architecture.

>
You can assume that s itself will be allocated exactly 1 byte.  Any
padding following it is part of the struct, but not part of s.
>
In general, compilers are allowed to insert arbitrary padding between
members or after the last member of a struct.  The purpose of this
padding is generally to meet alignment requirements, but the standard
doesn't place any restrictions on how little or how much padding is
used, as long as the members can be accessed.
>
So no, you can't portably make any assumptions about how much padding
is added after s.  But you generally shouldn't need to.  If you'll let
us know what you're trying to do, we can probably help you do it
*without* making any (or too many) non-portable assumptions.  If
you're trying to use the "struct hack", see question 2.6 in the
comp.lang.c FAQ, <http://c-faq.com/>.

Thanks Keith. I'll try the c-faq first next time. I just had
a look at 2.6 of c-faq, I'm surprised that what's written there
was exactly a discussion I had with a colleague. Taking the
code from the faq:

#include <stdlib.h>
#include <string.h>

#define MAXSIZE 100

struct name {
int namelen;
char namestr[MAXSIZE];
};

struct name *makename(char *newname)
{
struct name *ret =
malloc(sizeof(s truct name)-MAXSIZE+strlen( newname)+1);
/* +1 for \0 */
if(ret != NULL) {
ret->namelen = strlen(newname) ;
strcpy(ret->namestr, newname);
}

return ret;
}

The argument from my colleague was about the extra
byte (+1 for \0), why is it needed as padding is
always there? He assumed 32-bit alignment and
MAXSIZE=1 in our discussion. Is his argument always
right?

[Eric Sosman]

Re: Memory alignment

Keith Thompson wrote:

Lowell Gilbert <lgusenet@be-well.ilk.orgwri tes:

>Why Tea <ytlim1@gmail.c omwrites:

>>On Oct 3, 12:48 pm, danmat...@gmail .com wrote:

[...]

>>>Why would you want to declare a 1 char array to store 2 anyway?
>>Good question. This is found in some real embedded
>>code to make more efficient of the memory. As I understood
>>it, the last s[1] is just a placeholder as you can
>>allocate more memory when needed. For example:
>>>
>>my_struct = malloc(sizeof(m y_struct_t) + MY_PAYLOAD_STRI NG_SIZE);

>or even more likely, something more like
> my_struct = malloc(sizeof(m y_struct_t) + strlen(my->struct->s));

>
+ 1. The length returned by strlen() doesn't include the terminating '\0'.

The struct has `char s[1]' as its last element[*], and the
size of that element is already included in sizeof(my_struc t_t).
[*] Or so I assume. We started with a some_struct_t, but
morphed to my_struct_t somewhere in mid-dicussion without ever
seeing the type defined. Given the reference to "the last s[1]"
I think we can count on it still being present, though.

--
Eric.Sosman@sun .com

[CBFalconer]

Re: Memory alignment

Why Tea wrote:

Fred <fred.l.kleinsc hm...@boeing.co mwrote:

>Why Tea <ytl...@gmail.c omwrote:
>>

>>typedef struct some_struct {
>> int i;
>> short k,
>> int m;
>> char s[1];
>>} some_struct_t;

>>

.... snip ...

>

>but writing to s[1] is an error.

>
But strcpy(my->struct->s, "AB"); is OK if there is
padding. Isn't it not? Please go easy on me here.
I just want to understand how this really works...

Because you declared s to hold one byte. You have no idea what the
system is doing with any extra memory that it had to supply, or
even if it had to supply anything extra.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home .att.net>
Try the download section.

[lawrence.jones@siemens.com]

Re: Memory alignment

Antoninus Twink <nospam@nospam. invalidwrote:

>
Yes, of course, if there are two or more padding bytes at the end of the
struct then that's your memory to write to, whether it's on the stack if
my_struct is an automatic variable, or on the heap if you got the memory
from malloc().

No, the padding bytes are most definitely *not* yours to write to.
Although most implementations will let you get away with it, there are
implementations that do careful memory bounds checking and won't.
--
Larry Jones

My life needs a rewind/erase button. -- Calvin

[Ben Pfaff]

Re: Memory alignment

lawrence.jones@ siemens.com writes:

Antoninus Twink <nospam@nospam. invalidwrote:

>>
>Yes, of course, if there are two or more padding bytes at the end of the
>struct then that's your memory to write to, whether it's on the stack if
>my_struct is an automatic variable, or on the heap if you got the memory
>from malloc().

>
No, the padding bytes are most definitely *not* yours to write to.

Really?

struct { int a; } b;
memset(&b, 0, sizeof b); /* Bang? */
--
char a[]="\n .CJacehknorstu" ;int putchar(int);in t main(void){unsi gned long b[]
={0x67dffdff,0x 9aa9aa6a,0xa77f fda9,0x7da6aa6a ,0xa67f6aaa,0xa a9aa9f6,0x11f6} ,*p
=b,i=24;for(;p+ =!*p;*p/=4)switch(0[p]&3)case 0:{return 0;for(p--;i--;i--)case+
2:{i++;if(i)bre ak;else default:continu e;if(0)case 1:putchar(a[i&15]);break;}}}

[Eric Sosman]

Re: Memory alignment

lawrence.jones@ siemens.com wrote:

Antoninus Twink <nospam@nospam. invalidwrote:

>Yes, of course, if there are two or more padding bytes at the end of the
>struct then that's your memory to write to, whether it's on the stack if
>my_struct is an automatic variable, or on the heap if you got the memory
>from malloc().

>
No, the padding bytes are most definitely *not* yours to write to.
Although most implementations will let you get away with it, there are
implementations that do careful memory bounds checking and won't.

There's also the issue of copying one such struct to another
by assignment, or passing a struct instance as a function argument
or returning one as a function value. The implementation is not
obliged to preserve the values of padding bytes during these
operations.

--
Eric.Sosman@sun .com

[Keith Thompson]

Re: Memory alignment

Why Tea <ytlim1@gmail.c omwrites:

On Oct 3, 2:10 pm, Keith Thompson <ks...@mib.orgw rote:

>Why Tea <ytl...@gmail.c omwrites:

typedef struct some_struct
{
    int i;
    short k,
    int m;
    char s[1];
} some_struct_t;

>>

Assuming 16 bit or 32-bit alignment, can I assume that
s always gets 4 or 8 bytes of allocation due to padding
in the following? (I.e. s is either 4 or 8 characters long)

>>

some_struct_t *my_struct;
my_struct = malloc(sizeof(s ome_struct_t));

>>

BTW, this is on a PowerPC architecture.

>>
>You can assume that s itself will be allocated exactly 1 byte.  Any
>padding following it is part of the struct, but not part of s.
>>
>In general, compilers are allowed to insert arbitrary padding between
>members or after the last member of a struct.  The purpose of this
>padding is generally to meet alignment requirements, but the standard
>doesn't place any restrictions on how little or how much padding is
>used, as long as the members can be accessed.
>>
>So no, you can't portably make any assumptions about how much padding
>is added after s.  But you generally shouldn't need to.  If you'll let
>us know what you're trying to do, we can probably help you do it
>*without* making any (or too many) non-portable assumptions.  If
>you're trying to use the "struct hack", see question 2.6 in the
>comp.lang.c FAQ, <http://c-faq.com/>.

>
Thanks Keith. I'll try the c-faq first next time. I just had
a look at 2.6 of c-faq, I'm surprised that what's written there
was exactly a discussion I had with a colleague. Taking the
code from the faq:
>
#include <stdlib.h>
#include <string.h>
>
#define MAXSIZE 100
>
struct name {
int namelen;
char namestr[MAXSIZE];
};
>
struct name *makename(char *newname)
{
struct name *ret =
malloc(sizeof(s truct name)-MAXSIZE+strlen( newname)+1);
/* +1 for \0 */
if(ret != NULL) {
ret->namelen = strlen(newname) ;
strcpy(ret->namestr, newname);
}
>
return ret;
}
>
The argument from my colleague was about the extra
byte (+1 for \0), why is it needed as padding is
always there? He assumed 32-bit alignment and
MAXSIZE=1 in our discussion. Is his argument always
right?

It may happen to be true that it will work on all implementations . A
compiler will *probably* add enough padding to the end of the struct
to make it work. Even if not, a runtime library's malloc()
implementation will *probably* add enough padding to the allocated
space. And even if not, you're likely (but by no means certain) to be
able to get away with accessing memory just one byte beyond what's
been allocated, depending on what happens to be there. Finally, most
C implementations won't go out of their way to prevent you from
accessing memory beyond the declared bounds of an array (and if yours
does, you can't use the struct hack in the first place).

But the line of reasoning that lets you assume that you don't need to
explicitly allocate enough space for the terminating '\0' is
convoluted, weak, and system-specific. If I were writing the code,
I'd just allocate the byte and be done with it. Otherwise, every time
a problem shows up, I'd have to spend time confirming that the failure
to allocate that byte isn't the cause. It doesn't cost much to do it
right.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[CBFalconer]

Re: Memory alignment

lawrence.jones@ siemens.com wrote:

Antoninus Twink <nospam@nospam. invalidwrote:
>

>Yes, of course, if there are two or more padding bytes at the end
>of the struct then that's your memory to write to, whether it's
>on the stack if my_struct is an automatic variable, or on the heap
>if you got the memory from malloc().

>
No, the padding bytes are most definitely *not* yours to write to.
Although most implementations will let you get away with it, there
are implementations that do careful memory bounds checking and won't.

Bear in mind that Twink is a troll, whose objective is to disrupt
this newsgroup.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home .att.net>
Try the download section.

[Keith Thompson]

Re: Memory alignment

Eric Sosman <Eric.Sosman@su n.comwrites:

Keith Thompson wrote:

>Lowell Gilbert <lgusenet@be-well.ilk.orgwri tes:

>>Why Tea <ytlim1@gmail.c omwrites:
>>>On Oct 3, 12:48 pm, danmat...@gmail .com wrote:

>[...]

>>>>Why would you want to declare a 1 char array to store 2 anyway?
>>>Good question. This is found in some real embedded
>>>code to make more efficient of the memory. As I understood
>>>it, the last s[1] is just a placeholder as you can
>>>allocate more memory when needed. For example:
>>>>
>>>my_struct = malloc(sizeof(m y_struct_t) + MY_PAYLOAD_STRI NG_SIZE);
>>or even more likely, something more like
>> my_struct = malloc(sizeof(m y_struct_t) + strlen(my->struct->s));

>+ 1. The length returned by strlen() doesn't include the
>terminating '\0'.

>
The struct has `char s[1]' as its last element[*], and the
size of that element is already included in sizeof(my_struc t_t).

So it does.

Seeing strlen() in a computation of how much memory to allocate sets
off my alarm bells. If I were going to write something like that in
real code, it would be heavily commented.

[...]

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Why Tea]

Re: Memory alignment

On Oct 4, 4:40 am, Antoninus Twink <nos...@nospam. invalidwrote:

On  3 Oct 2008 at 18:33, Why Tea wrote:
>

typedef struct some_struct
{
    int i;
    short k,
    int m;
    char s[1];
} some_struct_t;

>

But strcpy(my->struct->s, "AB"); is OK if there is
padding. Isn't it not? Please go easy on me here.
I just want to understand how this really works...

>
Yes, of course, if there are two or more padding bytes at the end of the
struct then that's your memory to write to, whether it's on the stack if
my_struct is an automatic variable, or on the heap if you got the memory
from malloc().
>
You should be aware that the regulars here aren't interested in your
wish for a pragmatic answer that's true in practise: they'll just
bombard you with hypothetical answers that are true in theory.

Thanks Antoninus. I appreciate your answer. Eric S. is very
knowledgeable and I respect that. But to say "It's not OK at all"
categorically without considering cases when it COULD be OK
can give a wrong impression of the problem.

For example, if there is a system crash in an embedded system and
the dump shows it was due to a wild pointer. When you see a piece
of code that uses the struct hack (a term I just learned) and it
forgets to include the '\0' in the size allocated, you can't really
swear by the bible that it causes the crash as it over steps the
memory allocated with strcpy. I understood it's bad and dangerous
programming, but the question is can we be 100% sure that an
existing code like that causes memory corruption? Based on the
little I know, I don't think it's a simple answer of a YES or NO.
That's why I think Antoninus has given a more accurate answer.

can't swear by the bible
"It's not OK at all"

[Keith Thompson]

Re: Memory alignment

Why Tea <ytlim1@gmail.c omwrites:

On Oct 4, 4:40 am, Antoninus Twink <nos...@nospam. invalidwrote:

[snip]

>
Thanks Antoninus. I appreciate your answer. Eric S. is very
knowledgeable and I respect that. But to say "It's not OK at all"
categorically without considering cases when it COULD be OK
can give a wrong impression of the problem.
>
For example, if there is a system crash in an embedded system and
the dump shows it was due to a wild pointer. When you see a piece
of code that uses the struct hack (a term I just learned) and it
forgets to include the '\0' in the size allocated, you can't really
swear by the bible that it causes the crash as it over steps the
memory allocated with strcpy. I understood it's bad and dangerous
programming, but the question is can we be 100% sure that an
existing code like that causes memory corruption? Based on the
little I know, I don't think it's a simple answer of a YES or NO.
That's why I think Antoninus has given a more accurate answer.
>
can't swear by the bible
"It's not OK at all"

Um, did you read my answer, in which I specifically acknowledged that
you can probably get away with accessing padding bytes but also
explained why it's a bad idea to depend on it?

There is nothing "pragmatic" about encouraging you to assume that
there are a certain number of padding bytes at the end of a structure,
and that you can safely use them for whatever you want.

"Antoninus Twink" is a troll. Please do us all a favor and ignore
him.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
Nokia
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"