alligned memory allocation

[Kutty Banerjee]

Hi,
I ve got the following piece of code which does the role of allocating
aligned memory (not sure what aligned memory allocation is either).

void *
_align_calloc(s ize_t bytes, unsigned long alignment)
{
unsigned long ptr, buf;

ASSERT( alignment > 0 );

ptr = (unsigned long) calloc(bytes + alignment + sizeof(void *));
if (!ptr)
return NULL;

buf = (ptr + alignment + sizeof(void *)) & ~(unsigned long)(alignment -
1);
*(unsigned long *)(buf - sizeof(void *)) = ptr;

#ifdef DEBUG
/* mark the non-aligned area */
while ( ptr < buf - sizeof(void *) ) {
*(unsigned long *)ptr = 0xcdcdcdcd;
ptr += sizeof(unsigned long);
}
#endif

return (void *)buf;
}

Can somebody please explain what this code is doing? I m a little confused
with the memory being allocated to a non pointer data namely unsigned long
ptr ?


kutty

[Christian Bau]

Re: alligned memory allocation

In article <c26nfc$1pot$1@ bigboote.WPI.ED U>,
"Kutty Banerjee" <kuttyb@wpi.edu > wrote:
[color=blue]
> Hi,
> I ve got the following piece of code which does the role of allocating
> aligned memory (not sure what aligned memory allocation is either).
>
> void *
> _align_calloc(s ize_t bytes, unsigned long alignment)
> {
> unsigned long ptr, buf;
>
> ASSERT( alignment > 0 );
>
> ptr = (unsigned long) calloc(bytes + alignment + sizeof(void *));
> if (!ptr)
> return NULL;
>
> buf = (ptr + alignment + sizeof(void *)) & ~(unsigned long)(alignment -
> 1);
> *(unsigned long *)(buf - sizeof(void *)) = ptr;
>
> #ifdef DEBUG
> /* mark the non-aligned area */
> while ( ptr < buf - sizeof(void *) ) {
> *(unsigned long *)ptr = 0xcdcdcdcd;
> ptr += sizeof(unsigned long);
> }
> #endif
>
> return (void *)buf;
> }
>
> Can somebody please explain what this code is doing? I m a little confused
> with the memory being allocated to a non pointer data namely unsigned long
> ptr ?[/color]

It is a case of a clueless programmer trying to return a pointer that is
aligned in a certain way. I'll give you an overview what he is trying to
do and where he gets it wrong:

malloc and calloc return pointers that are properly aligned for _any_ C
type. Now there might be reasons why you would want a pointer that is
aligned to multiples of 128 bytes, for example, and malloc won't do
that.

So what he does is allocate a pointer that is bigger, lets say a pointer
"char* p", and in the case of 128 byte alignment you would add some
value from 0 to 127 to make it aligned on a multiple of 128 bytes. But
if you did just that and returned the result pointer, then you wouldn't
be able to free the memory later: To be able to call free, you must have
the original pointer. So you store the original pointer just in front of
the pointer that you return.

Now your original author does a few remarkably stupid things. He makes
assumptions that are completely unwarranted in portable C and actually
wrong on more and more computers that are in actual use. No wonder you
are confused :-(

First, he assumes that "alignment" is not only greater than zero, but
also a power of two, AND at least as large as the alignment requirement
of a void* value. Why doesn't he write this in the ASSERT? To test
whether x is a power of two: If you write x and x-1 in binary, then if x
is a power of two no bit will be set both in x and x-1. If x is nonzero
and not a power of two, then the highest bit of x is also set in x-1. So
you could write

ASSERT(align >= sizeof (void *) && (align & (align-1)) != 0);

Next, he assumes that you can cast a void* to unsigned long and perform
pointer arithmetic on the unsigned long. Bullshit. Take an Athlon 64 and
you may be surprised to see that void* = 64 bit, unsigned long = 32 bit,
and casting void* to unsigned long produces a whole lot of garbage. Take
16 bit DOS with "huge" memory model, and adding to an unsigned long to
do pointer arithmetic will produce garbage. Try it on a Cray and heaven
knows what you get. But in fact it is very easy to do what he wants in a
much more portable way:

char* p = calloc (bytes + alignment + sizeof (void *));
char* q = p + sizeof (void *) + alignment;
q -= ((unsigned long) q) % alignment;

// If you want your code unreadable like the original author,
// then change the last line to
// q -= ((unsigned long) q) & (alignment - 1);

Now it doesn't matter what the relative sizes of pointers and unsigned
long are.

The next mistake that he makes is that you can store an unsigned long if
you have space for a void*. Bad mistake. Just as you have systems where
sizeof (void *) > sizeof (unsigned long), you have systems where it is
the other way round. Buy a new Macintosh and you will get a machine that
uses 64 bit integers and 32 bit pointers. So he has this pointer "buf",
subtracts four byte = sizeof (void *), and stores an eight byte unsigned
long. Congratulations . The last four bytes of that unsigned long will be
overwritten by the caller as soon as they use the memory. Crash when you
try to free the memory later on.

Have fun with this guy's code. Expect the worst. People who think they
can program and can't are the worst kind.

[CBFalconer]

Re: alligned memory allocation

Christian Bau wrote:[color=blue]
> "Kutty Banerjee" <kuttyb@wpi.edu > wrote:
>[color=green]
> > I ve got the following piece of code which does the role of
> > allocating aligned memory (not sure what aligned memory
> > allocation is either).
> >
> > void *
> > _align_calloc(s ize_t bytes, unsigned long alignment)
> > {[/color][/color]
.... snip code ...[color=blue][color=green]
> >
> > Can somebody please explain what this code is doing? I m a
> > little confused with the memory being allocated to a non
> > pointer data namely unsigned long ptr ?[/color]
>
> It is a case of a clueless programmer trying to return a pointer
> that is aligned in a certain way. I'll give you an overview what
> he is trying to do and where he gets it wrong:
>[/color]
.... snip ...[color=blue]
>
> Have fun with this guy's code. Expect the worst. People who
> think they can program and can't are the worst kind.[/color]

The code is obviously not portable, and is also obviously intended
for system use (the _align... name is not in the user name
space). This is a problem that is simply not soluble with
portable code, so the only requirement is that it work, somehow or
other, with the original implementation. Your caustic criticisms
are probably undeserved.

To the OP: The internal workings of a particular system are often
wierd and wonderful, and highly non-standard. You are not
expected to understand them on the basis of the language.

--
Chuck F (cbfalconer@yah oo.com) (cbfalconer@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Christian Bau]

Re: alligned memory allocation

In article <40470747.70E9F BFE@yahoo.com>,
CBFalconer <cbfalconer@yah oo.com> wrote:
[color=blue]
> Christian Bau wrote:[color=green]
> > Have fun with this guy's code. Expect the worst. People who
> > think they can program and can't are the worst kind.[/color]
>
> The code is obviously not portable, and is also obviously intended
> for system use (the _align... name is not in the user name
> space). This is a problem that is simply not soluble with
> portable code, so the only requirement is that it work, somehow or
> other, with the original implementation. Your caustic criticisms
> are probably undeserved.[/color]

The point is that this code is unportable without any good reason. And
assuming that unsigned long and void* have the same size, today, is just
utterly stupid. Not the "it works everywhere except on the DeathStation
9000" kind of stupid, but the "I bought this brand new computer and my
code crashes all over the place" kind of stupid.

[CBFalconer]

Re: alligned memory allocation

Christian Bau wrote:[color=blue]
> CBFalconer <cbfalconer@yah oo.com> wrote:[color=green]
> > Christian Bau wrote:[/color]
>[color=green][color=darkred]
> > > Have fun with this guy's code. Expect the worst. People who
> > > think they can program and can't are the worst kind.[/color]
> >
> > The code is obviously not portable, and is also obviously intended
> > for system use (the _align... name is not in the user name
> > space). This is a problem that is simply not soluble with
> > portable code, so the only requirement is that it work, somehow or
> > other, with the original implementation. Your caustic criticisms
> > are probably undeserved.[/color]
>
> The point is that this code is unportable without any good reason.
> And assuming that unsigned long and void* have the same size, today,
> is just utterly stupid. Not the "it works everywhere except on the
> DeathStation 9000" kind of stupid, but the "I bought this brand new
> computer and my code crashes all over the place" kind of stupid.[/color]

You yourself pointed out a reason for it - to provide alignment to
larger values than does malloc. Unsigned long and void* may well
have the same size on that system, they do under DJGPP. I believe
gcc requires such system routines (although I don't approve).
This is probably not the forum to ask questions about it, although
I am hard pressed to think of a suitable newsgroup.

--
Chuck F (cbfalconer@yah oo.com) (cbfalconer@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Mark F. Haigh]

Re: alligned memory allocation

CBFalconer <cbfalconer@yah oo.com> wrote in message news:<40470747. 70E9FBFE@yahoo. com>...[color=blue]
> Christian Bau wrote:[/color]
<snip>[color=blue][color=green]
> >
> > Have fun with this guy's code. Expect the worst. People who
> > think they can program and can't are the worst kind.[/color]
>
> The code is obviously not portable, and is also obviously intended
> for system use (the _align... name is not in the user name
> space). This is a problem that is simply not soluble with
> portable code, so the only requirement is that it work, somehow or
> other, with the original implementation. Your caustic criticisms
> are probably undeserved.[/color]

I strongly disagree. As Christian pointed out, the original code
chooses unportability even though choosing (more) portability is
essentially free.

In fact, to me it's more unmaintainable. If you're doing something
like this, I immediately wonder why the author did not simply call
memalign then memset. Or simply wrote a small function to round the
alignment to the nearest power of two and then do a memalign and
memset. Now I just spent time that I shouldn't have had to spend.

Bad code is bad code, whether it works on the original implementation
or not. The mark of a bad programmer is that they write portable code
when there's a reason to. The mark of a good programmer is that they
write portable code when there is no reason not to.

Mark F. Haigh
mfhaigh@sbcglob al.net

[Christian Bau]

Re: alligned memory allocation

In article <4047BDAB.8053E 98@yahoo.com>,
CBFalconer <cbfalconer@yah oo.com> wrote:
[color=blue]
> Christian Bau wrote:[color=green]
> > CBFalconer <cbfalconer@yah oo.com> wrote:[color=darkred]
> > > Christian Bau wrote:[/color]
> >[color=darkred]
> > > > Have fun with this guy's code. Expect the worst. People who
> > > > think they can program and can't are the worst kind.
> > >
> > > The code is obviously not portable, and is also obviously intended
> > > for system use (the _align... name is not in the user name
> > > space). This is a problem that is simply not soluble with
> > > portable code, so the only requirement is that it work, somehow or
> > > other, with the original implementation. Your caustic criticisms
> > > are probably undeserved.[/color]
> >
> > The point is that this code is unportable without any good reason.
> > And assuming that unsigned long and void* have the same size, today,
> > is just utterly stupid. Not the "it works everywhere except on the
> > DeathStation 9000" kind of stupid, but the "I bought this brand new
> > computer and my code crashes all over the place" kind of stupid.[/color]
>
> You yourself pointed out a reason for it - to provide alignment to
> larger values than does malloc. Unsigned long and void* may well
> have the same size on that system, they do under DJGPP.[/color]
[color=blue]
> I believe
> gcc requires such system routines (although I don't approve).[/color]

And I hope sincerely that they have been written better.
[color=blue]
> This is probably not the forum to ask questions about it, although
> I am hard pressed to think of a suitable newsgroup.[/color]

char* p;
unsigned long x;

* (unsigned long *) (p - sizeof (void *)) = x;

is not non-portable, it is just plain stupid. The original poster was
confused _by the stupidity_ of the code. There were other errors in the
code that had nothing to do with portability, but just with stupidity.

[Mark F. Haigh]

Re: alligned memory allocation

Mark F. Haigh wrote:
[color=blue]
> CBFalconer <cbfalconer@yah oo.com> wrote in message news:<40470747. 70E9FBFE@yahoo. com>...
>[color=green]
>>Christian Bau wrote:[/color]
>
> <snip>
>[color=green][color=darkred]
>>>Have fun with this guy's code. Expect the worst. People who
>>>think they can program and can't are the worst kind.[/color]
>>
>>The code is obviously not portable, and is also obviously intended
>>for system use (the _align... name is not in the user name
>>space). This is a problem that is simply not soluble with
>>portable code, so the only requirement is that it work, somehow or
>>other, with the original implementation. Your caustic criticisms
>>are probably undeserved.[/color]
>
>
> I strongly disagree. As Christian pointed out, the original code
> chooses unportability even though choosing (more) portability is
> essentially free.
>
> In fact, to me it's more unmaintainable. If you're doing something
> like this, I immediately wonder why the author did not simply call
> memalign then memset. Or simply wrote a small function to round the
> alignment to the nearest power of two and then do a memalign and
> memset. Now I just spent time that I shouldn't have had to spend.
>
> Bad code is bad code, whether it works on the original implementation
> or not. The mark of a bad programmer is that they write portable code
> when there's a reason to. The mark of a good programmer is that they
> write portable code when there is no reason not to.
>[/color]

yeah, yeah, memalign not in C standard, or POSIX for that matter, yeah
yeah. My point remains essentially the same. If memalign, snprintf,
etc, are not implemented on your system, then implement 'em and use 'em.
Even at a glance, the entire _align_calloc thing the OP quoted should
jump out at you and say 'probably a bad idea'.

Some of these supposed systems programming guys come up with this kind
of thing and then throw casts around until the compiler shuts up. This
is neither weird nor wonderful, just annoying to those of us that have
to clean up afterwards.

Or maybe I'm just cynical, you decide.



Mark F. Haigh
mfhaigh@sbcglob al.net

[Dan Pop]

Re: alligned memory allocation

In <4047BDAB.8053E 98@yahoo.com> CBFalconer <cbfalconer@yah oo.com> writes:
[color=blue]
>You yourself pointed out a reason for it - to provide alignment to
>larger values than does malloc.[/color]

malloc provides maximally aligned blocks, *by definition*:

The pointer returned if the allocation succeeds
is suitably aligned so that it may be assigned to a pointer to
any type of object and then used to access such an object or an
^^^^^^^^^^^^^^^ ^^^
array of such objects in the space allocated (until the space
is explicitly deallocated).

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Dan.Pop@ifh.de

[CBFalconer]

Re: alligned memory allocation

Dan Pop wrote:[color=blue]
> CBFalconer <cbfalconer@yah oo.com> writes:
>[color=green]
>> You yourself pointed out a reason for it - to provide alignment to
>> larger values than does malloc.[/color]
>
> malloc provides maximally aligned blocks, *by definition*:
>
> The pointer returned if the allocation succeeds
> is suitably aligned so that it may be assigned to a pointer to
> any type of object and then used to access such an object or an
> ^^^^^^^^^^^^^^^ ^^^
> array of such objects in the space allocated (until the space
> is explicitly deallocated).[/color]

For the C language. There are entities in the world other than C
language objects, and sometimes C code is used to organize and
manipulate them.

--
Chuck F (cbfalconer@yah oo.com) (cbfalconer@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Dan Pop]

Re: alligned memory allocation

In <40486B7F.DEEEE BCF@yahoo.com> CBFalconer <cbfalconer@yah oo.com> writes:
[color=blue]
>Dan Pop wrote:[color=green]
>> CBFalconer <cbfalconer@yah oo.com> writes:
>>[color=darkred]
>>> You yourself pointed out a reason for it - to provide alignment to
>>> larger values than does malloc.[/color]
>>
>> malloc provides maximally aligned blocks, *by definition*:
>>
>> The pointer returned if the allocation succeeds
>> is suitably aligned so that it may be assigned to a pointer to
>> any type of object and then used to access such an object or an
>> ^^^^^^^^^^^^^^^ ^^^
>> array of such objects in the space allocated (until the space
>> is explicitly deallocated).[/color]
>
>For the C language. There are entities in the world other than C
>language objects, and sometimes C code is used to organize and
>manipulate them.[/color]

How is this possible, without treating them as C objects?

Dan
--
Dan Pop
DESY Zeuthen, RZ group
Email: Dan.Pop@ifh.de

[xarax]

Re: alligned memory allocation

"Dan Pop" <Dan.Pop@cern.c h> wrote in message
news:c2a98e$a5h $1@sunnews.cern .ch...[color=blue]
> In <40486B7F.DEEEE BCF@yahoo.com> CBFalconer <cbfalconer@yah oo.com> writes:
>[color=green]
> >Dan Pop wrote:[color=darkred]
> >> CBFalconer <cbfalconer@yah oo.com> writes:
> >>
> >>> You yourself pointed out a reason for it - to provide alignment to
> >>> larger values than does malloc.
> >>
> >> malloc provides maximally aligned blocks, *by definition*:
> >>
> >> The pointer returned if the allocation succeeds
> >> is suitably aligned so that it may be assigned to a pointer to
> >> any type of object and then used to access such an object or an
> >> ^^^^^^^^^^^^^^^ ^^^
> >> array of such objects in the space allocated (until the space
> >> is explicitly deallocated).[/color]
> >
> >For the C language. There are entities in the world other than C
> >language objects, and sometimes C code is used to organize and
> >manipulate them.[/color]
>
> How is this possible, without treating them as C objects?[/color]

Depends on the hardware on which the C program is
running, which means that the C code is not intended
to be portable to other platforms, yet still have
defined behavior and not use implementation-dependent
compiler features.

The IBM Mainframe has hardware structures of various
alignments, like 64-byte, 256-byte, 4096-byte,
1MB alignment, and larger. There is no C compiler
"object" that conforms to those alignments, yet
it's necessary to represent such requirements in
C code. That requires pointer<->integer conversions
and arithmetic that yield defined behavior. In such
cases, malloc() will not yield a properly aligned
"object". The usual approach is to "over allocate"
with malloc() and massage the pointer upward to an
appropriate alignment boundary, saving somewhere the
original pointer for a subsequent free().

A smart implementation of calloc() and malloc()
would examine the length to determine a plausible
maximum alignment and attempt to return a pointer
that conforms to the plausible max alignment, but
that is not a requirement. They only need to conform
to maximum alignment of a primitive C object (like
a "long long" or a "double", whatever is largest in
the implementation) .


--
----------------------------
Jeffrey D. Smith
Farsight Systems Corporation
24 BURLINGTON DRIVE
LONGMONT, CO 80501-6906

http://www.farsight-systems.com

z/Debug debugs your Systems/C programs running on IBM z/OS!
Are ISV upgrade fees too high? Check our custom product development!

[CBFalconer]

Re: alligned memory allocation

Dan Pop wrote:[color=blue]
> CBFalconer <cbfalconer@yah oo.com> writes:[color=green]
>> Dan Pop wrote:[color=darkred]
>>> CBFalconer <cbfalconer@yah oo.com> writes:
>>>
>>>> You yourself pointed out a reason for it - to provide alignment to
>>>> larger values than does malloc.
>>>
>>> malloc provides maximally aligned blocks, *by definition*:
>>>
>>> The pointer returned if the allocation succeeds
>>> is suitably aligned so that it may be assigned to a pointer to
>>> any type of object and then used to access such an object or an
>>> ^^^^^^^^^^^^^^^ ^^^
>>> array of such objects in the space allocated (until the space
>>> is explicitly deallocated).[/color]
>>
>> For the C language. There are entities in the world other than C
>> language objects, and sometimes C code is used to organize and
>> manipulate them.[/color]
>
> How is this possible, without treating them as C objects?[/color]

By taking advantage of system specific knowledge, and accepting
the non-portability of the result. Your system implementation of
malloc, free, and realloc is almost certainly a case in point. My
implementation of the same for DJGPP is certainly such. You can
find it on my site if curious.

--
Chuck F (cbfalconer@yah oo.com) (cbfalconer@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

[Chris Torek]

Re: alligned memory allocation

>>> CBFalconer <cbfalconer@yah oo.com> writes:[color=blue][color=green][color=darkred]
>>>> You yourself pointed out a reason for it - to provide alignment to
>>>> larger values than does malloc.[/color][/color][/color]
[color=blue][color=green]
>>Dan Pop wrote:[color=darkred]
>>> malloc provides maximally aligned blocks, *by definition*:[/color][/color][/color]
[snippage]
[color=blue]
>In <40486B7F.DEEEE BCF@yahoo.com> CBFalconer <cbfalconer@yah oo.com> writes:[color=green]
>>For the C language. There are entities in the world other than C
>>language objects, and sometimes C code is used to organize and
>>manipulate them.[/color][/color]

In article <news:c2a98e$a5 h$1@sunnews.cer n.ch>
Dan Pop <Dan.Pop@cern.c h> writes:[color=blue]
>How is this possible, without treating them as C objects?[/color]

By faking it, of course.

Typically one might build a struct or array that (by chance *and*
purpose both) corresponds to the hardware's requirements, then use
non-portable trickery to cause the existing mechanisms to produce
values that work with this.

The result is not guaranteed by the language, certainly, and often
even the compiler-vendor may not guarantee it (other than internally
if the vendor is also in the hardware or hardware-support business).
But if it works, and is more maintainable than (e.g.) writing the
entire thing in assembly, that may be enough to cause one to use it.

The fact that a "struct page_table" (for instance) can only be
allocated by the special "page table allocator", and:

struct page_table tmp;
... use &tmp ...

fails miserably, is relevant to someone who thinks the system is
written in ANSI/ISO C, but not so important one who knows it is
much more restricted than that. In other words, these non-C-objects
*sometimes* work like real C objects, and sometimes do not, and we
(I can speak for more than just myself here :-) ) pay this price
willingly for the value returned -- namely, some measure of
convenience. When it stops working well, we change the code
(and/or the compiler, in those cases where we control it).
--
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W) +1 801 277 2603
email: forget about it http://web.torek.net/torek/index.html
Reading email is like searching for food in the garbage, thanks to spammers.