May the size argument of operator new overflow?

[Angel Tsankov]

Hello!

Does the C++ standard define what happens when the size argument of void*
operator new(size_t size) cannot represent the total number of bytes to be
allocated? For example:

struct S
{
char a[64];
};

S* allocate(int size)
{
return new S[size]; // What happens here?
}

int main()
{
allocate(0x7FFF FFFF);
}

[Ian Collins]

Re: May the size argument of operator new overflow?

Angel Tsankov wrote:

Hello!
>
Does the C++ standard define what happens when the size argument of void*
operator new(size_t size) cannot represent the total number of bytes to be
allocated? For example:
>

size_t will always be wide enough to represent the maximum memory range
on a given system.

If the system can't supply the requested size, new throws std::bad_alloc.

--
Ian Collins.

[joseph cook]

Re: May the size argument of operator new overflow?

On Jun 18, 5:44 am, "Angel Tsankov" <fn42...@fmi.un i-sofia.bgwrote:

Hello!
>
Does the C++ standard define what happens when the size argument of void*
operator new(size_t size) cannot represent the total number of bytes to be
allocated? For example:

Yes. You cannot exceed numeric_limits< size_t>::max(). The same is
true for array size.

[Angel Tsankov]

Re: May the size argument of operator new overflow?

Hello!

>
Does the C++ standard define what happens when the size argument of void*
operator new(size_t size) cannot represent the total number of bytes to be
allocated? For example:

Yes. You cannot exceed numeric_limits< size_t>::max(). The same is
true for array size.

OK, but what happens in the example that you have cut off?

[Angel Tsankov]

Re: May the size argument of operator new overflow?

>Hello!

>>
>Does the C++ standard define what happens when the size argument of void*
>operator new(size_t size) cannot represent the total number of bytes to
>be
>allocated? For example:
>>

size_t will always be wide enough to represent the maximum memory range
on a given system.
>
If the system can't supply the requested size, new throws std::bad_alloc.

This is not an answer to the question what happens in the example you have
cut off.

[Daniel T.]

Re: May the size argument of operator new overflow?

On Jun 18, 8:44 am, "Angel Tsankov" <fn42...@fmi.un i-sofia.bgwrote:

If the system can't supply the requested size, new throws std::bad_alloc.

>
This is not an answer to the question what happens in the example you have
cut off.

Either the system will supply the requested size, or std::bad_alloc
will be thrown. That is what happens in the example that was cut off.

[James Kanze]

Re: May the size argument of operator new overflow?

On Jun 18, 11:44 am, "Angel Tsankov" <fn42...@fmi.un i-sofia.bgwrote:

Does the C++ standard define what happens when the size
argument of void* operator new(size_t size) cannot represent
the total number of bytes to be allocated?

For example:

struct S
{
char a[64];
};

S* allocate(int size)
{
return new S[size]; // What happens here?
}

int main()
{
allocate(0x7FFF FFFF);
}

Supposing that all values in an int can be represented in a
size_t (i.e. that size_t is unsigned int or larger---very, very
probably), then you should either get the memory, or get a
bad_alloc exception (which you don't catch). That's according
to the standard; a lot of implementations seem to have bugs
here.

--
James Kanze (GABI Software) email:james.kan ze@gmail.com
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34

[Kai-Uwe Bux]

Re: May the size argument of operator new overflow?

James Kanze wrote:

On Jun 18, 11:44 am, "Angel Tsankov" <fn42...@fmi.un i-sofia.bgwrote:

>Does the C++ standard define what happens when the size
>argument of void* operator new(size_t size) cannot represent
>the total number of bytes to be allocated?

>

>For example:

>

>struct S
>{
> char a[64];
>};

>

>S* allocate(int size)
>{
> return new S[size]; // What happens here?
>}

>

>int main()
>{
> allocate(0x7FFF FFFF);
>}

>
Supposing that all values in an int can be represented in a
size_t (i.e. that size_t is unsigned int or larger---very, very
probably), then you should either get the memory, or get a
bad_alloc exception (which you don't catch). That's according
to the standard; a lot of implementations seem to have bugs
here.

I think, you are missing a twist that the OP has hidden within his posting:
the size of S is at least 64. The number of S objects that he requests is
close to numeric_limits< size_t>::max(). So when new S[size] is translated
into raw memory allocation, the number of bytes (not the number of S
objects) requested might exceed numeric_limits< size_t>::max().

I think (based on my understanding of [5.3.4/12]) that in such a case, the
unsigned arithmetic will just silently overflow and you end up allocating a
probably unexpected amount of memory.


Best

Kai-Uwe Bux

[Bo Persson]

Re: May the size argument of operator new overflow?

Kai-Uwe Bux wrote:

James Kanze wrote:
>

>On Jun 18, 11:44 am, "Angel Tsankov" <fn42...@fmi.un i-sofia.bg>
>wrote:

>>Does the C++ standard define what happens when the size
>>argument of void* operator new(size_t size) cannot represent
>>the total number of bytes to be allocated?

>>

>>For example:

>>

>>struct S
>>{
>> char a[64];
>>};

>>

>>S* allocate(int size)
>>{
>> return new S[size]; // What happens here?
>>}

>>

>>int main()
>>{
>> allocate(0x7FFF FFFF);
>>}

>>
>Supposing that all values in an int can be represented in a
>size_t (i.e. that size_t is unsigned int or larger---very, very
>probably), then you should either get the memory, or get a
>bad_alloc exception (which you don't catch). That's according
>to the standard; a lot of implementations seem to have bugs
>here.

>
I think, you are missing a twist that the OP has hidden within his
posting: the size of S is at least 64. The number of S objects that
he requests is close to numeric_limits< size_t>::max(). So when new
S[size] is translated into raw memory allocation, the number of
bytes (not the number of S objects) requested might exceed
numeric_limits< size_t>::max().
>
I think (based on my understanding of [5.3.4/12]) that in such a
case, the unsigned arithmetic will just silently overflow and you
end up allocating a probably unexpected amount of memory.

Here is what one compiler does - catch the overflow and wrap it back
to numeric_limits< size_t>::max().

int main()
{
allocate(0x7FFF FFFF);
00401000 xor ecx,ecx
00401002 mov eax,7FFFFFFFh
00401007 mov edx,40h
0040100C mul eax,edx
0040100E seto cl
00401011 neg ecx
00401013 or ecx,eax
00401015 push ecx
00401016 call operator new[] (401021h)
0040101B add esp,4
}
0040101E xor eax,eax
00401020 ret


Bo Persson

[Jerry Coffin]

Re: May the size argument of operator new overflow?

In article <g3alej$tb$1@ai oe.org>, fn42551@fmi.uni-sofia.bg says...

Hello!
>
Does the C++ standard define what happens when the size argument of void*
operator new(size_t size) cannot represent the total number of bytes to be
allocated? For example:
>
struct S
{
char a[64];
};
>
S* allocate(int size)
{
return new S[size]; // What happens here?
}
>
int main()
{
allocate(0x7FFF FFFF);
}

Chances are pretty good that at some point, you get something like:

void *block = ::new(0x7FFFFFF F*64);

On an implementation with a 32-bit size_t, that'll wraparound, and it'll
attempt to allocate 0xffffffc0 bytes instead of 0x1fffffffc0 bytes.
Chances are that allocation will immediately fail since that number is
_barely_ short of 4 gigabytes, and no 32-bit system I know of wiil have
that much contiguous address space available.

If, OTOH, you picked numbers where the wraparound produced a relatively
small number, chances are that the allocation would succeed, but when
you attempted to access what appeared to be successfully allocated
memory, you'd quickly go past the end of the real allocation, and get
undefined behavior.

--
Later,
Jerry.

The universe is a figment of its own imagination.

[Angel Tsankov]

Re: May the size argument of operator new overflow?

>>>Does the C++ standard define what happens when the size

>>>argument of void* operator new(size_t size) cannot represent
>>>the total number of bytes to be allocated?
>>>
>>>For example:
>>>
>>>struct S
>>>{
>>> char a[64];
>>>};
>>>
>>>S* allocate(int size)
>>>{
>>> return new S[size]; // What happens here?
>>>}
>>>
>>>int main()
>>>{
>>> allocate(0x7FFF FFFF);
>>>}
>>>
>>Supposing that all values in an int can be represented in a
>>size_t (i.e. that size_t is unsigned int or larger---very, very
>>probably), then you should either get the memory, or get a
>>bad_alloc exception (which you don't catch). That's according
>>to the standard; a lot of implementations seem to have bugs
>>here.

>>
>I think, you are missing a twist that the OP has hidden within his
>posting: the size of S is at least 64. The number of S objects that
>he requests is close to numeric_limits< size_t>::max(). So when new
>S[size] is translated into raw memory allocation, the number of
>bytes (not the number of S objects) requested might exceed
>numeric_limits <size_t>::max() .

Thanks for pointing this out, I though it would be obvious to everyone.
The following example might be a little bit less confusing:

struct S
{
char a[64]; // Any size greater than 1 would do.
};

S* allocate(std::s ize_t size)
{
return new S[size]; // How many bytes of memory must the new operator
allocate if size equals std::numeric_li mits<size_t>::m ax()?
}

>I think (based on my understanding of [5.3.4/12]) that in such a
>case, the unsigned arithmetic will just silently overflow and you
>end up allocating a probably unexpected amount of memory.

>
Here is what one compiler does - catch the overflow and wrap it back to
numeric_limits< size_t>::max().
>
int main()
{
allocate(0x7FFF FFFF);
00401000 xor ecx,ecx
00401002 mov eax,7FFFFFFFh
00401007 mov edx,40h
0040100C mul eax,edx
0040100E seto cl
00401011 neg ecx
00401013 or ecx,eax
00401015 push ecx
00401016 call operator new[] (401021h)
0040101B add esp,4
}
0040101E xor eax,eax
00401020 ret

Yes, the size requested is rounded to the maximum allocatable size, but is
this standard-compliant behavior? And if it is, how is client code notified
of the rounding?

[Ian Collins]

Re: May the size argument of operator new overflow?

Angel Tsankov wrote:

>>Hello!
>>>
>>Does the C++ standard define what happens when the size argument of void*
>>operator new(size_t size) cannot represent the total number of bytes to
>>be
>>allocated? For example:
>>>

>size_t will always be wide enough to represent the maximum memory range
>on a given system.
>>
>If the system can't supply the requested size, new throws std::bad_alloc.

>
This is not an answer to the question what happens in the example you have
cut off.
>

What more is there to say other than "If the system can't supply the
requested size, new throws std::bad_alloc" ? If the system had 32GB
free, new would succeed, otherwise it would fail.

--
Ian Collins.

[Ian Collins]

Re: May the size argument of operator new overflow?

Angel Tsankov wrote:

[please don't snip attributions]

Bo Persson wrote:

>Here is what one compiler does - catch the overflow and wrap it back to
>numeric_limits <size_t>::max() .
>>
>int main()
>{
>allocate(0x7FF FFFFF);
>00401000 xor ecx,ecx
>00401002 mov eax,7FFFFFFFh
>00401007 mov edx,40h
>0040100C mul eax,edx
>0040100E seto cl
>00401011 neg ecx
>00401013 or ecx,eax
>00401015 push ecx
>00401016 call operator new[] (401021h)
>0040101B add esp,4
>}
>0040101E xor eax,eax
>00401020 ret

>
Yes, the size requested is rounded to the maximum allocatable size, but is
this standard-compliant behavior? And if it is, how is client code notified
of the rounding?
>

Your question has nothing to do with operator new() and everything to do
with integer overflow.

The reason some of us answered the way we did is probably because we are
used to systems where sizeof(int) == 4 and sizeof(size_t) == 8, so your
original code would simply have requested 32GB, not a lot on some systems.

--
Ian Collins.

[Paavo Helde]

Re: May the size argument of operator new overflow?

Jerry Coffin <jcoffin@taeus. comkirjutas:

In article <g3alej$tb$1@ai oe.org>, fn42551@fmi.uni-sofia.bg says...

>Hello!
>>
>Does the C++ standard define what happens when the size argument of
>void* operator new(size_t size) cannot represent the total number of
>bytes to be allocated? For example:
>>
>struct S
>{
> char a[64];
>};
>>
>S* allocate(int size)
>{
> return new S[size]; // What happens here?
>}
>>
>int main()
>{
> allocate(0x7FFF FFFF);
>}

>
Chances are pretty good that at some point, you get something like:
>
void *block = ::new(0x7FFFFFF F*64);
>
On an implementation with a 32-bit size_t, that'll wraparound, and
it'll attempt to allocate 0xffffffc0 bytes instead of 0x1fffffffc0
bytes. Chances are that allocation will immediately fail since that
number is _barely_ short of 4 gigabytes, and no 32-bit system I know
of wiil have that much contiguous address space available.
>
If, OTOH, you picked numbers where the wraparound produced a
relatively small number, chances are that the allocation would
succeed, but when you attempted to access what appeared to be
successfully allocated memory, you'd quickly go past the end of the
real allocation, and get undefined behavior.

The standard says that for too large allocations std::bad_alloc must be
thrown. In the user code there is no unsigned arithmetic done, thus no
wraparound can occur. I would say that if the implementation does not
check for the overflow and silently wraps the result, the implementation
does not conform to the standard. It is irrelevant if the implementation
uses unsigned arithmetics inside, or e.g. double.

I have not studied the standard in detail, so this is just my opinion how
it should work.

Best,
Paavo

[Bo Persson]

Re: May the size argument of operator new overflow?

Angel Tsankov wrote:

>>>>Does the C++ standard define what happens when the size
>>>>argument of void* operator new(size_t size) cannot represent
>>>>the total number of bytes to be allocated?
>>>>
>>>>For example:
>>>>
>>>>struct S
>>>>{
>>>> char a[64];
>>>>};
>>>>
>>>>S* allocate(int size)
>>>>{
>>>> return new S[size]; // What happens here?
>>>>}
>>>>
>>>>int main()
>>>>{
>>>> allocate(0x7FFF FFFF);
>>>>}
>>>>
>>>Supposing that all values in an int can be represented in a
>>>size_t (i.e. that size_t is unsigned int or larger---very, very
>>>probably), then you should either get the memory, or get a
>>>bad_alloc exception (which you don't catch). That's according
>>>to the standard; a lot of implementations seem to have bugs
>>>here.
>>>
>>I think, you are missing a twist that the OP has hidden within his
>>posting: the size of S is at least 64. The number of S objects
>>that he requests is close to numeric_limits< size_t>::max(). So
>>when new S[size] is translated into raw memory allocation, the
>>number of bytes (not the number of S objects) requested might
>>exceed numeric_limits< size_t>::max().

>
Thanks for pointing this out, I though it would be obvious to
everyone. The following example might be a little bit less
confusing:
struct S
{
char a[64]; // Any size greater than 1 would do.
};
>
S* allocate(std::s ize_t size)
{
return new S[size]; // How many bytes of memory must the new
operator allocate if size equals std::numeric_li mits<size_t>::m ax()?
}
>

>>I think (based on my understanding of [5.3.4/12]) that in such a
>>case, the unsigned arithmetic will just silently overflow and you
>>end up allocating a probably unexpected amount of memory.

>>
>Here is what one compiler does - catch the overflow and wrap it
>back to numeric_limits< size_t>::max().
>>
>int main()
>{
>allocate(0x7FF FFFFF);
>00401000 xor ecx,ecx
>00401002 mov eax,7FFFFFFFh
>00401007 mov edx,40h
>0040100C mul eax,edx
>0040100E seto cl
>00401011 neg ecx
>00401013 or ecx,eax
>00401015 push ecx
>00401016 call operator new[] (401021h)
>0040101B add esp,4
>}
>0040101E xor eax,eax
>00401020 ret

>
Yes, the size requested is rounded to the maximum allocatable size,
but is this standard-compliant behavior? And if it is, how is
client code notified of the rounding?

Requesting a numeric_limits< size_t>::max() allocation size is pretty
much assured to fail with a std::bad_alloc exception.


Bo Persson