xmalloc

[Keith Thompson]

Re: xmalloc

rpbg123@yahoo.c om (Roland Pibinger) writes:
[...]

The discussion basically boils down to one question: Is OOM an error
that reasonably can and should be handled by the application or is it
a fatal error?
The 'fatal error' advocates have already shown how they tackle the
problem. Now it's time for the other camp to demonstrate how OOM can
be consistently _handled_ throughout the program ('return NULL;' is
not enough).

Who says it can be handled *consistently*?

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Malcolm McLean]

Re: xmalloc


"Keith Thompson" <kst-u@mib.orgwrote in message
news:lnfy4hd4pu .fsf@nuthaus.mi b.org...

rpbg123@yahoo.c om (Roland Pibinger) writes:
[...]

>The discussion basically boils down to one question: Is OOM an error
>that reasonably can and should be handled by the application or is it
>a fatal error?
>The 'fatal error' advocates have already shown how they tackle the
>problem. Now it's time for the other camp to demonstrate how OOM can
>be consistently _handled_ throughout the program ('return NULL;' is
>not enough).

>
Who says it can be handled *consistently*?
>

That is part of the issue.
malloc() return NULL. The if condition following the malloc() is like a
yippy dog chasing a car. Now it's got an error, what is it going to do with
it?


--
Free games and programming goodies.

http://www.personal.leeds.ac.uk/~bgy1mm

[Eric Sosman]

Re: xmalloc

Roland Pibinger wrote:

On Sat, 23 Jun 2007 16:30:12 -0400, Eric Sosman wrote:

> Not all that strange. Half the programmers in the world
>are below-average.

>
which is at least half-true.
>

> But if you follow Malcolm's Maxims, even the above-average
>programmers will be helpless. Programs that deal with malloc()
>failure in a significant way may be a minority, but those that
>do it *need* to to do it. Malcolm argues, in essence, that
>such programs should not be written in C.

>
The discussion basically boils down to one question: Is OOM an error
that reasonably can and should be handled by the application or is it
a fatal error?
The 'fatal error' advocates have already shown how they tackle the
problem. Now it's time for the other camp to demonstrate how OOM can
be consistently _handled_ throughout the program ('return NULL;' is
not enough).

Already mentioned a few times in this thread:

buff = malloc(image_si ze);
if (buff == NULL) {
fprintf (stderr, "Image too large (%lu) to paste\n",
(unsigned long)image_size );
return;
}
/* read image into buff, insert in current document */

Here's another I think has been referred to:

must_have_mem = malloc(how_much );
if (must_have_mem == NULL) {
fprintf (stderr, "Out of memory; shutting down\n");
save_snapshot(s napshot_file);
exit (EXIT_FAILURE);
}
/* store "must have" data in allocated memory */

And here's still another (I don't remember whether it's
cropped up in this thread yet):

void *getmem(size_t bytes) {
void *new = malloc(bytes);
if (new == NULL && bytes 0) {
fprintf (stderr, "Failed to allocate %lu bytes\n",
(unsigned long)bytes);
free (emergency_stas h);
emergency_stash = NULL;
new = malloc(bytes);
if (new == NULL) {
fprintf (stderr, "You were warned ...!\n");
exit (EXIT_FAILURE);
}
fprintf (stderr, "Running on fumes: save your work "
"and exit soon!\n");
}
return NULL;
}

Even if out-of-memory is a "fatal error," it does not follow
that the program should have no opportunity to "die with dignity."
Have you made a will, Roland? If so, should the fact that many
people die intestate invalidate your will? If not, I certainly
don't want your intestacy to invalidate my will!

--
Eric Sosman
esosman@acm-dot-org.invalid

[Eric Sosman]

Re: xmalloc

Malcolm McLean wrote:

>
"Roland Pibinger" <rpbg123@yahoo. comwrote in message
news:467d734b.3 793825@news.uta net.at...

>On Sat, 23 Jun 2007 18:47:44 +0000, Richard Heathfield wrote:

>>The second problem is also quite easy to fix, and it's this - the
>>function calls exit(), thus taking the decision to quit the program out
>>of the hands of the programmer. Therefore, the programmer won't call
>>this function himself, and won't be able to call any other function in
>>the library that /does/ call it.

>>
>You can often see programs that check the return value of malloc but
>you hardly (practically never) see any code that actually handles the
>OOM (out-of-memory) condition (e.g. frees some pre-allocated memory).
>Strange, isn't it?
>Moreover, C libraries usually crash when the caller provides
>insufficient resources or invalid arguments. In that sense abort() is
>preferable to exit() for OOM.
>>

Normally if you've written the program in such a way that it makes a lot
of trival allocations, there is nothing you can do on failure, short of
calling a back-up routine that uses fixed buffers.
So you handle fail conditions for the few large legitimate allocations,
which might run the computer out of memory, and abort on the rest.

That is so bass-ackwards it boggles the mind.

Here's the question: Does your program have in-memory state
that is valuable? If so, the proximate cause of the program's
trouble is irrelevant; what matters is the preservation of that
valuable state.

You can declare a global "world" pointer, and then in atexit try to save
the state of the program, but there are two problems.

... one being that abort doesn't run atexit callbacks ...

Firstly if the OS
won't give you a few hundred bytes, it might well not open files for you
either. Secondly, all the code has to be very carefully written so that
an allocation failure never causes the state to lose coherence.
Otherwise you will crash when you try to write a null pointer, or worse,
write an image that looks right but is in fact subtly corrupted.
>
So there really isn't an easy answer.

Let's avoid all hard problems; they demoralize the weak.

So if you use recursive subroutines you've already got one potential
source of crashes on malicious or excessive input. That reduces the
motivation to fix malloc(), because you can't guarantee correct
operation anyway.

And the janitor might unplug the computer to plug in the
floor polisher, or the fire sprinklers might open up and flood
the motherboard, or a meteorite might smash the computer to
tiny bits, or the Borg might teleport it away for assimilation.
Does the existence of Failure Mode A dissuade you from taking
any precautions against Failure Mode B? "I don't wear seat
belts in cars, because avian flu might kill me anyhow." Pfui!
--
Eric Sosman
esosman@acm-dot-org.invalid

[SM Ryan]

Re: xmalloc

rpbg123@yahoo.c om (Roland Pibinger) wrote:
# On Sat, 23 Jun 2007 16:30:12 -0400, Eric Sosman wrote:
# Not all that strange. Half the programmers in the world
# >are below-average.
#
# which is at least half-true.

Presumes a normal distribution. While many processes show normal
distribution of results, many processes also do not. Human
populations which are deliberately selected for some trait can
be signficantly skewed.

Hidden assumptions--the claymore mines of programming.

--
SM Ryan http://www.rawbw.com/~wyrmwif/
I love the smell of commerce in the morning.

[CBFalconer]

Re: xmalloc

Malcolm McLean wrote:

"Keith Thompson" <kst-u@mib.orgwrote in message

>rpbg123@yahoo.c om (Roland Pibinger) writes:
>>
>[...]

>>The discussion basically boils down to one question: Is OOM an
>>error that reasonably can and should be handled by the
>>application or is it a fatal error? The 'fatal error' advocates
>>have already shown how they tackle the problem. Now it's time
>>for the other camp to demonstrate how OOM can be consistently
>>_handled_ throughout the program ('return NULL;' is not enough).

>>
>Who says it can be handled *consistently*?

>
That is part of the issue. malloc() return NULL. The if condition
following the malloc() is like a yippy dog chasing a car. Now it's
got an error, what is it going to do with it?

The answer is simple. The model for calling malloc is:

if (!(p = malloc(N * sizeof *p))) callerrorrecove ryetc();
else { /* all is well */
....
}

--
<http://www.cs.auckland .ac.nz/~pgut001/pubs/vista_cost.txt>
<http://www.securityfoc us.com/columnists/423>
<http://www.aaxnet.com/editor/edit043.html>
cbfalconer at maineline dot net



--
Posted via a free Usenet account from http://www.teranews.com

[Malcolm McLean]

Re: xmalloc


"Eric Sosman" <esosman@acm-dot-org.invalidwrot e in message

>

>So if you use recursive subroutines you've already got one potential
>source of crashes on malicious or excessive input. That reduces the
>motivation to fix malloc(), because you can't guarantee correct operation
>anyway.

>
And the janitor might unplug the computer to plug in the
floor polisher, or the fire sprinklers might open up and flood
the motherboard, or a meteorite might smash the computer to
tiny bits, or the Borg might teleport it away for assimilation.
Does the existence of Failure Mode A dissuade you from taking
any precautions against Failure Mode B? "I don't wear seat
belts in cars, because avian flu might kill me anyhow." Pfui!
>

Yes. That's actually a fundamental observation.

Let's say we've got a failure rate of 10% per annum. By eliminating cause B,
we can reduce that to 9.9%. Unfortunately the cost of the product goes up
10% as result. Almost certainly it is not worth it.
Now imagine the failure rate is 1% per annnum. By eliminating cause B, we
can reduce that to 0.9%. Again, cost rises by 10%.
In the final scenario, failure rate is 0.1% per annum. By eliminating cause
B we can reduc that to 0.01% per annum, again for the same price.

In the middle scenario eliminating feature B is quite attractive. In the
last one, almost certainly you will want to do it, because the you can claim
a nearly failure-free product. The only exception would be if customers care
about price and not about quality at all - a T-shirt with a slogan on it,
for instance.

The value of eliminating an error goes up as it accounts for a greater
proportion of overall errors.

--
Free games and programming goodies.

http://www.personal.leeds.ac.uk/~bgy1mm

[Roland Pibinger]

Re: xmalloc

On Sun, 24 Jun 2007 09:12:06 -0400, Eric Sosman wrote:

Already mentioned a few times in this thread:
>
> buff = malloc(image_si ze);
> if (buff == NULL) {
> fprintf (stderr, "Image too large (%lu) to paste\n",
> (unsigned long)image_size );
> return;
> }
> /* read image into buff, insert in current document */
>
Here's another I think has been referred to:
>
> must_have_mem = malloc(how_much );
> if (must_have_mem == NULL) {
> fprintf (stderr, "Out of memory; shutting down\n");
> save_snapshot(s napshot_file);
> exit (EXIT_FAILURE);
> }
> /* store "must have" data in allocated memory */

The above approachs do not handle OOM ('handle' in the meaning of
'remedy'). They pass the problem to the caller or exit the program.
The second approach would soon be refactored into a malloc_or_exit( )
function.

And here's still another (I don't remember whether it's
>cropped up in this thread yet):
>
> void *getmem(size_t bytes) {
> void *new = malloc(bytes);
> if (new == NULL && bytes 0) {
> fprintf (stderr, "Failed to allocate %lu bytes\n",
> (unsigned long)bytes);
> free (emergency_stas h);
> emergency_stash = NULL;
> new = malloc(bytes);
> if (new == NULL) {
> fprintf (stderr, "You were warned ...!\n");
> exit (EXIT_FAILURE);
> }
> fprintf (stderr, "Running on fumes: save your work "
> "and exit soon!\n");
> }
> return NULL;
> }

This actually is an attempt to cope with OOM. But, as demonstrated,
the possibilities are very limited. You cannot do much when the
primary source of your program is exhausted.

Even if out-of-memory is a "fatal error," it does not follow
>that the program should have no opportunity to "die with dignity."
>Have you made a will, Roland?

I consider to use

void *xmalloc (size_t bytes) {
return getmem (bytes);
}

which, IMHO, elegently combines the best of all approaches posted so
far in this thread ;-)

In general I prefer: 'Repair what you can — but when you must fail,
fail noisily and as soon as possible'
(http://www.catb.org/~esr/writings/ta...html#id2878538)


--
Roland Pibinger
"Software development has been, is, and will remain fundamentally hard" - Grady Booch

[Keith Thompson]

Re: xmalloc

"Malcolm McLean" <regniztar@btin ternet.comwrite s:

"Keith Thompson" <kst-u@mib.orgwrote in message
news:lnfy4hd4pu .fsf@nuthaus.mi b.org...

>rpbg123@yahoo.c om (Roland Pibinger) writes:
>[...]

>>The discussion basically boils down to one question: Is OOM an error
>>that reasonably can and should be handled by the application or is it
>>a fatal error?
>>The 'fatal error' advocates have already shown how they tackle the
>>problem. Now it's time for the other camp to demonstrate how OOM can
>>be consistently _handled_ throughout the program ('return NULL;' is
>>not enough).

>>
>Who says it can be handled *consistently*?
>>

That is part of the issue.
malloc() return NULL. The if condition following the malloc() is like
a yippy dog chasing a car. Now it's got an error, what is it going to
do with it?

The "yippy dog chasing a car" analogy applies equally well when
malloc() returns a non-null result. Now you've got a valid pointer to
allocated memory; what are you going to do with it?

The answer is: do whatever makes sense in the context of the program.

Determining the answers to such questions is called "programmin g";
it's a non-trivial activity.

--
Keith Thompson (The_Other_Keit h) kst-u@mib.org <http://www.ghoti.net/~kst>
San Diego Supercomputer Center <* <http://users.sdsc.edu/~kst>
"We must do something. This is something. Therefore, we must do this."
-- Antony Jay and Jonathan Lynn, "Yes Minister"

[Eric Sosman]

Re: xmalloc

Roland Pibinger wrote:

On Sun, 24 Jun 2007 09:12:06 -0400, Eric Sosman wrote:

> Already mentioned a few times in this thread:
>>
> buff = malloc(image_si ze);
> if (buff == NULL) {
> fprintf (stderr, "Image too large (%lu) to paste\n",
> (unsigned long)image_size );
> return;
> }
> /* read image into buff, insert in current document */
>>
> Here's another I think has been referred to:
>>
> must_have_mem = malloc(how_much );
> if (must_have_mem == NULL) {
> fprintf (stderr, "Out of memory; shutting down\n");
> save_snapshot(s napshot_file);
> exit (EXIT_FAILURE);
> }
> /* store "must have" data in allocated memory */

>
The above approachs do not handle OOM ('handle' in the meaning of
'remedy'). They pass the problem to the caller or exit the program.
The second approach would soon be refactored into a malloc_or_exit( )
function.

Where did "remedy" creep into the picture? Both of the
above sketches acknowledge that the program is unable to
carry out some operation in the absence of sufficient memory,
but both "handle" the lack by taking a substitute action.
The first might be found in something like an interactive
editor, where the user gets a chance to say "Forty gig? Oh,
right, that's the original super-high-resolution file, not
the low-res image I actually intended to use." The second
does something that seems to have escaped your notice, to
wit, it calls save_snapshot() to preserve the valuable state
before pulling the plug. Neither strategy could be implemented
if malloc() just decided to blow up the program on its own whim.

> And here's still another (I don't remember whether it's
>cropped up in this thread yet):
>>
> void *getmem(size_t bytes) {
> void *new = malloc(bytes);
> if (new == NULL && bytes 0) {
> fprintf (stderr, "Failed to allocate %lu bytes\n",
> (unsigned long)bytes);
> free (emergency_stas h);
> emergency_stash = NULL;
> new = malloc(bytes);
> if (new == NULL) {
> fprintf (stderr, "You were warned ...!\n");
> exit (EXIT_FAILURE);
> }
> fprintf (stderr, "Running on fumes: save your work "
> "and exit soon!\n");
> }
> return NULL;
> }

>
This actually is an attempt to cope with OOM. But, as demonstrated,
the possibilities are very limited. You cannot do much when the
primary source of your program is exhausted.

How much you can do depends on how much you have made
provision for. That in turn depends on your analysis of how
much memory the program requires for "last gasp" operations.
At a PPOE this feature of the program received regular Q/A:
a whole bunch of state was loaded into memory, and then the
test harness induced artificial malloc failures; if the
program's emergency stash wasn't enough to sustain the last
gasp, a bug got filed.

> Even if out-of-memory is a "fatal error," it does not follow
>that the program should have no opportunity to "die with dignity."
>Have you made a will, Roland?

>
I consider to use
>
void *xmalloc (size_t bytes) {
return getmem (bytes);
}
>
which, IMHO, elegently combines the best of all approaches posted so
far in this thread ;-)
>
In general I prefer: 'Repair what you can — but when you must fail,
fail noisily and as soon as possible'
(http://www.catb.org/~esr/writings/ta...html#id2878538)

"When you must fail" is not blanket advice to give up
the ghost at the first sign of adversity. You might also
note that all three of the sketches I provided made noise.

--
Eric Sosman
esosman@acm-dot-org.invalid

[Nick Keighley]

Re: xmalloc

On 23 Jun, 20:06, "Malcolm McLean" <regniz...@btin ternet.comwrote :

"Richard Heathfield" <r...@see.sig.i nvalidwrote in message
news:kbednX2M0N uh9-DbnZ2dnUVZ8qOtn Z2d@bt.com...

Malcolm McLean said:

<snip>

In which case would call regular malloc()
However you are doubling the cost of the software with that strategy, all
because a 2GB machine might refuse to give a few hundred bytes of memory. If
it is running a life-support machine, fair enough, but most software
doesn't.
>
On the other hand you could argue that xmalloc() is a bad idea, because no
library that uses it could ever find its way into a life-support machine.
On the other hand, you could argue that the life support machine is more
likely to blow a fuse than to exit in xmalloc(), so you need a backup
anyway.

I assume "life-support machine" is a meta syntactic variable for
safety/mission critical software.

Just how important does a program have to be before its given
LSM status and doesn't crash without doing *some* error handling?

- actual life support machine
- anti-lock brakes
- communications equipment used by emergency services
- communications equipment used to report aircraft movements
- your bank account
- your word-processor
- your central heating system
- the game you've been playing for a week
- unix filter
- post to clc


--
Nick Keighley

[Christopher Benson-Manica]

Re: xmalloc

Ian Collins <ian-news@hotmail.co mwrote:

It could be enhanced by providing a (cough) global failure function, say
xmalloc_failure , that gets called when malloc returns NULL:

void* (*xmalloc_failu re)(size_t) = &default_xmallo c_failure;

Why make it a global function?

static void* (*xmalloc_failu re)( size_t );

void register_xmallo c_failfunc( void* (*failfunc)(siz e_t) ) {
xmalloc_failure = failfunc;
}

There's no need to expose xmalloc_failure or default_xmalloc _failure
to client code, and I would personally prefer not to.

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gma il.com | don't, I need to know. Flames welcome.

[Christopher Benson-Manica]

Re: xmalloc

Malcolm McLean <regniztar@btin ternet.comwrote :

void *xmalloc(int sz)
{

assert(sz >= 0);
if(sz == 0)
sz = 1;

All right, I give up. Why should xmalloc() quietly prohibit zero-byte
allocations? If a client asks for zero bytes, why not just *give* it
zero bytes? For that matter, given the above code, what is wrong with

if( sz <= 0 ) {
sz=1;
}

? You seem to be assuming that xmalloc()'s clients aren't very bright
anyway.

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gma il.com | don't, I need to know. Flames welcome.

[Christopher Benson-Manica]

Re: xmalloc

Harald van D?k <truedfx@gmail. comwrote:

malloc(0) can make sense, but is allowed by the standard to either return
NULL or non-NULL on success. Because of this, xmalloc(0) calls malloc(1) so
that a successfull allocation will always return non-NULL.

I take about half of this point. While the code as written did indeed
need to distinguish between failure to allocate non-zero bytes, and
successful allocation of zero bytes (a fact which I missed), it's not
clear to me that xmalloc()'s clients gain anything by essentially
having all calls to malloc(0) replaced by malloc(1). The pointer
returned by malloc(0), if not NULL, can't be used to access an object
anyway, so from xmalloc()'s clients' perspectives, it doesn't much
matter whether xmalloc(0) returns NULL or not:

void *xmalloc( size_t sz ) {
if( sz == 0 ) {
return NULL;
}
/* ... */
}

May as well save the call to malloc(0), if you ask me.

--
C. Benson Manica | I *should* know what I'm talking about - if I
cbmanica(at)gma il.com | don't, I need to know. Flames welcome.

[Harald van =?UTF-8?B?RMSzaw==?=]

Re: xmalloc

Christopher Benson-Manica wrote:

Harald van D?k <truedfx@gmail. comwrote:
>

>malloc(0) can make sense, but is allowed by the standard to either return
>NULL or non-NULL on success. Because of this, xmalloc(0) calls malloc(1)
>so that a successfull allocation will always return non-NULL.

>
I take about half of this point. While the code as written did indeed
need to distinguish between failure to allocate non-zero bytes, and
successful allocation of zero bytes (a fact which I missed), it's not
clear to me that xmalloc()'s clients gain anything by essentially
having all calls to malloc(0) replaced by malloc(1). The pointer
returned by malloc(0), if not NULL, can't be used to access an object
anyway, so from xmalloc()'s clients' perspectives, it doesn't much
matter whether xmalloc(0) returns NULL or not:
>
void *xmalloc( size_t sz ) {
if( sz == 0 ) {
return NULL;
}
/* ... */
}
>
May as well save the call to malloc(0), if you ask me.

It depends on the project. Sometimes it matters to have different calls to
malloc(0) return values that do not compare equal, sometimes it doesn't.