Expanding buffer in C

[James Harris]

Do you remember a previous discussion on this newsgroup as at

http://groups.google.com/group/comp.lang.c/browse_frm/thread/cb502ce8414cea06/bb738c4f91113b4e

It concerned code to manage a buffer, expanding it as needed semi
automatically. I've incorporated suggestions people made and published
the code at

http://codewiki.wikispaces.com/xbuf.c

(For the impatient check the last two links in the table of contents
for the code. The rest is documentation.)

The code is intended to be fast. It's uses include reading an
arbitrary-length line from an input stream - a safe gets() replacement
if you like - but it is more flexible than line reading functions and
is intended to be useful in more cases than just line reading.

--NB. Code and documentation are on a wiki. If you see things that
should be changed feel free to change them.

Apart from the code itself what do you think of the concept of
publishing it on a wiki...?

James

[Gene]

Re: Expanding buffer in C

On Nov 15, 8:12 pm, James Harris <james.harri... @googlemail.com >
wrote:

Do you remember a previous discussion on this newsgroup as at
>
 http://groups.google.com/group/comp....hread/cb502ce8....
>
It concerned code to manage a buffer, expanding it as needed semi
automatically. I've incorporated suggestions people made and published
the code at
>
 http://codewiki.wikispaces.com/xbuf.c
>
(For the impatient check the last two links in the table of contents
for the code. The rest is documentation.)
>
The code is intended to be fast. It's uses include reading an
arbitrary-length line from an input stream - a safe gets() replacement
if you like - but it is more flexible than line reading functions and
is intended to be useful in more cases than just line reading.
>
--NB. Code and documentation are on a wiki. If you see things that
should be changed feel free to change them.
>
Apart from the code itself what do you think of the concept of
publishing it on a wiki...?
>
James

Nice presentation. You'll probably get less heap fragmentation for
programs that use lots of buffers if all of them are at set standard
sizes. So your new_size computation would become something like

new_size = current_size;
while (new_size < requested_size)
new_size = new_size * FACTOR;

NB. Your choice of default 3 / 2 for FACTOR is interesting. I've
been using this for many years because the standard 2 seemed
extravagant.

[James Harris]

Re: Expanding buffer in C - how choose each size increase?

On 18 Nov, 17:53, Gene <gene.ress...@g mail.comwrote:


....

Is there a better way to increase the buffer size? Especially as the
address space fills up should the increase factor be reduced, and how?

>

Anyone been down this road?

>
Garbage collectors deal with this when they need to increase the pool
size when not much VM is left. There isn't a single best policy
because behavior depends heavily on the OS.

Agreed. The more I think about this the less I think there is any kind
of one-size-fits-all solution. One advantage of distributing as source
code is that people can vary the code to suit their situation. I've
kept the original code as it was but have documented some alternatives
as comments within the code.

http://codewiki.wikispaces.com/xbuf.c

The commented alternatives are purely illustrative and are untested. I
hope I've got the C syntax etc correct. Either way they should be
enough to illustrate the options to anyone using the code. If you,
dear reader, see an error feel free to fix it or let me know what I've
got wrong and I'll fix it.

James

[CBFalconer]

Re: Expanding buffer in C - how choose each size increase?

James Harris wrote:

Gene <gene.ress...@g mail.comwrote:
>

.... snip ...

>

>Garbage collectors deal with this when they need to increase the
>pool size when not much VM is left. There isn't a single best
>policy because behavior depends heavily on the OS.

>
Agreed. The more I think about this the less I think there is any
kind of one-size-fits-all solution. One advantage of distributing
as source code is that people can vary the code to suit their
situation. I've kept the original code as it was but have
documented some alternatives as comments within the code.

In general you should adjust the mechanism to fit the expected
use. As an example, my ggets routine expands the buffer in
constant increments (of 128), because its primary use is expected
to be interactive input, and those strings are not normally
unlimited. However my hashlib expands its table by a factor of two
each time, resulting in constant average overhead, and basically
eliminating consideration of table size. In each case there are
possible penalties.

--
[mail]: Chuck F (cbfalconer at maineline dot net)
[page]: <http://cbfalconer.home .att.net>
Try the download section.