STL and over 2GB memory

[W Randolph Franklin]

Hi,

Does STL work on a 32-bit machine when a process's total memory is
over 2GB? E.g., a Pentium allows the process memory to be as
large as 3GB.

Ditto Boost multi arrays.

Are there any other C++ problems that I ought to know about? I've
actually bought half a dozen C++, STL, and FAQ books and don't
happen to remember this ever being mentioned.

Thanks.

[Peter van Merkerk]

Re: STL and over 2GB memory

> Does STL work on a 32-bit machine when a process's total memory is[color=blue]
> over 2GB? E.g., a Pentium allows the process memory to be as
> large as 3GB.[/color]

That question cannot be answered without knowing which implementation of
the standard library your are using. The ISO C++ standard says nothing
about the architecture or the organisation process memory of the target
platform.. Nor does the standard dictate how the standard library is to
be implemented. It is up to implementers of the standard library to
decide how to deal with the issues you mentioned. Maybe the
documentation that came with your standard library implementation can
provide you with an answer, or otherwise the developers of that library
may be able help you.
[color=blue]
> Ditto Boost multi arrays.
>
> Are there any other C++ problems that I ought to know about? I've
> actually bought half a dozen C++, STL, and FAQ books and don't
> happen to remember this ever being mentioned.[/color]

If they are about _standard_ C++ they can't because it is beyond the
scope of the language. It is an implementation issue.

--
Peter van Merkerk
peter.van.merke rk(at)dse.nl

[Mike Wahler]

Re: STL and over 2GB memory

W Randolph Franklin <wrf+usenet1102 @ecse.rpi.edu> wrote in message
news:rDUQa.4685 7$EQ5.25660@twi ster.nyroc.rr.c om...[color=blue]
> Hi,
>
> Does STL work on a 32-bit machine when a process's total memory is
> over 2GB?[/color]

If someone has created an implementation for that
machine, sure.
[color=blue]
>E.g., a Pentium allows the process memory to be as
> large as 3GB.[/color]

That's irrelevant to a platform-specific language
like C++.
[color=blue]
>
> Ditto Boost multi arrays.[/color]

Ask about boost stuff on their mailing list.
[color=blue]
>
> Are there any other C++ problems that I ought to know about?[/color]

None of the above issues are about C++, thus they're
not 'C++ problems'. The problem is your misconception.
[color=blue]
> I've
> actually bought half a dozen C++, STL, and FAQ books and don't
> happen to remember this ever being mentioned.[/color]

You never saw "things like this" discussed in a C++
book because C++ is a platform-indepenedent language,
which does not depend upon or specify such platform
details.

-Mike

[Sarah Thompson]

Re: STL and over 2GB memory



Mike Wahler wrote:[color=blue][color=green]
>> I've
>>actually bought half a dozen C++, STL, and FAQ books and don't
>>happen to remember this ever being mentioned.[/color]
>
>
> You never saw "things like this" discussed in a C++
> book because C++ is a platform-indepenedent language,
> which does not depend upon or specify such platform
> details.[/color]

This is an interesting point, worthy of expansion. C++ is indeed a
platform independent language, whose specification does a good job of
avoiding platform-specific language details. However, it is not correct
to state that C++ does not depend upon platform details. If one is
writing code with portability in mind, then striving for exactly this
kind of platform independence is a worthy goal. However, the semantics
of any C++ program are entirely dependent on the underlying architecture.

As a trivial example:

int main()
{
unsigned int x = 0xFFFFFFFF;
x++;

if(x == 0)
{
// 32 bit integers, probably.
return 0;
}
else
{
// Some other sized integers, probably.
return 1;
}
}

The above program is legal C++ (and for that matter legal C, but let's
stay on topic here). However, its runtime semantics critically depend
upon the details of its implementation. The language standard may have
nothing much to say about the word length of integers, but the above
program will behave completely differently depending on whether it is
compiled for a 32 bit or a 64 bit architecture.

C++, therefore, can not be regarded as truly platform independent, since
the semantics of quite simple code is entirely dependent on the details
of the implementation. With clever programming, it is often possible to
get around such limitations, but the language doesn't give you this for
free.

Arguably, if the C++ standard really did standardise data types, giving
them well-defined meanings (e.g. int8, unsigned16, ieeefloat, etc.) with
a fully specified semantics, this would make the language *more*
portable. The existing definition is sufficiently wooly that programming
for portability can often be extremely challenging.

Yes, this may be regarded as a little heretical, but I'm not trolling
here -- this is an important issue. I've written more than one C++
library with portability in mind (e.g. the signal/slot library on my web
site, as well as a new lexical analysis/parsing library that I'll
release sometime when I've finally gotten around to finishing its
documentation), and can't say I find C++ terribly helpful in this regard.

Sarah

--
----------------------------------------------
/ __ + / Sarah Thompson **** /
/ (_ _ _ _ |_ / * /
/ __)(_|| (_|| ) / sarah@telergy.c om * /
/ + / http://findatlantis.com/ /
----------------------------------------------

[Michiel Salters]

Re: STL and over 2GB memory

W Randolph Franklin <wrf+usenet1102 @ecse.rpi.edu> wrote in message news:<rDUQa.468 57$EQ5.25660@tw ister.nyroc.rr. com>...[color=blue]
> Hi,
>
> Does STL work on a 32-bit machine when a process's total memory is
> over 2GB? E.g., a Pentium allows the process memory to be as
> large as 3GB.[/color]

STL the concept, yes. Your STL implementation, may be. However if
it doesn't you can fix things by provinding your own allocator
capable of addressing 3GB. The allocator is the only part that
deals with memory allocations.

Regards,
--
Michiel Salters

[David Cattarin]

Re: STL and over 2GB memory

Sarah Thompson <sarah@telergy. remove.this.bit .com> wrote in message news:<bf22l5$9p 8$1$8300dec7@ne ws.demon.co.uk> ...[color=blue]
> Mike Wahler wrote:[color=green][color=darkred]
> >> I've
> >>actually bought half a dozen C++, STL, and FAQ books and don't
> >>happen to remember this ever being mentioned.[/color]
> >
> >
> > You never saw "things like this" discussed in a C++
> > book because C++ is a platform-indepenedent language,
> > which does not depend upon or specify such platform
> > details.[/color]
>
> This is an interesting point, worthy of expansion. C++ is indeed a
> platform independent language, whose specification does a good job of
> avoiding platform-specific language details. However, it is not correct
> to state that C++ does not depend upon platform details. If one is
> writing code with portability in mind, then striving for exactly this
> kind of platform independence is a worthy goal. However, the semantics
> of any C++ program are entirely dependent on the underlying architecture.[/color]
[snip]
[color=blue]
> C++, therefore, can not be regarded as truly platform independent,[/color]
[snip]

Actually, you mean a C++ program, the language is platform
independent.

[snip][color=blue]
> Arguably, if the C++ standard really did standardise data types, giving
> them well-defined meanings (e.g. int8, unsigned16, ieeefloat, etc.) with
> a fully specified semantics, this would make the language *more*
> portable. The existing definition is sufficiently wooly that programming
> for portability can often be extremely challenging.[/color]

Well, I think you are missing a key point. C++, and C, were defined to
be run on as many platforms and evironments as possible. That includes
embedded systems and systems where you could have bit sizes of 9, 12,
and even 40 bits. To avoid penalizing such systems, the standard
avoids mandating byte and type sizes; for example, whether a char
(byte) is 8 bits or not, and even whether it is signed or unsigned.
Also note that with such systems, mandating the sizes does not
necessarily simplify portability.

As a result, the programmer is responsible for determining how
portable a program is, which is as it should be. As for me, I can't
think of any program that I'd want to run on every conceivable
platform. But there are classes of platforms (32/64 bit UNIX and
WinNT) that I do want to program for. I make my design choices
accordingly.
[color=blue]
> Yes, this may be regarded as a little heretical, but I'm not trolling
> here -- this is an important issue. I've written more than one C++
> library with portability in mind (e.g. the signal/slot library on my web
> site, as well as a new lexical analysis/parsing library that I'll
> release sometime when I've finally gotten around to finishing its
> documentation), and can't say I find C++ terribly helpful in this regard.[/color]

Well, I agree with much of what you say, but IMO, C++ can run on more
platforms and is a truly platform independent language because it does
avoid "hardware" requirements.

Dave

[Ron Natalie]

Re: STL and over 2GB memory


"Michiel Salters" <Michiel.Salter s@cmg.nl> wrote in message news:cefd6cde.0 307160037.55040 a61@posting.goo gle.com...
[color=blue]
> STL the concept, yes. Your STL implementation, may be. However if
> it doesn't you can fix things by provinding your own allocator
> capable of addressing 3GB. The allocator is the only part that
> deals with memory allocations.[/color]

Well the other problem is making sure the size_type can hold numbers
that big. (By the way, pentiums have no problem with 4GB address
spaces, it's only windows that insists on the kernel and the user rings
sharing the 4Gigs that leads to the 2 or 3 gb limit).

[Ron Natalie]

Re: STL and over 2GB memory


"W Randolph Franklin" <wrf+usenet1102 @ecse.rpi.edu> wrote in message news:CAfRa.1231 75$8B.114411@tw ister.nyroc.rr. com...[color=blue]
> According to Ron Natalie <ron@sensor.com >:[color=green]
> >
> > "Michiel Salters" <Michiel.Salter s@cmg.nl> wrote in message
> > news:cefd6cde.0 307160037.55040 a61@posting.goo gle.com...
> >[color=darkred]
> > > STL the concept, yes. Your STL implementation, may be. However if
> > > it doesn't you can fix things by provinding your own allocator
> > > capable of addressing 3GB. The allocator is the only part that
> > > deals with memory allocations.[/color]
> >
> > Well the other problem is making sure the size_type can hold numbers
> > that big. (By the way, pentiums have no problem with 4GB address
> > spaces, it's only windows that insists on the kernel and the user rings
> > sharing the 4Gigs that leads to the 2 or 3 gb limit).[/color]
>
> No, linux has the same problem. My uninformed feeling is that it
> would be rather difficult for an operating system on a Pentium to
> let the user have more than 3GB of VM while still having any
> security.[/color]

I didn't say the problem was unique to WIN32. What does security
have to do with it? It's a performance/convenience thing to keep the
user space mapped while in kernel mode, but it is not required.
[color=blue]
> No one else has expressed a concern about something that had been
> worrying me, which is negative addresses if the pointers were
> signed ints. I could easily imagine that causing problems for
> sloppy code.[/color]

Pointers aren't ints at all. I fail to understand what difference that makes.
You're not likely to have an issue with pointers. What you're likely to
have issue with is the sizes of things (which will be some sort of integer).
Addresses don't get negative (at least not on any machine I've ever used).
There were a couple with the address breaks for kernel vs. user hard coded
into them but the Pentium isn't one of these.
[color=blue]
> Nevertheless, if the authors have that knowledge, including some
> references to a few specific widely used implementations , and
> including info about the tools' limits, would increase the books'
> usefulness, especially since that info is not to be obtained
> anywhere else.[/color]

What usefulness? What limitations? All implementations are going to
have limitations, but it's not part of the language. You're going to have to
consult the individual implementation. Just gecause G++ on win32 supports
one thing doesn't mean VC++ on the same platform will have the same limitations
etc...
[color=blue]
> BTW, I had thought that the goal of boost was to become part of
> standard C++, as STL basically is now, and that it has an overlap
> of personnel. Could someone correct me on this?[/color]

Some of boost has merit for inclusion, some of it doesn't. As far as I know
there is no intention by the standards committee to take boost as a whole.

[Sarah Thompson]

Re: STL and over 2GB memory



David Cattarin wrote:[color=blue][color=green]
>>Arguably, if the C++ standard really did standardise data types, giving
>>them well-defined meanings (e.g. int8, unsigned16, ieeefloat, etc.) with
>>a fully specified semantics, this would make the language *more*
>>portable. The existing definition is sufficiently wooly that programming
>>for portability can often be extremely challenging.[/color]
>
> Well, I think you are missing a key point. C++, and C, were defined to
> be run on as many platforms and evironments as possible. That includes
> embedded systems and systems where you could have bit sizes of 9, 12,
> and even 40 bits. To avoid penalizing such systems, the standard
> avoids mandating byte and type sizes; for example, whether a char
> (byte) is 8 bits or not, and even whether it is signed or unsigned.
> Also note that with such systems, mandating the sizes does not
> necessarily simplify portability.
>
> As a result, the programmer is responsible for determining how
> portable a program is, which is as it should be. As for me, I can't
> think of any program that I'd want to run on every conceivable
> platform. But there are classes of platforms (32/64 bit UNIX and
> WinNT) that I do want to program for. I make my design choices
> accordingly.[/color]

There are really two choices here. Either standardise types in the
language, so that programs have a well-defined semantics regardless of
platform, or don't standardise types, and have no standardised semantics
either. C++ makes the wrong choice, in my opinion.

It *should not* be necessary to go to considerable lengths to guarantee
portability. Programs should be portable by default, unless they are
specifically written to interface with something nonstandard (low level
drivers, a GUI, etc.). This is not the case with C++.
[color=blue][color=green]
>>Yes, this may be regarded as a little heretical, but I'm not trolling
>>here -- this is an important issue. I've written more than one C++
>>library with portability in mind (e.g. the signal/slot library on my web
>>site, as well as a new lexical analysis/parsing library that I'll
>>release sometime when I've finally gotten around to finishing its
>>documentation ), and can't say I find C++ terribly helpful in this regard.[/color]
>
>
> Well, I agree with much of what you say, but IMO, C++ can run on more
> platforms and is a truly platform independent language because it does
> avoid "hardware" requirements.[/color]

C++ runs on lots of platforms mostly because lots of platforms have C++
compilers implemented for them, which itself is a consequence of the
language's popularity. This probably owes more to its C heritage, and to
C's own enormous market share in the late 80s and early 90s, than it
does to any specific features.

C++ doesn't avoid hardware requirements, because its semantics are
entirely hardware dependent, whether or not you choose to use
nonstandard libraries. In this sense, it is pushing the point a bit to
truthfully call the language genuinely platform independent, or for that
matter particularly portable. It's just that it happens to be supported
on lots of platforms -- this isn't the same thing. Moving code between
architectures involves quite a lot of pot luck -- however much effort
has been put into making the code portable, *something* typically breaks
if the code is non-trivial.

It is possible to write very portable C++. I know, I've written lots of
it myself. My argument is that such code is by no means easy to write,
nor is it the default for 'typical' code, because of C++'s incompletely
standardised semantics.

Frankly, being able to compile unmodified code for 9 bit bytes or 56 bit
machine words (say) is all very well, but this is something that is so
spectacularly rare that sacrificing genuine portability for this is not
really a supportable design choice.

Whoops, too late. We're stuck with it now! :-)

--
----------------------------------------------
/ __ + / Sarah Thompson **** /
/ (_ _ _ _ |_ / * /
/ __)(_|| (_|| ) / sarah@telergy.c om * /
/ + / http://findatlantis.com/ /
----------------------------------------------

[David Cattarin]

Re: STL and over 2GB memory

Sarah Thompson <sarah@telergy. remove.this.bit .com> wrote in message news:<bf4409$2r s$1$8300dec7@ne ws.demon.co.uk> ...[color=blue]
> David Cattarin wrote:[color=green][color=darkred]
> >>Arguably, if the C++ standard really did standardise data types, giving
> >>them well-defined meanings (e.g. int8, unsigned16, ieeefloat, etc.) with
> >>a fully specified semantics, this would make the language *more*
> >>portable. The existing definition is sufficiently wooly that programming
> >>for portability can often be extremely challenging.[/color]
> >
> > Well, I think you are missing a key point. C++, and C, were defined to
> > be run on as many platforms and evironments as possible. That includes
> > embedded systems and systems where you could have bit sizes of 9, 12,
> > and even 40 bits. To avoid penalizing such systems, the standard
> > avoids mandating byte and type sizes; for example, whether a char
> > (byte) is 8 bits or not, and even whether it is signed or unsigned.
> > Also note that with such systems, mandating the sizes does not
> > necessarily simplify portability.
> >
> > As a result, the programmer is responsible for determining how
> > portable a program is, which is as it should be. As for me, I can't
> > think of any program that I'd want to run on every conceivable
> > platform. But there are classes of platforms (32/64 bit UNIX and
> > WinNT) that I do want to program for. I make my design choices
> > accordingly.[/color]
>
> There are really two choices here. Either standardise types in the
> language, so that programs have a well-defined semantics regardless of
> platform, or don't standardise types, and have no standardised semantics
> either. C++ makes the wrong choice, in my opinion.[/color]

Fair enough. I think the standard is well-specified given what they
wanted to achieve: to allow programmers to write systems for
practically any platform.
[color=blue]
> It *should not* be necessary to go to considerable lengths to guarantee
> portability. Programs should be portable by default, unless they are
> specifically written to interface with something nonstandard (low level
> drivers, a GUI, etc.). This is not the case with C++.[/color]

Now I don't really agree with your claim that one has to go to
"considerab le length to guarntee portability". I've written plenty of
code that is portable with little effort. Sure, somethimes I have to
be aware of the pitfalls, but it really is not that hard. And
sometimes it also takes some planning.

I currently have well over 100K LOC that compiles as 32bit or 64bit on
several versions of 5 different UNIX platforms and also WinNT and
Win2K. Any portability issues are as you noted above, which is outside
the scope of the language.

I know that doesn't prove anything, but I do think the difficulty of
the task is overstated.
[color=blue][color=green][color=darkred]
> >>Yes, this may be regarded as a little heretical, but I'm not trolling
> >>here -- this is an important issue. I've written more than one C++
> >>library with portability in mind (e.g. the signal/slot library on my web
> >>site, as well as a new lexical analysis/parsing library that I'll
> >>release sometime when I've finally gotten around to finishing its
> >>documentation ), and can't say I find C++ terribly helpful in this regard.[/color]
> >
> >
> > Well, I agree with much of what you say, but IMO, C++ can run on more
> > platforms and is a truly platform independent language because it does
> > avoid "hardware" requirements.[/color]
>
> C++ runs on lots of platforms mostly because lots of platforms have C++
> compilers implemented for them, which itself is a consequence of the
> language's popularity. This probably owes more to its C heritage, and to
> C's own enormous market share in the late 80s and early 90s, than it
> does to any specific features.[/color]

Fair enough, but I think you aren't considering embedded systems.
[color=blue]
> C++ doesn't avoid hardware requirements,[/color]

I think there are very few places in the standard that could be
considered hardware requirements. Did you have anything specifically
in mind?
[color=blue]
> because its semantics are
> entirely hardware dependent,[/color]

An implementation or program is hardware dependent. The standard
typically notes potential portability problems with phrases such as
"implementa tion-defined behavior", "unspecifie d behavior" and
sometimes even "undefined behavior".
[color=blue]
> whether or not you choose to use
> nonstandard libraries. In this sense, it is pushing the point a bit to
> truthfully call the language genuinely platform independent, or for that
> matter particularly portable.[/color]

Perhaps we are both splitting hairs, but I think the C++ language--not
programs--is platform independent. That's why the standard makes
careful use of the terms noted above.
[color=blue]
> It's just that it happens to be supported
> on lots of platforms -- this isn't the same thing. Moving code between
> architectures involves quite a lot of pot luck -- however much effort
> has been put into making the code portable, *something* typically breaks
> if the code is non-trivial.[/color]

I don't really agree with that either. If you are peforming
bit-twiddling then you should be wary of portability problems. Also if
you rely on ints and longs being the same size. Complex code isn't
necessarily non-portable.
[color=blue]
> It is possible to write very portable C++. I know, I've written lots of
> it myself. My argument is that such code is by no means easy to write,
> nor is it the default for 'typical' code, because of C++'s incompletely
> standardised semantics.[/color]

I think the standard is pretty well-specified, but since we have two
different viewpoints about this, I can agree to disagree.
[color=blue]
> Frankly, being able to compile unmodified code for 9 bit bytes or 56 bit
> machine words (say) is all very well, but this is something that is so
> spectacularly rare that sacrificing genuine portability for this is not
> really a supportable design choice.[/color]

Take a look around you. Embedded systems are not rare. Cell phones are
one of the most visible embedded markets. Admittedly, odd bit sizes
are becomming less frequent, but they are still present.
[color=blue]
> Whoops, too late. We're stuck with it now! :-)[/color]

Yep. And a good thing too. :)
Dave