std::string vs. Unicode UTF-8

[Wolfgang Draxinger]

I understand that it is perfectly possible to store UTF-8 strings
in a std::string, however doing so can cause some implicaions.
E.g. you can't count the amount of characters by length() |
size(). Instead one has to iterate through the string, parse all
UTF-8 multibytes and count each multibyte as one character.

To address this problem the GTKmm bindings for the GTK+ toolkit
have implemented a own string class Glib::ustring
<http://tinyurl.com/bxpu4> which takes care of UTF-8 in strings.

The question is, wouldn't it be logical to make std::string
Unicode aware in the next STL version? I18N is an important
topic nowadays and I simply see no logical reason to keep
std::string as limited as it is nowadays. Of course there is
also the wchar_t variant, but actually I don't like that.

Wolfgang Draxinger
--

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[Niels Dybdahl]

Re: std::string vs. Unicode UTF-8

> The question is, wouldn't it be logical to make std::string[color=blue]
> Unicode aware in the next STL version? I18N is an important
> topic nowadays and I simply see no logical reason to keep
> std::string as limited as it is nowadays. Of course there is
> also the wchar_t variant, but actually I don't like that.[/color]

It is much easier to handle unicode strings with wchar_t internally and
there is much less confusion about whether the string is ANSI or UTF8
encoded. So I have started using wchar_t wherever I can and I only use UTF8
for external communication.

Niels Dybdahl


---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[John Harrison]

Re: std::string vs. Unicode UTF-8

Wolfgang Draxinger wrote:[color=blue]
> I understand that it is perfectly possible to store UTF-8 strings
> in a std::string, however doing so can cause some implicaions.
> E.g. you can't count the amount of characters by length() |
> size(). Instead one has to iterate through the string, parse all
> UTF-8 multibytes and count each multibyte as one character.
>
> To address this problem the GTKmm bindings for the GTK+ toolkit
> have implemented a own string class Glib::ustring
> <http://tinyurl.com/bxpu4> which takes care of UTF-8 in strings.
>
> The question is, wouldn't it be logical to make std::string
> Unicode aware in the next STL version? I18N is an important
> topic nowadays and I simply see no logical reason to keep
> std::string as limited as it is nowadays. Of course there is
> also the wchar_t variant, but actually I don't like that.
>
> Wolfgang Draxinger[/color]

UTF-8 is only an encoding, why to you think a strings internal to the
program should be represented as UTF-8? Makes more sense to me to
translate to or from UTF-8 when you input or output strings from your
program. C++ already has the framework in place for that.

john

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[peter.koch.larsen@gmail.com]

Re: std::string vs. Unicode UTF-8


Wolfgang Draxinger wrote:[color=blue]
> I understand that it is perfectly possible to store UTF-8 strings
> in a std::string, however doing so can cause some implicaions.
> E.g. you can't count the amount of characters by length() |
> size(). Instead one has to iterate through the string, parse all
> UTF-8 multibytes and count each multibyte as one character.[/color]

Correct. Also you can't print it or anything else.
[color=blue]
>
> To address this problem the GTKmm bindings for the GTK+ toolkit
> have implemented a own string class Glib::ustring
> <http://tinyurl.com/bxpu4> which takes care of UTF-8 in strings.[/color]

Ok.
[color=blue]
>
> The question is, wouldn't it be logical to make std::string
> Unicode aware in the next STL version?[/color]
It already is - using e.g. wchar_t.[color=blue]
> I18N is an important
> topic nowadays and I simply see no logical reason to keep
> std::string as limited as it is nowadays.[/color]
It is not limited.[color=blue]
>Of course there is
> also the wchar_t variant, but actually I don't like that.[/color]

So you'd like to have Unicode support. And you realize you already have
it. But you don't like it. Why?[color=blue]
>
> Wolfgang Draxinger
> --
>[/color]
/Peter

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[Bob Hairgrove]

Re: std::string vs. Unicode UTF-8

On Tue, 13 Sep 2005 04:20:30 GMT, wdraxinger@dark stargames.de
(Wolfgang Draxinger) wrote:
[color=blue]
>I understand that it is perfectly possible to store UTF-8 strings
>in a std::string, however doing so can cause some implicaions.
>E.g. you can't count the amount of characters by length() |
>size(). Instead one has to iterate through the string, parse all
>UTF-8 multibytes and count each multibyte as one character.[/color]

Not only that, but substr(), operator[] etc. pose equally
"interestin g" problems.
[color=blue]
>To address this problem the GTKmm bindings for the GTK+ toolkit
>have implemented a own string class Glib::ustring
><http://tinyurl.com/bxpu4> which takes care of UTF-8 in strings.
>
>The question is, wouldn't it be logical to make std::string
>Unicode aware in the next STL version? I18N is an important
>topic nowadays and I simply see no logical reason to keep
>std::string as limited as it is nowadays. Of course there is
>also the wchar_t variant, but actually I don't like that.
>
>Wolfgang Draxinger[/color]

People use std::string in many different ways. You can even store
binary data with embedded null characters in it. I don't know for
sure, but I believe there are already proposals in front of the C++
standards committee for what you suggest. In the meantime, it might
make more sense to use a third-party UTF-8 string class if that is
what you mainly use it for. IBM has released the ICU library as open
source, for example, and it is widely used these days.

--
Bob Hairgrove
NoSpamPlease@Ho me.com

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[benben]

Re: std::string vs. Unicode UTF-8


"Wolfgang Draxinger" <wdraxinger@dar kstargames.de> wrote in message
news:q2egv2-lf.ln1@darkstar games.dnsalias. net...[color=blue]
>I understand that it is perfectly possible to store UTF-8 strings
> in a std::string, however doing so can cause some implicaions.
> E.g. you can't count the amount of characters by length() |
> size(). Instead one has to iterate through the string, parse all
> UTF-8 multibytes and count each multibyte as one character.
>
> To address this problem the GTKmm bindings for the GTK+ toolkit
> have implemented a own string class Glib::ustring
> <http://tinyurl.com/bxpu4> which takes care of UTF-8 in strings.
>
> The question is, wouldn't it be logical to make std::string
> Unicode aware in the next STL version? I18N is an important
> topic nowadays and I simply see no logical reason to keep
> std::string as limited as it is nowadays. Of course there is
> also the wchar_t variant, but actually I don't like that.
>
> Wolfgang Draxinger[/color]

That's why people have std::wstring :)

Ben


---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[Pete Becker]

Re: std::string vs. Unicode UTF-8

Wolfgang Draxinger wrote:[color=blue]
> I understand that it is perfectly possible to store UTF-8 strings
> in a std::string, however doing so can cause some implicaions.
> E.g. you can't count the amount of characters by length() |
> size(). Instead one has to iterate through the string, parse all
> UTF-8 multibytes and count each multibyte as one character.[/color]

Yup. That's what happens when you use the wrong tool.
[color=blue]
>
> The question is, wouldn't it be logical to make std::string
> Unicode aware in the next STL version? I18N is an important
> topic nowadays and I simply see no logical reason to keep
> std::string as limited as it is nowadays.[/color]

There's much more to internationaliz ation than Unicode. Requiring
std::string to be Unicode aware (presumably that means UTF-8 aware)
would impose implementation overhead that's not needed for the kinds of
things it was designed for, like the various ISO 8859 code sets. In
general, neither string nor wstring knows anything about multi-character
encodings. That's for efficiency. Do the translation on input and output.

[color=blue]
> Of course there is
> also the wchar_t variant, but actually I don't like that.
>[/color]

That's unfortunate, since it's exactly what wchar_t and wstring were
designed for. What is your objection to them?

--

Pete Becker
Dinkumware, Ltd. (http://www.dinkumware.com)

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[msalters]

Re: std::string vs. Unicode UTF-8


Wolfgang Draxinger schreef:
[color=blue]
> I understand that it is perfectly possible to store UTF-8 strings
> in a std::string, however doing so can cause some implicaions.
> E.g. you can't count the amount of characters by length() |
> size(). Instead one has to iterate through the string, parse all
> UTF-8 multibytes and count each multibyte as one character.[/color]

Usually correct, but not always. A char is a byte in C++, but
a byte might not be an octet. UTF-8 is of course octet-based.
[color=blue]
> The question is, wouldn't it be logical to make std::string
> Unicode aware in the next STL version? I18N is an important
> topic nowadays and I simply see no logical reason to keep
> std::string as limited as it is nowadays. Of course there is
> also the wchar_t variant, but actually I don't like that.[/color]

wchar_t isn't always Unicode, either. There's a proposal to add an
extra unicode char type, and that probably will include std::ustring

However, that is probably a 20+bit type. Unicode itself assigns
numbers to characters, and the numbers have exceeded 65536.
UTF-x means Unicode Transformation Format - x. These formats
map each number to one or more x-bit values. E.g. UTF-8 maps
the number of each unicode character to an octet sequence,
with the additional property that the 0 byte isn't used for
anything but number 0.

Now, these formats are intended for data transfer and not data
processing. That in turn means UTF-8 should go somewhere in
<iostream>, if it's added.

HTH,
Michiel Salters

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[kanze]

Re: std::string vs. Unicode UTF-8

msalters wrote:[color=blue]
> Wolfgang Draxinger schreef:[/color]

[...][color=blue]
> However, that is probably a 20+bit type. Unicode itself
> assigns numbers to characters, and the numbers have exceeded
> 65536. UTF-x means Unicode Transformation Format - x. These
> formats map each number to one or more x-bit values.
> E.g. UTF-8 maps the number of each unicode character to an
> octet sequence, with the additional property that the 0 byte
> isn't used for anything but number 0.[/color]

It has a lot more additional properties than that. Like the
fact that you can immediately tell whether a byte is a single
byte character, the first byte of a multibyte sequence, or a
following byte in a multibyte sequence, without looking beyond
just that byte.
[color=blue]
> Now, these formats are intended for data transfer and not data
> processing. That in turn means UTF-8 should go somewhere in
> <iostream>, if it's added.[/color]

I don't know where you find that these formats are intended just
for data transfer. Depending on what the code is doing (and the
text it has to deal with), the ideal solution may be UTF-8,
UTF-16 or UTF-32. For most of what I do, UTF-8 would be more
appropriate, including internally, than any of the other
formats. (It's also required in some cases.)

--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[Dave Rahardja]

Re: std::string vs. Unicode UTF-8

On 14 Sep 2005 14:40:21 GMT, "kanze" <kanze@gabi-soft.fr> wrote:

[color=blue][color=green]
>> Now, these formats are intended for data transfer and not data
>> processing. That in turn means UTF-8 should go somewhere in
>> <iostream>, if it's added.[/color]
>
>I don't know where you find that these formats are intended just
>for data transfer. Depending on what the code is doing (and the
>text it has to deal with), the ideal solution may be UTF-8,
>UTF-16 or UTF-32. For most of what I do, UTF-8 would be more
>appropriate, including internally, than any of the other
>formats. (It's also required in some cases.)[/color]

RFC 3629 says it this way:

"ISO/IEC 10646 and Unicode define several encoding forms of their
common repertoire: UTF-8, UCS-2, UTF-16, UCS-4 and UTF-32. In an
encoding form, each character is represented as one or more encoding
units. All standard UCS encoding forms except UTF-8 have an encoding
unit larger than one octet, making them hard to use in many current
applications and protocols that assume 8 or even 7 bit characters."

Note that UTF-8 is intended to _encode_ a larger space, its primary purpose
being the compatibily of the encoded format with "applicatio ns and protocols"
that assume 8- or 7-bit characters. This suggests to me that UTF-8 was devised
so that Unicode text can be _passed through_ older protocols that only
understand 8- or 7-bit characters by encoding it at the input, and later
decoding it at the output to recover the original data.

If you want to _manipulate_ Unicode characters, however, why not deal with
them in their native, unencoded space? wchar_t is guaranteed to be wide enough
to contain all characters in all supported locales in the implementation, and
each character will have an equal size in memory.

-dr

[msalters]

Re: std::string vs. Unicode UTF-8


kanze schreef:
[color=blue]
> msalters wrote:[color=green]
> > Wolfgang Draxinger schreef:[/color]
>
> [...][color=green]
> > However, that is probably a 20+bit type. Unicode itself
> > assigns numbers to characters, and the numbers have exceeded
> > 65536. UTF-x means Unicode Transformation Format - x. These
> > formats map each number to one or more x-bit values.
> > E.g. UTF-8 maps the number of each unicode character to an
> > octet sequence, with the additional property that the 0 byte
> > isn't used for anything but number 0.[/color]
>
> It has a lot more additional properties than that. Like the
> fact that you can immediately tell whether a byte is a single
> byte character, the first byte of a multibyte sequence, or a
> following byte in a multibyte sequence, without looking beyond
> just that byte.[/color]

Yep, that makes scanning through a byte sequence a lot easier.
However, that's not very important for std::string. .substr()
can't do anything useful with it. For .c_str(), the non-null
property is important.
Of course, for an utf8string type, these additional properties
make implementations a lot easier. UTF8 is quite a good encoding
actually.
[color=blue][color=green]
> > Now, these formats are intended for data transfer and not data
> > processing. That in turn means UTF-8 should go somewhere in
> > <iostream>, if it's added.[/color]
>
> I don't know where you find that these formats are intended just
> for data transfer. Depending on what the code is doing (and the
> text it has to deal with), the ideal solution may be UTF-8,
> UTF-16 or UTF-32. For most of what I do, UTF-8 would be more
> appropriate, including internally, than any of the other
> formats. (It's also required in some cases.)[/color]

Getting a substring, uppercasing, finding characters, replacing
characters: all common string operations, but non-trivial in UTF8
Saving to file, sending over TCP/IP, or to mobile devices: all
common I/O operations, and UTF8 makes it easy.

Regards,
Michiel Salters

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[kanze]

Re: std::string vs. Unicode UTF-8

msalters wrote:[color=blue]
> kanze schreef:[/color]

[...][color=blue][color=green]
> > I don't know where you find that these formats are intended
> > just for data transfer. Depending on what the code is doing
> > (and the text it has to deal with), the ideal solution may
> > be UTF-8, UTF-16 or UTF-32. For most of what I do, UTF-8
> > would be more appropriate, including internally, than any of
> > the other formats. (It's also required in some cases.)[/color][/color]
[color=blue]
> Getting a substring, uppercasing, finding characters,
> replacing characters: all common string operations, but
> non-trivial in UTF8.[/color]

I said "for most of what I do". Comparing for equality, using
as keys in std::set or an unordered_set, for example. UTF-8
works fine, and because it uses less memory, it will result in
better overall performance (less cache misses, less paging,
etc.).

In other cases, I've been dealing with binary input, with
embedded UTF-8 strings. Which means that I cannot translate
directly on input, only once I've parsed the binary structure
enough to know where the strings are located. In the last
application, the strings were just user names and passwords --
again, no processing which wouldn't work just fine in UTF-8.

Imagine a C++ compiler. The only place UTF-8 might cause some
added difficulty is when scanning a symbol -- and even there, I
can imagine some fairly simply solutions. For all of the
rest... the critical delimiters can all be easily recognized in
UTF-8, and of course, once past scanning, we're talking about
symbol table management, and perhaps concatenation (to generate
mangled names), but they're both easily done in UTF-8. All in
all, I think a C++ compiler would be a good example of an
application where using UTF-8 as the internal encoding would
make sense.
[color=blue]
> Saving to file, sending over TCP/IP, or to mobile devices: all
> common I/O operations, and UTF8 makes it easy.[/color]

The external world is byte oriented. That's for sure. UTF-8
(or some other 8 bit format) is definitly required for external
use. But there are numerous cases where UTF-8 is also a good
choice for internal use as well; why bother with the conversions
and the added memory overhead if it doesn't buy you anything?

--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientier ter Datenverarbeitu ng
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34


---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[lancediduck@nyc.rr.com]

Re: std::string vs. Unicode UTF-8

UTF-8 is already in iostream. Just about any platform, when use set you
locale to something with "utf8" support, then your libraries codecvt
facet will likely convert the utf8 to whatever wide char type your
platform supports.
Which on some platforms is 16 byte, and others is 32.

But the main trouble that C++ programmers have with unicode is that
they still want to use it just like arrays of ASCII encoded characters
that you can send to a console command line. That won't work. At the
very least, Unicode assumes that it will be displayed on a graphical
terminal. And there is certainly no "one to one correspondence"
between the characters rendered by the device and what you see encoded
in your Unicode string.
And don't even ask about Unicode regular expressions or "equality
comparison"-- Consider JavaScript,
var a='Hello';
var b=' World!';
if ((a+b) == 'Hello world!')
The conditional expression really means "encode in UTF16LE, normalize
each string using Unicode Normalization Form 3, and then do a byte by
byte comparison and return true if they match"

Just like ASCII is not a better way of doing Morse Code, Unicode is not
a better ASCII, but something way different.

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[Dietmar Kuehl]

Re: std::string vs. Unicode UTF-8

Pete Becker wrote:[color=blue]
> That's unfortunate, since it's exactly what wchar_t and wstring were
> designed for. What is your objection to them?[/color]

Well, 'wchar_t' and 'wstring' were designed at a time when Unicode
was still pretending that they use 16-bit characters and that each
Unicode character consists of a single 16-bit character. Neither of
these two properties holds: Unicode is [currently] a 20-bit encoding
and a Unicode character can consist of multiple such 20-bit entities
for combining characters.
--
<mailto:dietmar _kuehl@yahoo.co m> <http://www.dietmar-kuehl.de/>
<http://www.eai-systems.com> - Efficient Artificial Intelligence

---
[ comp.std.c++ is moderated. To submit articles, try just posting with ]
[ your news-reader. If that fails, use mailto:std-c++@ncar.ucar.e du ]
[ --- Please see the FAQ before posting. --- ]
[ FAQ: http://www.jamesd.demon.co.uk/csc/faq.html ]

[Mirek Fidler]

Re: std::string vs. Unicode UTF-8

Dietmar Kuehl wrote:[color=blue]
> Pete Becker wrote:
>[color=green]
>>That's unfortunate, since it's exactly what wchar_t and wstring were
>>designed for. What is your objection to them?[/color]
>
>
> Well, 'wchar_t' and 'wstring' were designed at a time when Unicode
> was still pretending that they use 16-bit characters and that each
> Unicode character consists of a single 16-bit character. Neither of
> these two properties holds: Unicode is [currently] a 20-bit encoding
> and a Unicode character can consist of multiple such 20-bit entities[/color]
^^^^^^^^^^^^^^^ ^

16-bit?

Mirek