UTF-8 locales

**Christian Bau** · Nov 14 '05, 08:25 PM

Re: UTF-8 locales

In article <1111520992.946 923.264520@z14g 2000cwz.googleg roups.com>,
saroj.yadav@gma il.com wrote:
[color=blue]
> As I understand it (correct me, if I am wrong) Unicode came into
> picture so that a document containing multiple language characters can
> be supported like somebody can write a document comparing Korean and
> Chinese in French language.
>
> Now, I am looking at all UNIX platforms and seems like all Unix (AIX,
> HP, Solaris) platforms support Unicode by supporting language/region
> specific UTF-8 locales like fr_FR.UTF-8, ja_JP.UTF-8, ko_KR.UTF-8 etc.
>
> Now in order to use UTF-8 for Japanese, I have to set locale to
> ja_JP.UTF-8. To use UTF-8 for Korean, I have to set locale to
> ko_KR.UTF-8.
>
> With this approach it's not possible to mix multiple language
> characters. Doesn't this defeat the whole purpose of Unicode ?
> Am I missing something ?[/color]

A locale describes much more than just the characters.

There are for example the rules for formatting a number. Threethousand
twohundred and six hundredths is 3,200.06 in one locale and 3.200,06 in
another and 3'200.06 in a third locale.

There are rules how to display monetary values. There are rules how to
sort text. There are rules how to display dates. They all will be
different between locales.

**Spacen Jasset** · Nov 14 '05, 08:25 PM

Re: UTF-8 locales

saroj.yadav@gma il.com wrote:[color=blue]
> As I understand it (correct me, if I am wrong) Unicode came into
> picture so that a document containing multiple language characters can
> be supported like somebody can write a document comparing Korean and
> Chinese in French language.
>
> Now, I am looking at all UNIX platforms and seems like all Unix (AIX,
> HP, Solaris) platforms support Unicode by supporting language/region
> specific UTF-8 locales like fr_FR.UTF-8, ja_JP.UTF-8, ko_KR.UTF-8 etc.
>
> Now in order to use UTF-8 for Japanese, I have to set locale to
> ja_JP.UTF-8. To use UTF-8 for Korean, I have to set locale to
> ko_KR.UTF-8.
>
> With this approach it's not possible to mix multiple language
> characters. Doesn't this defeat the whole purpose of Unicode ?
> Am I missing something ?
>
> Thanks in advance for any insight you can provide.
>[/color]

I am not certain how this works on Unix platforms. However I suspect
that the locale has an effect on local conventions as the previous
poster mentioned. I.e date formats and what not.

You still should be able to display any language no matter what locale
you have set, i.e in ja_JP.UTF-8 you should still be able to display
French an Chinese and so on. Try it. ( You must have fonts capable of
mapping those characters of course, if it doesn't work try changing font
- look on the web for names of the fonts that support the glyphs that
are needed )

**CBFalconer** · Nov 14 '05, 08:26 PM

Re: UTF-8 locales

saroj.yadav@gma il.com wrote:[color=blue]
>
> As I understand it (correct me, if I am wrong) Unicode came into
> picture so that a document containing multiple language characters
> can be supported like somebody can write a document comparing
> Korean and Chinese in French language.
>
> Now, I am looking at all UNIX platforms and seems like all Unix
> (AIX, HP, Solaris) platforms support Unicode by supporting
> language/region specific UTF-8 locales like fr_FR.UTF-8,
> ja_JP.UTF-8, ko_KR.UTF-8 etc.[/color]

.... snip ...

This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
solaris, etc. in its name. c.l.c deals with the actual portable
language as defined in the C standard. As soon as you have to
specify a system or compiler, you become off-topic.

--
"I conclude that there are two ways of constructing a software
design: One way is to make it so simple that there are obviously
no deficiencies and the other way is to make it so complicated
that there are no obvious deficiencies." -- C. A. R. Hoare

**Kevin Bracey** · Nov 14 '05, 08:26 PM

Re: UTF-8 locales

In message <4240C332.1A275 887@yahoo.com>
CBFalconer <cbfalconer@yah oo.com> wrote:
[color=blue]
> saroj.yadav@gma il.com wrote:[color=green]
> >
> > Now, I am looking at all UNIX platforms and seems like all Unix
> > (AIX, HP, Solaris) platforms support Unicode by supporting
> > language/region specific UTF-8 locales like fr_FR.UTF-8,
> > ja_JP.UTF-8, ko_KR.UTF-8 etc.[/color]
>
> ... snip ...
>
> This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
> solaris, etc. in its name. c.l.c deals with the actual portable
> language as defined in the C standard. As soon as you have to
> specify a system or compiler, you become off-topic.[/color]

Which is of course why the whole locale system and wchar_t are effectively
useless for portable programming. Everything about them is
implementation-defined, so you can't even talk about them in comp.*.c :(

Is there anywhere where one could discuss "best practice" or standardisation
for these facilities?

--
Kevin Bracey, Principal Software Engineer
Tematic Ltd Tel: +44 (0) 1223 503464
182-190 Newmarket Road Fax: +44 (0) 1728 727430
Cambridge, CB5 8HE, United Kingdom WWW: http://www.tematic.com/

**Michael Mair** · Nov 14 '05, 08:26 PM

Re: UTF-8 locales

Kevin Bracey wrote:[color=blue]
> In message <4240C332.1A275 887@yahoo.com>
> CBFalconer <cbfalconer@yah oo.com> wrote:
>
>[color=green]
>>saroj.yadav@g mail.com wrote:
>>[color=darkred]
>>>Now, I am looking at all UNIX platforms and seems like all Unix
>>>(AIX, HP, Solaris) platforms support Unicode by supporting
>>>language/region specific UTF-8 locales like fr_FR.UTF-8,
>>>ja_JP.UTF-8, ko_KR.UTF-8 etc.[/color]
>>
>>... snip ...
>>
>>This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
>>solaris, etc. in its name. c.l.c deals with the actual portable
>>language as defined in the C standard. As soon as you have to
>>specify a system or compiler, you become off-topic.[/color]
>
>
> Which is of course why the whole locale system and wchar_t are effectively
> useless for portable programming. Everything about them is
> implementation-defined, so you can't even talk about them in comp.*.c :(
>
> Is there anywhere where one could discuss "best practice" or standardisation
> for these facilities?[/color]

Ack.
I really would be interested in that, too.

Cheers
Michael
--
E-Mail: Mine is an /at/ gmx /dot/ de address.

**CBFalconer** · Nov 14 '05, 08:27 PM

Re: UTF-8 locales

Michael Mair wrote:[color=blue]
> Kevin Bracey wrote:[color=green]
>> CBFalconer <cbfalconer@yah oo.com> wrote:[color=darkred]
>>> saroj.yadav@gma il.com wrote:
>>>
>>>> Now, I am looking at all UNIX platforms and seems like all Unix
>>>> (AIX, HP, Solaris) platforms support Unicode by supporting
>>>> language/region specific UTF-8 locales like fr_FR.UTF-8,
>>>> ja_JP.UTF-8, ko_KR.UTF-8 etc.
>>>
>>>... snip ...
>>>
>>> This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
>>> solaris, etc. in its name. c.l.c deals with the actual portable
>>> language as defined in the C standard. As soon as you have to
>>> specify a system or compiler, you become off-topic.[/color]
>>
>> Which is of course why the whole locale system and wchar_t are
>> effectively useless for portable programming. Everything about
>> them is implementation-defined, so you can't even talk about them
>> in comp.*.c :(
>>
>> Is there anywhere where one could discuss "best practice" or
>> standardisation for these facilities?[/color]
>
> Ack.
> I really would be interested in that, too.[/color]

I would consider comp.programmin g quite suitable for this sort of
general internationaliz ation discussion.

--
Chuck F (cbfalconer@yah oo.com) (cbfalconer@wor ldnet.att.net)
Available for consulting/temporary embedded and systems.
<http://cbfalconer.home .att.net> USE worldnet address!

**websnarf@gmail.com** · Nov 14 '05, 08:31 PM

Re: UTF-8 locales

Kevin Bracey wrote:[color=blue]
> CBFalconer <cbfalconer@yah oo.com> wrote:[color=green]
> > saroj.yadav@gma il.com wrote:[color=darkred]
> > > Now, I am looking at all UNIX platforms and seems like all Unix
> > > (AIX, HP, Solaris) platforms support Unicode by supporting
> > > language/region specific UTF-8 locales like fr_FR.UTF-8,
> > > ja_JP.UTF-8, ko_KR.UTF-8 etc.[/color]
> >
> > ... snip ...
> >
> > This is off-topic in c.l.c. Try a newsgroup with unix, posix, aix,
> > solaris, etc. in its name. c.l.c deals with the actual portable
> > language as defined in the C standard. As soon as you have to
> > specify a system or compiler, you become off-topic.[/color]
>
> Which is of course why the whole locale system and wchar_t are
> effectively useless for portable programming. Everything about
> them is implementation-defined, so you can't even talk about
> them in comp.*.c :([/color]

Yeah, amazing he doesn't see the irony of his own statement. The
wchar_t types and functions are only portable in the very narrow scope
of "I can compile that source code to do something" sense. But since
the definition of wchar_t is so open ended, system dependent, and in
fact *NOT* universal in any real sense, there's no way you can actually
use it for any truly portable applications.

And there's no way to somehow "augment" the specification, or use these
types/function in some special way to *make* them portable. There are
no ifdef tricks or anything like that -- you're just screwed. The
whole wchar_t idea literally has no relevant functional content.

C's whole misconception of what it means to be portable, is in fact
what *prevents* it from being portable where it counts, on issues like
localization.
[color=blue]
> Is there anywhere where one could discuss "best practice" or
> standardisation for these facilities?[/color]

Well more importantly, portability in general, really can't be
discussed in comp.lang.c. I lurk on comp.programmin g, so you can
always throw questions about portability there.

---
Paul Hsieh

Pobox is now part of Fastmail

http://www.pobox.com/~qed/

Pobox has been discontinued as a separate service, and all existing customers moved to the Fastmail platform.

The Better String Library

http://bstring.sf.net/