Where is the ucs-32 codec?

[beni.cherniavsky@gmail.com]

Python seems to be missing a UCS-32 codec, even in wide builds (not
that it the build should matter).
Is there some deep reason or should I just contribute a patch?

If it's just a bug, should I call the codec 'ucs-32' or 'utf-32'? Or
both (aliased)?
There should be '-le' and '-be' variats, I suppose. Should there be a
variant without explicit endianity, using a BOM to decide (like
'utf-16')?
And it should combine surrogates into valid characters (on all builds),
like the 'utf-8' codec does, right?

--
Beni Cherniavsky <cben@users.sf. net>, who can only read email on
weekends.

[Erik Max Francis]

Re: Where is the ucs-32 codec?

beni.cherniavsk y@gmail.com wrote:
[color=blue]
> Python seems to be missing a UCS-32 codec, even in wide builds (not
> that it the build should matter).
> Is there some deep reason or should I just contribute a patch?
>
> If it's just a bug, should I call the codec 'ucs-32' or 'utf-32'? Or
> both (aliased)?
> There should be '-le' and '-be' variats, I suppose. Should there be a
> variant without explicit endianity, using a BOM to decide (like
> 'utf-16')?
> And it should combine surrogates into valid characters (on all builds),
> like the 'utf-8' codec does, right?[/color]

Note that UTF-32 is UCS-4. UCS-32 ("Universial Character Set in 32
octets") wouldn't make much sense.

Not that Python has a UCS-4 encoding available either. I'm really not
sure why.

--
Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Could it be / That we need loving to survive
-- Neneh Cherry

[Méta-MCI]

Re: Where is the ucs-32 codec?

Hi!

Look at: http://cjkpython.berlios.de (iconvcodec)

(Serge Orlov has built a version for Python 2.4 "special for me"; thanks to
him).


@-salutations
--
Michel Claveau

[Martin v. Löwis]

Re: Where is the ucs-32 codec?

beni.cherniavsk y@gmail.com wrote:[color=blue]
> Python seems to be missing a UCS-32 codec, even in wide builds (not
> that it the build should matter).
> Is there some deep reason or should I just contribute a patch?[/color]

The only reason is that nobody has needed one so far, and because
it is quite some work to do if done correctly. Why do you need it?
[color=blue]
> There should be '-le' and '-be' variats, I suppose. Should there be a
> variant without explicit endianity, using a BOM to decide (like
> 'utf-16')?[/color]

Right.
[color=blue]
> And it should combine surrogates into valid characters (on all builds),
> like the 'utf-8' codec does, right?[/color]

Right.

Also, it should support the incremental interface (as any multi-byte
codec should).

If you want it complete, it should also support line-oriented input.
Notice that .readline/.readlines is particularly difficult to implement,
as you can't rely on the underlying stream's .readline implementation
to provide meaningful results.

While we are discussing problems: there also is the issue whether
..readline/.readlines should take the additional Unicode linebreak
characters into account (e.g. U+2028, U+2029), and if so, whether
that should be restricted to "universal newlines" mode.

Regards,
Martin

[Erik Max Francis]

Re: Where is the ucs-32 codec?

Martin v. Löwis wrote:
[color=blue]
> The only reason is that nobody has needed one so far, and because
> it is quite some work to do if done correctly. Why do you need it?[/color]

Why would it be "quite some work"? Converting from UTF-16 to UTF-32 is
pretty straightforward , and UTF-16 is already supported.

--
Erik Max Francis && max@alcyone.com && http://www.alcyone.com/max/
San Jose, CA, USA && 37 20 N 121 53 W && AIM erikmaxfrancis
Democritus may have come from Abdera, but he was no dummy.
-- Carl Sagan

[Martin v. Löwis]

Re: Where is the ucs-32 codec?

Erik Max Francis wrote:[color=blue][color=green]
>> The only reason is that nobody has needed one so far, and because
>> it is quite some work to do if done correctly. Why do you need it?[/color]
>
> Why would it be "quite some work"? Converting from UTF-16 to UTF-32 is
> pretty straightforward , and UTF-16 is already supported.[/color]

I would like to see it correct, unlike the current UTF-16 codec. Perhaps
whoever contributes an UTF-32 codec could also deal with the defects of
the UTF-16 codec.

Regards,
Martin

[cben@users.sf.net]

Re: Where is the ucs-32 codec?

Méta-MCI wrote:[color=blue]
> Hi!
>
> Look at: http://cjkpython.berlios.de (iconvcodec)
>
> (Serge Orlov has built a version for Python 2.4 "special for me"; thanks to
> him).
>[/color]
Thanks for the pointer.
iconvcodec should do the job, but I still want a native implementation
to be included with any python.

[cben@users.sf.net]

Re: Where is the ucs-32 codec?

Martin v. Löwis wrote:[color=blue]
> Erik Max Francis wrote:[color=green][color=darkred]
> >> The only reason is that nobody has needed one so far, and because
> >> it is quite some work to do if done correctly. Why do you need it?[/color]
> >[/color][/color]
Somebody asked me about generating UTF-32 (he didn't have choice of the
output format).
I was about to propose the obvious ``u.encode('utf-32')`` but
discovered it's missing.
Someone proposed 'unicode-internal' but it depends on the build and is
an ugly answer.
Next time, I want Guido's Time Machine to just work, so I have to fix
this ;-).
[color=blue][color=green]
> > Why would it be "quite some work"? Converting from UTF-16 to UTF-32 is
> > pretty straightforward , and UTF-16 is already supported.[/color]
>
> I would like to see it correct, unlike the current UTF-16 codec. Perhaps
> whoever contributes an UTF-32 codec could also deal with the defects of
> the UTF-16 codec.
>[/color]
Now this is interesting, as I hoped to base my code on UTF-16 (and
perhaps UTF-8 for combining surrogates)... Can you elaborate?

I could attempt to fix UTF-16 as well but I don't have the expertise to
choose the right behaviour,
so you'll have to specify precisely what it should do (that it doesn't
do now).

[Fredrik Lundh]

Re: Where is the ucs-32 codec?

cben@users.sf.n et wrote:
[color=blue]
> Somebody asked me about generating UTF-32 (he didn't have choice of the
> output format). I was about to propose the obvious ``u.encode('utf-32')``
> but discovered it's missing.[/color]

hint 1:
[color=blue][color=green][color=darkred]
>>> u = u"Hello"
>>> a = array.array("I" , map(ord, u))
>>> a.tostring()[/color][/color][/color]
'H\x00\x00\x00e \x00\x00\x00l\x 00\x00\x00l\x00 \x00\x00o\x00\x 00\x00'[color=blue][color=green][color=darkred]
>>> a.byteswap()
>>> a.tostring()[/color][/color][/color]
'\x00\x00\x00H\ x00\x00\x00e\x0 0\x00\x00l\x00\ x00\x00l\x00\x0 0\x00o'

hint 2:
[color=blue][color=green][color=darkred]
>>> import sys
>>> sys.byteorder[/color][/color][/color]
'little'

</F>

[Martin v. Löwis]

Re: Where is the ucs-32 codec?

cben@users.sf.n et wrote:[color=blue][color=green]
>> I would like to see it correct, unlike the current UTF-16 codec. Perhaps
>> whoever contributes an UTF-32 codec could also deal with the defects of
>> the UTF-16 codec.
>>[/color]
> Now this is interesting, as I hoped to base my code on UTF-16 (and
> perhaps UTF-8 for combining surrogates)... Can you elaborate?[/color]

The codec doesn't do line-oriented input correctly (i.e. readline);
it raises NotImplementedE rror.

Regards,
Martin