byte count unicode string

[willie]

Marc 'BlackJack' Rintsch:

>In <mailman.313.11 58732191.10491. python-l...@python.org >, willie wrote:

># What's the correct way to get the
># byte count of a unicode (UTF-8) string?
># I couldn't find a builtin method
># and the following is memory inefficient.

>ustr = "example\xC2\x9 D".decode('U TF-8')

>num_chars = len(ustr) # 8

>buf = ustr.encode('UT F-8')

>num_bytes = len(buf) # 9

>That is the correct way.

# Apologies if I'm being dense, but it seems
# unusual that I'd have to make a copy of a
# unicode string, converting it into a byte
# string, before I can determine the size (in bytes)
# of the unicode string. Can someone provide the rational
# for that or correct my misunderstandin g?

# Thanks.

[John Machin]

Re: byte count unicode string


willie wrote:

Marc 'BlackJack' Rintsch:
>

>In <mailman.313.11 58732191.10491. python-l...@python.org >, willie wrote:

># What's the correct way to get the
># byte count of a unicode (UTF-8) string?
># I couldn't find a builtin method
># and the following is memory inefficient.

>

>ustr = "example\xC2\x9 D".decode('U TF-8')

>

>num_chars = len(ustr) # 8

>

>buf = ustr.encode('UT F-8')

>

>num_bytes = len(buf) # 9

>

>That is the correct way.

>
>
# Apologies if I'm being dense, but it seems
# unusual that I'd have to make a copy of a
# unicode string, converting it into a byte
# string, before I can determine the size (in bytes)
# of the unicode string. Can someone provide the rational
# for that or correct my misunderstandin g?
>

You initially asked "What's the correct way to get the byte countof a
unicode (UTF-8) string".

It appears you meant "How can I find how many bytes there are in the
UTF-8 representation of a Unicode string without manifesting the UTF-8
representation? ".

The answer is, "You can't", and the rationale would have to be that
nobody thought of a use case for counting the length of the UTF-8 form
but not creating the UTF-8 form. What is your use case?

Cheers,
John

[MonkeeSage]

Re: byte count unicode string

John Machin wrote:

The answer is, "You can't", and the rationale would have to be that
nobody thought of a use case for counting the length of the UTF-8 form
but not creating the UTF-8 form. What is your use case?

Playing DA here, what if you need to send the byte-count on a server
via a header, but need the utf8 representation for the actual data?

Regards,
Jordan

[Diez B. Roggisch]

Re: byte count unicode string

MonkeeSage schrieb:

John Machin wrote:

>The answer is, "You can't", and the rationale would have to be that
>nobody thought of a use case for counting the length of the UTF-8 form
>but not creating the UTF-8 form. What is your use case?

>
Playing DA here, what if you need to send the byte-count on a server
via a header, but need the utf8 representation for the actual data?

So what - you need it in the end, don't you?

The runtime complexity of the calculation will be the same - you have to
consider each character, so its O(n).

Of course you will roughly double the memory consumption - the original
unicode being represented as UCS2 or UCS4.

But then - if that really is a problem, how would you work with that
string anyway?

So you have to resort to slicing and computing the size of the parts,
which will remedy that easily.

Diez

[Duncan Booth]

Re: byte count unicode string

"MonkeeSage " <MonkeeSage@gma il.comwrote:

John Machin wrote:

>The answer is, "You can't", and the rationale would have to be that
>nobody thought of a use case for counting the length of the UTF-8 form
>but not creating the UTF-8 form. What is your use case?

>
Playing DA here, what if you need to send the byte-count on a server
via a header, but need the utf8 representation for the actual data?

Then you still need both the data and its length. John asked for an example
where you need only the length and not the data itself.

I guess you could invent something like inserting a string into a database
which has fixed size fields, silently truncates fields which are too long
and stores the strings internally in utf-8 but only accepts ucs-2 in its
interface. Pretty far fetched, but if it exists I suspect that an extra
utf-8 encoding here or there is the least of your problems.

[Paul Rubin]

Re: byte count unicode string

Duncan Booth <duncan.booth@i nvalid.invalidw rites:

I guess you could invent something like inserting a string into a database
which has fixed size fields, silently truncates fields which are too long
and stores the strings internally in utf-8 but only accepts ucs-2 in its
interface. Pretty far fetched, but if it exists I suspect that an extra
utf-8 encoding here or there is the least of your problems.

More direct would be to add an option to the http parser to return the
utf8 received from the browser as a byte array still in utf8, instead
of decoding it so that it needs to be re-encoded before insertion into
the database. A lot of the time, the application doesn't need to look
at the string anyway.

[MonkeeSage]

Re: byte count unicode string

OK, so the devil always loses. ;P

Regards,
Jordan

[Virgil Dupras]

Re: byte count unicode string


MonkeeSage wrote:

OK, so the devil always loses. ;P
>
Regards,
Jordan

Huh? The devil always loses? *turns TV on, watches the news, turns TV
off* Nope, buddy. Quite the contrary.