unicode and socket

[zyqnews@163.net]

hello all,
I am new in Python. And I have got a problem about unicode.
I have got a unicode string, when I was going to send it out throuth a
socket by send(), I got an exception. How can I send the unicode string
to the remote end of the socket as it is without any conversion of
encode, so the remote end of the socket will receive unicode string?

Thanks

[aurora]

Re: unicode and socket

You could not. Unicode is an abstract data type. It must be encoded into
octets in order to send via socket. And the other end must decode the
octets to retrieve the unicode string. Needless to say the encoding scheme
must be consistent and understood by both ends.


On 18 Feb 2005 11:03:46 -0800, <zyqnews@163.ne t> wrote:
[color=blue]
> hello all,
> I am new in Python. And I have got a problem about unicode.
> I have got a unicode string, when I was going to send it out throuth a
> socket by send(), I got an exception. How can I send the unicode string
> to the remote end of the socket as it is without any conversion of
> encode, so the remote end of the socket will receive unicode string?
>
> Thanks
>[/color]

[Irmen de Jong]

Re: unicode and socket

aurora wrote:[color=blue]
> You could not. Unicode is an abstract data type. It must be encoded
> into octets in order to send via socket. And the other end must decode
> the octets to retrieve the unicode string. Needless to say the encoding
> scheme must be consistent and understood by both ends.[/color]

So use pickle.

--Irmen

[Irmen de Jong]

Re: unicode and socket

Irmen de Jong wrote:[color=blue]
> aurora wrote:
>[color=green]
>> You could not. Unicode is an abstract data type. It must be encoded
>> into octets in order to send via socket. And the other end must
>> decode the octets to retrieve the unicode string. Needless to say the
>> encoding scheme must be consistent and understood by both ends.[/color]
>
>
> So use pickle.
>
> --Irmen[/color]

Well, on second thought: don't use pickle.
If all you want to transfer is unicode strings (or normal strings)
it's safer to just encode them to, say, UTF-8, transfer
that octet stream across, and on the other side, decode the
UTF-8 octets back into a unicode string.


--Irmen

[Lion Kimbro]

Re: unicode and socket

You probably want to use UTF-16 or UTF-8 on both sides of the socket.

See http://www.python.org/moin/Unicode for more information.

So, we have a Unicode string...[color=blue][color=green][color=darkred]
>>> mystring=u'eggs and ham'
>>> mystring[/color][/color][/color]
u'eggs and ham'

Now, we want to send it over:[color=blue][color=green][color=darkred]
>>> to_send=mystrin g.encode('utf-8')
>>> to_send[/color][/color][/color]
'eggs and ham'

It's encoded in UTF-8 now.

On the other side, (result=to_send ,) we decode:
[color=blue][color=green][color=darkred]
>>> result=received .decode('utf-8')
>>> result[/color][/color][/color]
u'eggs and ham'

You have transfered a unicode string. {:)}=

[zyqnews@163.net]

Re: unicode and socket

It's really funny, I cannot send a unicode stream throuth socket with
python while all the other languages as perl,c and java can do it.
then, how about converting the unicode string to a binary stream? It is
possible to send a binary through socket with python?

[Serge Orlov]

Re: unicode and socket

zyqnews@163.net wrote:[color=blue]
> It's really funny, I cannot send a unicode stream throuth socket with
> python while all the other languages as perl,c and java can do it.[/color]

You may really start laughing loudly <wink> after you find out that you
can send arbitrary python objects over sockets. If you want language
specific way of sending objects, see Irmen's first answer: use pickle.
[color=blue]
> then, how about converting the unicode string to a binary stream?[/color]

Sure, there are already three answers in this thread that suggest you
to do that. Use encode method of unicode strings.
[color=blue]
> It is possible to send a binary through socket with python?[/color]

Sure. If it wouldn't be possible to send bytes through sockets with Python
what else do you think could be sent? Perhaps you're confused that
bytes are stored in byte strings in Python, which are often called strings in
documentation and conversations? It will be fixed in Python 3.0, but
these days you have to store bytes in str type.

Serge.

[aurora]

Re: unicode and socket

On 18 Feb 2005 19:10:36 -0800, <zyqnews@163.ne t> wrote:
[color=blue]
> It's really funny, I cannot send a unicode stream throuth socket with
> python while all the other languages as perl,c and java can do it.
> then, how about converting the unicode string to a binary stream? It is
> possible to send a binary through socket with python?
>[/color]

I was answering your specific question:

"How can I send the unicode string to the remote end of the socket as it
is without any conversion of encode"

The answer is you could not. Not that you cannot sent unicode but you have
to encode it. The same applies to perl, c or Java. The only difference is
the detail of how strings get encoded.

There are a few posts suggest various means. Or you can check out
codecs.getwrite r() which closer resembles Java's way.

[Fredrik Lundh]

Re: unicode and socket

anonymous coward <zyqnews@163.ne t> wrote:
[color=blue]
> It's really funny, I cannot send a unicode stream throuth socket with
> python while all the other languages as perl,c and java can do it.[/color]

Are you sure you understand what Unicode is, and how sockets work?

Sockets are used to transfer byte streams. If you want to transfer
a python-level object, you have to decide how to encode it as a
byte stream. For integers, you have to decide whether to use a single
byte, a string of decimal ascii characters, netstring syntax, etc. For
text, you have to decide what character encoding to use. For arbitrary
objects, you have to decide what serialisation protocol to use. etc.

(and yes, the same applies to all other languages. Java sockets and C
sockets are no different from Python sockets...)

</F>

[Christos TZOTZIOY Georgiou]

Re: unicode and socket

On 18 Feb 2005 19:10:36 -0800, rumours say that zyqnews@163.net might have
written:
[color=blue]
>It's really funny, I cannot send a unicode stream throuth socket with
>python while all the other languages as perl,c and java can do it.[/color]

I don't know about perl. What I think you mean by unicode in C most probably is
the wchar_t, which is Unicode encoded as 'ucs-2' or 'utf-16' (little or big
endian, depending on your platform) or maybe a 4-byte int, for which I don't
know a Python equivalent. And I /assume/ in Java that Unicode is equivalent to
'utf-16' encoded strings when input/output.

Perhaps Unicode encoded as 'utf-16' is what you're after. However, Unicode
encoded as 'utf-8' (like others also suggested) might be what you /should/ be
using, given that this encoding has some attractive properties (no null bytes,
no spurious control characters etc).

Don't interpret as weakness the explicitness requested from Python.
--
TZOTZIOY, I speak England very best.
"Be strict when sending and tolerant when receiving." (from RFC1958)
I really should keep that in mind when talking with people, actually...