Unicode/ascii encoding nightmare

[Hendrik van Rooyen]

Re: Unicode/ascii encoding nightmare

"John Machin" <sjmachin@lexic on.netwrote:

8<---------------------------------------

I strongly suggest that you read the docs *FIRST*, and don't "tinker"
at all.
>
HTH,
John

This is *good* advice - its unlikely to be followed though, as the OP is prolly
just like most of us - you unpack the stuff out of the box and start assembling
it, and only towards the end, when it wont fit together, do you read the manual
to see where you went wrong...

- Hendrik

[Andrea Griffini]

Re: Unicode/ascii encoding nightmare

John Machin wrote:

Indeed yourself.

What does the above mean ?

Have you ever considered reading posts in
chronological order, or reading all posts in a thread?

I do no think people read posts in chronological order;
it simply doesn't make sense. I also don't think many
do read threads completely, but only until the issue is
clear or boredom kicks in.

Your nice "double whammy" post was enough to clarify
what happened to the OP, I just wanted to make a bit
more explicit what you meant; my poor english also
made me understand that you were just "suspecting "
such an error, so I verified and posted the result.

That your "suspect" was a sarcastic remark could be
clear only when reading the timewise "former" reply
that however happened to be lower in the thread tree
in my newsreader; fact that pushed it into the "not
worth reading" area.

It might help
you avoid writing posts with non-zero information content.

Why should I *avoid* writing posts with *non-zero*
information content ? Double whammy on negation or
still my poor english kicking in ? :-)

Suppose you didn't post the double whammy message,
and suppose someone else made it seven minutes later
than your other post. I suppose that in this case
the message would be a zero content noise (and not
the precious pearl of wisdom it is because it
comes from you).

Cheers,
John

Andrea

[Paul Boddie]

Re: Unicode/ascii encoding nightmare

Thomas W wrote:

Ok, I've cleaned up my code abit and it seems as if I've
encoded/decoded myself into a corner ;-).

Yes, you may encounter situations where you have some string, you
"decode" it (ie. convert it to Unicode) using one character encoding,
but then you later "encode" it (ie. convert it back to a plain string)
using a different character encoding. This isn't a problem on its own,
but if you then take that plain string and attempt to convert it to
Unicode again, using the same input encoding as before, you'll be
misinterpreting the contents of the string.

This "round tripping" of character data is typical of Web applications:
you emit a Web page in one encoding, the fields in the forms are
represented in that encoding, and upon form submission you receive this
data. If you then process the form data using a different encoding,
you're misinterpreting what you previously emitted, and when you emit
this data again, you compound the error.

My understanding of unicode has room for improvement, that's for sure. I got some pointers
and initial code-cleanup seem to have removed some of the strange results I got, which
several of you also pointed out.

Converting to Unicode for processing is a "best practice" that you seem
to have adopted, but it's vital that you use character encodings
consistently. One trick, that can be used to mitigate situations where
you have less control over the encoding of data given to you, is to
attempt to convert to Unicode using an encoding that is "conservati ve"
with regard to acceptable combinations of byte sequences, such as
UTF-8; if such a conversion fails, it's quite possible that another
encoding applies, such as ISO-8859-1, and you can try that. Since
ISO-8859-1 is a "liberal" encoding, in the sense that any byte value
or combination of byte values is acceptable, it should only be used as
a last resort.

However, it's best to have a high level of control over character
encodings rather than using tricks to avoid considering representation
issues carefully.

Paul

[Cliff Wells]

Re: Unicode/ascii encoding nightmare

On Tue, 2006-11-07 at 08:10 +0200, Hendrik van Rooyen wrote:

"John Machin" <sjmachin@lexic on.netwrote:
>
8<---------------------------------------
>

I strongly suggest that you read the docs *FIRST*, and don't "tinker"
at all.

This is *good* advice - its unlikely to be followed though, as the OP is prolly
just like most of us - you unpack the stuff out of the box and start assembling
it, and only towards the end, when it wont fit together, do you read the manual
to see where you went wrong...

I fall right into this camp(fire). I'm always amazed and awed at people
who actually read the docs *thoroughly* before starting. I know some
people do but frankly, unless it's a step-by-step tutorial, I rarely
read the docs beyond getting a basic understanding of what something
does before I start "tinkering" .

I've always been a firm believer in the Chinese proverb:

I hear and I forget
I see and I remember
I do and I understand

Of course, I usually just skip straight to the third step and try to
work backwards as needed. This usually works pretty well but when it
doesn't it fails horribly. Unfortunately (for me), working from step
one rarely works at all, so that's the boat I'm stuck in.

I've always been a bit miffed at the RTFM crowd (and somewhat jealous, I
admit). I *do* RTFM, but as often as not the fine manual confuses as
much as clarifies. I'm not convinced this is the result of poor
documentation so much as that I personally have a different mental
approach to problem-solving than the others who find documentation
universally enlightening. I also suspect that I'm not alone in my
approach and that the RTFM crowd is more than a little close-minded
about how others might think about and approach solving problems and
understanding concepts.

Also, much documentation (including the Python docs) tends to be
reference-manual style. This is great if you *already* understand the
problem and just need details, but does about as much for
*understanding* as a dictionary does for learning a language. When I'm
perusing the Python reference manual, I usually find that 10 lines of
example code are worth 1000 lines of function descriptions and
cross-references.

Just my $0.02.

Regards,
Cliff

[Cliff Wells]

Re: Unicode/ascii encoding nightmare

On Mon, 2006-11-06 at 15:47 -0800, John Machin wrote:

Gabriel Genellina wrote:

At Monday 6/11/2006 20:34, Robert Kern wrote:

>John Machin wrote:
Indeed yourself. Have you ever considered reading posts in
chronological order, or reading all posts in a thread?
>
>That presumes that messages arrive in chronological order and
>transmission s are
>instantaneou s. Neither are true.

Sometimes I even got the replies *before* the original post comes.

>
What is in question is the likelihood that message B can appear before
message A, when both emanate from the same source, and B was sent about
7 minutes after A.

Usenet, email, usenet/email gateways, internet in general... all in
all, pretty likely. I've often seen replies to my posts long before my
own post shows up. In fact, I've seen posts not show up for several
hours.

Regards,
Cliff