UTF-8 garbage characters

**Arne** · Jul 23 '05, 10:16 PM

Re: UTF-8 garbage characters

*lawrence* skrev 2004-10-01 09:34:[color=blue]
> I'd love to ask why this page is not rendering correctly in Safari on
> a Macintosh but I suspect someone will tell me to validate the page
> first. Nevertheless, if anyone sees an obvious reason that I'm
> missing, I'd like to know. It looks like a missing div tag but I
> can't see one.
>
> http://www.krubner.com/
>
>
> Let's move on to a question that might be answerable. If i copy and
> paste non-UTF-8 characters to the page, and then send out a UTF-8
> charset UTF-8 header then I'll get the garbage characters that I'm
> seeing?[/color]

If you set charset UTF-8 in the header, you also have to save the
document as UTF-8 before uploading it. I belive it's saved in i default
charset, probably iso-8859-1

--
/Arne

**lawrence** · Jul 23 '05, 10:17 PM

Re: UTF-8 garbage characters

Arne <arne@luras.n u> wrote in message news:<SH87d.462 5$d5.36294@news b.telia.net>...[color=blue]
> *lawrence* skrev 2004-10-01 09:34:[color=green]
> > I'd love to ask why this page is not rendering correctly in Safari on
> > a Macintosh but I suspect someone will tell me to validate the page
> > first. Nevertheless, if anyone sees an obvious reason that I'm
> > missing, I'd like to know. It looks like a missing div tag but I
> > can't see one.
> >
> > http://www.krubner.com/
> >
> >
> > Let's move on to a question that might be answerable. If i copy and
> > paste non-UTF-8 characters to the page, and then send out a UTF-8
> > charset UTF-8 header then I'll get the garbage characters that I'm
> > seeing?[/color]
>
> If you set charset UTF-8 in the header, you also have to save the
> document as UTF-8 before uploading it. I belive it's saved in i default
> charset, probably iso-8859-1[/color]

Since sending out UTF-8 headers, I've started to get a lot of garbage
characters on the page. This despit hitting it with encode_UTF-8 in
PHP. The problem is I don't know what charset the text is in a head of
time, so I can't do any kind of proper conversion on the text. The web
is too heterogenous. I wonder how Blogger and TypePad tackle this
problem? Is it just luck that so few of their users end up with
garbage characters? Or is it because they standardize on a different
charset?

**Arne** · Jul 23 '05, 10:17 PM

Re: UTF-8 garbage characters

*lawrence* skrev 2004-10-02 02:13:[color=blue]
> Arne <arne@luras.n u> wrote in message news:<SH87d.462 5$d5.36294@news b.telia.net>...[color=green]
>> *lawrence* skrev 2004-10-01 09:34:[color=darkred]
>> > I'd love to ask why this page is not rendering correctly in Safari on
>> > a Macintosh but I suspect someone will tell me to validate the page
>> > first. Nevertheless, if anyone sees an obvious reason that I'm
>> > missing, I'd like to know. It looks like a missing div tag but I
>> > can't see one.
>> >
>> > http://www.krubner.com/
>> >
>> >
>> > Let's move on to a question that might be answerable. If i copy and
>> > paste non-UTF-8 characters to the page, and then send out a UTF-8
>> > charset UTF-8 header then I'll get the garbage characters that I'm
>> > seeing?[/color]
>>
>> If you set charset UTF-8 in the header, you also have to save the
>> document as UTF-8 before uploading it. I belive it's saved in i default
>> charset, probably iso-8859-1[/color]
>
> Since sending out UTF-8 headers, I've started to get a lot of garbage
> characters on the page. This despit hitting it with encode_UTF-8 in
> PHP. The problem is I don't know what charset the text is in a head of
> time, so I can't do any kind of proper conversion on the text. The web
> is too heterogenous. I wonder how Blogger and TypePad tackle this
> problem? Is it just luck that so few of their users end up with
> garbage characters? Or is it because they standardize on a different
> charset?[/color]

What editor do you use for building and editing the pages?

In the editor I use, I can choose whatever encoding I want to use, and
on saving the file is saved in that encoding. If I open a file I can
change the encoding the same way, with just a click on a button in the
editors toolbar. In Windows XP you can also choose encoding for files
when saving them in Notebook.

Because I save the files in UTF-8 I don't manually need to do any
conversion in the text, when writing it. My guess is that Blogger and
TypePad is done the same way, the posts that the user write is saved as
UTF-8 and since that is a unicod, all kind of characters and languages
can be used.

--
/Arne

**lawrence** · Jul 23 '05, 10:18 PM

Re: UTF-8 garbage characters

Arne <arne@luras.n u> wrote in message news:<%9u7d.473 2$d5.37631@news b.telia.net>...[color=blue][color=green]
> > Since sending out UTF-8 headers, I've started to get a lot of garbage
> > characters on the page. This despit hitting it with encode_UTF-8 in
> > PHP. The problem is I don't know what charset the text is in a head of
> > time, so I can't do any kind of proper conversion on the text. The web
> > is too heterogenous. I wonder how Blogger and TypePad tackle this
> > problem? Is it just luck that so few of their users end up with
> > garbage characters? Or is it because they standardize on a different
> > charset?[/color]
>
> What editor do you use for building and editing the pages?[/color]

Most people use Microsoft Internet Explorer to build pages on their
weblogs. That is, they log in and type some text in a TEXTAREA and hit
"Post" and then there words appear as a new page on the web.

The problem arises when they copy and paste from other places. The
biggest problems, I think, arise when they copy text off a webpage
that uses another web page encoding.

You can check see for yourself here:

404 Not Found

http://www.publicdomainsoftware.org/designer/

You'll need a username and password to login. Use these:

username: designer
password: designer123

Link for login at bottom.

[color=blue]
> Because I save the files in UTF-8 I don't manually need to do any
> conversion in the text, when writing it. My guess is that Blogger and
> TypePad is done the same way, the posts that the user write is saved as
> UTF-8 and since that is a unicod, all kind of characters and languages
> can be used.[/color]

I guess that is the theory that now needs to be tested. My suspicion
is the opposite of yours - they are using something other than UTF-8,
which is why garbage characters appear if you copy something from them
and paste it into a web page that is using UTF-8.

**Daniel R. Tobias** · Jul 23 '05, 10:19 PM

Re: UTF-8 garbage characters

lkrubner@geocit ies.com (lawrence) wrote in message news:<da7e68e8. 0410041836.216a 8135@posting.go ogle.com>...[color=blue]
> Arne <arne@luras.n u> wrote in message news:<%9u7d.473 2$d5.37631@news b.telia.net>...
> Most people use Microsoft Internet Explorer to build pages on their
> weblogs. That is, they log in and type some text in a TEXTAREA and hit
> "Post" and then there words appear as a new page on the web.[/color]

I would have thought that bloggers would be the sort to be clueful
enough to use decent browsers, like Mozilla, instead.

--
Dan

**Arne** · Jul 23 '05, 10:19 PM

Re: UTF-8 garbage characters

*Daniel R. Tobias* skrev 2004-10-05 22:49:[color=blue]
> lkrubner@geocit ies.com (lawrence) wrote in message news:<da7e68e8. 0410041836.216a 8135@posting.go ogle.com>...[color=green]
>> Arne <arne@luras.n u> wrote in message news:<%9u7d.473 2$d5.37631@news b.telia.net>...
>> Most people use Microsoft Internet Explorer to build pages on their
>> weblogs. That is, they log in and type some text in a TEXTAREA and hit
>> "Post" and then there words appear as a new page on the web.[/color]
>
> I would have thought that bloggers would be the sort to be clueful
> enough to use decent browsers, like Mozilla, instead.[/color]

Ok. I see noe what Lawrence's problem is. The browser used don't matter
here. When testing the blog I used som words from my native language
(Swedish) and writed this:

The problem with charset UTF-8 on pages with forms for e.g. guestbooks,
formmail and bloggs is that writing in a non-english language can give
garbage characters from the letters that is not represented in the
english language. That's because what is writed in the text box don't
get encoded, as text done with HTML editors does.

So the best may be to use an other charset (is ISO-8859-1 the best?) for
pages with forms and textbox. ISO-8859-1 cover a lot of languages with
"strange" characters, but far from all, so the problem may not totally
be solved.

You can write the message in a separate software (sutch as Notebook) and
then encode foreign letters to entities. As an example the letter "å"
(latin small letter a with ring above) to å but then, will the
å look like "å" å or on the page after posting?

--
/Arne

**Pierre Goiffon** · Jul 23 '05, 10:19 PM

Re: UTF-8 garbage characters

"Arne" <arne@luras.n u> a écrit dans le message de
news:4tE8d.1051 80$dP1.388564@n ewsc.telia.net[color=blue]
> The problem with charset UTF-8 on pages with forms for e.g.
> guestbooks, formmail and bloggs is that writing in a non-english
> language can give garbage characters from the letters that is not
> represented in the english language. That's because what is writed in
> the text box don't get encoded, as text done with HTML editors does.[/color]

I really can't understand your post. A server that sends a form to a client
with the appropriate charset headers should get in return all the users
input encoded in that charset. If the form is sent with a UTF-8 header, you
should get all the characters encoded in UTF-8. And so user could input any
character included in Unicode.

**Alan J. Flavell** · Jul 23 '05, 10:19 PM

Re: UTF-8 garbage characters

On Wed, 6 Oct 2004, Pierre Goiffon wrote:
[color=blue]
> "Arne" <arne@luras.n u> a écrit dans le message de
> news:4tE8d.1051 80$dP1.388564@n ewsc.telia.net[color=green]
> > The problem with charset UTF-8 on pages with forms for e.g.
> > guestbooks, formmail and bloggs is that writing in a non-english
> > language can give garbage characters from the letters that is not
> > represented in the english language. That's because what is writed in
> > the text box don't get encoded, as text done with HTML editors does.[/color]
>
> I really can't understand your post. A server that sends a form to a client
> with the appropriate charset headers should get in return all the users
> input encoded in that charset.[/color]

"should" sounds right, but it isn't always going to work, depending on
all kinds of browser bugs and oddities, to say nothing of the
uncertainty of what should happen with method GET (which /officially/
only supports us-ascii).
[color=blue]
> If the form is sent with a UTF-8 header, you should get all the
> characters encoded in UTF-8. And so user could input any character
> included in Unicode.[/color]

On the other hand, the combination of user errors, browser bugs and
plain malice means that your server *could* get presented with all
kinds of rubbish: you might not be able to do anything useful with it,
but you better program defensively to ensure it won't do you any harm.
(E.g you better validate any byte-sequences that are supposed to be
utf-8 to make sure they really are - the unicode spec effectively
mandates such validation, and failure to do so could be rated as a
security exposure).

I haven't worked this page over in a while, but it gives a general
flavour of what I found, reviewed against what the specifications say
(and - also important - what they /don't/ say):

404 Not Found

http://ppewww.ph.gla.ac.uk/~flavell/charset/form-i18n.html

h.t.h

**Andreas Prilop** · Jul 23 '05, 10:19 PM

Re: UTF-8 garbage characters

On Wed, 6 Oct 2004, Pierre Goiffon wrote:
[color=blue]
> A server that sends a form to a client
> with the appropriate charset headers should get in return all the users
> input encoded in that charset. If the form is sent with a UTF-8 header, you
> should get all the characters encoded in UTF-8. And so user could input any
> character included in Unicode.[/color]

Try
<http://google.com/search?q=%EA%E5 %F1%DC%F4%E9%EF .%ED&ie=ISO-8859-7&oe=UTF-8>
with Netscape 4.x.

--
Top-posting.
What's the most irritating thing on Usenet?

**Arne** · Jul 23 '05, 10:20 PM

Re: UTF-8 garbage characters

*Pierre Goiffon* skrev 2004-10-06 10:29:[color=blue]
> "Arne" <arne@luras.n u> a écrit dans le message de
> news:4tE8d.1051 80$dP1.388564@n ewsc.telia.net[color=green]
>> The problem with charset UTF-8 on pages with forms for e.g.
>> guestbooks, formmail and bloggs is that writing in a non-english
>> language can give garbage characters from the letters that is not
>> represented in the english language. That's because what is writed in
>> the text box don't get encoded, as text done with HTML editors does.[/color]
>
> I really can't understand your post. A server that sends a form to a client
> with the appropriate charset headers should get in return all the users
> input encoded in that charset. If the form is sent with a UTF-8 header, you
> should get all the characters encoded in UTF-8. And so user could input any
> character included in Unicode.
>[/color]

Yes, the servers *should* do that, but I have yet to see a server that
does. Google is the exeption to this.

When I write the Swedish word "räksmörgås " to Google I can see in the
search string code on the output page that Google encoded it to
r%C3%A4ksm%C3%B 6rg%C3%A5s and that is encoded back to "räksmörgås " in
the links Google finds. But that is still to happend in other servers.

E.g many websites by Swedes (in Swedish or other languages) use the free
guestbook services find on Internet (guestbook files is on a remote
server). Mostly they are using the charset ISO-8859-1 but if they use
UTF-8 the garbage is those guestbooks also.

--
/Arne

**Andreas Prilop** · Jul 23 '05, 10:20 PM

Re: UTF-8 garbage characters

On Wed, 6 Oct 2004, Arne wrote:
[color=blue]
> When I write the Swedish word "räksmörgås " to Google[/color]

"Google" is no URL!
There's <http://www.google.se/webhp?oe=ISO-8859-1> and
there's <http://www.google.se/webhp?oe=UTF-8> .
[color=blue]
> I can see in the
> search string code on the output page that Google encoded it to
> r%C3%A4ksm%C3%B 6rg%C3%A5s[/color]

Not necessarily. You can also have
<http://www.google.se/search?ie=ISO-8859-1&oe=ISO-8859-1&q=r%E4ksm%F6r g%E5s>

--
Top-posting.
What's the most irritating thing on Usenet?

**Arne** · Jul 23 '05, 10:20 PM

Re: UTF-8 garbage characters

*Andreas Prilop* skrev 2004-10-06 17:53:
[color=blue]
> On Wed, 6 Oct 2004, Arne wrote:
>[color=green]
>> When I write the Swedish word "räksmörgås " to Google[/color]
>
> "Google" is no URL![/color]

Did I said that? It's a Search service, or what ever you like to call
it, on the Internet. Goofle is used to search URL's from a webpage.
[color=blue]
> There's <http://www.google.se/webhp?oe=ISO-8859-1> and
> there's <http://www.google.se/webhp?oe=UTF-8> .[/color]

I only type in www.google.se (or google.com) to get the page with the
search form, as I guess most people do. And in all cases the meta tag is
<meta http-equiv="content-type" content="text/html; charset=UTF-8">

BTW, I can give you http://www.google.se/webhp?hl=ISO-8859-1 also, but
that don't change the charset, only the prefered language from Swedish
to English on the search page.

[color=blue][color=green]
>> I can see in the
>> search string code on the output page that Google encoded it to
>> r%C3%A4ksm%C3%B 6rg%C3%A5s[/color]
>
> Not necessarily. You can also have
> <http://www.google.se/search?ie=ISO-8859-1&oe=ISO-8859-1&q=r%E4ksm%F6r g%E5s>
>[/color]

What are you trying to prof? Why should I use URL's like that? :-)

--
/Arne

**Arne** · Jul 23 '05, 10:20 PM

Re: UTF-8 garbage characters

*Arne* skrev 2004-10-06 19:16:
[color=blue]
> it, on the Internet. Goofle is used to search URL's from a webpage.[/color]
^
Bah, damn typos! Google!!! :-)

--
/Arne

**Pierre Goiffon** · Jul 23 '05, 10:20 PM

Re: UTF-8 garbage characters

"Arne" <arne@luras.n u> a écrit dans le message de
news:OOT8d.5121 $d5.41952@newsb .telia.net[color=blue][color=green]
>> I really can't understand your post. A server that sends a form to a
>> client with the appropriate charset headers should get in return all
>> the users input encoded in that charset.[/color][/color]
(...)

Alan, Andreas, Arne, thanks for your answers.
Well I saw on different technical documents that the GET method is really
buggy (characters sent back by the browser to the server aren't always
encoded using the appropriate charset). But I never heard of such problems
with the POST method ? Does anyone made a few sample tests with it ?

Andreas , I didn't understand what to see on the web page you indicated :[color=blue]
> Try
>[/color]
<http://google.com/search?q=%EA%E5%F1...859-7&oe=UTF-8[color=blue]
>
> with Netscape 4.x.[/color]
I open this using Netscape 4.75 and, well, I can't really read arabic and
russian alphabet but everything seems ok compared with Mozilla 1.7.3 (apart
for the arabic characters in the title).

UTF-8 garbage characters

UTF-8 garbage characters

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment