UTF8: file_put_contents doesn't seem to write UTF8 content properly

[amygdala]

Hi,

I'm trying to let PHP write a 'sitemap.xml' sitemap for Google and other
searchengines. It's working, except that the content in the XML file doesn't
seem to be UTF8. (Which it should be, judging by the information given on
Google's webmaster helpcenter).

The way I test to see if the content is UTF8, is by opening the XML file in
notepad and choose 'save as...'. Normally the coding option should be set to
UTF8, but now it just shows ANSI.

This is what I have tried to write UTF8 content with:

file_put_conten ts( '.' . SITEMAP_FILE, utf8_encode(
$this->sitemapForCraw lers ) );
....and...
file_put_conten ts( '.' . SITEMAP_FILE, iconv( "ISO-8859-1", "UTF8",
$this->sitemapForCraw lers ) );

....where...
SITEMAP_FILE is the filename constant
....and...
$this->sitemapForCraw lers is the string with XML data

With the last attempt I even got an error saying:

Wrong charset, conversion from `ISO-8859-1' to `UTF8' is not allowed in...


Any adeas of how I can make this work?

Thanks for the input.

[Andy Hassall]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly

On Wed, 13 Jun 2007 22:25:44 +0200, "amygdala" <noreply@norepl y.comwrote:

>I'm trying to let PHP write a 'sitemap.xml' sitemap for Google and other
>searchengine s. It's working, except that the content in the XML file doesn't
>seem to be UTF8. (Which it should be, judging by the information given on
>Google's webmaster helpcenter).
>
>The way I test to see if the content is UTF8, is by opening the XML file in
>notepad and choose 'save as...'. Normally the coding option should be set to
>UTF8, but now it just shows ANSI.

Well, that's not a foolproof method...

>This is what I have tried to write UTF8 content with:
>
>file_put_conte nts( '.' . SITEMAP_FILE, utf8_encode(
>$this->sitemapForCraw lers ) );
>...and...
>file_put_conte nts( '.' . SITEMAP_FILE, iconv( "ISO-8859-1", "UTF8",
>$this->sitemapForCraw lers ) );
>
>...where...
>SITEMAP_FILE is the filename constant
>...and...
>$this->sitemapForCraw lers is the string with XML data
>
>With the last attempt I even got an error saying:
>
>Wrong charset, conversion from `ISO-8859-1' to `UTF8' is not allowed in...
>
>Any adeas of how I can make this work?

Start from the beginning; what character set encoding is the original data in?
The error implies that it's not ISO-8859-1 (which does have some gaps where
characters aren't valid...)

--
Andy Hassall :: andy@andyh.co.u k :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

[C.]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly

On 13 Jun, 21:25, "amygdala" <nore...@norepl y.comwrote:

>
The way I test to see if the content is UTF8, is by opening the XML file in
notepad and choose 'save as...'. Normally the coding option should be set to
UTF8, but now it just shows ANSI.
>

ROFL.

Try inserting a BOM in front of the content.

C.

[amygdala]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly


"Andy Hassall" <andy@andyh.co. ukschreef in bericht
news:nbn073pqso 001bsrhpkmpcqvn joulkjjlb@4ax.c om...

On Wed, 13 Jun 2007 22:25:44 +0200, "amygdala" <noreply@norepl y.com>
wrote:
>

>>I'm trying to let PHP write a 'sitemap.xml' sitemap for Google and other
>>searchengines . It's working, except that the content in the XML file
>>doesn't
>>seem to be UTF8. (Which it should be, judging by the information given on
>>Google's webmaster helpcenter).
>>
>>The way I test to see if the content is UTF8, is by opening the XML file
>>in
>>notepad and choose 'save as...'. Normally the coding option should be set
>>to
>>UTF8, but now it just shows ANSI.

>
Well, that's not a foolproof method...

I was afraid of that.

>>This is what I have tried to write UTF8 content with:
>>
>>file_put_cont ents( '.' . SITEMAP_FILE, utf8_encode(
>>$this->sitemapForCraw lers ) );
>>...and...
>>file_put_cont ents( '.' . SITEMAP_FILE, iconv( "ISO-8859-1", "UTF8",
>>$this->sitemapForCraw lers ) );
>>
>>...where...
>>SITEMAP_FIL E is the filename constant
>>...and...
>>$this->sitemapForCraw lers is the string with XML data
>>
>>With the last attempt I even got an error saying:
>>
>>Wrong charset, conversion from `ISO-8859-1' to `UTF8' is not allowed in...
>>
>>Any adeas of how I can make this work?

>
Start from the beginning; what character set encoding is the original data
in?
The error implies that it's not ISO-8859-1 (which does have some gaps
where
characters aren't valid...)

Well... I discovered the 'Set Code Page...' option in UltraEdit, the main
editor I use to code PHP. And it tells me my PHP code files are encoded in
'1252 (ANSI - Latin I)'. So, now my next question is... what would be the
correct first parameter for the iconv function to tell it that the original
data is '1252 (ANSI - Latin I)'. I've tried numerous stings, which include:

'1252 (ANSI - Latin I)'
'1252'
'1252 ANSI'
'1252-ANSI'
'ANSI-1252'
'ANSI 1252'

....and variations.

Is there any iconv encoding table with acceptable encodings I can consult?
Also, isn't '1252 (ANSI - Latin I)' just a pimped version of ISO-8859-1?

Although I'm still curious of this. Please read my reply to C. also.

Thanks.

[amygdala]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly


"C." <colin.mckinnon @gmail.comschre ef in bericht
news:1181771230 .035994.248850@ e9g2000prf.goog legroups.com...

On 13 Jun, 21:25, "amygdala" <nore...@norepl y.comwrote:

>>
>The way I test to see if the content is UTF8, is by opening the XML file
>in
>notepad and choose 'save as...'. Normally the coding option should be set
>to
>UTF8, but now it just shows ANSI.
>>

>
ROFL.

Yes, very amusing. :-/

Try inserting a BOM in front of the content.

Ok, I did a little research on BOM. And came up with information that tells
me the BOM isn't particularly necessary for UTF-8. Then I ran a simple test
with utf8_encode:

<?php
echo utf8_encode( 'ï' );
?>

Which output looks like it works just fine:

ï

Since I've concluded (see my reply to Andy Hassall) that my files are
encoded in '1252 (ANSI - Latijn I)' and this test file was also '1252
(ANSI - Latijn I)' I guess it works. And I don't necessary have to provide a
BOM and don't have to resort to iconv. Correct?

Or am I missing something vital here?

Thanks.

[Andy Hassall]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly

On Thu, 14 Jun 2007 03:39:34 +0200, "amygdala" <noreply@norepl y.comwrote:

>Start from the beginning; what character set encoding is the original data
>in?
>The error implies that it's not ISO-8859-1 (which does have some gaps
>where
>characters aren't valid...)

>
>Well... I discovered the 'Set Code Page...' option in UltraEdit, the main
>editor I use to code PHP. And it tells me my PHP code files are encoded in
>'1252 (ANSI - Latin I)'.

Well... again, that's not foolproof. It's generally not possible to
definitively detect the encoding of a file. You can work out whether it's
impossible to be in a particular encoding (invalid characters or byte
sequences), and you can make some guesses on character distribution or
spellings of words, but unless it's tagged in some way (like HTML and XML, or
through another channel like HTTP headers) then it's not certain.

"Windows Codepage 1252" is a Windows character set encoding that is similar,
but not exactly the same as ISO-8859-1. It (1252) differs on the location of
the Euro character, and has a few extra characters in a range that is reserved
in ISO-8859-1.

Do you have any Euro currency symbols in the file?

>So, now my next question is... what would be the
>correct first parameter for the iconv function to tell it that the original
>data is '1252 (ANSI - Latin I)'. I've tried numerous stings, which include:
>
>'1252 (ANSI - Latin I)'
>'1252'
>'1252 ANSI'
>'1252-ANSI'
>'ANSI-1252'
>'ANSI 1252'
>
>...and variations.
>
>Is there any iconv encoding table with acceptable encodings I can consult?

http://www.gnu.org/software/libiconv/

You possibly want:

CP1252

>Also, isn't '1252 (ANSI - Latin I)' just a pimped version of ISO-8859-1?

I should read the entire message before typing ;-)

--
Andy Hassall :: andy@andyh.co.u k :: http://www.andyh.co.uk
http://www.andyhsoftware.co.uk/space :: disk and FTP usage analysis tool

[Willem Bogaerts]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly

I'm trying to let PHP write a 'sitemap.xml' sitemap for Google and other

searchengines. It's working, except that the content in the XML file doesn't
seem to be UTF8. (Which it should be, judging by the information given on
Google's webmaster helpcenter).

How can you tell? YOU tell the system what encoding is used. The system
rarely tells you, as bytes can be perfectly valid text in a lot of
encodings and look very different in each of them.

Even if the system tells you, it usually does so separately from the
text itself. Which is obvious, because you need the encoding to be able
to read the text! In webpages and e-mail, for example, headers are used
to set the encoding of the data.

I suggest you search the net for encodings and how to work with them.
This is a good start:

http://www.joelonsoftware.com/articles/Unicode.html

Good luck with the onions,
--
Willem Bogaerts

Application smith
Kratz B.V.

http://www.kratz.nl/

[amygdala]

Re: UTF8: file_put_conten ts doesn't seem to write UTF8 content properly


"Willem Bogaerts" <w.bogaerts@kra tz.maardanzonde rditstuk.nlschr eef in
bericht news:4672410c$0 $331$e4fe514c@n ews.xs4all.nl.. .

>I'm trying to let PHP write a 'sitemap.xml' sitemap for Google and other
>searchengine s. It's working, except that the content in the XML file
>doesn't
>seem to be UTF8. (Which it should be, judging by the information given on
>Google's webmaster helpcenter).

>
How can you tell? YOU tell the system what encoding is used. The system
rarely tells you, as bytes can be perfectly valid text in a lot of
encodings and look very different in each of them.

Yes, you're right of course, what was I thinking.

Even if the system tells you, it usually does so separately from the
text itself. Which is obvious, because you need the encoding to be able
to read the text! In webpages and e-mail, for example, headers are used
to set the encoding of the data.
>
I suggest you search the net for encodings and how to work with them.
This is a good start:
>
http://www.joelonsoftware.com/articles/Unicode.html

Great article. Thanks for the pointer.

Good luck with the onions,

Hopefully that won't be necessary anymore.
Cheers.

--
Willem Bogaerts
>
Application smith
Kratz B.V.
http://www.kratz.nl/