Character Problem...

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • frizzle

    #1

    Character Problem...

    Hi there!

    I have (for sofar) a small problem:

    I have a site, with mySQL-backend, and an online
    admin system.

    The site is in three languages: 1 of them is German.
    My poblem concerns the 'ß' (without quotes), and
    probably more characters.

    When an admin enters it in a textfield, and then
    writes it into the DB, i use mysql_real_esca pe_string()
    to prevent any mistakes. When an admin wants to edit
    the record with the 'ß' in it, it's displayed just right
    in the textfield, but when someone views it through a
    regular page, they see & # 9 4 6 ; (without spaces...).
    To display DB-contents in a page, i use
    htmlentities(st ripslashes($str ing)).

    Where is the flaw?

    Thanks in advance!

  • Brion Vibber

    #2
    Re: Character Problem...

    frizzle wrote:[color=blue]
    > When an admin wants to edit
    > the record with the 'ß' in it, it's displayed just right
    > in the textfield, but when someone views it through a
    > regular page, they see & # 9 4 6 ; (without spaces...).[/color]

    β is an HTML numeric representation of the the Greek lowercase
    letter beta (β). If you want the German letter ß, somebody's typing the
    wrong thing in.

    -- brion vibber (brion @ pobox.com)

    Comment

    • frizzle

      #3
      Re: Character Problem...

      Ok, my bad... :$

      Anyway, my concern is the transformation before it's put into
      the DB. Normal special chars like <, &, >, and ", etc. enter the DB
      perfect....

      Exept this Greek character and the ß and probably more are messed
      up...

      Comment

      • Michael Fesser

        #4
        Re: Character Problem...

        .oO(frizzle)
        [color=blue]
        >To display DB-contents in a page, i use
        >htmlentities(s tripslashes($st ring)).[/color]

        Why stripslashes()? And why htmlentities()?
        htmlspecialchar s() should be more than enough.

        Micha

        Comment

        • frizzle

          #5
          Re: Character Problem...

          Too bad, i still get the same error... :(

          (Btw, does htmlspecialchar s strip slashes?)

          Comment

          • Brion Vibber

            #6
            Re: Character Problem...

            frizzle wrote:[color=blue]
            > Too bad, i still get the same error... :([/color]

            Your problem is likely because the wrong character is *already* in the
            database. No matter how you display it, it's still going to be wrong.

            If your pages are in an 8-bit encoding (as opposed to UTF-8), some web
            browsers will transform Unicode characters outside the 8-bit encoding's
            range into numbered character references such as β when submitting
            form data.

            If you then output the β *raw* into page output, you will see the
            Greek letter lowercase beta. If you output it escaped by htmlentities()
            or htmlspecialchar s(), then you will see the "β" string. (View
            Source will show "&amp;#946; ")

            Since you say it looks ok in your admin system, this may indicate that
            your admin system is outputting data without HTML encoding; if true this
            may open you to a cross-site scripting attack where a malicious user
            could submit data which will execute JavaScript when your administrator
            loads up a page with that data. The JavaScript could for instance send
            login credentials to the malicious user, or submit forms back to the
            admin system to delete or alter data in your system.
            [color=blue]
            > (Btw, does htmlspecialchar s strip slashes?)[/color]

            No, and that would be very surprising if it did.

            If you have to stripslashes on output, then either of these two may be true:
            * You have magic_quotes_ru ntime on, which will corrupt all data when
            reading it from the database, text files, or various other sources by
            adding slashes. In this case, your data in the database would be correct
            but you are getting extra slashes when it's read into your script.
            * You are over-escaping your input to the database. For instance, if you
            have magic_quotes_gp c on, and you take $_POST contents and run it
            directly through addslashes() before inserting it into an SQL statement.
            In this case, the data in your database is wrong and contains extra slashes.

            Generally, you should probably have magic_quotes_ru ntime off (it's not
            well documented enough to be predictable as far as what it affects), and
            should either be in a position to enforce magic_quotes_gp c setting or
            detect it and accordingly remove slashes from form input, then add
            slashes (once!) when creating your SQL strings.

            -- brion vibber (brion @ pobox.com)

            Comment

            • frizzle

              #7
              Re: Character Problem...

              Wow, thanks for the explanation!

              When i enter the data, and i view through php-myAdmin i see the code.
              To enter it into the db i use mysql_escape_st ring. This to prevent
              people with bad intentions from dropping my db for instance...

              Comment

              • frizzle

                #8
                Re: Character Problem...

                By the way, running phpinfo() a discovered the following:
                Directive Local Value Master Value
                magic_quotes_gp c On On
                magic_quotes_ru ntime Off Off
                magic_quotes_sy base Off Off


                Frizzle

                Comment

                • Andy Hassall

                  #9
                  Re: Character Problem...

                  On Tue, 08 Mar 2005 14:42:50 -0800, Brion Vibber <brion@pobox.co m> wrote:
                  [color=blue]
                  >frizzle wrote:[color=green]
                  >> Too bad, i still get the same error... :([/color]
                  >
                  >Your problem is likely because the wrong character is *already* in the
                  >database. No matter how you display it, it's still going to be wrong.
                  >
                  >If your pages are in an 8-bit encoding (as opposed to UTF-8), some web
                  >browsers will transform Unicode characters outside the 8-bit encoding's
                  >range into numbered character references such as β when submitting
                  >form data.[/color]

                  Sounds like this is the case - if you don't specify character encodings,
                  there's various points in the chain that will assume ISO-8859-1 which does not
                  contain 'GREEK SMALL LETTER BETA' (U+03B2) a.k.a. β since it's not needed
                  for the Western European languages that ISO-8859-1 targets (it contains 'SMALL
                  GERMAN LETTER SHARP s' U+00DF [a.k.a. "Eszett"] which looks similar to a Greek
                  beta, but isn't one).

                  The world probably ought to catch up and default to ISO-8859-15 instead of
                  ISO-8859-1 but that would be the subject of a separate rant ;-)
                  [color=blue]
                  >If you then output the β *raw* into page output, you will see the
                  >Greek letter lowercase beta. If you output it escaped by htmlentities()
                  >or htmlspecialchar s(), then you will see the "β" string. (View
                  >Source will show "&amp;#946; ")[/color]

                  --
                  Andy Hassall / <andy@andyh.co. uk> / <http://www.andyh.co.uk >
                  <http://www.andyhsoftwa re.co.uk/space> Space: disk usage analysis tool

                  Comment

                  • Brion Vibber

                    #10
                    Re: Character Problem...

                    frizzle wrote:[color=blue]
                    > By the way, running phpinfo() a discovered the following:
                    > Directive Local Value Master Value
                    > magic_quotes_gp c On On
                    > magic_quotes_ru ntime Off Off
                    > magic_quotes_sy base Off Off[/color]

                    Ok, this means that when someone types "It's good" into a form and
                    submits it, your $_GET/$_POST/$_REQUEST/$_COOKIE variables will contain
                    a slashed version: "It\'s good".

                    If you then take this value and run it through mysql_escape_st ring(),
                    you end up with "It\\\'s good", which gets put into an SQL statement
                    something like "INSERT INTO blah VALUES ('It\\\'s good')".

                    The database engine understands the first level of slashes to keep the
                    string together, and inserts the value "It\'s good" into the table. When
                    you read it back out from the database ("SELECT * FROM blah"), you get
                    back "It\'s good", which contains too many slashes.


                    What you probably should be doing is to detect whether magic_quotes_gp c
                    is on when you first get your input from $_GET/$_POST/$_REQUEST/etc, and
                    use stripslashes() there to get the _correct_ "It's good" value. (It's
                    unfortunate that these things are optional and sometimes on/sometimes
                    off like that; it makes it unnecessarily hard to write solid, safe,
                    portable PHP code.)

                    Then when you use mysql_escape_st ring() to make your SQL statement, it
                    will be correctly escaped ("INSERT INTO blah VALUES('It\'s good')"), the
                    correct value will be stored ("It's good"), and the correct value will
                    be read back and displayed ("It's good").

                    This is separate from running htmlentities() or htmlspecialchar s() on
                    output, which you still need to do anyway when outputting text into an
                    HTML page.

                    -- brion vibber (brion @ pobox.com)

                    Comment

                    • frizzle

                      #11
                      Re: Character Problem...

                      Wow! Problem Solved! :)

                      I missed out something earlier in the replies, i deleted the
                      original Greek character from the DB, and inserted 'Ze German' one..
                      And now it looks just fine.

                      I only have small problems yet with the escaping quotes etc. part.
                      But i don't expect texts to contain lots of quotes etc. behind each
                      other...

                      Well thanks anyway people, a lot of interesting information in here...

                      Thanks people!

                      Comment

                      • Markku Uttula

                        #12
                        Re: Character Problem...

                        Andy Hassall wrote:[color=blue]
                        > The world probably ought to catch up and default to ISO-8859-15
                        > instead of ISO-8859-1 but that would be the subject of a separate
                        > rant ;-)[/color]

                        What would this solve? ISO-8859-15 is a very tiny addition to
                        ISO-8859-1 ... see http://www.cs.tut.fi/~jkorpela/latin9.html for more
                        details. The world would be better if everybody started using UTF-8 as
                        default. I know this would bring many difficulties, but I also believe
                        that the pros outweight the cons in this matter.

                        --
                        Markku Uttula

                        Comment

                        • Andy Hassall

                          #13
                          Re: Character Problem...

                          On Wed, 9 Mar 2005 17:00:01 +0200, "Markku Uttula"
                          <markku.uttula@ disconova.com> wrote:
                          [color=blue]
                          >Andy Hassall wrote:[color=green]
                          >> The world probably ought to catch up and default to ISO-8859-15
                          >> instead of ISO-8859-1 but that would be the subject of a separate
                          >> rant ;-)[/color]
                          >
                          >What would this solve? ISO-8859-15 is a very tiny addition to
                          >ISO-8859-1 ... see http://www.cs.tut.fi/~jkorpela/latin9.html for more
                          >details.[/color]

                          It's just the the Euro currency symbol that I'm thinking of really - quite
                          important for a Western European character set!
                          [color=blue]
                          > The world would be better if everybody started using UTF-8 as
                          >default. I know this would bring many difficulties, but I also believe
                          >that the pros outweight the cons in this matter.[/color]

                          Unfortunately the cons are significant, even if they do outweigh the pros.

                          --
                          Andy Hassall / <andy@andyh.co. uk> / <http://www.andyh.co.uk >
                          <http://www.andyhsoftwa re.co.uk/space> Space: disk usage analysis tool

                          Comment

                          Working...