Unicode Character Issue

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Samuel

    #1

    Unicode Character Issue

    Hi

    I am trying to read text files that are saved in ANSI format with Unicode
    characters such as French e German big S etc, and as I read the file these
    characters appear as squares etc.

    I know that if the file would be saved as Unicode this wouldn't be a
    problem.

    The question is whether there is an option that when I create the Stream
    Reader the application will recognize it as Unicode characters.


    Thank you,
    Samuel


  • Joe Duchtel

    #2
    Re: Unicode Character Issue

    Hello -

    There is an overloaded constructor that takes an Encoding (e.g.
    System.Text.Uni codeEncoding). Visual Basic treats all strings as
    Unicode so if you do a ReadLine() to a String it should work.

    Joe



    On Jun 4, 12:04 pm, "Samuel" <samuel.shul... @ntlworld.comwr ote:
    Hi
    >
    I am trying to read text files that are saved in ANSI format with Unicode
    characters such as French e German big S etc, and as I read the file these
    characters appear as squares etc.
    >
    I know that if the file would be saved as Unicode this wouldn't be a
    problem.
    >
    The question is whether there is an option that when I create the Stream
    Reader the application will recognize it as Unicode characters.
    >
    Thank you,
    Samuel

    Comment

    • Joe Duchtel

      #3
      Re: Unicode Character Issue

      Hello -

      There is an overloaded constructor that takes an Encoding (e.g.
      System.Text.Uni codeEncoding). Visual Basic treats its String as
      Unicode so you should be find if you use ReadLine().

      Joe



      On Jun 4, 12:04 pm, "Samuel" <samuel.shul... @ntlworld.comwr ote:
      Hi
      >
      I am trying to read text files that are saved in ANSI format with Unicode
      characters such as French e German big S etc, and as I read the file these
      characters appear as squares etc.
      >
      I know that if the file would be saved as Unicode this wouldn't be a
      problem.
      >
      The question is whether there is an option that when I create the Stream
      Reader the application will recognize it as Unicode characters.
      >
      Thank you,
      Samuel

      Comment

      • Joergen Bech

        #4
        Re: Unicode Character Issue


        Here is a small, working example. Disregard
        the ReadBinary stuff. That was just something
        from another project and could be written
        smaller/cleaner/etc.

        Regards,

        Joergen Bech

        ---snip---

        Option Explicit On
        Option Strict On

        Imports System.IO
        Imports System.Text
        Imports System

        Public Class Form1

        Private Sub Form1_Load(ByVa l sender As System.Object, ByVal e As
        System.EventArg s) Handles MyBase.Load
        Dim ansi() As Byte = ReadBinary("d:\ ansi.txt")
        Dim s As String =
        Encoding.GetEnc oding("iso-8859-1").GetString(a nsi)
        TextBox1.Text = s

        ' Dim sb As New StringBuilder
        ' For Each b As Byte In ansi
        ' sb.Append(Chr(b ))
        ' Next
        ' TextBox1.Text = sb.ToString
        End Sub

        Public Shared Function ReadBinary(ByVa l file As String) As Byte()
        Dim errorInformatio n As String = String.Empty
        Dim result As Byte() = ReadBinary(file , errorInformatio n)
        If String.IsNullOr Empty(errorInfo rmation) Then
        Return result
        Else
        Throw New IOException(err orInformation)
        End If
        End Function

        Public Shared Function ReadBinary(ByVa l file As String, _
        ByRef errorInformatio n As
        String) As Byte()

        Try
        Dim fInfo As New FileInfo(file)
        Dim numBytes As Long = fInfo.Length
        Dim fStream As New FileStream(file , FileMode.Open,
        FileAccess.Read )
        Dim br As New BinaryReader(fS tream)
        Dim data() As Byte = br.ReadBytes(CI nt(numBytes))

        br.Close()
        fStream.Close()

        Return data

        Catch ex As Exception
        errorInformatio n = ex.Message
        Return Nothing

        End Try

        End Function



        On Wed, 4 Jun 2008 17:04:48 +0100, "Samuel"
        <samuel.shulman @ntlworld.comwr ote:
        >Hi
        >
        >I am trying to read text files that are saved in ANSI format with Unicode
        >characters such as French e German big S etc, and as I read the file these
        >characters appear as squares etc.
        >
        >I know that if the file would be saved as Unicode this wouldn't be a
        >problem.
        >
        >The question is whether there is an option that when I create the Stream
        >Reader the application will recognize it as Unicode characters.
        >
        >
        >Thank you,
        >Samuel
        >

        Comment

        • Samuel

          #5
          Re: Unicode Character Issue

          Unicode didn't work but 'Default' did

          Thank you,
          Samuel


          "Joe Duchtel" <duchtel@gmail. comwrote in message
          news:aeb93576-8529-4e33-9817-891a7ce233a5@79 g2000hsk.google groups.com...
          Hello -

          There is an overloaded constructor that takes an Encoding (e.g.
          System.Text.Uni codeEncoding). Visual Basic treats all strings as
          Unicode so if you do a ReadLine() to a String it should work.

          Joe



          On Jun 4, 12:04 pm, "Samuel" <samuel.shul... @ntlworld.comwr ote:
          Hi
          >
          I am trying to read text files that are saved in ANSI format with Unicode
          characters such as French e German big S etc, and as I read the file these
          characters appear as squares etc.
          >
          I know that if the file would be saved as Unicode this wouldn't be a
          problem.
          >
          The question is whether there is an option that when I create the Stream
          Reader the application will recognize it as Unicode characters.
          >
          Thank you,
          Samuel

          Comment

          • Cor Ligthert[MVP]

            #6
            Re: Unicode Character Issue

            Samuel,

            Are you sure that the program that is showing you the characters is able to
            read and show those characters.

            You would not be the first to make the wrong conclussion because of this.

            By instance Windows mail (Outlook Express) will not show these characters
            as it is not send in HTML format.

            Cor

            "Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
            news:OgGc3ylxIH A.1980@TK2MSFTN GP02.phx.gbl...
            Hi
            >
            I am trying to read text files that are saved in ANSI format with Unicode
            characters such as French e German big S etc, and as I read the file these
            characters appear as squares etc.
            >
            I know that if the file would be saved as Unicode this wouldn't be a
            problem.
            >
            The question is whether there is an option that when I create the Stream
            Reader the application will recognize it as Unicode characters.
            >
            >
            Thank you,
            Samuel
            >

            Comment

            • Samuel

              #7
              Re: Unicode Character Issue

              Certainly, because when I save the same document in Unicode format I can
              read it without any problem

              Thank you,
              Samuel


              "Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
              news:C3D1F178-9792-49A9-85FF-9E95E1F86A27@mi crosoft.com...
              Samuel,
              >
              Are you sure that the program that is showing you the characters is able
              to read and show those characters.
              >
              You would not be the first to make the wrong conclussion because of this.
              >
              By instance Windows mail (Outlook Express) will not show these characters
              as it is not send in HTML format.
              >
              Cor
              >
              "Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
              news:OgGc3ylxIH A.1980@TK2MSFTN GP02.phx.gbl...
              >Hi
              >>
              >I am trying to read text files that are saved in ANSI format with Unicode
              >characters such as French e German big S etc, and as I read the file
              >these characters appear as squares etc.
              >>
              >I know that if the file would be saved as Unicode this wouldn't be a
              >problem.
              >>
              >The question is whether there is an option that when I create the Stream
              >Reader the application will recognize it as Unicode characters.
              >>
              >>
              >Thank you,
              >Samuel
              >>
              >

              Comment

              • Cor Ligthert[MVP]

                #8
                Re: Unicode Character Issue

                Samuel,

                By the way what do you mean with Ansi format.

                Wikipedia does not know it.

                Cor

                "Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                news:ejn6x5xxIH A.6096@TK2MSFTN GP06.phx.gbl...
                Certainly, because when I save the same document in Unicode format I can
                read it without any problem
                >
                Thank you,
                Samuel
                >
                >
                "Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
                news:C3D1F178-9792-49A9-85FF-9E95E1F86A27@mi crosoft.com...
                >Samuel,
                >>
                >Are you sure that the program that is showing you the characters is able
                >to read and show those characters.
                >>
                >You would not be the first to make the wrong conclussion because of this.
                >>
                >By instance Windows mail (Outlook Express) will not show these
                >characters as it is not send in HTML format.
                >>
                >Cor
                >>
                >"Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                >news:OgGc3ylxI HA.1980@TK2MSFT NGP02.phx.gbl.. .
                >>Hi
                >>>
                >>I am trying to read text files that are saved in ANSI format with
                >>Unicode characters such as French e German big S etc, and as I read the
                >>file these characters appear as squares etc.
                >>>
                >>I know that if the file would be saved as Unicode this wouldn't be a
                >>problem.
                >>>
                >>The question is whether there is an option that when I create the Stream
                >>Reader the application will recognize it as Unicode characters.
                >>>
                >>>
                >>Thank you,
                >>Samuel
                >>>
                >>
                >
                >

                Comment

                • Samuel

                  #9
                  Re: Unicode Character Issue

                  Open notepad and look in the list of the Encoding - did I have to say
                  encoding

                  Regards,
                  Samuel


                  "Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
                  news:5C9AE26A-DE85-42E3-81B1-9FD92A12606E@mi crosoft.com...
                  Samuel,
                  >
                  By the way what do you mean with Ansi format.
                  >
                  Wikipedia does not know it.
                  >
                  Cor
                  >
                  "Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                  news:ejn6x5xxIH A.6096@TK2MSFTN GP06.phx.gbl...
                  >Certainly, because when I save the same document in Unicode format I can
                  >read it without any problem
                  >>
                  >Thank you,
                  >Samuel
                  >>
                  >>
                  >"Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
                  >news:C3D1F17 8-9792-49A9-85FF-9E95E1F86A27@mi crosoft.com...
                  >>Samuel,
                  >>>
                  >>Are you sure that the program that is showing you the characters is able
                  >>to read and show those characters.
                  >>>
                  >>You would not be the first to make the wrong conclussion because of
                  >>this.
                  >>>
                  >>By instance Windows mail (Outlook Express) will not show these
                  >>characters as it is not send in HTML format.
                  >>>
                  >>Cor
                  >>>
                  >>"Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                  >>news:OgGc3ylx IHA.1980@TK2MSF TNGP02.phx.gbl. ..
                  >>>Hi
                  >>>>
                  >>>I am trying to read text files that are saved in ANSI format with
                  >>>Unicode characters such as French e German big S etc, and as I read the
                  >>>file these characters appear as squares etc.
                  >>>>
                  >>>I know that if the file would be saved as Unicode this wouldn't be a
                  >>>problem.
                  >>>>
                  >>>The question is whether there is an option that when I create the
                  >>>Stream Reader the application will recognize it as Unicode characters.
                  >>>>
                  >>>>
                  >>>Thank you,
                  >>>Samuel
                  >>>>
                  >>>
                  >>
                  >>
                  >

                  Comment

                  • Joergen Bech

                    #10
                    Re: Unicode Character Issue


                    In the context of the original post, Wikipedia does indeed have a few
                    articles describing the codes:



                    Regards,

                    Joergen Bech



                    On Thu, 5 Jun 2008 19:16:23 +0200, "Cor Ligthert[MVP]"
                    <notmyfirstname @planet.nlwrote :
                    >Samuel,
                    >
                    >By the way what do you mean with Ansi format.
                    >
                    >Wikipedia does not know it.
                    >
                    >Cor
                    >
                    >"Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                    >news:ejn6x5xxI HA.6096@TK2MSFT NGP06.phx.gbl.. .
                    >Certainly, because when I save the same document in Unicode format I can
                    >read it without any problem
                    >>
                    >Thank you,
                    >Samuel
                    >>
                    >>
                    >"Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
                    >news:C3D1F17 8-9792-49A9-85FF-9E95E1F86A27@mi crosoft.com...
                    >>Samuel,
                    >>>
                    >>Are you sure that the program that is showing you the characters is able
                    >>to read and show those characters.
                    >>>
                    >>You would not be the first to make the wrong conclussion because of this.
                    >>>
                    >>By instance Windows mail (Outlook Express) will not show these
                    >>characters as it is not send in HTML format.
                    >>>
                    >>Cor
                    >>>
                    >>"Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                    >>news:OgGc3ylx IHA.1980@TK2MSF TNGP02.phx.gbl. ..
                    >>>Hi
                    >>>>
                    >>>I am trying to read text files that are saved in ANSI format with
                    >>>Unicode characters such as French e German big S etc, and as I read the
                    >>>file these characters appear as squares etc.
                    >>>>
                    >>>I know that if the file would be saved as Unicode this wouldn't be a
                    >>>problem.
                    >>>>
                    >>>The question is whether there is an option that when I create the Stream
                    >>>Reader the application will recognize it as Unicode characters.
                    >>>>
                    >>>>
                    >>>Thank you,
                    >>>Samuel
                    >>>>
                    >>>
                    >>
                    >>

                    Comment

                    • Cor Ligthert[MVP]

                      #11
                      Re: Unicode Character Issue

                      Samuel,

                      There is often spoken about ANSI when is meant the code table for
                      characters, but ANSI does give as much information as when you ask for about
                      a problem with a car and tell that it is an European brand.

                      The used bit format, or tell that you are using a code table, or even ASCII
                      although most people are then in fact talking about extended ASCII which is
                      for English and Dutch code table 437 gives much more information. While
                      using true ASCII which only has 7 bits characters (in fact 26 alphabetic
                      ones in upper and lower case).

                      As you are encoding between those, it will always take the characters as are
                      in the resulting code.

                      Cor

                      "Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                      news:OHeXILzxIH A.2292@TK2MSFTN GP03.phx.gbl...
                      Open notepad and look in the list of the Encoding - did I have to say
                      encoding
                      >
                      Regards,
                      Samuel
                      >
                      >
                      "Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
                      news:5C9AE26A-DE85-42E3-81B1-9FD92A12606E@mi crosoft.com...
                      >Samuel,
                      >>
                      >By the way what do you mean with Ansi format.
                      >>
                      >Wikipedia does not know it.
                      >>
                      >Cor
                      >>
                      >"Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                      >news:ejn6x5xxI HA.6096@TK2MSFT NGP06.phx.gbl.. .
                      >>Certainly, because when I save the same document in Unicode format I can
                      >>read it without any problem
                      >>>
                      >>Thank you,
                      >>Samuel
                      >>>
                      >>>
                      >>"Cor Ligthert[MVP]" <notmyfirstname @planet.nlwrote in message
                      >>news:C3D1F1 78-9792-49A9-85FF-9E95E1F86A27@mi crosoft.com...
                      >>>Samuel,
                      >>>>
                      >>>Are you sure that the program that is showing you the characters is
                      >>>able to read and show those characters.
                      >>>>
                      >>>You would not be the first to make the wrong conclussion because of
                      >>>this.
                      >>>>
                      >>>By instance Windows mail (Outlook Express) will not show these
                      >>>characters as it is not send in HTML format.
                      >>>>
                      >>>Cor
                      >>>>
                      >>>"Samuel" <samuel.shulman @ntlworld.comsc hreef in bericht
                      >>>news:OgGc3yl xIHA.1980@TK2MS FTNGP02.phx.gbl ...
                      >>>>Hi
                      >>>>>
                      >>>>I am trying to read text files that are saved in ANSI format with
                      >>>>Unicode characters such as French e German big S etc, and as I read
                      >>>>the file these characters appear as squares etc.
                      >>>>>
                      >>>>I know that if the file would be saved as Unicode this wouldn't be a
                      >>>>problem.
                      >>>>>
                      >>>>The question is whether there is an option that when I create the
                      >>>>Stream Reader the application will recognize it as Unicode characters.
                      >>>>>
                      >>>>>
                      >>>>Thank you,
                      >>>>Samuel
                      >>>>>
                      >>>>
                      >>>
                      >>>
                      >>
                      >
                      >

                      Comment

                      Working...