Junk characters when using StreamReader and StreamWriter

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Rob

    #1

    Junk characters when using StreamReader and StreamWriter

    Hi,
    I have a VB.Net application that parses an HTML file. This file was an
    MS Word document that was saved as web page.

    My application removes all unnecessary code generated by MS Word and
    does some custom formatting needed by my client.

    I use a StreamReader to read in the file...regular expressions to parse
    and clean up the file...and a StreamWriter to write the new file. On
    some HTML files that I parse, I get this character "á" showing up in a
    lot of places.
    I've used different types of encoding for my StreamReader and
    StreamWriter which works sometimes but then my application doesn't work
    when I parse a French HTML file.

    This is what I'm doing:

    Dim filename As String = myfileinfo.Orig inalFileName
    Dim sr As New StreamReader(fi lename, System.Text.Enc oding.Default)

    Dim textstream As String = sr.ReadToEnd()
    'close the stream...don't need it anymore
    sr.Close()

    Dim newtext As String

    'send the stream for cleaning
    newtext = CleanHTML(texts tream)

    Dim OutputFileName As String = myfileinfo.NewF olderPathToFile

    Dim fs As New FileStream(Outp utFileName, FileMode.Create ,
    FileAccess.Writ e)

    Dim sw As New StreamWriter(fs , System.Text.Enc oding.Default)

    sw.WriteLine(ne wtext)
    sw.Close()

    These are the results:

    <p class="MsoToc1" ><span
    ><a href="#_Toc1690 72483"><span>99 (15)0<span>á </span>INSTALMENT
    PROGRAM</span></a></span></p>

    <p class="MsoToc1" ><span
    ><a href="#_Toc1690 72484"><span>99 (15)1<span>á </span>OVERVIEW OF THE
    INSTALMENT PROGRAM</span></a></span></p>

    <p class="MsoNorma l" ><span>áá </span>OP:<span>á
    </span>BB<span>áá áááááááááá </span>ACCT:<span >á </span>123456789
    YR:<span>á </span>2004<span> ááááááááááááááá ááááááá </span>PG:<span>á
    </span>1 of 1<span>áááááááá áááááááááá </span>26FEB
    2004 USER-ID</p>

    I also have different characters representing quotes such as "ô" and "ö"
    (not shown here).

    When I do a "Save as Web Page" from MS Word 2000, the characters aren't
    there but they show up after runnung through my program.

    I'm assuming they represent characters that it doesn't understand but
    does anyone know how to reslove this?


    Thanks
    Rob



    *** Sent via Developersdex http://www.developersdex.com ***
  • Patrice

    #2
    Re: Junk characters when using StreamReader and StreamWriter

    Don't call CleanHTML function to see if it still happens. If yes, this is
    likely an encoding problem (Word could perhaps save the file using an utf-8
    encoding). If not it is something in your CleanHTML code...

    --
    Patrice

    "Rob" <robert@hotmail .coma écrit dans le message de news:
    uHjrtF0sHHA.141 6@TK2MSFTNGP06. phx.gbl...
    Hi,
    I have a VB.Net application that parses an HTML file. This file was an
    MS Word document that was saved as web page.
    >
    My application removes all unnecessary code generated by MS Word and
    does some custom formatting needed by my client.
    >
    I use a StreamReader to read in the file...regular expressions to parse
    and clean up the file...and a StreamWriter to write the new file. On
    some HTML files that I parse, I get this character "á" showing up in a
    lot of places.
    I've used different types of encoding for my StreamReader and
    StreamWriter which works sometimes but then my application doesn't work
    when I parse a French HTML file.
    >
    This is what I'm doing:
    >
    Dim filename As String = myfileinfo.Orig inalFileName
    Dim sr As New StreamReader(fi lename, System.Text.Enc oding.Default)
    >
    Dim textstream As String = sr.ReadToEnd()
    'close the stream...don't need it anymore
    sr.Close()
    >
    Dim newtext As String
    >
    'send the stream for cleaning
    newtext = CleanHTML(texts tream)
    >
    Dim OutputFileName As String = myfileinfo.NewF olderPathToFile
    >
    Dim fs As New FileStream(Outp utFileName, FileMode.Create ,
    FileAccess.Writ e)
    >
    Dim sw As New StreamWriter(fs , System.Text.Enc oding.Default)
    >
    sw.WriteLine(ne wtext)
    sw.Close()
    >
    These are the results:
    >
    <p class="MsoToc1" ><span
    >><a href="#_Toc1690 72483"><span>99 (15)0<span>á </span>INSTALMENT
    PROGRAM</span></a></span></p>
    >
    <p class="MsoToc1" ><span
    >><a href="#_Toc1690 72484"><span>99 (15)1<span>á </span>OVERVIEW OF THE
    INSTALMENT PROGRAM</span></a></span></p>
    >
    <p class="MsoNorma l" ><span>áá </span>OP:<span>á
    </span>BB<span>áá áááááááááá </span>ACCT:<span >á </span>123456789
    YR:<span>á </span>2004<span> ááááááááááááááá ááááááá </span>PG:<span>á
    </span>1 of 1<span>áááááááá áááááááááá </span>26FEB
    2004 USER-ID</p>
    >
    I also have different characters representing quotes such as "ô" and "ö"
    (not shown here).
    >
    When I do a "Save as Web Page" from MS Word 2000, the characters aren't
    there but they show up after runnung through my program.
    >
    I'm assuming they represent characters that it doesn't understand but
    does anyone know how to reslove this?
    >
    >
    Thanks
    Rob
    >
    >
    >
    *** Sent via Developersdex http://www.developersdex.com ***

    Comment

    • Joergen Bech

      #3
      Re: Junk characters when using StreamReader and StreamWriter


      á is just a single-byte ASCII character and Word just puts it into
      the html file without encoding.

      Why would you want to use System.Text.Enc oding.Default?

      Then you do not know what you are getting. You might get something
      to work - until you run the application somewhere else in the world.

      If you write the data out without going through the CleanHTML
      function, do you then get a file which byte for byte is identical to
      the original?

      Regards,

      Joergen Bech



      On Wed, 20 Jun 2007 06:46:38 -0700, Rob <robert@hotmail .comwrote:
      >Hi,
      >I have a VB.Net application that parses an HTML file. This file was an
      >MS Word document that was saved as web page.
      >
      >My application removes all unnecessary code generated by MS Word and
      >does some custom formatting needed by my client.
      >
      >I use a StreamReader to read in the file...regular expressions to parse
      >and clean up the file...and a StreamWriter to write the new file. On
      >some HTML files that I parse, I get this character "á" showing up in a
      >lot of places.
      >I've used different types of encoding for my StreamReader and
      >StreamWriter which works sometimes but then my application doesn't work
      >when I parse a French HTML file.
      >
      >This is what I'm doing:
      >
      >Dim filename As String = myfileinfo.Orig inalFileName
      >Dim sr As New StreamReader(fi lename, System.Text.Enc oding.Default)
      >
      >Dim textstream As String = sr.ReadToEnd()
      >'close the stream...don't need it anymore
      >sr.Close()
      >
      >Dim newtext As String
      >
      >'send the stream for cleaning
      >newtext = CleanHTML(texts tream)
      >
      >Dim OutputFileName As String = myfileinfo.NewF olderPathToFile
      >
      >Dim fs As New FileStream(Outp utFileName, FileMode.Create ,
      >FileAccess.Wri te)
      >
      >Dim sw As New StreamWriter(fs , System.Text.Enc oding.Default)
      >
      >sw.WriteLine(n ewtext)
      >sw.Close()
      >
      >These are the results:
      >
      ><p class="MsoToc1" ><span
      >><a href="#_Toc1690 72483"><span>99 (15)0<span>á </span>INSTALMENT
      >PROGRAM</span></a></span></p>
      >
      ><p class="MsoToc1" ><span
      >><a href="#_Toc1690 72484"><span>99 (15)1<span>á </span>OVERVIEW OF THE
      >INSTALMENT PROGRAM</span></a></span></p>
      >
      ><p class="MsoNorma l" ><span>áá </span>OP:<span>á
      ></span>BB<span>áá áááááááááá </span>ACCT:<span >á </span>123456789
      >YR:<span>á </span>2004<span> ááááááááááááááá ááááááá </span>PG:<span>á
      ></span>1 of 1<span>áááááááá áááááááááá </span>26FEB
      >2004 USER-ID</p>
      >
      >I also have different characters representing quotes such as "ô" and "ö"
      >(not shown here).
      >
      >When I do a "Save as Web Page" from MS Word 2000, the characters aren't
      >there but they show up after runnung through my program.
      >
      >I'm assuming they represent characters that it doesn't understand but
      >does anyone know how to reslove this?
      >
      >
      >Thanks
      >Rob
      >
      >
      >
      >*** Sent via Developersdex http://www.developersdex.com ***

      Comment

      • Herfried K. Wagner [MVP]

        #4
        Re: Junk characters when using StreamReader and StreamWriter

        "Joergen Bech @ post1.tele.dk>" <jbech<NOSPAMNO SPAMschrieb:
        á is just a single-byte ASCII character and Word just puts it into
        the html file without encoding.
        >
        Why would you want to use System.Text.Enc oding.Default?
        It will return the system's default Windows ANSI codepage. However, the
        encoding used to encode the file must be used to decode the file too.

        --
        M S Herfried K. Wagner
        M V P <URL:http://dotnet.mvps.org/>
        V B <URL:http://dotnet.mvps.org/dotnet/faqs/>

        Comment

        • Joergen Bech

          #5
          Re: Junk characters when using StreamReader and StreamWriter

          On Wed, 20 Jun 2007 22:09:55 +0200, "Herfried K. Wagner [MVP]"
          <hirf-spam-me-here@gmx.atwrot e:
          >"Joergen Bech @ post1.tele.dk>" <jbech<NOSPAMNO SPAMschrieb:
          >á is just a single-byte ASCII character and Word just puts it into
          >the html file without encoding.
          >>
          >Why would you want to use System.Text.Enc oding.Default?
          >
          >It will return the system's default Windows ANSI codepage. However, the
          >encoding used to encode the file must be used to decode the file too.
          Yes, that is what I meant. If you grab the system code page it
          could be anything - not necessarily suitable for reading a particular
          html file.

          Perhaps the charset information found in any Word-generated
          html file might be of more use here?

          /Joergen Bech



          Comment

          • Rob

            #6
            Re: Junk characters when using StreamReader and StreamWriter


            Thanks for your input guys, I think I've got it.
            I've ran the program without calling the CleanHTML function and it
            worked fine...so it must be the encoding when running this function.

            The reason I used System.Text.Enc oding.Default for the StreamReader and
            StreamWriter is because any other encoding wouldn't work with both
            english and french documents. When I used UTF8 for both...the french
            side would remove french characters altogether so I used Default for the
            StreamReader and UTF8 for the StreamWriter and it seems to be working
            fine for both english and french documents.

            I should mention that this application is only going to be used in house
            and only used on english and french documents so I think I should be
            fine.

            Thanks again


            *** Sent via Developersdex http://www.developersdex.com ***

            Comment

            Working...