manipulating files with ado.net

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • dougancil
    Contributor
    • Apr 2010
    • 347

    manipulating files with ado.net

    I have a web page that does a bulk insert of a csv file to a database but I want to be able to look at the file before it's uploaded to make sure its in the proper format. A few of the things that I'm looking for are

    Correct phone number format (if it doesnt have an area code, one needs to be given to it ... this will all be done with a 512 area code)
    Correct number of fields for the database. Right now there are only

    ID
    Patient Name
    Doctors Name
    Appointment Time
    Appointment Date
    Current Time
    Result

    ID, Current Time and Result are all supplied either by the database or the program I'm running in the background so they don't have to be manipulated in any way.

    And finally the datetime is formatted as yyyymmdd:hhmm
    and I will need all incoming data to comply with that.

    I know that data can be read with ado.net but this is new to me. Does any one have any kind of idea as to what would be the best way to go about this?

    Thank you

    Doug

    Here is my code for the page:

    Code:
    Imports System.IO
    Imports System.Data
    Imports System.Data.SqlClient
    Partial Class _Default
        Inherits System.Web.UI.Page
        Protected Sub Page_Load(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.Load
    
        End Sub
        Protected Sub Submit1_Click(ByVal sender As Object, ByVal e As System.EventArgs)
            Dim SaveLocation = "\\path\testCSV.csv"
            If UploadFile(SaveLocation) Then
                'the file was uploaded: now try saving it to the database
                SaveToDatabase(SaveLocation)
            End If
        End Sub
        Private Function UploadFile(ByVal SavePath As String) As Boolean
            Dim fileWasUploaded As Boolean = False 'indicates whether or not the file was uploaded
    
            'Checking if the file upload control contains a file
            If Not File1.PostedFile Is Nothing And File1.PostedFile.ContentLength > 0 Then
                Try
                    'checking if it was .txt file BEFORE UPLOADING IT!
                    'You used to upload it first...but the file could be a virus
                    If File1.FileName = ("doug.csv") = False Then
                        'The file is not the expected type...do not upload it
                        'just post the validation message
                        message.Text = "Sorry, thats not the correct file."
                        message2.Text = "Please locate and upload 'doug.csv'"
                    Else
                        'The file is a .txt file
                        'checking to see if the file exists already
                        'If it does exist Deleting the existing one so that the new one can be created
                        If IO.File.Exists(SavePath) Then
                            IO.File.Delete(SavePath)
                        End If
    
                        'Now upload the file (save it to your server)
                        File1.PostedFile.SaveAs(SavePath)
    
                        'After saving it check to see if it exists
                        If File.Exists(SavePath) Then
                            'Upload was sucessful
                            message.Text = "Thank you for your submission."
                            fileWasUploaded = True
                        Else
                            'the file was not saved
                            message.Text = "Unable to save the file."
                        End If
                    End If
    
                Catch Exc As Exception
                    'We encountered a problem
                    message.Text = "Your file was not in the correct format. Please contact MSB Customer Service at 512-467-5200."
                End Try
            Else
                'No file was selected for uploading
                message.Text = "Please select a file to upload."
            End If
            Return fileWasUploaded
        End Function
    
        Private Sub SaveToDatabase(ByVal SavePath As String)
            Try
                Dim sqlQueryText As String = _
                  "BULK INSERT dialerresults " + _
                  "FROM '" & SavePath & "' " + _
                  "WITH ( FIELDTERMINATOR = ',' , ROWTERMINATOR = '\n' )"
    
    
                ' and bulk import the data:   
                'If ConfigurationManager.ConnectionStrings("Dialerresults") IsNot Nothing Then
                'Dim connection As String = ConfigurationManager.ConnectionStrings("Dialerresults").ConnectionString
                Dim connection As String = "data source=10.2.1.40;initial catalog=IVRDialer;uid=xxxxx;password=xxxxxx;"
    
                Using con As New SqlConnection(connection)
                    con.Open()
                    ' execute the bulk import   
                    Using cmd As New SqlCommand(sqlQueryText, con)
    
                        cmd.ExecuteNonQuery()
    
                    End Using
                End Using
    
                'Else
                'message.Text="ConfigurationManager.ConnectionStrings('Dialerresults') is Nothing!"
                'End If
            Catch ex As Exception
                message.Text = "Your file was not in the correct format. Please contact MSB Customer Service at 512-467-5200."
    
            End Try
        End Sub
    
    End Class
    Last edited by Frinavale; Aug 4 '10, 08:57 PM. Reason: Removed any sensitive data
  • Christopher Nigro
    New Member
    • Jul 2010
    • 53

    #2
    I'm not sure which part you need help with exactly. If it is reading the file via ADO.NET then you would do something like:

    Code:
     string connString = @"Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\Documents and Settings\cnigro\Desktop" +
                      @";Extended Properties=""Text;HDR=Yes;FMT=Delimited\""";
    DataTable tbl = new DataTable();
    OleDbConnection conn = new OleDbConnection(connString);
    OleDbCommand comm = new OleDbCommand("SELECT * FROM Test.csv", conn);
    OleDbDataAdapter adapter = new OleDbDataAdapter(comm);
    adapter.Fill(tbl);
    This will select all the SDV data and fill a DataTable that you can use to carry out any logic that needs to be done.

    Comment

    • dougancil
      Contributor
      • Apr 2010
      • 347

      #3
      Christoper,

      Using ado.net to get information from a file is new to me so I really have no idea how to read the data and then manipulate it to match a preset format (as I described in my other post.) So any suggestion that you may have on how best to manipulate the data as I'm reading it would be useful.

      Thank you

      Doug

      Comment

      • Christopher Nigro
        New Member
        • Jul 2010
        • 53

        #4
        Hi Doug,

        The code I showed will give a data table. You can iterate the rows and columns to check the values. I cannot be more specific because I do not know how you want to handle everything. You may want to use a default value if you find an error or you may want to skip the row entirely - I don't know.

        Here is how you can iterate:

        Code:
        foreach (DataRow row in tbl.Rows)
        {
            foreach (DataColumn col in tbl.Columns)
            {
                // DO SOMETHING TO CHECK THE VALUE OF: row[col.ColumnName]
            }
        }
        You probably would not want to use the inner loop: you'd likely refernce the values explicitely like:
        Code:
        if (row["PatientName"].ToString().Length > 100)
        {
            //do something
        }

        Comment

        • dougancil
          Contributor
          • Apr 2010
          • 347

          #5
          Christopher,

          At this point, this is a customer by customer basis and I suspect that they will all be different in their own way. So what you're saying is that for each value that I want/need to check (ie phone number, field names, etc) that I can read the file, but say for example I get a file that doesnt have the correct data in the phone number field

          Say I'm looking for 512-xxx-xxxx and the incoming data doesnt have that then I'd just write the if statement to reflect that explicitely and then add that to the .tostring?

          As I said this is all new to me so I'm learning as I go. I'm not used to having to manipulate data on the fly.

          Comment

          • Christopher Nigro
            New Member
            • Jul 2010
            • 53

            #6
            Hi Doug,

            Here is an example of iterating all the rows and checking the phone number field in each row:

            Code:
            //USE A REGULAR EXPRESSION TO CHECK PHONE FORMAT
            Regex rePhone = new Regex(@"^512-\d{3}-\d{4}$");
            
            foreach (DataRow row in tbl.Rows) 
            { 
            	//CHECK PHONE NUMBER FORMAT
            	if (!rePhone.IsMatch((row["PhoneNumber"].ToString()))
            	{
            		//HANDLE INVALID PHONE NUMBER FORMAT HERE
            	} 
            
            	//NOW CHECK THE OTHER FIELD VALUES ACCORDINGLY...
            }
            You would need to add the other validations and the actions taken when invalid data is encountered.



            HTH,
            Chris

            Comment

            • dougancil
              Contributor
              • Apr 2010
              • 347

              #7
              Chris,

              I did have one question. I noticed this line:

              Code:
              "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=C:\Documents and Settings\cnigro\Desktop
              in your oledb string. Since I'm new to oledb, what does that part of the string signify?

              Comment

              • dougancil
                Contributor
                • Apr 2010
                • 347

                #8
                Oh and one other thing, since I'll have to do iterations for quite a few things, would it be easier to use XML and regular expressions rather than the way I'm going. What I'm fearful of is that my page will become excessively bloated and I'd like to keep it as streamlined as possible if I could.

                Comment

                • Christopher Nigro
                  New Member
                  • Jul 2010
                  • 53

                  #9
                  Hi,

                  The Provider is set to the JET provider. You use JET when connecting to Access, Excel or text files.

                  The Datasource is set to my desktop, which is where the CSV file is that I am connecting to.

                  Take a look at http://www.connectionstrings.com and http://www.connectionstrings.com/textfile specifically.


                  Thanks,
                  Chris

                  Comment

                  • Christopher Nigro
                    New Member
                    • Jul 2010
                    • 53

                    #10
                    Originally posted by dougancil
                    Oh and one other thing, since I'll have to do iterations for quite a few things, would it be easier to use XML and regular expressions rather than the way I'm going. What I'm fearful of is that my page will become excessively bloated and I'd like to keep it as streamlined as possible if I could.


                    I'd use regular expressions where they are called for as I showed in my example. What are you considering using XML for?

                    Comment

                    • dougancil
                      Contributor
                      • Apr 2010
                      • 347

                      #11
                      Chris,

                      As far as the xml ... I'm not sure what I'd use it for, I just want to make sure that I'm writing this page the most efficient way possible. It just make debugging it and troubleshooting it easier. One other question, all of my files will need a leading comma because my ID field is first in my database, I'm sure I can use a regular expression to check for that and add it correct?

                      Comment

                      • Christopher Nigro
                        New Member
                        • Jul 2010
                        • 53

                        #12
                        Since you are checking a DataColumn in a DataRow, JET may deduce that it is an integer field. In that case, you'd probably want to check that it is not equal to DbNull.value. If it treats it as a string, you could use a regexp to check that it is a number.

                        Comment

                        • dougancil
                          Contributor
                          • Apr 2010
                          • 347

                          #13
                          Chris,

                          Thank you for your help this far. I'm learning about Regex and really appreciate the guidance as I'm doing so. I finally got a sample of the data that I'll be parsing. You can see from the example that there is a lot of superfluous data that I won't be needing.
                          Last edited by Frinavale; Aug 4 '10, 08:55 PM.

                          Comment

                          • Christopher Nigro
                            New Member
                            • Jul 2010
                            • 53

                            #14
                            Hi,

                            You will not be able to treat that file as a data source and query it with ADO.NET: it is not a CSV file and it is not tabular. You will have to read it as a file and program some sort of structure to work with.

                            Comment

                            • dougancil
                              Contributor
                              • Apr 2010
                              • 347

                              #15
                              CSV File

                              Chris,

                              I am unable to upload a .csv file to this forum, so I had to upload it as a text file. Also it shows you all of the superfluous information that I won't need. Really the only information that I will need from the document is date/time, appointment date, phone number (of the 3 listed, I will need the first one,) Patient's Name and Doctors name. Everything else I can discard but it has to be in the following order,

                              Phonenumber
                              Patient Name
                              Dr Name
                              Appt Date
                              Appt Time

                              (as this is how my database is currently laid out.)

                              Comment

                              Working...