Is Dump Complete?

Collapse
This topic is closed.
X
X
Collapse
 
  • Page of 1
  • Time
  • Show
Clear All
new posts
Previous template Next
  • Demetri
    #1

    Is Dump Complete?

    We have a file that is an extract file that represents data from several
    different systems (main frame, etc). This data varies and so from day to day
    the file that gets generated has variant size of data. Sometimes is just 5
    meg, other times is can be 10 gig. This means that the time it takes for the
    file to be completed varies also. In other words it is un-predictable what
    the size will be.

    Now, the problem is, when the file is complete we need to kick off another
    process that parses the file as well as other tasks that are not related but
    can not kick off until the file is done being generated.

    So, how do we know the file is done being written? The app that generates
    the file is a 3rd party and is no-longer supported by the vendor.

    I've considered writting code that will attempt to open the file and if the
    app still has a lock on it then an exception will get thrown and voila we
    know it isn't done yet. That however, is un-acceptable jerry rigging. This
    needs to be production worthy code and done properly.

    So my question to you guys, in C#, what sort of solution would you provide
    the problem of knowing when the file is done?

    --
    -Brycen
    Tags: None
    • Willy Denoyette [MVP]
      #2
      Re: Is Dump Complete?


      "Demetri" <Demetri@discus sions.microsoft .comwrote in message
      news:B9A80175-1C7C-4F77-861C-7A1160F43E96@mi crosoft.com...
      | We have a file that is an extract file that represents data from several
      | different systems (main frame, etc). This data varies and so from day to
      day
      | the file that gets generated has variant size of data. Sometimes is just 5
      | meg, other times is can be 10 gig. This means that the time it takes for
      the
      | file to be completed varies also. In other words it is un-predictable what
      | the size will be.
      |
      | Now, the problem is, when the file is complete we need to kick off another
      | process that parses the file as well as other tasks that are not related
      but
      | can not kick off until the file is done being generated.
      |
      | So, how do we know the file is done being written? The app that generates
      | the file is a 3rd party and is no-longer supported by the vendor.
      |
      | I've considered writting code that will attempt to open the file and if
      the
      | app still has a lock on it then an exception will get thrown and voila we
      | know it isn't done yet. That however, is un-acceptable jerry rigging. This
      | needs to be production worthy code and done properly.
      |

      Unfortunately it's the only way to handle this, you try to open the file, if
      it fails because of a sharing violation, you wait a couple of seconds and
      retry, what makes you think that this isn't production worthy?
      To handle this more "nicely", you need two cooparating applications, one
      producer and one consumer. The producer uses a semaphore or mutex to signal
      the consumer that he's has finished producing the data, the consumer wait's
      for the semaphore to be signaled.
      The real professional solution is to have both consumer and producer to run
      simultaniously, this by using region locks or semaphores to signal data
      ready to be consumed.

      Willy.



        Comment

        • Jack Robertson
          #3
          Re: Is Dump Complete?

          In addition to what Willy said (which is absolutely correct), is the file so
          tightly controlled that you can guarantee no 3rd-party will ever access it
          besides the programs that are supposed to (e.g., can a user unexpectly open
          it up, can it be accessed via some backup program that runs when not
          expected, etc.). It's unlikely so a sharing violation can potentially happen
          when you're not expecting it. Morevoer, since you can't synchronize access
          given that it's created by a 3rd-party vendor (using the methods recommended
          by Willy), what other choice do you have but to check for sharing
          violations? To do it more efficiently however (rather than explicitly
          checking every few seconds), perhaps you can leverage a class like
          "FileSystemWatc her" (though I've never had a need to use it myself). Watch
          the file in question IOW. If no activity has taken place for a given period
          of time then you may be able to assume the file is now free (so try to a
          open it exclusively presumably and if it fails because of a sharing
          violation then start your wait over again). Food for thought anyway.


            Comment

            Previous template Next
            Working...