Windows Services - FileWatcher

[tshad]

What exactly is FileWatcher doing? When you drop 100 files in a folder it
is watching, it normally will fire of the event 100 times.

In my case, I do all my processing on the first event so I don't need to
reinitialize all my variables for each file. Normally 50 - 100 files will
be dropped in the folder at a time.

If I get an error, in my code it moves out to my try/catch code in the
service - handle it and then exit. Then the next event should fire (and
normally it will). Sometimes however it will just stop firing the event.
It is still running, which I know because the next time I drop a file in the
folder it fires the event.

How is it keeping the events? Is it storing them up as events in a
multiclass delegate fashion? If so, is there a way to see how many more
events are left to fire.

I need to figure out what is happening and how the events could stop firing.
At the moment, I am going to add a loop to keep checking until all the files
are done to fix the problem.

Btw, I have a it pointing at a function when the event fires. I assume that
the next event won't fire until I exit that function. Correct?

My code is:

*************** *************** *************** *************** *******
Imports System.IO
Imports FieldNameMapSet up

Public Class ServiceApp

Private fileWatcher As FileSystemWatch er
Private fieldNameSetup As FieldNameMapSet up.FieldNameMap Setup

Protected Overrides Sub OnStart(ByVal args() As String)
Try
System.Diagnost ics.EventLog.Wr iteEntry(Me.Ser viceName,
Me.ServiceName + " started.", _
System.Diagnost ics.EventLogEnt ryType.Informat ion)

'Use the following line to slow down the onstart to allow you to
attach to the process.
'System.Threadi ng.Thread.Sleep (15000)

InitializeFileW atcher()

Catch ex As Exception
MyException.Pub lish(ex)
End Try
End Sub

Protected Overrides Sub OnStop()
System.Diagnost ics.EventLog.Wr iteEntry(Me.Ser viceName, Me.ServiceName
+ " stopped.", System.Diagnost ics.EventLogEnt ryType.Informat ion)
End Sub

Private Sub InitializeFileW atcher()
fileWatcher = New FileSystemWatch er()
fileWatcher.Pat h = MySettings.XMLF ilePath
fileWatcher.Not ifyFilter = NotifyFilters.F ileName
fileWatcher.Fil ter = "*.*"
AddHandler fileWatcher.Cre ated, AddressOf HandleAppraisal s
fileWatcher.Ena bleRaisingEvent s = True
End Sub

Private Sub HandleAppraisal s(ByVal o As Object, ByVal e As
FileSystemEvent Args)

Try

fieldNameSetup = New FieldNameMapSet up.FieldNameMap Setup()
fieldNameSetup. CheckForApprais als()

Catch exc As Exception
fieldNameSetup. MoveFiles(Field NameMapSetup.Fi eldNameMapSetup .fileInProcess,
FieldNameMapSet up.MySettings.E xceptionFilePat h)
Logging.WriteTo Log(MySettings. LogFilePath, exc.Message + "Writing
to: " + FieldNameMapSet up.MuSettings.E xceptionFilePat h + _
Path.GetFileNam e(FieldNameMapS etup.FieldNameM apSetup.fileInP rocess))
End Try
End Sub

End Class
*************** *************** *************** *************** ********

Thanks,

Tom

[Peter Duniho]

Re: Windows Services - FileWatcher

On Tue, 24 Jun 2008 14:33:15 -0700, tshad <tshad@dslextre me.comwrote:

[...]
How is it keeping the events? Is it storing them up as events in a
multiclass delegate fashion? If so, is there a way to see how many more
events are left to fire.

It has a private buffer it uses. The buffer can fill, at which point no
more events will be noted until the buffer's contents are consumed by
handling the event.

I need to figure out what is happening and how the events could stop
firing.

You then should probably read the detailed explanation found on MSDN, in
the main page for the class:

http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx

This appears to me to have as much detailed information as a client of the
class would really need.

At the moment, I am going to add a loop to keep checking until all the
files
are done to fix the problem.

I wouldn't put the loop in your event handler. You should process the
event as quickly as possible. If you expect to do some non-trivial
processing on the data, put it in a queue somewhere and let a different
thread deal with it. But yes, other than that caveat it seems reasonable
to me for you to have code somewhere (but not in the event handler for the
FileSystemWatch er class) that iterates over the files to verify whatever
eventual status you want to verify. The FileSystemWatch er cannot and does
not guarantee notification of every single event.

Btw, I have a it pointing at a function when the event fires. I assume
that
the next event won't fire until I exit that function. Correct?

Based on the warning in the description of the class to make your event
handler return as quickly as possible, I infer that to be correct. But!
Even so, it's not an assumption I would make. I couldn't find any
explicit documentation that guarantees you that you won't have multiple
events raised asynchronously to each other, and so this behavior _could_
change in the future (in particular, it'd be one way for the .NET
designers to address the issue of limited buffer space for the file system
event data, by just always passing it off to you as fast as it can).

By the way, if you're going to post VB code, you probably ought to be
posting your question in a VB newsgroup instead of a C# newsgroup.
Conversely, if you really want to post your question here, make sure your
code is C# code.

Pete

[tshad]

Re: Windows Services - FileWatcher


"Peter Duniho" <NpOeStPeAdM@nn owslpianmk.comw rote in message
news:op.uc9wpdw 68jd0ej@petes-computer.local. ..

On Tue, 24 Jun 2008 14:33:15 -0700, tshad <tshad@dslextre me.comwrote:
>

>[...]
>How is it keeping the events? Is it storing them up as events in a
>multiclass delegate fashion? If so, is there a way to see how many more
>events are left to fire.

>
It has a private buffer it uses. The buffer can fill, at which point no
more events will be noted until the buffer's contents are consumed by
handling the event.
>

>I need to figure out what is happening and how the events could stop
>firing.

>
You then should probably read the detailed explanation found on MSDN, in
the main page for the class:

http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx

>
This appears to me to have as much detailed information as a client of the
class would really need.
>

Except that it might be nice to know how many files it has in its private
buffer.

>At the moment, I am going to add a loop to keep checking until all the
>files
>are done to fix the problem.

>
I wouldn't put the loop in your event handler. You should process the
event as quickly as possible. If you expect to do some non-trivial
processing on the data, put it in a queue somewhere and let a different
thread deal with it. But yes, other than that caveat it seems reasonable
to me for you to have code somewhere (but not in the event handler for the
FileSystemWatch er class) that iterates over the files to verify whatever
eventual status you want to verify. The FileSystemWatch er cannot and does
not guarantee notification of every single event.

That's true. But you can apparently change the size of the buffer but
without knowing how many files are in the buffer, that is pretty hard to
guage. If it were only keeping 10 files that is one thing but if it is
keeping 100, the fact that it overflows doesn't bother me. It does say that
the size is 8K by default and 16 bytes per event (plus size of filename). I
assume this means it would hold upwards of about 200 files. As it says, it
would lose track of changes in the folder so you couldn't get the specific
file that changed or was created - but blanket notification is really all I
need since I am just getting a list of files in the folders myself. If the
buffer holds 200and I sent 500 - it would end up with 200 in the buffer,
which is fine. But that doesn't seem to be what is happening. It seems to
dump the buffer at some point.

>

>Btw, I have a it pointing at a function when the event fires. I assume
>that
>the next event won't fire until I exit that function. Correct?

>
Based on the warning in the description of the class to make your event
handler return as quickly as possible, I infer that to be correct. But!
Even so, it's not an assumption I would make. I couldn't find any
explicit documentation that guarantees you that you won't have multiple
events raised asynchronously to each other, and so this behavior _could_
change in the future (in particular, it'd be one way for the .NET
designers to address the issue of limited buffer space for the file system
event data, by just always passing it off to you as fast as it can).
>

That is probably true but in my case the code it is handling is pretty
extensive and I am building many collections at the start of the program and
only want to do it once.

I don't want another file to be handled before the previous one is
finished..

As you say, maybe the the private buffer is getting dumped after period of
time. In my case, on my clients machine each file could take up to 5-10
minutes to process. If 500 files were dumped into the folder at once it
could take 40-80 hours to process all the files. And other files could get
dumped in the folder during that time. so maybe there is a time limit
before it decides to dump the buffer. I know that I may have processed 200
hundred files and it may fire the event fine or it may just stop until
another file gets put in the folder.

Even if I put this in another thread that call the HandleAppraisal s
function - I would have the same problem. I don't want the program starting
another thread to process. I definately don't want 500 threads running at
once.

One way around the problem would be to re-call the HandleAppraisal s function
after the first one is finished, just to make sure all the files have been
processed.

Then it won't matter what happens to the event buffer. If it is still
there, it will just call HandleAppraisal s x number of times and find that
there are no files in the folder (or will process any files that my have
shown up in the folder). This would guarantee that all the files are
processed.

By the way, if you're going to post VB code, you probably ought to be
posting your question in a VB newsgroup instead of a C# newsgroup.
Conversely, if you really want to post your question here, make sure your
code is C# code.

Sorry.

You're right. I should have posted it in the VB group. I forgot it was in
VB. I actually have the service written in VB, but the code it calls is in
C# - so I just missed that.

Thanks,

Tom

>
Pete

[Peter Duniho]

Re: Windows Services - FileWatcher

On Thu, 26 Jun 2008 09:29:01 -0700, tshad <tshad@dslextre me.comwrote:

[...]
Are you suggesting that I have my own list (array or collection) that is
filled by the Filewatcher event and another thread that does the
processing?

I'd use a Queue<T>. But yes, the general idea is that the event handler
would simply add the event data to a collection somewhere, and a different
thread would be reading that data.

Not sure how I would sync this but that might be an idea (if that was
what
you meant).

Synchronization could be simply through use of the Monitor class and/or a
lock() statement. The Monitor class includes semantics for waking a
thread up. So the consumer thread would probably just call Monitor.Wait()
when it's run out of things to do, and the event handler would call
Monitor.Pulse() (using the same synchronization object, of course) when
it's added something to the queue.

Obviously, you'll want to make sure your processing thread doesn't hold
the lock except when it's actually trying to get something from the
queue. So it'd (for example) call Monitor.Exit() before doing any lengthy
processing not involving the queue itself, and call Monitor.Enter() when
it wants to reacquire the lock for the queue to get the next item in the
queue.

Actually, there might be 2 extra threads. The original thread that
FileWatcher runs on and maintains my own array. One that FileWatcher
fires
up (if it isn't already running) that will check to see if there are
anymore
files to handle and will fire up another thread that does the processing.
When the last thread is done, it would end and the 1st thread will check
to
see if there are anymore files in the folder to process and fire another
thread (again) that would process any files left. This would keep on
going
until there are no more files in the folder.

I don't see a need to have two extra threads. If you're only ever going
to have one thread processing a file at a time, you might as well make
that the same thread that consumes the queue and is otherwise monitoring
the directory to identify files that need to be processed.

[...] My confusion is how I could lose 190 events
which is why sometimes I end up with files sitting in the folder and the
filewatcher waiting for another file to get dropped in the folder.
>
What should happen is that after I have had 3 or 4 events - all the files
should now be processed and there should be about 190 events, each one
checking the folder and finding nothing there and exit. After each event
exits another one should fire, find nothing exit until all 190 events are
done. But this doesn't seem to be happening.

Well, I can't answer that question. If for no other reason than that I
really know very little about how you've designed your use of the
FileSystemWatch er, and what processing you're doing. For all I know, you
simply haven't calculated the theoretical capacity of the FSW buffer
correctly. It doesn't help that I don't have very much first-hand
experience with using the FSW class, and so don't have a lot of first-hand
practical advice to offer. The fact is, FSW is not 100% reliable so
you'll simply have to design your own code to account for that.

Pete

[tshad]

Re: Windows Services - FileWatcher

Peter Duniho wrote:

On Thu, 26 Jun 2008 09:29:01 -0700, tshad <tshad@dslextre me.com>
wrote:

>[...]
>Are you suggesting that I have my own list (array or collection)
>that is filled by the Filewatcher event and another thread that does
>the processing?

>
I'd use a Queue<T>. But yes, the general idea is that the event
handler would simply add the event data to a collection somewhere,
and a different thread would be reading that data.
>

>Not sure how I would sync this but that might be an idea (if that was
>what
>you meant).

>
Synchronization could be simply through use of the Monitor class
and/or a lock() statement. The Monitor class includes semantics for
waking a thread up. So the consumer thread would probably just call
Monitor.Wait() when it's run out of things to do, and the event
handler would call Monitor.Pulse() (using the same synchronization
object, of course) when it's added something to the queue.
>
Obviously, you'll want to make sure your processing thread doesn't
hold the lock except when it's actually trying to get something from
the queue. So it'd (for example) call Monitor.Exit() before doing
any lengthy processing not involving the queue itself, and call
Monitor.Enter() when it wants to reacquire the lock for the queue to
get the next item in the queue.
>

>Actually, there might be 2 extra threads. The original thread that
>FileWatcher runs on and maintains my own array. One that FileWatcher
>fires
>up (if it isn't already running) that will check to see if there are
>anymore
>files to handle and will fire up another thread that does the
>processing. When the last thread is done, it would end and the 1st
>thread will check to
>see if there are anymore files in the folder to process and fire
>another thread (again) that would process any files left. This
>would keep on going
>until there are no more files in the folder.

>
I don't see a need to have two extra threads. If you're only ever
going to have one thread processing a file at a time, you might as
well make that the same thread that consumes the queue and is
otherwise monitoring the directory to identify files that need to be
processed.

>[...] My confusion is how I could lose 190 events
>which is why sometimes I end up with files sitting in the folder and
>the filewatcher waiting for another file to get dropped in the
>folder. What should happen is that after I have had 3 or 4 events - all
>the
>files should now be processed and there should be about 190 events,
>each one checking the folder and finding nothing there and exit. After
>each event exits another one should fire, find nothing exit
>until all 190 events are done. But this doesn't seem to be
>happening.

>
Well, I can't answer that question. If for no other reason than that
I really know very little about how you've designed your use of the
FileSystemWatch er, and what processing you're doing. For all I know,
you simply haven't calculated the theoretical capacity of the FSW
buffer correctly. It doesn't help that I don't have very much
first-hand experience with using the FSW class, and so don't have a
lot of first-hand practical advice to offer. The fact is, FSW is not
100% reliable so you'll simply have to design your own code to
account for that.

That is true. It isn't 100% reliable.

I moved 61 files in the Filewatcher folder, so I should get 61 events.

I am running my program that increases a static variable which is an
eventKtr by in my event program. This eventKtr tells us how many times the
event was raised.

So each time the event is called it calls my program and my program
increases the eventKtr by 1. Then my program handles all the files it finds
in the folder. When done, it does another pass to see if there are any
other programs in the folder and handles them. Then it displays the
eventKtr and exits the problem.

At this point, the fileWatcher should fire another event (it has 60 left).

When it exited the first time the eventKtr was one - so my program was never
called again. And another event doesn't get called.

I can only surmise 2 reasons for this.

1. There is timer attached to the events. If it takes too long to raise the
event, it gets dumped.
2. All the events were raised, but the program somehow knows that the 1st
program is running and just ignores the event.

My concern was that it would run the program concurrently - but it doesn't
seem to.

Thanks,

Tom

>
Pete

[Peter Duniho]

Re: Windows Services - FileWatcher

On Sun, 29 Jun 2008 22:54:48 -0700, tshad <tfs@dslextreme .comwrote:

That is true. It isn't 100% reliable.
>
I moved 61 files in the Filewatcher folder, so I should get 61 events.

Those two statements are mutually exclusive of each other. I hope that
you can see why that is.

[...]
I can only surmise 2 reasons for this.
>
1. There is timer attached to the events. If it takes too long to raise
the
event, it gets dumped.
2. All the events were raised, but the program somehow knows that the 1st
program is running and just ignores the event.

I can think of lots of other possibilities as well. But the actual
underlying reason really doesn't matter much. The fact is, your code must
be able to deal with this condition.

Now, I believe that you could reduce the likelihood of the condition
occurring by fixing your code so that it doesn't abuse the FSW event being
raised. That is, comply with the admonition in the documentation to keep
the event handler simpler and make sure it returns as close to immediately
as you can make it.

But even if you do that, there simply will never be a guarantee that for
every single file system change, your event handler will be called.

My concern was that it would run the program concurrently - but it
doesn't
seem to.

I think between the fact that the docs warn you to complete your event
handler as quickly as possible, and the fact that a multi-threaded
event-raising mechanism would introduce other problems (for example, if it
uses the thread pool it could increase the changes of a thread pool
deadlock, and if it doesn't use the thread pool, it would either have to
maintain its own thread pool, or it would run the chance of creating an
excessive number of threads), you can probably count on the events not
being raised concurrently.

That said, the docs don't specifically promise that behavior. So the
safest thing might be to not assume non-concurrent events being raised.
Your option. :)

Pete

[tshad]

Re: Windows Services - FileWatcher


"Peter Duniho" <NpOeStPeAdM@nn owslpianmk.comw rote in message
news:op.udjtiqk 18jd0ej@petes-computer.local. ..

On Sun, 29 Jun 2008 22:54:48 -0700, tshad <tfs@dslextreme .comwrote:
>

>That is true. It isn't 100% reliable.
>>
>I moved 61 files in the Filewatcher folder, so I should get 61 events.

>
Those two statements are mutually exclusive of each other. I hope that
you can see why that is.

I agree.

That was what I was saying. Either one or the other. Can't really think of
another.

>

>[...]
>I can only surmise 2 reasons for this.
>>
>1. There is timer attached to the events. If it takes too long to raise
>the
>event, it gets dumped.
>2. All the events were raised, but the program somehow knows that the 1st
>program is running and just ignores the event.

>
I can think of lots of other possibilities as well. But the actual
underlying reason really doesn't matter much. The fact is, your code must
be able to deal with this condition.
>

It does.

If, in fact, the event is being called but not executed (for whatever
reason) - that is exactly what I want to happen.

I am not sure you understand what I am doing.

If 650 files are dropped in the folder at once (which was done last night) -
I am handling them all at one time. I look at the folder, get a list of
files that are there and then process that list. Took about 45 minutes last
night.

When done (still in the event), I check again to see if there are any more
files there - if there I process them. I keep doing this until there are no
more files in them.

What was happening with a small number of files (15 for example), was I
would get 14 events raised when I was was done. Which is fine. I would
just check the folder for files - find it empty ane exit.

What happened last night, however, was that I finished my 45 minute run -
which worked fine (all files were processed) - but I never got another
event. I should have gotten at least one event.

Here is the start and end of the file:

6/29/2008 11:17:54 PM Starting new instance of HandleAppraisal s -
EventStarts: 1 Name: 512953.XML
6/29/2008 11:17:54 PM Starting loop #1 in HandleAppraisal s with
fileInProcess =

....

6/29/2008 11:51:08 PM Ending loop #10 in HandleAppraisal s with fileInProcess
=
6/29/2008 11:51:08 PM Events finished: 1
6/29/2008 11:51:08 PM Exiting HandleAppraisal s - EventStarts: 1 Name:
512953.XML

Here I started at 11:17 and finished at 11:51. The "Loop #" is the number
of times I am checking to see if there are anymore files in the folder and
still in the same event. Apparently, it checked 10 times.

It finished the event whith EventStarts at 1. So only one event was raised
(or at least my function was only called once).

The first 2 lines in my function are ( I realize this is VB - but that is
what my service is written in but the rest of the code is in C#):

Private Sub HandleAppraisal s(ByVal o As Object, ByVal e As
FileSystemEvent Args)

AppSettings.num berOfEventStart s = AppSettings.num berOfEventStart s + 1

As you can see the EventStarts is updating the Static variable the very
first line in my code.

So it only gets there once.

So either the event doesn't get raised again (and there should be about
150-200 of them - accounting for buffer overflows - definately more than
one), or it gets raised and the function doesn't get called for some reason.

Now, I believe that you could reduce the likelihood of the condition
occurring by fixing your code so that it doesn't abuse the FSW event being
raised. That is, comply with the admonition in the documentation to keep
the event handler simpler and make sure it returns as close to immediately
as you can make it.

I could. But then I would need set up another task, as we talked about and
this seems to be working fine. Just not sure what is happening here.

>
But even if you do that, there simply will never be a guarantee that for
every single file system change, your event handler will be called.
>

Don't want that to happen. As I mention, I would have already handled all
the files.

What I had thought about doing was to disabling the raising of events for a
second when leaving my function (if that would clear the events - not sure
if it would):

fileWatcher.Ena bleRaisingEvent s = False

Then start it back up again:
fileWatcher.Ena bleRaisingEvent s = True

Then leave the function.

But if I am losing the events anyway, this would not be necessary.

>My concern was that it would run the program concurrently - but it
>doesn't
>seem to.

>
I think between the fact that the docs warn you to complete your event
handler as quickly as possible, and the fact that a multi-threaded
event-raising mechanism would introduce other problems (for example, if it
uses the thread pool it could increase the changes of a thread pool
deadlock, and if it doesn't use the thread pool, it would either have to
maintain its own thread pool, or it would run the chance of creating an
excessive number of threads), you can probably count on the events not
being raised concurrently.
>

That does seem to be the case - which means it is working fine. Just
confused as to where the events went.

Thanks,

Tom

That said, the docs don't specifically promise that behavior. So the
safest thing might be to not assume non-concurrent events being raised.
Your option. :)
>
Pete

[Peter Duniho]

Re: Windows Services - FileWatcher

On Mon, 30 Jun 2008 08:30:23 -0700, tshad <tshad@dslextre me.comwrote:

[...]
What happened last night, however, was that I finished my 45 minute run -
which worked fine (all files were processed) - but I never got another
event. I should have gotten at least one event.

You keep using that word "should". I don't think that word means what you
think it means. We've already established that FSW is unreliable. There
is no valid statement that combines the word "should" with some assertion
about what events are raised.

[...]
That does seem to be the case - which means it is working fine. Just
confused as to where the events went.

As I mentioned, there are any number of reasons that the FSW might fail to
deliver notification of some specific file system event. If you insist on
writing an event handler that doesn't return for 45 minutes, you will
simply exacerbate those reasons, making them more likely. MUCH more
likely.

I'm afraid I really don't see the point in refusing to fix the event
handler but continuing to wonder why the handler isn't called for each and
every file system change.

Pete