Reading the first file from a directory.

[Craig Vermeer]

Hi All,

I have a program that's using the file system as a queuing mechanism,
and it's consuming an inordinate amount of CPU time when the file system
queue gets all that large (any more than a thousand or so messages in
the queue).

I know why. It's because my code to grab a single message off the queue
looks something like this :

//m_folder is a DirectoryInfo object
FileInfo file = m_folder.GetFil es("*.xml")[0];

So, it's pulling info about every file in the queue _every time_ it
wants to get a message from the queue.

Where I'm struggling, though, is finding a more efficient way to get the
first file in the directory.

One approach I can think of is to do some in-memory caching of the list
of files, so that I don't read them from disk every time. However,
there's a fair amount of effort involved there, because my program
reading the messages from the queue is multithreaded.

So, before I go down that route I thought I'd ask around. Anyone know
of a very efficient way to just pull a single file from a directory?

Thanks,
Craig

[John Bailo]

Re: Reading the first file from a directory.


Sounds like what you need is a variation of the dir command switches.

From the command line, you can do:

dir /od

To sort by the data timestamp and then you could pick off just the first
file. Maybe DirectoryInfo will let you sort by date timestamp.


Craig Vermeer wrote:[color=blue]
> Hi All,
>
> I have a program that's using the file system as a queuing mechanism,
> and it's consuming an inordinate amount of CPU time when the file system
> queue gets all that large (any more than a thousand or so messages in
> the queue).
>
> I know why. It's because my code to grab a single message off the queue
> looks something like this :
>
> //m_folder is a DirectoryInfo object
> FileInfo file = m_folder.GetFil es("*.xml")[0];
>
> So, it's pulling info about every file in the queue _every time_ it
> wants to get a message from the queue.
>
> Where I'm struggling, though, is finding a more efficient way to get the
> first file in the directory.
>
> One approach I can think of is to do some in-memory caching of the list
> of files, so that I don't read them from disk every time. However,
> there's a fair amount of effort involved there, because my program
> reading the messages from the queue is multithreaded.
>
> So, before I go down that route I thought I'd ask around. Anyone know
> of a very efficient way to just pull a single file from a directory?
>
> Thanks,
> Craig[/color]


--
Get the new

http://www.you-read-it-here-first.com

content from Texeme Textcasting Technology

[John Bailo]

Re: Reading the first file from a directory.

John Bailo wrote:[color=blue]
>
> Sounds like what you need is a variation of the dir command switches.
>
> From the command line, you can do:[/color]

You could use this:

http://www.shanebauer.com/Weblog/PermaLink,guid,b9cd286d-e147-4a25-aa49-bde888a11433.aspx

I'm assuming you are doing a FIFO queue so getting the earliest date
would work for you.
[color=blue]
>
> dir /od
>
> To sort by the data timestamp and then you could pick off just the first
> file. Maybe DirectoryInfo will let you sort by date timestamp.
>
>
> Craig Vermeer wrote:
>[color=green]
>> Hi All,
>>
>> I have a program that's using the file system as a queuing mechanism,
>> and it's consuming an inordinate amount of CPU time when the file
>> system queue gets all that large (any more than a thousand or so
>> messages in the queue).
>>
>> I know why. It's because my code to grab a single message off the
>> queue looks something like this :
>>
>> //m_folder is a DirectoryInfo object
>> FileInfo file = m_folder.GetFil es("*.xml")[0];
>>
>> So, it's pulling info about every file in the queue _every time_ it
>> wants to get a message from the queue.
>>
>> Where I'm struggling, though, is finding a more efficient way to get
>> the first file in the directory.
>>
>> One approach I can think of is to do some in-memory caching of the
>> list of files, so that I don't read them from disk every time.
>> However, there's a fair amount of effort involved there, because my
>> program reading the messages from the queue is multithreaded.
>>
>> So, before I go down that route I thought I'd ask around. Anyone know
>> of a very efficient way to just pull a single file from a directory?
>>
>> Thanks,
>> Craig[/color]
>
>
>[/color]


--
Get the new

http://www.you-read-it-here-first.com

content from Texeme Textcasting Technology

[Gerard]

RE: Reading the first file from a directory.

Hi Craig,

have you considered having a service that uses FileSystemWatch er to maintain
a listing of the files in your directory?
The list could be as simple as a text file in that directory through to any
level of sophistication you might like to best fit your overall applciation
architecture.
It would at least allow you to go directly to a single point for your next
file.

Good Luck
Gerard


"Craig Vermeer" wrote:
[color=blue]
> Hi All,
>
> I have a program that's using the file system as a queuing mechanism,
> and it's consuming an inordinate amount of CPU time when the file system
> queue gets all that large (any more than a thousand or so messages in
> the queue).
>
> I know why. It's because my code to grab a single message off the queue
> looks something like this :
>
> //m_folder is a DirectoryInfo object
> FileInfo file = m_folder.GetFil es("*.xml")[0];
>
> So, it's pulling info about every file in the queue _every time_ it
> wants to get a message from the queue.
>
> Where I'm struggling, though, is finding a more efficient way to get the
> first file in the directory.
>
> One approach I can think of is to do some in-memory caching of the list
> of files, so that I don't read them from disk every time. However,
> there's a fair amount of effort involved there, because my program
> reading the messages from the queue is multithreaded.
>
> So, before I go down that route I thought I'd ask around. Anyone know
> of a very efficient way to just pull a single file from a directory?
>
> Thanks,
> Craig
>[/color]

[Stephany Young]

Re: Reading the first file from a directory.

I would be inclined to use a combination of a FileSystemWatch er component
and a Queue object.

At application startup, start the FileSystemWatch er to wtach for new files
in your target folder, then use the GetFiles method and enqueue each
filename to the Queue object.

When your application wants to get a message from the queue simply

- Check the length of the Queue object (the number of entries)

- Dequeue the first entry into a string (filename)

- Check for the existence of the file

- Process it if it exists

The check for the existence of the file will cater for the potential for the
same filename to get put in the Queue object twice if the FileSystemWatch er
happens to fire bewteen the time you start it and the time you execute the
GetFiles method.

In my view it is best to do it that way round because if you do it the other
way round the is the potential for a file to be missed of it is created
between executing the GetFiles method and starting the FileSystemWatch er.


"Craig Vermeer" <vermeeca@sentd otcom.nospam> wrote in message
news:%23vXiURqT GHA.4956@TK2MSF TNGP09.phx.gbl. ..[color=blue]
> Hi All,
>
> I have a program that's using the file system as a queuing mechanism, and
> it's consuming an inordinate amount of CPU time when the file system queue
> gets all that large (any more than a thousand or so messages in the
> queue).
>
> I know why. It's because my code to grab a single message off the queue
> looks something like this :
>
> //m_folder is a DirectoryInfo object
> FileInfo file = m_folder.GetFil es("*.xml")[0];
>
> So, it's pulling info about every file in the queue _every time_ it wants
> to get a message from the queue.
>
> Where I'm struggling, though, is finding a more efficient way to get the
> first file in the directory.
>
> One approach I can think of is to do some in-memory caching of the list of
> files, so that I don't read them from disk every time. However, there's a
> fair amount of effort involved there, because my program reading the
> messages from the queue is multithreaded.
>
> So, before I go down that route I thought I'd ask around. Anyone know of
> a very efficient way to just pull a single file from a directory?
>
> Thanks,
> Craig[/color]