alternative file.copy

[luis molina Micasoft]

it seems that when i do file.copy the svchost.exe is hanged, i mean if i make
40 threads of file.copy , 40 copys of files at same time the system is going
down and stop responding, this is when i'm working with cifs (shares). there
is another solution to copy files than file.copy in .net?

[Herfried K. Wagner [MVP]]

Re: alternative file.copy

"luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com >
schrieb:[color=blue]
> it seems that when i do file.copy the svchost.exe is hanged, i
> mean if i make 40 threads of file.copy[/color]

Reduce the number of threads...

--
Herfried K. Wagner [MVP]
<URL:http://dotnet.mvps.org/>

[Cor Ligthert]

Re: alternative file.copy

Luis,

You make me curious, are you using threads to copy files, and then what is
the reason for that?

Cor

"luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com >
....[color=blue]
> it seems that when i do file.copy the svchost.exe is hanged, i mean if i
> make
> 40 threads of file.copy , 40 copys of files at same time the system is
> going
> down and stop responding, this is when i'm working with cifs (shares).
> there
> is another solution to copy files than file.copy in .net?[/color]

[Jay B. Harlow [MVP - Outlook]]

Re: alternative file.copy

Luis,
There are any number of alternatives to file.copy, some that work better
then others.

However! I'm not so sure that File.Copy is the problem as much as 40
File.Copy running at one time is.

To copy a file, you (or the Framework) needs open the source file, open the
destination file, read of chunk of data, write a chunk of data, repeat
reading & writing until there is no more data.

Depending on the size of the files & the size of the chunks, I would expect
40 copies to bring a system to its knees... I would expect adding network
shares to the mix would only compound the problem

Are you using the ThreadPool or creating the threads yourself.

I would consider using a thread safe Queue to hold the copy requests and
only start a handful of Workers. Each worker would get one copy request from
the Queue, copy the file, then get another request.

I would also consider using Asynchronous File I/O to copy the contents with
Stream.BeginRea d & Stream.BeginWri te, allowing the overlapping of the reads
& writes, which may require reducing the number of Workers.

I'm not sure which ones, I suspect there are a handful of Performance
Counters that you could use to monitor the process and dynamically adjust
the number of Workers to match your system performance. System not being
stressed increase workers, system being stressed decrease workers... I know
MSDN magazine a few years ago had an article on how to use Performance
Counters to adjust number of workers...

Hope this helps
Jay

"luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com > wrote
in message news:932D6B30-E1C9-46F7-837B-804EC47EF270@mi crosoft.com...[color=blue]
> it seems that when i do file.copy the svchost.exe is hanged, i mean if i
> make
> 40 threads of file.copy , 40 copys of files at same time the system is
> going
> down and stop responding, this is when i'm working with cifs (shares).
> there
> is another solution to copy files than file.copy in .net?[/color]

[luis molina Micasoft]

Re: alternative file.copy

yes you got me, im not using a threadpool, and im trying to copy 20 or 40
files from uncs shares at same time with a independent thread for each one of
them, im gonna try with the threadpool and the asyncronous io, thanks for all.

"Jay B. Harlow [MVP - Outlook]" wrote:
[color=blue]
> Luis,
> There are any number of alternatives to file.copy, some that work better
> then others.
>
> However! I'm not so sure that File.Copy is the problem as much as 40
> File.Copy running at one time is.
>
> To copy a file, you (or the Framework) needs open the source file, open the
> destination file, read of chunk of data, write a chunk of data, repeat
> reading & writing until there is no more data.
>
> Depending on the size of the files & the size of the chunks, I would expect
> 40 copies to bring a system to its knees... I would expect adding network
> shares to the mix would only compound the problem
>
> Are you using the ThreadPool or creating the threads yourself.
>
> I would consider using a thread safe Queue to hold the copy requests and
> only start a handful of Workers. Each worker would get one copy request from
> the Queue, copy the file, then get another request.
>
> I would also consider using Asynchronous File I/O to copy the contents with
> Stream.BeginRea d & Stream.BeginWri te, allowing the overlapping of the reads
> & writes, which may require reducing the number of Workers.
>
> I'm not sure which ones, I suspect there are a handful of Performance
> Counters that you could use to monitor the process and dynamically adjust
> the number of Workers to match your system performance. System not being
> stressed increase workers, system being stressed decrease workers... I know
> MSDN magazine a few years ago had an article on how to use Performance
> Counters to adjust number of workers...
>
> Hope this helps
> Jay
>
> "luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com > wrote
> in message news:932D6B30-E1C9-46F7-837B-804EC47EF270@mi crosoft.com...[color=green]
> > it seems that when i do file.copy the svchost.exe is hanged, i mean if i
> > make
> > 40 threads of file.copy , 40 copys of files at same time the system is
> > going
> > down and stop responding, this is when i'm working with cifs (shares).
> > there
> > is another solution to copy files than file.copy in .net?[/color]
>
>
>[/color]

[Cor Ligthert]

Re: alternative file.copy

Jay,

I am quiet sure that using threads for diskcopying is useless. It gives you
even more threadmanaging processtime, and you even have even to check in a
strange way that you are not using twice the same filename in multiple
threads with the change that it will blow up your program because you cannot
control a part of this process.

Cor

[Jay B. Harlow [MVP - Outlook]]

Re: alternative file.copy

Cor,[color=blue]
> I am quiet sure that using threads for diskcopying is useless.[/color]
I'm suspect, for you, you are correct! :-|
[color=blue]
> you even have even to check in a strange way that you are not using twice
> the same filename in multiple[/color]
Which is what the Thread Safe Queue is for. Here I am referring to a
System.Collecti ons.Queue object. Then using either Queue.Synchroni zed to
create a thread safe Queue or encapsulate the Queue in your own class with
SyncLock statements to make it Thread Safe.

Seeing as a Queue is a First In First Out (FIFO) construct you would put all
the copy requests (an object with source name & destination name properties)
into the Queue. The workers would then read a request & process the copy.
[color=blue]
> the change that it will blow up your program because you cannot control a
> part of this process.[/color]
My change (using a Thread Safe Queue) is a standard pattern for
multi-threading, its the pattern that ThreadPool is based on (as suggested
by the method ThreadPool.Queu eUserWorkItem)

Asynchronous File I/O is a standard pattern within the Framework.

http://msdn.microsoft.com/library/default.asp?url=/library/en-us/cpguide/html/cpconasynchronousfileio.asp

I hope you realize my real suggestion to Luis was to limit the number of
requests, I was then offering alternatives that he may not have considered.

I do agree that one needs to be more careful writing multi-threaded
applications or not using Multi-threading, however as you reread the Luis's
original question, you will find he is already using Multi-threading!

Hope this helps
Jay


"Cor Ligthert" <notmyfirstname @planet.nl> wrote in message
news:uwJW2oStEH A.2536@TK2MSFTN GP11.phx.gbl...[color=blue]
> Jay,
>
> I am quiet sure that using threads for diskcopying is useless. It gives
> you even more threadmanaging processtime, and you even have even to check
> in a strange way that you are not using twice the same filename in
> multiple threads with the change that it will blow up your program because
> you cannot control a part of this process.
>
> Cor
>[/color]

[Jay B. Harlow [MVP - Outlook]]

Re: alternative file.copy

Luis,
Using the ThreadPool may not help per se.

My point is the number of requests going on. The ThreadPool will effectively
limit you to 25 requests. 25 requests may still be too many.

If I used the ThreadPool I would still try to limit the number of actual
copy requests going on...

Hope this helps
Jay

"luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com > wrote
in message news:1A287F1D-F3D4-4EFD-8917-BAE800A507FB@mi crosoft.com...[color=blue]
> yes you got me, im not using a threadpool, and im trying to copy 20 or 40
> files from uncs shares at same time with a independent thread for each one
> of
> them, im gonna try with the threadpool and the asyncronous io, thanks for
> all.
>
> "Jay B. Harlow [MVP - Outlook]" wrote:
>[color=green]
>> Luis,
>> There are any number of alternatives to file.copy, some that work better
>> then others.
>>
>> However! I'm not so sure that File.Copy is the problem as much as 40
>> File.Copy running at one time is.
>>
>> To copy a file, you (or the Framework) needs open the source file, open
>> the
>> destination file, read of chunk of data, write a chunk of data, repeat
>> reading & writing until there is no more data.
>>
>> Depending on the size of the files & the size of the chunks, I would
>> expect
>> 40 copies to bring a system to its knees... I would expect adding network
>> shares to the mix would only compound the problem
>>
>> Are you using the ThreadPool or creating the threads yourself.
>>
>> I would consider using a thread safe Queue to hold the copy requests and
>> only start a handful of Workers. Each worker would get one copy request
>> from
>> the Queue, copy the file, then get another request.
>>
>> I would also consider using Asynchronous File I/O to copy the contents
>> with
>> Stream.BeginRea d & Stream.BeginWri te, allowing the overlapping of the
>> reads
>> & writes, which may require reducing the number of Workers.
>>
>> I'm not sure which ones, I suspect there are a handful of Performance
>> Counters that you could use to monitor the process and dynamically adjust
>> the number of Workers to match your system performance. System not being
>> stressed increase workers, system being stressed decrease workers... I
>> know
>> MSDN magazine a few years ago had an article on how to use Performance
>> Counters to adjust number of workers...
>>
>> Hope this helps
>> Jay
>>
>> "luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com >
>> wrote
>> in message news:932D6B30-E1C9-46F7-837B-804EC47EF270@mi crosoft.com...[color=darkred]
>> > it seems that when i do file.copy the svchost.exe is hanged, i mean if
>> > i
>> > make
>> > 40 threads of file.copy , 40 copys of files at same time the system is
>> > going
>> > down and stop responding, this is when i'm working with cifs (shares).
>> > there
>> > is another solution to copy files than file.copy in .net?[/color]
>>
>>
>>[/color][/color]

[Nak]

Re: alternative file.copy

Hi Luis,

This is probably going to sound stupid, but arent multiple asynchronous
file operations slow because of hard drive technology? I'll give you an
example, copying a load of MP3's (50mb) from one drive to the other chugs
along quite happy, then when you throw just 1 more file copy to happen at
the same time the entire operation gets slowed down, and my understanding
for this is because the hard drive can't read / write that many sectors at
the same time.

Probably not helpful, but I thought I'd say it anyway! :-)

Nick.

"luis molina Micasoft" <luismolinaMica soft@discussion s.microsoft.com > wrote
in message news:932D6B30-E1C9-46F7-837B-804EC47EF270@mi crosoft.com...[color=blue]
> it seems that when i do file.copy the svchost.exe is hanged, i mean if i
> make
> 40 threads of file.copy , 40 copys of files at same time the system is
> going
> down and stop responding, this is when i'm working with cifs (shares).
> there
> is another solution to copy files than file.copy in .net?[/color]