Problems with file IO in a thread, and shutil

[Roopesh]

Hi,

I have a multithreaded application. There are two threads, T1 and T2.
Suppose that there are two folders A, B. Thread T1 fetches data from
network and creates files in folder A and after creating each file, it
moves the file to folder B, using shutil.move().

Thread T2, takes files from folder B and processes it.
Note: Only the thread T1 has access to the files in folder A.

psuedo code
=========
class T1(thread):
def run():
self.download()

def download():
data = from_network
filename = os.path.join ( "A", "file1")

f = open ( filename, "w")
for d in data:
f.write ( d + "\n" )
f.flush()
f.close()

shutil.move(os. path.join ( "A", "file1"), os.path.join("B ",
"file1"))


class T2(thread):
run()
process(listdir (os.path.join(" B")))

All the files has similar contents. But in some cases(very rare), when
thread T1 tries to move the newly created file from folder A to folder
B, an exception occurs (on shutil.move()).

The exception is WindowsError(32 , 'The process cannot access the file
because it is being used by another process'). I looked inside the
shutil.move. What I found was due to this WindowsError, the usual
os.rename() inside shutil.move() fails and the file is copied using
copy2(src, dst). Finally os.unlink() also failed. (So though copying
occurred, deleting the source file failed)

I am not getting why this error comes. Has anyone faced similar
situation? Please help me.

Thanks and Regards
Roopesh

[MRAB]

Re: Problems with file IO in a thread, and shutil

On Jun 25, 4:11 pm, Roopesh <roopesh....@gm ail.comwrote:

Hi,
>
I have a multithreaded application. There are two threads, T1 and T2.
Suppose that there are two folders A, B.  Thread T1 fetches data from
network and creates files in folder A and after creating each file, it
moves the file to folder B, using shutil.move().
>
Thread T2, takes files from folder B and processes it.
Note: Only the thread T1 has access to the files in folder A.
>
psuedo code
=========
class T1(thread):
   def run():
     self.download()
>
  def download():
     data = from_network
     filename = os.path.join ( "A", "file1")
>
     f = open ( filename, "w")
     for d in data:
        f.write ( d + "\n" )
     f.flush()
     f.close()
>
    shutil.move(os. path.join ( "A", "file1"),  os.path.join("B ",
"file1"))
>
class T2(thread):
  run()
      process(listdir (os.path.join(" B")))
>
All the files has similar contents. But in some cases(very rare), when
thread T1 tries to move the newly created file from folder A to folder
B, an exception occurs (on shutil.move()).
>
The exception is WindowsError(32 , 'The process cannot access the file
because it is being used by another process'). I looked inside the
shutil.move. What I found was due to this WindowsError, the usual
os.rename() inside shutil.move() fails and the file is copied using
copy2(src, dst). Finally os.unlink() also failed. (So though copying
occurred, deleting the source file failed)
>
I am not getting why this error comes. Has anyone faced similar
situation? Please help me.
>
Thanks and Regards
Roopesh

Do you have any anti-virus/anti-spyware software installed? (If not,
why not? :-))

It might be that your anti-virus/anti-spyware software is seeing the
new file appear and scanning it. If that happens just when
shutil.move() attempts to move it then the move will fail. You could
have your code wait for a while and then try again.

[Roopesh]

Re: Problems with file IO in a thread, and shutil

Thanks for the reply. I did testing in a clean system, were anti virus/
spyware is not installed. It still gave this problem, in say 1 out of
1000 cases.

By any chance would it be possible that the Windows OS has not
completed writing to the file even after file.flush() and file.close()
is called?

Thanks
Roopesh

[MRAB]

Re: Problems with file IO in a thread, and shutil

On Jun 26, 8:06 am, Roopesh <roopesh....@gm ail.comwrote:

Thanks for the reply. I did testing in a clean system, were anti virus/
spyware is not installed. It still gave this problem, in say 1 out of
1000 cases.
>
By any chance would it be possible that the Windows OS has not
completed writing to the file even after file.flush() and file.close()
is called?
>
Thanks
Roopesh

Maybe it's a Windows service, eg the indexing service or generating
thumbnails. Anyway, retrying after a delay would still be worth it.

[Roopesh]

Re: Problems with file IO in a thread, and shutil

Maybe it's a Windows service, eg the indexing service or generating

thumbnails. Anyway, retrying after a delay would still be worth it.

Yes, this thing works :-) Thanks a lot.

Thanks
Roopesh