Safely renaming a file without overwriting

[Steven D'Aprano]

I want to rename a file, but only if the destination file name doesn't
already exist.

I can do this:

if os.path.exists( dest):
# skip file, raise an exception, make a backup...
do_something_el se()
else:
os.rename(src, dest)


But on a multi-user system, it is possible that dest is created in the
time period between checking if it exists and attempting the rename.

Is there any way to prevent this? Or do I just try to keep the check and
the rename as close together as possible, minimizing the chances and
hoping for the best?


--
Steven.

[Wolfgang Draxinger]

Re: Safely renaming a file without overwriting

Steven D'Aprano wrote:

I want to rename a file, but only if the destination file name
doesn't already exist.
>
I can do this:
>
if os.path.exists( dest):
# skip file, raise an exception, make a backup...
do_something_el se()
else:
os.rename(src, dest)
>
>
But on a multi-user system, it is possible that dest is created
in the time period between checking if it exists and attempting
the rename.
>
Is there any way to prevent this? Or do I just try to keep the
check and the rename as close together as possible, minimizing
the chances and hoping for the best?

1: Open the file with os.open

2: Lock the file exclusively -no other process can now access
it.

3: Use rename to rename the file; this causes a file system level
implicit unlink of the old file (it dissappears from the file
system) but the opening process can still access it.

4: close the file -the lock is removed and the rename
finalized.

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867
GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E

[Diez B. Roggisch]

Re: Safely renaming a file without overwriting

Wolfgang Draxinger schrieb:

Steven D'Aprano wrote:
>

>I want to rename a file, but only if the destination file name
>doesn't already exist.
>>
>I can do this:
>>
>if os.path.exists( dest):
> # skip file, raise an exception, make a backup...
> do_something_el se()
>else:
> os.rename(src, dest)
>>
>>
>But on a multi-user system, it is possible that dest is created
>in the time period between checking if it exists and attempting
>the rename.
>>
>Is there any way to prevent this? Or do I just try to keep the
>check and the rename as close together as possible, minimizing
>the chances and hoping for the best?

>
1: Open the file with os.open
>
2: Lock the file exclusively -no other process can now access
it.
>
3: Use rename to rename the file; this causes a file system level
implicit unlink of the old file (it dissappears from the file
system) but the opening process can still access it.
>
4: close the file -the lock is removed and the rename
finalized.

Where does that help for new files? The OP was right in assuming that a
race condition could occur when he tests for a file & then tries to
create it, as in the meantime it could have been created.

Diez

[Wolfgang Draxinger]

Re: Safely renaming a file without overwriting

Diez B. Roggisch wrote:

>1: Open the file with os.open
>>
>2: Lock the file exclusively -no other process can now
>access it.
>>
>3: Use rename to rename the file; this causes a file system
>level implicit unlink of the old file (it dissappears from the
>file system) but the opening process can still access it.
>>
>4: close the file -the lock is removed and the rename
>finalized.

The open is to happen on the new file name with O_CREAT | O_EXCL
flags. Sorry, I forgot that to mention explicitly. However I
have not tried it yet, but it should work.

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867
GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E

[Chetan]

Re: Safely renaming a file without overwriting

Steven D'Aprano <steve@REMOVE.T HIS.cybersource .com.auwrites:

I want to rename a file, but only if the destination file name doesn't
already exist.
>
I can do this:
>
if os.path.exists( dest):
# skip file, raise an exception, make a backup...
do_something_el se()
else:
os.rename(src, dest)
>
>
But on a multi-user system, it is possible that dest is created in the
time period between checking if it exists and attempting the rename.
>
Is there any way to prevent this? Or do I just try to keep the check and
the rename as close together as possible, minimizing the chances and
hoping for the best?
>
>
--
Steven.

The answer, unfortunately, depends on the platform. I haven't tried, but it
looks like rename() will fail on Win32 if the file already exists. On Unix, you
can use link to rename the file - which will not overwrite the file if it
exists. Then use unlink to remove the src file.

Chetan

[Steven D'Aprano]

Re: Safely renaming a file without overwriting

On Sat, 28 Oct 2006 13:38:14 +0200, Wolfgang Draxinger wrote:

>But on a multi-user system, it is possible that dest is created
>in the time period between checking if it exists and attempting
>the rename.
>>
>Is there any way to prevent this? Or do I just try to keep the
>check and the rename as close together as possible, minimizing
>the chances and hoping for the best?

>
1: Open the file with os.open

Open "the" file? There are potentially two files -- the source and
destination. I only want to do the rename if the destination
*doesn't* exist, so there is no destination file to open. How will it help
me to lock the source file? Have I misunderstood?


--
Steven.

[Steven D'Aprano]

Re: Safely renaming a file without overwriting

On Sat, 28 Oct 2006 16:48:37 +0200, Diez B. Roggisch wrote:

Where does that help for new files? The OP was right in assuming that a
race condition could occur when he tests for a file & then tries to
create it, as in the meantime it could have been created.

Ah! "Race condition" -- that was the term I was looking for ... now maybe
I'll have some better results with Google.



--
Steven.

[Wolfgang Draxinger]

Re: Safely renaming a file without overwriting

Steven D'Aprano wrote:

Open "the" file? There are potentially two files -- the source
and destination. I only want to do the rename if the
destination *doesn't* exist, so there is no destination file to
open. How will it help me to lock the source file? Have I
misunderstood?

I forgot to say, to open the source file with O_CREAT | O_EXCL.
The open will fail if the file already exists. By locking the
file no other process will be able to access it, but the process
that holds the lock can do with it anything. This includes to
rename an existing file to the name of the opened one - the
previously created placeholder file will get unlinked before,
but there is _probably_ no way that any process can intercept
this. It is an interesting thing, that files that are opened
remain fully usable and accessible if you unlink them as long
you don't close them. You have to close it, to remove all
remains of it.

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867
GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E

[Steven D'Aprano]

Re: Safely renaming a file without overwriting

On Sun, 29 Oct 2006 00:29:06 +0200, Wolfgang Draxinger wrote:

Steven D'Aprano wrote:
>

>Open "the" file? There are potentially two files -- the source
>and destination. I only want to do the rename if the
>destination *doesn't* exist, so there is no destination file to
>open. How will it help me to lock the source file? Have I
>misunderstoo d?

>
I forgot to say, to open the source file with O_CREAT | O_EXCL.
The open will fail if the file already exists.

But the source file always exists, otherwise there is nothing to rename!
Do you mean, open the destination filename?



--
Steven.

[Wolfgang Draxinger]

Re: Safely renaming a file without overwriting

Steven D'Aprano wrote:

But the source file always exists, otherwise there is nothing
to rename! Do you mean, open the destination filename?

Of course I meant the destination file. Someone please spill some
ice chilled water over me to get me awake again. Time to go to
bed :-P before I make more dumb mistakes/typos...

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867
GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E

[Diez B. Roggisch]

Re: Safely renaming a file without overwriting

Wolfgang Draxinger schrieb:

Steven D'Aprano wrote:
>

>But the source file always exists, otherwise there is nothing
>to rename! Do you mean, open the destination filename?

>
Of course I meant the destination file. Someone please spill some
ice chilled water over me to get me awake again. Time to go to
bed :-P before I make more dumb mistakes/typos...

But that doesn't help. Opening the destination file that way will only
make a difference if _both_ processes that fight over that filename work
that way. Which might be possible in the actual case, but not as a
general recipe.

The link/unlink trick of Chetan sounds reasonable, though.

[Wolfgang Draxinger]

Re: Safely renaming a file without overwriting

Diez B. Roggisch wrote:

The link/unlink trick of Chetan sounds reasonable, though.

It will work only if both source and destination are on the same
file system, which means you can't move a file between mount
points - at least that's the way how it's defined by POSIX; NTFS
supports hard links and soft links, too but I don't know if
those got the same constraints.

Wolfgang Draxinger
--
E-Mail address works, Jabber: hexarith@jabber .org, ICQ: 134682867
GPG key FP: 2FC8 319E C7D7 1ADC 0408 65C6 05F5 A645 1FD3 BD3E