cannot open file in write mode, no such file or directory

[haynesc@gmail.com]

Hi,

I'm having a problem where when trying to open a file in write mode, I
get an IOError stating no such file or directory. I'm calling an
external program which takes an input file and produces an output file
repeatedly, simulating the input file separately for each replicate.
The error occurs when trying to open the input file to write out the
new data. The problem is difficult to reproduce since it only shows up
once every few thousand replicates. I've tried using both os.system
and os.popen to invoke the external program. Originally I was running
this on cygwin, but also tried under windows.

I'm confused as to why the error would state no such file when opening
in write mode, it should just create the file if it can't find it. I
imagine it has something to do with the external program (its mostly
likely not written very well), but I'm hoping theres a solution to this
that doesn't involve modifying that program. Is it possible that the
file isn't being closed properly? If so, how can I ensure it is
available?

Thanks

[Kartic]

Re: cannot open file in write mode, no such file or directory

> I'm having a problem where when trying to open a file in write mode,
I[color=blue]
> get an IOError stating no such file or directory. I'm calling an
> external program which takes an input file and produces an output[/color]
file[color=blue]
> repeatedly, simulating the input file separately for each replicate.
> The error occurs when trying to open the input file to write out the
> new data. The problem is difficult to reproduce since it only shows[/color]
up[color=blue]
> once every few thousand replicates. I've tried using both os.system[/color]

I am afraid you need to give more information that just IOError,
calling an external program.

Please post the exact message including the input filename at the time
the program went down. If you don't print the filename, please modify
your program to do so.

A shot in the dark solution to your problem might be that you are
dynamically generating a filename and that filename probably contains
characters not allowed by the local file system OR you generate a path
that does not exist. For open() to work with the 'w' flag, the path
where the file you say should be created should exist.

Thanks,
-Kartic

[Steve Holden]

Re: cannot open file in write mode, no such file or directory

Kartic wrote:[color=blue][color=green]
>>I'm having a problem where when trying to open a file in write mode,[/color]
>
> I
>[color=green]
>>get an IOError stating no such file or directory. I'm calling an
>>external program which takes an input file and produces an output[/color]
>
> file
>[color=green]
>>repeatedly, simulating the input file separately for each replicate.
>>The error occurs when trying to open the input file to write out the
>>new data. The problem is difficult to reproduce since it only shows[/color]
>
> up
>[color=green]
>>once every few thousand replicates. I've tried using both os.system[/color]
>
>
> I am afraid you need to give more information that just IOError,
> calling an external program.
>
> Please post the exact message including the input filename at the time
> the program went down. If you don't print the filename, please modify
> your program to do so.
>
> A shot in the dark solution to your problem might be that you are
> dynamically generating a filename and that filename probably contains
> characters not allowed by the local file system OR you generate a path
> that does not exist. For open() to work with the 'w' flag, the path
> where the file you say should be created should exist.
>[/color]
Another low-probability cause is that you are trying to create files in
a non-existent directory.

regards
Steve

[haynesc@gmail.com]

Re: cannot open file in write mode, no such file or directory

Kartic wrote:[color=blue][color=green]
> > I'm having a problem where when trying to open a file in write[/color][/color]
mode,[color=blue]
> I[color=green]
> > get an IOError stating no such file or directory. I'm calling an
> > external program which takes an input file and produces an output[/color]
> file[color=green]
> > repeatedly, simulating the input file separately for each[/color][/color]
replicate.[color=blue][color=green]
> > The error occurs when trying to open the input file to write out[/color][/color]
the[color=blue][color=green]
> > new data. The problem is difficult to reproduce since it only[/color][/color]
shows[color=blue]
> up[color=green]
> > once every few thousand replicates. I've tried using both[/color][/color]
os.system[color=blue]
>
> I am afraid you need to give more information that just IOError,
> calling an external program.
>
> Please post the exact message including the input filename at the[/color]
time[color=blue]
> the program went down. If you don't print the filename, please modify
> your program to do so.
>
> A shot in the dark solution to your problem might be that you are
> dynamically generating a filename and that filename probably[/color]
contains[color=blue]
> characters not allowed by the local file system OR you generate a[/color]
path[color=blue]
> that does not exist. For open() to work with the 'w' flag, the path
> where the file you say should be created should exist.
>
> Thanks,
> -Kartic[/color]

Sorry, here is the exact error:

Traceback (most recent call last):
File "hapSim.py" , line 415, in ?
run(REPLICATES, dsSelection, permuteStatus, sigThreshold
File "hapSim.py" , line 354, in run
createInput(dt)
File "hapSim.py" , line 178, in createInput
inF = open(IN_FNAME, "w")
IOError: [Errno 2] No such file or directory: 'prog.input'

I am using the same file name repeatedly, it works fine the first few
thousand times, but eventually gives this error.

[Kartic]

Re: cannot open file in write mode, no such file or directory

Could you please post your entire program, if possible?

Thanks!

[John Machin]

Re: cannot open file in write mode, no such file or directory


haynesc@gmail.c om wrote:[color=blue]
> Kartic wrote:[color=green][color=darkred]
> > > I'm having a problem where when trying to open a file in write[/color][/color]
> mode,[color=green]
> > I[color=darkred]
> > > get an IOError stating no such file or directory. I'm calling an
> > > external program which takes an input file and produces an output[/color]
> > file[color=darkred]
> > > repeatedly, simulating the input file separately for each[/color][/color]
> replicate.[color=green][color=darkred]
> > > The error occurs when trying to open the input file to write out[/color][/color]
> the[color=green][color=darkred]
> > > new data. The problem is difficult to reproduce since it only[/color][/color]
> shows[color=green]
> > up[color=darkred]
> > > once every few thousand replicates. I've tried using both[/color][/color]
> os.system[color=green]
> >
> > I am afraid you need to give more information that just IOError,
> > calling an external program.
> >
> > Please post the exact message including the input filename at the[/color]
> time[color=green]
> > the program went down. If you don't print the filename, please[/color][/color]
modify[color=blue][color=green]
> > your program to do so.
> >
> > A shot in the dark solution to your problem might be that you are
> > dynamically generating a filename and that filename probably[/color]
> contains[color=green]
> > characters not allowed by the local file system OR you generate a[/color]
> path[color=green]
> > that does not exist. For open() to work with the 'w' flag, the path
> > where the file you say should be created should exist.
> >
> > Thanks,
> > -Kartic[/color]
>
> Sorry, here is the exact error:
>
> Traceback (most recent call last):
> File "hapSim.py" , line 415, in ?
> run(REPLICATES, dsSelection, permuteStatus, sigThreshold
> File "hapSim.py" , line 354, in run
> createInput(dt)
> File "hapSim.py" , line 178, in createInput
> inF = open(IN_FNAME, "w")
> IOError: [Errno 2] No such file or directory: 'prog.input'
>
> I am using the same file name repeatedly, it works fine the first few
> thousand times, but eventually gives this error.[/color]

1. Exactly how many is "few thousand"? Is it the same number each time?
Does the problem still happen if you don't run the external program,
but just create the input file a few thousand times?

2. Which version of Windows are you using, and what type of filesystem?
When it dies, how many files exist in the directory where you are
trying to create "prog.input "? Reason for asking: I vaguely recall
problems with FAT-type filesystems where there was a rather low limit
on the number of files that could be recorded in a directory, PLUS a
rather misleading "errno" being returned when one hit the limit.

[David Bolen]

Re: cannot open file in write mode, no such file or directory

haynesc@gmail.c om writes:
[color=blue]
> I'm having a problem where when trying to open a file in write mode, I
> get an IOError stating no such file or directory. I'm calling an
> external program which takes an input file and produces an output file
> repeatedly, simulating the input file separately for each replicate.
> The error occurs when trying to open the input file to write out the
> new data. The problem is difficult to reproduce since it only shows up
> once every few thousand replicates. I've tried using both os.system
> and os.popen to invoke the external program. Originally I was running
> this on cygwin, but also tried under windows.[/color]

You might be hitting a race condition where the OS is still
considering the file to be in use when you get around to rewriting it,
even if the using application has just exited. I've run into similar
problems when trying to rename temporary files under NT based systems.

The problem can be obscured because some of the Win32-specific IO
errors can turn into more generic IOError exceptions at the Python
level due to incomplete mappings available for all Win32 errors. In
particular, a lot of Win32-layer failures turn into EINVAL errno's at
the C RTL level, which Python in turn translates to ENOENT (which is
the file not found). So the IOError exception at the Python level can
be misleading.

Since it sounds like you can reproduce the problem relatively easily
(just run your application several thousand times), a quick check for
this condition would be to trap the IOError, delay a few seconds (say
5-10 to be absolutely sure, although in the cases I've run into 2-3 is
generally more than enough), and retry the operation. If that
succeeds, then this might be the issue you're hitting.

-- David