tempfile.NamedTemporaryFile wont work

[Imbaud Pierre]

On suse 9.3, tempfile.NamedT emporaryFile() doesnt work as expected.
(I found a permanent workaround, so I dont ask for help)
I expected to write to a file, and access it thru a shell command.
This code, in a loop:
tf = tempfile.NamedT emporaryFile()
tfName = tf.name
#tf.seek(0) # rewind the file
tf.write(chunk) ; tf.flush()
print >sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists( tfName)])
subprocess.Pope n(['strings', tfName])

Symptom: the file does not always exist, after the call to
NamedTemporaryF ile(). Or at least its not seen by the strings command,
or by os.path.exists.

I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

[Bjoern Schliessmann]

Re: tempfile.NamedT emporaryFile wont work

Imbaud Pierre wrote:

tf = tempfile.NamedT emporaryFile()
tfName = tf.name
[...]
print >sys.stderr, '%s: %s' % (tfName, ['no',
'yes'][os.path.exists( tfName)])
subprocess.Pope n(['strings', tfName])

Just out of curiosity: Why did you assign tf.name to tfname?

Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Regards,


Björn

--
BOFH excuse #149:

Dew on the telephone lines.

[Steven D'Aprano]

Re: tempfile.NamedT emporaryFile wont work

On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:

On suse 9.3, tempfile.NamedT emporaryFile() doesnt work as expected.

[snip]

Symptom: the file does not always exist, after the call to
NamedTemporaryF ile(). Or at least its not seen by the strings command,
or by os.path.exists.
>
I guess the bug is pretty much os dependent, or even filesystem
dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
python interface. Or did I miss something?
Shame, I didnt even try to check for a python bug tracker.

I can verify this problem occurs on Fedora Core 5 too:

import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedT emporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists( tfName):
print 'pre-check: %s not there' % tfName
subprocess.Pope n(['strings', tfName])
if not os.path.exists( tfName):
print 'post-check: %s not there' % tfName


And here is a typical run, with the boring bits removed for ease of
reading:

>>test(30)

0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++


Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.

Is it possible this is a bug in strings?


--
Steven.

[Steven D'Aprano]

Re: tempfile.NamedT emporaryFile wont work

On Sun, 19 Nov 2006 13:18:39 +0100, Bjoern Schliessmann wrote:

Imbaud Pierre wrote:
>

> tf = tempfile.NamedT emporaryFile()
> tfName = tf.name
>[...]
> print >sys.stderr, '%s: %s' % (tfName, ['no',
>'yes'][os.path.exists( tfName)])
> subprocess.Pope n(['strings', tfName])

>
Just out of curiosity: Why did you assign tf.name to tfname?
>
Hypothetically, if tf.name changed, tfname wouldn't follow since
strings are immutable.

Well, yes, but if tf.name changed, that won't change the file name on disk
either:

>>tf = tempfile.NamedT emporaryFile()
>>tf.name

'/tmp/tmpYVV1Ij'

>>os.path.exist s(tf.name)

True

>>oldname = tf.name
>>tf.name = "/tmp/something"
>>os.path.exist s(tf.name)

False

>>os.path.exist s(oldname)

True


I'm guessing that binding tf.name to tfName is a micro-optimization. In a
very tight loop, name lookups can take considerable time, and one
optimization can be to reduce the number of lookups:

method = something.metho d
while 1:
something.metho d # needs at least two lookups
method # needs a single lookup


--
Steve.

[Peter Otten]

Re: tempfile.NamedT emporaryFile wont work

Steven D'Aprano wrote:

On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:
>

>On suse 9.3, tempfile.NamedT emporaryFile() doesnt work as expected.

[snip]
>

>Symptom: the file does not always exist, after the call to
>NamedTemporary File(). Or at least its not seen by the strings command,
>or by os.path.exists.
>>
>I guess the bug is pretty much os dependent, or even filesystem
>dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
>python interface. Or did I miss something?
>Shame, I didnt even try to check for a python bug tracker.

>
I can verify this problem occurs on Fedora Core 5 too:
>
import os
import sys
import tempfile
import subprocess
def test(n):
chunk = ': +++ abcd +++'
for i in xrange(n):
tf = tempfile.NamedT emporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(str(i) + chunk)
tf.flush()
if not os.path.exists( tfName):
print 'pre-check: %s not there' % tfName
subprocess.Pope n(['strings', tfName])
if not os.path.exists( tfName):
print 'post-check: %s not there' % tfName
>
>
And here is a typical run, with the boring bits removed for ease of
reading:
>

>>>test(30)

0: +++ abcd +++
1: +++ abcd +++
[ more of the same ]
14: +++ abcd +++
strings: '/tmp/tmpOALbx9': No such file
16: +++ abcd +++
17: +++ abcd +++
18: +++ abcd +++
[ more of the same ]
27: +++ abcd +++
strings: /tmp/tmpdc52Nz: No such file or directory
29: +++ abcd +++
>
>
Curiouser and curiouser... not only does os.path.exist always report the
temp file as existing (at least in my tests), even when strings can't find
it, but strings returns different error messages.
>
Is it possible this is a bug in strings?

What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
therefore you may enter the second iteration -- which implicitly closes the
temporary file -- before strings actually tries to access it. Use call()
and everything should be fine.

Peter

[Imbaud Pierre]

Re: tempfile.NamedT emporaryFile wont work

Steven D'Aprano a écrit :

On Sun, 19 Nov 2006 13:18:39 +0100, Bjoern Schliessmann wrote:

>>Imbaud Pierre wrote:
>>
>>

>> tf = tempfile.NamedT emporaryFile()
>> tfName = tf.name
>>>[...]
>> print >sys.stderr, '%s: %s' % (tfName, ['no',
>>>'yes'][os.path.exists( tfName)])
>> subprocess.Pope n(['strings', tfName])

>>
>>Just out of curiosity: Why did you assign tf.name to tfname?
>>
>>Hypotheticall y, if tf.name changed, tfname wouldn't follow since
>>strings are immutable.

Well, yes, but if tf.name changed, that won't change the file name on disk
either:

>>>>tf = tempfile.NamedT emporaryFile()
>>>>tf.name

'/tmp/tmpYVV1Ij'

>>>>os.path.exi sts(tf.name)

True

>>>>oldname = tf.name
>>>>tf.name = "/tmp/something"
>>>>os.path.exi sts(tf.name)

False

>>>>os.path.exi sts(oldname)

True


I'm guessing that binding tf.name to tfName is a micro-optimization.

indeed. And I dont see why tf.name would change.
In a

very tight loop, name lookups can take considerable time, and one
optimization can be to reduce the number of lookups:

method = something.metho d
while 1:
something.metho d # needs at least two lookups
method # needs a single lookup

[Imbaud Pierre]

Re: tempfile.NamedT emporaryFile wont work

Peter Otten a écrit :

Steven D'Aprano wrote:

>>On Sun, 19 Nov 2006 13:11:13 +0100, Imbaud Pierre wrote:
>>
>>

>>>On suse 9.3, tempfile.NamedT emporaryFile() doesnt work as expected.

>>
>>[snip]
>>
>>

>>>Symptom: the file does not always exist, after the call to
>>>NamedTempora ryFile(). Or at least its not seen by the strings command,
>>>or by os.path.exists.
>>>
>>>I guess the bug is pretty much os dependent, or even filesystem
>>>dependent (Im on reiserfs). Maybe the os is buggy, maybe, somehow, the
>>>python interface. Or did I miss something?
>>>Shame, I didnt even try to check for a python bug tracker.

>>
>>I can verify this problem occurs on Fedora Core 5 too:
>>
>>import os
>>import sys
>>import tempfile
>>import subprocess
>>def test(n):
>>chunk = ': +++ abcd +++'
>>for i in xrange(n):
>>tf = tempfile.NamedT emporaryFile()
>>tfName = tf.name
>>tf.seek(0)
>>tf.write(str( i) + chunk)
> tf.flush()
>>if not os.path.exists( tfName):
>>print 'pre-check: %s not there' % tfName
>>subprocess.Po pen(['strings', tfName])
>>if not os.path.exists( tfName):
>>print 'post-check: %s not there' % tfName
>>
>>
>>And here is a typical run, with the boring bits removed for ease of
>>reading:
>>
>>

>>>>>test(30)

>>
>>0: +++ abcd +++
>>1: +++ abcd +++
> [ more of the same ]
>>14: +++ abcd +++
>>strings: '/tmp/tmpOALbx9': No such file
>>16: +++ abcd +++
>>17: +++ abcd +++
>>18: +++ abcd +++
> [ more of the same ]
>>27: +++ abcd +++
>>strings: /tmp/tmpdc52Nz: No such file or directory
>>29: +++ abcd +++
>>
>>
>>Curiouser and curiouser... not only does os.path.exist always report the
>>temp file as existing (at least in my tests), even when strings can't find
>>it, but strings returns different error messages.
>>
>>Is it possible this is a bug in strings?

What /you/ are seeing is not a bug, I think. Popen() is asynchronous,
therefore you may enter the second iteration -- which implicitly closesthe
temporary file -- before strings actually tries to access it. Use call()
and everything should be fine.

Thanks A LOT, works fine, I feel kind of silly; your diagnostic is
pretty obvious, afterward... I felt uneasy not closing the Popen, but
it worked, so why bother? Its so easy to make ugly code!