Problem flushing stderr in embedded python

[Farshid Lashkari]

Hi,

My application has python embedded into it. I noticed that when I run any
python code the output is buffered and doesn't get flushed until my
application exits. To fix this I simply flush sys.stdout and sys.stderr
every once in while by using the following code:

//Get handle to python stdout file and flush it
PyObject *pyStdout = PySys_GetObject ("stdout");
if(pyStdout && PyFile_Check(py Stdout)) {
PyObject *result = PyObject_CallMe thod(pyStdout," flush",NULL);
Py_XDECREF(resu lt);
}

//Get handle to python stderr file and flush it
PyObject *pyStderr = PySys_GetObject ("stderr");
if(pyStderr && PyFile_Check(py Stderr)) {
PyObject *result = PyObject_CallMe thod(pyStderr," flush",NULL);
Py_XDECREF(resu lt);
}

This works for stdout but not for stderr. In order to flush stderr I have to
do:

PyRun_SimpleStr ing("sys.stderr .flush()");

Does anybody know why my first method doesn't work with stderr? I checked
and PyObject_CallMe thod is being called for both objects and no errors are
occuring. The solution I have is acceptable, but I would still like to know
why my first method didn't work.

BTW, I'm running python 2.3 on Windows XP

Thanks,

Farshid

[caroundw5h]

Re: Problem flushing stderr in embedded python

"Farshid Lashkari" <lashkariNO@SPA Mworldviz.com> wrote in message news:<VoRjd.33$ iY3.7@trnddc01> ...[color=blue]
> Hi,
>
> My application has python embedded into it. I noticed that when I run any
> python code the output is buffered and doesn't get flushed until my
> application exits. To fix this I simply flush sys.stdout and sys.stderr
> every once in while by using the following code:
>
> //Get handle to python stdout file and flush it
> PyObject *pyStdout = PySys_GetObject ("stdout");
> if(pyStdout && PyFile_Check(py Stdout)) {
> PyObject *result = PyObject_CallMe thod(pyStdout," flush",NULL);
> Py_XDECREF(resu lt);
> }
>
> //Get handle to python stderr file and flush it
> PyObject *pyStderr = PySys_GetObject ("stderr");
> if(pyStderr && PyFile_Check(py Stderr)) {
> PyObject *result = PyObject_CallMe thod(pyStderr," flush",NULL);
> Py_XDECREF(resu lt);
> }
>
> This works for stdout but not for stderr. In order to flush stderr I have to
> do:
>
> PyRun_SimpleStr ing("sys.stderr .flush()");
>
> Does anybody know why my first method doesn't work with stderr? I checked
> and PyObject_CallMe thod is being called for both objects and no errors are
> occuring. The solution I have is acceptable, but I would still like to know
> why my first method didn't work.
>
> BTW, I'm running python 2.3 on Windows XP
>
> Thanks,
>
> Farshid[/color]
I'm sorry to notice that you haven't received any responses back to
your questions even though you have found a workaround. I've noticed
this trend quite a lot in the python community and having learned
python first and now C(in order to understand python better) I'm
frankly disspapointed. I believe python and C are a wonderful marriage
but I suppose because many ppl come to python as their intro language
they info on extending and embedding is limited. I myself haven't
started using the python api but will soon, I hope I for one will be
someone who will be able to provide an answer next time.

Good luck.

BTW: from a C perspective. I don't understand your problem though.
stdout and stderr are the normal output device for your system -
usally your screen. I don't see why you should have to call one via
python sys module.
Perhaps because one doesn't normally flush(stderr) explicity but
flush(stdout).

interested to know as well.

[Farshid Lashkari]

Re: Problem flushing stderr in embedded python

I don't understand it from a C perspective either. I poked around the source
code for python and found where it is flushing the file. It is simply
calling fflush on the FILE handle. Also, I'm somewhat bewildered as to why
running the string "sys.stderr.flu sh()" is different than directly calling
the flush method of the stderr object. Oh well, I guess some things are
better left unknown.

"caroundw5h " <caroundw5h@yah oo.com> wrote in message
news:aa1af1ad.0 411082048.6432d 122@posting.goo gle.com...[color=blue]
> "Farshid Lashkari" <lashkariNO@SPA Mworldviz.com> wrote in message[/color]
news:<VoRjd.33$ iY3.7@trnddc01> ...[color=blue][color=green]
> > Hi,
> >
> > My application has python embedded into it. I noticed that when I run[/color][/color]
any[color=blue][color=green]
> > python code the output is buffered and doesn't get flushed until my
> > application exits. To fix this I simply flush sys.stdout and sys.stderr
> > every once in while by using the following code:
> >
> > //Get handle to python stdout file and flush it
> > PyObject *pyStdout = PySys_GetObject ("stdout");
> > if(pyStdout && PyFile_Check(py Stdout)) {
> > PyObject *result = PyObject_CallMe thod(pyStdout," flush",NULL);
> > Py_XDECREF(resu lt);
> > }
> >
> > //Get handle to python stderr file and flush it
> > PyObject *pyStderr = PySys_GetObject ("stderr");
> > if(pyStderr && PyFile_Check(py Stderr)) {
> > PyObject *result = PyObject_CallMe thod(pyStderr," flush",NULL);
> > Py_XDECREF(resu lt);
> > }
> >
> > This works for stdout but not for stderr. In order to flush stderr I[/color][/color]
have to[color=blue][color=green]
> > do:
> >
> > PyRun_SimpleStr ing("sys.stderr .flush()");
> >
> > Does anybody know why my first method doesn't work with stderr? I[/color][/color]
checked[color=blue][color=green]
> > and PyObject_CallMe thod is being called for both objects and no errors[/color][/color]
are[color=blue][color=green]
> > occuring. The solution I have is acceptable, but I would still like to[/color][/color]
know[color=blue][color=green]
> > why my first method didn't work.
> >
> > BTW, I'm running python 2.3 on Windows XP
> >
> > Thanks,
> >
> > Farshid[/color]
> I'm sorry to notice that you haven't received any responses back to
> your questions even though you have found a workaround. I've noticed
> this trend quite a lot in the python community and having learned
> python first and now C(in order to understand python better) I'm
> frankly disspapointed. I believe python and C are a wonderful marriage
> but I suppose because many ppl come to python as their intro language
> they info on extending and embedding is limited. I myself haven't
> started using the python api but will soon, I hope I for one will be
> someone who will be able to provide an answer next time.
>
> Good luck.
>
> BTW: from a C perspective. I don't understand your problem though.
> stdout and stderr are the normal output device for your system -
> usally your screen. I don't see why you should have to call one via
> python sys module.
> Perhaps because one doesn't normally flush(stderr) explicity but
> flush(stdout).
>
> interested to know as well.[/color]

[Donn Cave]

Re: Problem flushing stderr in embedded python

Quoth "Farshid Lashkari" <lashkariNO@SPA Mworldviz.com>:
| I don't understand it from a C perspective either. I poked around the source
| code for python and found where it is flushing the file. It is simply
| calling fflush on the FILE handle. Also, I'm somewhat bewildered as to why
| running the string "sys.stderr.flu sh()" is different than directly calling
| the flush method of the stderr object. Oh well, I guess some things are
| better left unknown.

It could be different if the stderr object - i.e., PySys_GetObject ("stderr") -
is actually not sys.stderr. I don't understand why, but am not familiar
with PySys_*, so I offer that as a hypothesis you might be able to test.
By the way, since as you observed it's only calling fflush(), I think you
could leave Python out of it altogether and do this with just
fflush(stdout);
fflush(stderr);

Donn Cave, donn@u.washingt on.edu

[Farshid Lashkari]

Re: Problem flushing stderr in embedded python

Here's the code for PySys_GetObject . It's getting the object from the sys
dictionary, so it should be the correct object.

PyObject *
PySys_GetObject (char *name)
{
PyThreadState *tstate = PyThreadState_G et();
PyObject *sd = tstate->interp->sysdict;
if (sd == NULL)
return NULL;
return PyDict_GetItemS tring(sd, name);
}

I've tried calling fflush on stderr, but that doesn't work either.

I've encountered another weird problem also. If my python code causes an
error and I don't flush stderr, then I get a runtime error when my app
exits. It also prints out:

Exception exceptions.Name Error: "global name 'y' is not defined" in 'garbage
collection' ignored
Fatal Python error: unexpected exception during garbage collection.

Does this give anybody a clue as to what's happening?

"Donn Cave" <donn@drizzle.c om> wrote in message
news:1100065836 .697971@yasure. ..[color=blue]
> Quoth "Farshid Lashkari" <lashkariNO@SPA Mworldviz.com>:
> | I don't understand it from a C perspective either. I poked around the[/color]
source[color=blue]
> | code for python and found where it is flushing the file. It is simply
> | calling fflush on the FILE handle. Also, I'm somewhat bewildered as to[/color]
why[color=blue]
> | running the string "sys.stderr.flu sh()" is different than directly[/color]
calling[color=blue]
> | the flush method of the stderr object. Oh well, I guess some things are
> | better left unknown.
>
> It could be different if the stderr object - i.e.,[/color]
PySys_GetObject ("stderr") -[color=blue]
> is actually not sys.stderr. I don't understand why, but am not familiar
> with PySys_*, so I offer that as a hypothesis you might be able to test.
> By the way, since as you observed it's only calling fflush(), I think you
> could leave Python out of it altogether and do this with just
> fflush(stdout);
> fflush(stderr);
>
> Donn Cave, donn@u.washingt on.edu[/color]

[Donn Cave]

Re: Problem flushing stderr in embedded python

Quoth "Farshid Lashkari" <lashkariNO@SPA Mworldviz.com>:
....
| I've tried calling fflush on stderr, but that doesn't work either.

OK, this would seem to mean that your sys.stderr has been replaced
with something else. I guess I don't know for sure what you can
count on in Windows, but I would have to expect that fflush on stderr
really does work - that is, it flushes stderr's buffer.

| I've encountered another weird problem also. If my python code causes an
| error and I don't flush stderr, then I get a runtime error when my app
| exits. It also prints out:
|
| Exception exceptions.Name Error: "global name 'y' is not defined" in 'garbage
| collection' ignored
| Fatal Python error: unexpected exception during garbage collection.
|
| Does this give anybody a clue as to what's happening?

Doesn't do anything for me, but you may find out what that is when you
locate the code for the faux stderr you're (not) flushing. I would be
more worried about this 'y' thing in principle, though, because it sounds
like a problem in the C (or whatever) to Python interface that could have
other unpleasant consequences.

Donn Cave, donn@drizzle.co m

[Farshid Lashkari]

Re: Problem flushing stderr in embedded python

Well I've somewhat solved my problem. First off, the 'y' problem was just an
error I purposely created in order to trigger an exception. My application
uses a bunch of calls to PyObject_Call to run python code. After each of
these calls I added:

if(PyErr_Occurr ed()) PyErr_Print();

This will force the errors to be printed to stderr, and now calling the
flush method on the stderr object works. I still don't understand why
running the string "sys.stderr.flu sh()" is different. Maybe it forces any
current errors to be printed and then flushes them too. My app doesn't
replace the python stderr object either.

Anyways, thanks to everybody for your help and suggestions.

-Farshid

"Donn Cave" <donn@drizzle.c om> wrote in message
news:1100156058 .567069@yasure. ..[color=blue]
> Quoth "Farshid Lashkari" <lashkariNO@SPA Mworldviz.com>:
> ...
> | I've tried calling fflush on stderr, but that doesn't work either.
>
> OK, this would seem to mean that your sys.stderr has been replaced
> with something else. I guess I don't know for sure what you can
> count on in Windows, but I would have to expect that fflush on stderr
> really does work - that is, it flushes stderr's buffer.
>
> | I've encountered another weird problem also. If my python code causes an
> | error and I don't flush stderr, then I get a runtime error when my app
> | exits. It also prints out:
> |
> | Exception exceptions.Name Error: "global name 'y' is not defined" in[/color]
'garbage[color=blue]
> | collection' ignored
> | Fatal Python error: unexpected exception during garbage collection.
> |
> | Does this give anybody a clue as to what's happening?
>
> Doesn't do anything for me, but you may find out what that is when you
> locate the code for the faux stderr you're (not) flushing. I would be
> more worried about this 'y' thing in principle, though, because it sounds
> like a problem in the C (or whatever) to Python interface that could have
> other unpleasant consequences.
>
> Donn Cave, donn@drizzle.co m[/color]