Question about StringIO

[Frank Millman]

Hi all

I understand that StringIO creates a file-like object in memory.

Is it possible to invoke another program, using os.system() or
os.popen(), and use the < redirect operator, so that the other program
reads my StringIO object as its input?

I will provide more details if required, but hopefully this is enough
for a simple yes or no answer, and if so, how.

BTW, I have tried using popen2() and passing my data via stdin, but the
other program (psql) does not react well to this - again, I will give
more info if necessary.

Thanks

Frank Millman

[Steve Holden]

Re: Question about StringIO

Frank Millman wrote:[color=blue]
> Hi all
>
> I understand that StringIO creates a file-like object in memory.
>
> Is it possible to invoke another program, using os.system() or
> os.popen(), and use the < redirect operator, so that the other program
> reads my StringIO object as its input?
>
> I will provide more details if required, but hopefully this is enough
> for a simple yes or no answer, and if so, how.
>
> BTW, I have tried using popen2() and passing my data via stdin, but the
> other program (psql) does not react well to this - again, I will give
> more info if necessary.
>[/color]
Unfortunately the StringIO module only creates instances inside the
process they are called: these objects have no existence to the
operating system or to other processes, and so can't be used for
inter-process communication.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Diez B. Roggisch]

Re: Question about StringIO

Frank Millman wrote:[color=blue]
> Hi all
>
> I understand that StringIO creates a file-like object in memory.
>
> Is it possible to invoke another program, using os.system() or
> os.popen(), and use the < redirect operator, so that the other program
> reads my StringIO object as its input?[/color]

No. Processes don't share memory - thus you have to either use a temp
file, or pipes.
[color=blue]
> BTW, I have tried using popen2() and passing my data via stdin, but the
> other program (psql) does not react well to this - again, I will give
> more info if necessary.[/color]

Better do so :)

Diez

[Frank Millman]

Re: Question about StringIO


Diez B. Roggisch wrote:[color=blue]
> Frank Millman wrote:[color=green]
> > Hi all
> >
> > I understand that StringIO creates a file-like object in memory.
> >
> > Is it possible to invoke another program, using os.system() or
> > os.popen(), and use the < redirect operator, so that the other program
> > reads my StringIO object as its input?[/color]
>
> No. Processes don't share memory - thus you have to either use a temp
> file, or pipes.
>[color=green]
> > BTW, I have tried using popen2() and passing my data via stdin, but the
> > other program (psql) does not react well to this - again, I will give
> > more info if necessary.[/color]
>
> Better do so :)
>
> Diez[/color]

Thanks, Steve and Diez, for the replies. I didn't think it was
possible, but it was worth asking :-)

I will try to explain my experience with popen() briefly.

I have some sql scripts to create tables, indexes, procedures, etc. At
present there are about 50 scripts, but this number will grow. I have
been running them manually so far. Now I want to automate the process.

I am supporting PostgreSQL and MS SQL Server, and the syntax is
slightly different in some cases. Rather than maintain two sets of
scripts, I prefix some lines with -pg- or -ms- to indicate the
platform, and then use Python to parse the scripts and generate a
correct output for each platform, passing it to 'psql' and 'osql'
respectively, using popen().

I have had a few problems, but it would take too long to describe them
all, and I just want a working solution, so I will focus on my latest
attempt.

I run through all the scripts and create a StringIO object with the
string I want to pass. It is about 250 000 bytes long. If I run psql
using popen(), and pass it the string via stdin, it works fine, but I
get all the messages on the screen. If I do the same, but end the
command with ' > fjm 2>&1' it works correctly and the messages end up
in the file fjm, which is about 40 000 bytes long. If I run it with
popen4(), it starts ok, but then hangs about 1/4 of the way through.
Exactly the same happens on MSW. It seems to be hitting a limit on the
size of the stdout file - is that possible?

For my purposes, I will be happy to use popen() and a choice of no
redirection, redirect to a file, or redirect to /dev/null. The question
about popen4() is therefore academic, though I would be interested to
know the answer.

BTW, is there an equivalent of /dev/null on MSW?

Thanks in advance for any suggestions.

Frank

[Benjamin Niemann]

Re: Question about StringIO

Frank Millman wrote:
[color=blue]
> I will try to explain my experience with popen() briefly.
>
> I have some sql scripts to create tables, indexes, procedures, etc. At
> present there are about 50 scripts, but this number will grow. I have
> been running them manually so far. Now I want to automate the process.
>
> I am supporting PostgreSQL and MS SQL Server, and the syntax is
> slightly different in some cases. Rather than maintain two sets of
> scripts, I prefix some lines with -pg- or -ms- to indicate the
> platform, and then use Python to parse the scripts and generate a
> correct output for each platform, passing it to 'psql' and 'osql'
> respectively, using popen().
>
> I have had a few problems, but it would take too long to describe them
> all, and I just want a working solution, so I will focus on my latest
> attempt.
>
> I run through all the scripts and create a StringIO object with the
> string I want to pass. It is about 250 000 bytes long. If I run psql
> using popen(), and pass it the string via stdin, it works fine, but I
> get all the messages on the screen. If I do the same, but end the
> command with ' > fjm 2>&1' it works correctly and the messages end up
> in the file fjm, which is about 40 000 bytes long. If I run it with
> popen4(), it starts ok, but then hangs about 1/4 of the way through.
> Exactly the same happens on MSW. It seems to be hitting a limit on the
> size of the stdout file - is that possible?
>
> For my purposes, I will be happy to use popen() and a choice of no
> redirection, redirect to a file, or redirect to /dev/null. The question
> about popen4() is therefore academic, though I would be interested to
> know the answer.[/color]

That's probably a deadlock as described in
<http://docs.python.org/lib/popen2-flow-control.html>
[color=blue]
> BTW, is there an equivalent of /dev/null on MSW?[/color]

Dunno - but as a last resort, you could create a tempfile with a unique name
(to be sure, not to override any existing data), dump your output there and
later os.unlink() it...

--
Benjamin Niemann
Email: pink at odahoda dot de
WWW: http://www.odahoda.de/

[Diez B. Roggisch]

Re: Question about StringIO

> Thanks, Steve and Diez, for the replies. I didn't think it was[color=blue]
> possible, but it was worth asking :-)
>
> I will try to explain my experience with popen() briefly.
>
> I have some sql scripts to create tables, indexes, procedures, etc. At
> present there are about 50 scripts, but this number will grow. I have
> been running them manually so far. Now I want to automate the process.
>
> I am supporting PostgreSQL and MS SQL Server, and the syntax is
> slightly different in some cases. Rather than maintain two sets of
> scripts, I prefix some lines with -pg- or -ms- to indicate the
> platform, and then use Python to parse the scripts and generate a
> correct output for each platform, passing it to 'psql' and 'osql'
> respectively, using popen().[/color]

Why don't youn use te python DB-Api instead?


Regards,

Diez

[Frank Millman]

Re: Question about StringIO

Benjamin Niemann wrote:[color=blue]
> Frank Millman wrote:
>[color=green]
> > I will try to explain my experience with popen() briefly.
> >
> > I run through all the scripts and create a StringIO object with the
> > string I want to pass. It is about 250 000 bytes long. If I run psql
> > using popen(), and pass it the string via stdin, it works fine, but I
> > get all the messages on the screen. If I do the same, but end the
> > command with ' > fjm 2>&1' it works correctly and the messages end up
> > in the file fjm, which is about 40 000 bytes long. If I run it with
> > popen4(), it starts ok, but then hangs about 1/4 of the way through.
> > Exactly the same happens on MSW. It seems to be hitting a limit on the
> > size of the stdout file - is that possible?
> >[/color]
>
> That's probably a deadlock as described in
> <http://docs.python.org/lib/popen2-flow-control.html>
>[/color]

Thanks for this pointer. I have read it, but I don't think it applies
to my situation, as it talks about 'reading' from the child's stdout
while the child is 'writing' to stderr. I am not doing that, or at
least not consciously. Here is a code snippet. 's' is a StringIO object
that contains my input. It is about 6000 lines/250000 bytes long.

-------------

sql_stdin,sql_s tdout = os.popen4('psql -U %s -d %s' % (user,database) )

sql_stdin.write lines(s.readlin es())
s.close()
sql_stdin.close ()

-------------

It starts, and then hangs, after processing about 6% of my input.

If I add ' > fjm 2>&1' to the command, it works, so it is definitely
connected with the child writing to stdout/stderr.

I tried storing my input in a list, and passing ''.join(s), but it had
the same result. I also looped over the list and wrote one line at a
time to sql_stdin - same result.

I can work around this, so a solution is not critical. However, it
would be nice to know if this is a limitation of popen4(), or if I am
doing something wrong.
[color=blue][color=green]
> > BTW, is there an equivalent of /dev/null on MSW?[/color]
>
> Dunno - but as a last resort, you could create a tempfile with a unique name
> (to be sure, not to override any existing data), dump your output there and
> later os.unlink() it...
>[/color]

A quick google revealed the answer - there is a device called NUL which
achieves the same purpose.

Thanks

Frank

[Frank Millman]

Re: Question about StringIO

Diez B. Roggisch wrote:[color=blue][color=green]
> > Thanks, Steve and Diez, for the replies. I didn't think it was
> > possible, but it was worth asking :-)
> >
> > I will try to explain my experience with popen() briefly.
> >
> > I have some sql scripts to create tables, indexes, procedures, etc. At
> > present there are about 50 scripts, but this number will grow. I have
> > been running them manually so far. Now I want to automate the process.
> >
> > I am supporting PostgreSQL and MS SQL Server, and the syntax is
> > slightly different in some cases. Rather than maintain two sets of
> > scripts, I prefix some lines with -pg- or -ms- to indicate the
> > platform, and then use Python to parse the scripts and generate a
> > correct output for each platform, passing it to 'psql' and 'osql'
> > respectively, using popen().[/color]
>
> Why don't youn use te python DB-Api instead?
>
>
> Regards,
>
> Diez[/color]

My scripts are used to create the tables in the database. I didn't
think that DB-API covered that. However, even if it did, I don't think
it would handle differences such as the following.

For Unicode support, PostgreSQL uses the character-set specified when
the database is created. SQL Server allows you to specify it for each
column, using the datatype NCHAR and NVARCHAR instead of CHAR and
VARCHAR.

PostgreSQL uses data types called DATE and TIMESTAMP. SQL Server uses
DATETIME (it also uses TIMESTAMP, but that is used for something else).

Both DBMS's have the concept of a column which is automatically
assigned a 'next number' each time a row is created, but the syntax for
defining the column is completely different.

PostgreSQL allows the use of a WHERE clause when creating an INDEX,
which is useful if you only want to index a subset of a table.

SQL Server has the concept of a CLUSTERED INDEX, whereby it stores the
rows physically in index sequence. It defaults to using a clustered
index for the primary key. Often this is not what you want, so it is
desirable to specify the primary key as NONCLUSTERED, and then specify
a CLUSTERED index for a more frequently used column.

These are just a few of the differences, but you get the idea. If there
is a better way to do this in a cross-platform manner, I would love to
know how.

Thanks

Frank

[Diez B. Roggisch]

Re: Question about StringIO

> Thanks for this pointer. I have read it, but I don't think it applies[color=blue]
> to my situation, as it talks about 'reading' from the child's stdout
> while the child is 'writing' to stderr.[/color]

But that is exactly the point: the psql blocks because you don't read
away the buffered data. Start a thread, read that stdout/stderr and see
if things go smoothly.

Diez

[Diez B. Roggisch]

Re: Question about StringIO

> My scripts are used to create the tables in the database. I didn't[color=blue]
> think that DB-API covered that.[/color]

The DB-Api covers executin arbirary SQL - either DDL or DML. It is
surely centered around DML, but that doesn't mean that its not usabel to
issue "create ..." statements.
[color=blue]
>However, even if it did, I don't think
> it would handle differences such as the following.[/color]

<snip>

All that has nocthing to do with teh API - you'd still need your
differentiated DDL - but the communication with the programs would go away.

Diez

[Frank Millman]

Re: Question about StringIO


Diez B. Roggisch wrote:[color=blue][color=green]
> > Thanks for this pointer. I have read it, but I don't think it applies
> > to my situation, as it talks about 'reading' from the child's stdout
> > while the child is 'writing' to stderr.[/color]
>
> But that is exactly the point: the psql blocks because you don't read
> away the buffered data. Start a thread, read that stdout/stderr and see
> if things go smoothly.
>
> Diez[/color]

Of course (kicks himself), it is obvious now that you have explained
it. I tried your suggestion and it works perfectly.

Many thanks

Frank

[Frank Millman]

Re: Question about StringIO


Diez B. Roggisch wrote:[color=blue][color=green]
> > My scripts are used to create the tables in the database. I didn't
> > think that DB-API covered that.[/color]
>
> The DB-Api covers executin arbirary SQL - either DDL or DML. It is
> surely centered around DML, but that doesn't mean that its not usabel to
> issue "create ..." statements.
>[color=green]
> >However, even if it did, I don't think
> > it would handle differences such as the following.[/color]
>
> <snip>
>
> All that has nocthing to do with teh API - you'd still need your
> differentiated DDL - but the communication with the programs would go away.
>
> Diez[/color]

I understand. It certainly gives me an alternative approach - I will
experiment to see which suits my purpose best.

Many thanks for your assistance.

Frank