Why does StringIO discard its initial value?

[Leif K-Brooks]

When StringIO gets an initial value passed to its constructor, it seems
to discard it after the first call to .write(). For instance:
[color=blue][color=green][color=darkred]
>>> from StringIO import StringIO
>>> buffer = StringIO('foo')
>>> buffer.getvalue ()[/color][/color][/color]
'foo'[color=blue][color=green][color=darkred]
>>> buffer.write('b ar')
>>> buffer.getvalue ()[/color][/color][/color]
'bar'[color=blue][color=green][color=darkred]
>>> buffer.write('b az')
>>> buffer.getvalue ()[/color][/color][/color]
'barbaz'

The obvious workaround is to call buffer.write() with the initial value
instead of passing it to StringIO's constructor, so this issue doesn't
bother me very much, but I'm still curious about it. Is this the
expected behavior, and why it isn't mentioned in the docs if so?

[jepler@unpythonic.net]

Re: Why does StringIO discard its initial value?

Maybe this short interactive session can give you an idea why.
[color=blue][color=green][color=darkred]
>>> from StringIO import StringIO
>>> b = StringIO("12345 6789")
>>> b.tell()[/color][/color][/color]
0[color=blue][color=green][color=darkred]
>>> b.write("abc")
>>> b.getvalue()[/color][/color][/color]
'abc456789'[color=blue][color=green][color=darkred]
>>> b.tell()[/color][/color][/color]
3

StringIO seems to operate like a file opened with "r+" (If I've got my modes
right): it is opened for reading and writing, and positioned at the beginning.
In my example, the write of 3 bytes overwrites the first 3 bytes of the file
and leaves the rest intact. In your example your first write overwrote the
whole initial contents of the file, so you couldn't notice this effect.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.6 (GNU/Linux)

iD8DBQFCWWqGJd0 1MZaTXX0RAvSuAJ 9lSChyzOej2TkqL uoaWpzxopOUPACf Qv8D
lWmB6rReTFep5sY MwanqF7I=
=t4F9
-----END PGP SIGNATURE-----

[Raymond Hettinger]

Re: Why does StringIO discard its initial value?

[Leif K-Brooks][color=blue]
> The obvious workaround is to call buffer.write() with the initial value
> instead of passing it to StringIO's constructor,[/color]

More than just a workaround, it is the preferred approach.
That makes is easier to switch to cStringIO where initialized objects are
read-only.


[color=blue]
> Is this the
> expected behavior[/color]

Yes.


[color=blue]
> , and why it isn't mentioned in the docs if so?[/color]

Per your request, the docs have been updated.



Raymond Hettinger

[David Fraser]

Re: Why does StringIO discard its initial value?

Raymond Hettinger wrote:[color=blue]
> [Leif K-Brooks]
>[color=green]
>>The obvious workaround is to call buffer.write() with the initial value
>>instead of passing it to StringIO's constructor,[/color]
>
>
> More than just a workaround, it is the preferred approach.
> That makes is easier to switch to cStringIO where initialized objects are
> read-only.
>[/color]

Others may find this helpful ; it's a pure Python wrapper for cStringIO
that makes it behave like StringIO in not having initialized objects
readonly. Would it be an idea to extend cStringIO like this in the
standard library? It shouldn't lose performance if used like a standard
cStringIO, but it prevents frustration :-)

David

class StringIO:
def __init__(self, buf = ''):
if not isinstance(buf, (str, unicode)):
buf = str(buf)
self.len = len(buf)
self.buf = cStringIO.Strin gIO()
self.buf.write( buf)
self.buf.seek(0 )
self.pos = 0
self.closed = 0

def __iter__(self):
return self

def next(self):
if self.closed:
raise StopIteration
r = self.readline()
if not r:
raise StopIteration
return r

def close(self):
"""Free the memory buffer.
"""
if not self.closed:
self.closed = 1
del self.buf, self.pos

def isatty(self):
if self.closed:
raise ValueError, "I/O operation on closed file"
return False

def seek(self, pos, mode = 0):
if self.closed:
raise ValueError, "I/O operation on closed file"
self.buf.seek(p os, mode)
self.pos = self.buf.tell()

def tell(self):
if self.closed:
raise ValueError, "I/O operation on closed file"
return self.pos

def read(self, n = None):
if self.closed:
raise ValueError, "I/O operation on closed file"
if n == None:
r = self.buf.read()
else:
r = self.buf.read(n )
self.pos = self.buf.tell()
return r

def readline(self, length=None):
if self.closed:
raise ValueError, "I/O operation on closed file"
if length is not None:
r = self.buf.readli ne(length)
else:
r = self.buf.readli ne(length)
self.pos = self.buf.tell()
return r

def readlines(self) :
if self.closed:
raise ValueError, "I/O operation on closed file"
lines = self.buf.readli nes()
self.pos = self.buf.tell()
return lines

def truncate(self, size=None):
if self.closed:
raise ValueError, "I/O operation on closed file"
self.buf.trunca te(size)
self.pos = self.buf.tell()
self.buf.seek(0 , 2)
self.len = self.buf.tell()
self.buf.seek(s elf.pos)

def write(self, s):
if self.closed:
raise ValueError, "I/O operation on closed file"
origpos = self.buf.tell()
self.buf.write( s)
self.pos = self.buf.tell()
if origpos + len(s) > self.len:
self.buf.seek(0 , 2)
self.len = self.buf.tell()
self.buf.seek(s elf.pos)

def writelines(self , lines):
if self.closed:
raise ValueError, "I/O operation on closed file"
self.buf.writel ines(lines)
self.pos = self.buf.tell()
self.buf.seek(0 , 2)
self.len = self.buf.tell()
self.buf.seek(s elf.pos)

def flush(self):
if self.closed:
raise ValueError, "I/O operation on closed file"
self.buf.flush( )

def getvalue(self):
if self.closed:
raise ValueError, "I/O operation on closed file"
return self.buf.getval ue()

[Raymond Hettinger]

Re: Why does StringIO discard its initial value?

[David Fraser][color=blue]
> Others may find this helpful ; it's a pure Python wrapper for cStringIO
> that makes it behave like StringIO in not having initialized objects
> readonly. Would it be an idea to extend cStringIO like this in the
> standard library? It shouldn't lose performance if used like a standard
> cStringIO, but it prevents frustration :-)[/color]

IMO, that would be a step backwards. Initializing the object and then
writing to it is not a good practice. The cStringIOAPI needs to be as
file-like as possible. With files, we create an emtpy object and then
starting writing (the append mode for existing files is a different story).
Good code ought to maintain that parallelism so that it is easier to
substitute a real file for a writeable cStringIO object.

This whole thread (except for the documentation issue which has been
fixed) is about fighting the API rather than letting it be a guide to good
code.

If there were something wrong with the API, Guido would have long
since fired up the time machine and changed the timeline so that all
would be as right as rain ;-)


Raymond Hettinger

[David Fraser]

Re: Why does StringIO discard its initial value?

Raymond Hettinger wrote:[color=blue]
> [David Fraser]
>[color=green]
>>Others may find this helpful ; it's a pure Python wrapper for cStringIO
>>that makes it behave like StringIO in not having initialized objects
>>readonly. Would it be an idea to extend cStringIO like this in the
>>standard library? It shouldn't lose performance if used like a standard
>>cStringIO, but it prevents frustration :-)[/color]
>
>
> IMO, that would be a step backwards. Initializing the object and then
> writing to it is not a good practice. The cStringIOAPI needs to be as
> file-like as possible. With files, we create an emtpy object and then
> starting writing (the append mode for existing files is a different story).
> Good code ought to maintain that parallelism so that it is easier to
> substitute a real file for a writeable cStringIO object.
>
> This whole thread (except for the documentation issue which has been
> fixed) is about fighting the API rather than letting it be a guide to good
> code.
>
> If there were something wrong with the API, Guido would have long
> since fired up the time machine and changed the timeline so that all
> would be as right as rain ;-)[/color]

But surely the whole point of files is that you can do more than either
creating a new file or appending to an existing one (seek, write?)

The reason I wrote this was to enable manipulating zip files inside zip
files, in memory. This is on translate.sourc eforge.net - I wanted to
manipulate Mozilla XPI files, and replace file contents etc. within the
XPI. The XPI files are zip format that contains jars inside (also zip
format). I needed to alter the contents of files within the inner zip files.

The zip classes in Python can handle adding files but not replacing
them. The cStringIO is as described above.

So I created extensions to the zipfile.ZipFile class that allow it to
delete existing files, and add them again with new contents (thus
replacing them).

And I created wStringIO so that I could do this all inplace on the
existing zip files.

This all required some extra hacking because of the dual-layer zip files.

But all this as far as I see would have been really tricky using the
existing zipfile and cStringIO classes, which both assume (conceptually)
that files are either readable or new or merely appendable (for zipfile).

The problem for me was not that cStringIO classes are too similar to
files, it was that they are too dissimilar. All of this would work with
either StringIO (but too slow) or real files (but I needed it in memory
because of the zipfiles being inside other zip files).

Am I missing something?

David