Bad marshal data

[Michael McGarry]

Hi,

I am using the marshal module in python to save a data structure to a
file. It does not appear to be portable. The data is saved on a Linux
machine. Loading that same data on a Mac gives me the bad marshal data
message.

Is this data really not portable or I am doing something wrong here. I
thought the whole point of using Python and similar cross platform
scripting languages to write once run everywhere, does that not hold
for data too?

Marshal should save the data in a readable text format, but I guess it
does not.

Any help would be appreciated,

Michael

[Paul Rubin]

Re: Bad marshal data

"Michael McGarry" <michael.mcgarr y@gmail.com> writes:[color=blue]
> Marshal should save the data in a readable text format, but I guess it
> does not.
>
> Any help would be appreciated,[/color]

RTFM. Marshal is not intended for what you're doing. Use Pickle,
which is.

[Alex Martelli]

Re: Bad marshal data

Michael McGarry <michael.mcgarr y@gmail.com> wrote:
[color=blue]
> I am using the marshal module in python to save a data structure to a
> file. It does not appear to be portable. The data is saved on a Linux
> machine. Loading that same data on a Mac gives me the bad marshal data
> message.[/color]

If you're using identical versions of Python and exactly the same bits
in both cases, this should not happen. For example, I run on Linux,
with Python 2.4.1:
[color=blue][color=green][color=darkred]
>>> foo=open('foo', 'wb')
>>> marshal.dump({1 :23, 2:[3,4]}, foo)
>>> foo.close()[/color][/color][/color]

Then I scp foo to my Mac, and there, again with Python 2.4.1:
[color=blue][color=green][color=darkred]
>>> import marshal
>>> marshal.load(op en('foo', 'rb'))[/color][/color][/color]
{1: 23, 2: [3, 4]}

It would be OK if either version was, say, 2.4.2 (compatibility is
guaranteed across bugfix-only releases), but not, e.g., 2.3.5 (no
guarantee across minor-releases).

[color=blue]
> Is this data really not portable or I am doing something wrong here. I
> thought the whole point of using Python and similar cross platform
> scripting languages to write once run everywhere, does that not hold
> for data too?[/color]

If you need portability across Python releases, you can use pickle.
marshal is a lower-level module, intended for simple (essentially
elementary) data, to be serialized with min size, max speed.
[color=blue]
> Marshal should save the data in a readable text format, but I guess it
> does not.[/color]

It most definitely should *NOT* use "readable text format", as that
would make it impossible to minimize size and maximize speed. pickle
uses an (unreadable but portable) text format by default, but the binary
format (and even more, the mode 2, most-modern pickle format) are
generally better, unless you need portability to very old Python
releases. (sharing binary data across machines is no big deal anyway;
and you can always b64-encode it if you must send it by email...).


Alex

[Michael McGarry]

Re: Bad marshal data

Pickle is working well for me. I do not need speed or small file size.
Flexibility is more important for me. If speed was important I would
write the app in C.

[Alex Martelli]

Re: Bad marshal data

Michael McGarry <michael.mcgarr y@gmail.com> wrote:
[color=blue]
> Pickle is working well for me. I do not need speed or small file size.
> Flexibility is more important for me. If speed was important I would
> write the app in C.[/color]

Coding an app in C which writes very large text files would probably be
the wrong choice for speed if the app was I/O bound (as is likely when
the files get large enough, or get pushed through some narrow bandwidth
bottleneck, e.g. on a network filesystem): using Python and writing
small binary files instead might easily get better optimization with
less effort.


Alex