Marshal Obj is String or Binary?

[Mike]

Re: Marshal Obj is String or Binary?

> Even faster would be to write your code in assembly, and dump that[color=blue]
> ridiculously bloated database and just write everything to raw bytes on
> an unformatted disk. Of course, it might take the programmer a thousand
> times longer to actually write the program, and there will probably be
> hundreds of bugs in it, but the important thing is that you'll save three
> or four milliseconds at runtime.[/color]
[color=blue]
> Right?[/color]

Correct. I didn't quite see the issue as assembly vs. python, having
direct translation to programming hours. The structure in mind is meant
to act as a dictionary to extend my db with a few table fields that
could vary from one record to another and won't be queried for.
Considering everytime my record is loaded, it pickle or marshal data
has to be decoded, I figured the faster alternative should be better.
With the incompatibility issue, I figured the day I upgrade my python,
I would write a python script to upgrade the data. I take my word back.
[color=blue]
> Your database either can handle binary data, or it can't.[/color]

It can. It's my web framework that doesn't.
[color=blue]
> If it can, then just use pickle with a binary protocol and be done with it.[/color]

That I will do.
[color=blue]
> Either way, you have to find a way to translate your Python data
> structures into something that you can feed to the database. Your database
> can't automatically suck data structures out of Python's working memory!
> So why re-invent the wheel? marshal is not recommended, but if you can
> live with the limitations of marshal then it might do the job. But trying
> to optimise code that hasn't even been written yet is a sure way to
> trouble.[/color]

Thanks. Will do.

Regards,
Mike

[Mike]

Re: Marshal Obj is String or Binary?

> Even faster would be to write your code in assembly, and dump that[color=blue]
> ridiculously bloated database and just write everything to raw bytes on
> an unformatted disk. Of course, it might take the programmer a thousand
> times longer to actually write the program, and there will probably be
> hundreds of bugs in it, but the important thing is that you'll save three
> or four milliseconds at runtime.[/color]
[color=blue]
> Right?[/color]

Correct. I didn't quite see the issue as assembly vs. python, having
direct translation to programming hours. The structure in mind is meant
to act as a dictionary to extend my db with a few table fields that
could vary from one record to another and won't be queried for.
Considering everytime my record is loaded, it pickle or marshal data
has to be decoded, I figured the faster alternative should be better.
With the incompatibility issue, I figured the day I upgrade my python,
I would write a python script to upgrade the data. I take my word back.
[color=blue]
> Your database either can handle binary data, or it can't.[/color]

It can. It's my web framework that doesn't.
[color=blue]
> If it can, then just use pickle with a binary protocol and be done with it.[/color]

That I will do.
[color=blue]
> Either way, you have to find a way to translate your Python data
> structures into something that you can feed to the database. Your database
> can't automatically suck data structures out of Python's working memory!
> So why re-invent the wheel? marshal is not recommended, but if you can
> live with the limitations of marshal then it might do the job. But trying
> to optimise code that hasn't even been written yet is a sure way to
> trouble.[/color]

Thanks. Will do.

Regards,
Mike

[Mike]

Re: Marshal Obj is String or Binary?

> Well, the Cookbook isn't an exhaustive list of everything you can do[color=blue]
> with Python, it's just a record of some of the things people *have* done.[/color]

Considering I am a newbie, it's a good start for me...
[color=blue]
> I presume your database has no datatype that will store binary data of
> indeterminate length? Clearly that would be the most satisfactory solution.[/color]

PostgreSQL. I think the only two thing it doesn't do is wash my car and
code my software. Well, that's up until you use it in conjunction with
Django, then the only work left is to wash my car, which I can't care
less either. We'll wait for some rain :)

Mike

[Steve Holden]

Re: Marshal Obj is String or Binary?

Mike wrote:[color=blue][color=green]
>>Well, the Cookbook isn't an exhaustive list of everything you can do
>>with Python, it's just a record of some of the things people *have* done.[/color]
>
>
> Considering I am a newbie, it's a good start for me...
>
>[color=green]
>>I presume your database has no datatype that will store binary data of
>>indetermina te length? Clearly that would be the most satisfactory solution.[/color]
>
>
> PostgreSQL. I think the only two thing it doesn't do is wash my car and
> code my software. Well, that's up until you use it in conjunction with
> Django, then the only work left is to wash my car, which I can't care
> less either. We'll wait for some rain :)
>[/color]
So this question was primarily theoretical, right?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Mike]

Re: Marshal Obj is String or Binary?

> So this question was primarily theoretical, right?

Theoretical? not really Steve. I wanted to use django's wonderful db
framework to save a structure into my postgresql. Except there is no
direct BLOB support for it yet. There, I was trying to explore my
options with saving this structure in clear text.

Thanks,
Mike

[Steve Holden]

Re: Marshal Obj is String or Binary?

Mike wrote:[color=blue][color=green]
>>So this question was primarily theoretical, right?[/color]
>
>
> Theoretical? not really Steve. I wanted to use django's wonderful db
> framework to save a structure into my postgresql. Except there is no
> direct BLOB support for it yet. There, I was trying to explore my
> options with saving this structure in clear text.
>[/color]
Right, NOW I understand.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Paul Rubin]

Re: Marshal Obj is String or Binary?

"Mike" <mike12mike12@h otmail.com> writes:[color=blue]
> Correct. I didn't quite see the issue as assembly vs. python, having
> direct translation to programming hours.... I figured the day I
> upgrade my python, I would write a python script to upgrade the
> data. I take my word back.[/color]

Writing that script sounds potentially capable of consuming some
programming hours. If the marshal format changes, it might not be
easy to have the old and new marshal modules in the same Python
instance. You might need two separate interpreters (old and new), to
demarshal your objects in the old interpreter and communicate them
somehow to the new interpreter (e.g. through pickle and sockets) for
re-marshalling. Migrating a database is a big pain in the neck
already without such extra complication.