is there a safe marshaler?

[Irmen de Jong]

Pickle and marshal are not safe. They can do harmful
things if fed maliciously constructed data.
That is a pity, because marshal is fast.
I need a fast and safe (secure) marshaler.
Is xdrlib the only option?
I would expect that it is fast and safe because
it (the xdr spec) has been around for so long.

Or are there better options (perhaps 3rd party libraries)?

Thanks

Irmen.

[Pierre Barbier de Reuille]

Re: is there a safe marshaler?

Irmen de Jong a écrit :[color=blue]
> Pickle and marshal are not safe. They can do harmful
> things if fed maliciously constructed data.
> That is a pity, because marshal is fast.
> I need a fast and safe (secure) marshaler.
> Is xdrlib the only option?
> I would expect that it is fast and safe because
> it (the xdr spec) has been around for so long.
>
> Or are there better options (perhaps 3rd party libraries)?
>
> Thanks
>
> Irmen.[/color]

What exactly do you mean by "safe" ? Do you want to ensure your objects
cannot receive corrupted data ? Do you want to ensure no code will be
evaluated during the unmarshalling ?

Please, be more precise,

Pierre

[guido@python.org]

Re: is there a safe marshaler?

Irmen de Jong wrote:[color=blue]
> Pickle and marshal are not safe. They can do harmful
> things if fed maliciously constructed data.
> That is a pity, because marshal is fast.[/color]

I think marshal could be fixed; the only unsafety I'm aware of is that
it doesn't always act rationally when confronted with incorrect input
like bad type codes or truncated input. It only receives instances of
the built-in types and it never executes user code as a result of
unmarshalling.

Perhaps someone would be interested in submitting a patch to the
unmarshalling code? Since this is a security fix we'd even accept a fix
for 2.3.
[color=blue]
> I need a fast and safe (secure) marshaler.
> Is xdrlib the only option?
> I would expect that it is fast and safe because
> it (the xdr spec) has been around for so long.[/color]

I don't expect that to be particularly fast, since it mostly operates
at Python speed. I think it could be safe but I would still do a
thorough code review if I were you -- the code is older than my
awareness of the vulnerabilities inherent in this kind of remote data
transfer.

--Guido

[Irmen de Jong]

Re: is there a safe marshaler?

Pierre Barbier de Reuille wrote:[color=blue]
> Irmen de Jong a écrit :
>[color=green]
>> Pickle and marshal are not safe. They can do harmful
>> things if fed maliciously constructed data.
>> That is a pity, because marshal is fast.
>> I need a fast and safe (secure) marshaler.
>> Is xdrlib the only option?
>> I would expect that it is fast and safe because
>> it (the xdr spec) has been around for so long.
>>
>> Or are there better options (perhaps 3rd party libraries)?
>>
>> Thanks
>>
>> Irmen.[/color]
>
>
> What exactly do you mean by "safe" ? Do you want to ensure your objects
> cannot receive corrupted data ? Do you want to ensure no code will be
> evaluated during the unmarshalling ?[/color]

"safe (secure)"
But to be more precise, let's look at the security warning that
is in the marshal documentation:
"The marshal module is not intended to be secure against erroneous or
maliciously constructed data. Never unmarshal data received from an
untrusted or unauthenticated source."

So essentially I want the opposite of that ;-)

I want a marshalar that is okay to use where the data it processes
comes from unknown, external sources (untrusted). It should not crash
on corrupt data and it should not execute arbitrary code when
unmarshaling, so that it is safe against hacking attempts.

Oh, preferrably, it should be fast :)
Some XML-ish thing may be secure but is likely to be not fast at all.

Ideally it should be able to transfer user defined Python types,
but if it is like marshal (can only marshal builtin types) that's
okay too.

--Irmen

[Irmen de Jong]

Re: is there a safe marshaler?

Hello Guido

guido@python.or g wrote:[color=blue]
> Irmen de Jong wrote:
>[color=green]
>>Pickle and marshal are not safe. They can do harmful
>>things if fed maliciously constructed data.
>>That is a pity, because marshal is fast.[/color]
>
>
> I think marshal could be fixed; the only unsafety I'm aware of is that
> it doesn't always act rationally when confronted with incorrect input
> like bad type codes or truncated input. It only receives instances of
> the built-in types and it never executes user code as a result of
> unmarshalling.[/color]

So it is not vulnerable in the way that pickle is? That's a start.
The security warning in the marsal doc then makes it sound worse than
it is...
[color=blue]
> Perhaps someone would be interested in submitting a patch to the
> unmarshalling code? Since this is a security fix we'd even accept a fix
> for 2.3.[/color]

That would be nice indeed :)

[color=blue][color=green]
>>I need a fast and safe (secure) marshaler.
>>Is xdrlib the only option?
>>I would expect that it is fast and safe because
>>it (the xdr spec) has been around for so long.[/color]
>
>
> I don't expect that to be particularly fast, since it mostly operates
> at Python speed.[/color]

Ah, I wasn't aware that xdrlib was implemented in Python :)
I thought it used a (standard?) C-implementation.
But I now see that it's a Python module (utilizing struct).
[color=blue]
> I think it could be safe but I would still do a
> thorough code review if I were you -- the code is older than my
> awareness of the vulnerabilities inherent in this kind of remote data
> transfer.[/color]

Thanks for the warning.

--Irmen de Jong

[PA]

XDR? (was Re: is there a safe marshaler?)


On Feb 10, 2005, at 15:01, Irmen de Jong wrote:
[color=blue]
> Is xdrlib the only option?
> I would expect that it is fast and safe because
> it (the xdr spec) has been around for so long.[/color]

XDR? Like Sun's "XDR: External Data Representation standard"?

http://www.faqs.org/rfcs/rfc1014.html
http://www.faqs.org/rfcs/rfc1832.html

How does XDR copes with Unicode these days?

Alternatively, perhaps there is a ASN.1 DER library in python?

http://asn1.elibel.tm.fr/en/standards/index.htm

Cheers

--
PA, Onnay Equitursay

http://alt.textdrive.com/

[Alan Kennedy]

Re: is there a safe marshaler?

[Irmen de Jong][color=blue]
> Pickle and marshal are not safe. They can do harmful
> things if fed maliciously constructed data.
> That is a pity, because marshal is fast.
> I need a fast and safe (secure) marshaler.[/color]

Hi Irmen,

I'm not necessarily proposing a solution to your problem, but am
interested in your requirement. Is this for pyro?

In the light of pyro, would something JSON be suitable for your need? I
only came across it a week ago (when someone else posted about it here
on c.l.py), and am intrigued by it.

http://json.org

What I find particularly intriguing is the JSON-RPC protocol, which
looks like a nice lightweight alternative to XML-RPC.

http://oss.metaparadigm.com/jsonrpc/

Also interesting is the browser embeddable JSON-RPC client written in
javascript, for which you can see a demo here

http://oss.metaparadigm.com/jsonrpc/demos.html

I thought you might be interested.

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Alan Kennedy]

Re: is there a safe marshaler?

[Alan Kennedy][color=blue]
> What I find particularly intriguing is the JSON-RPC protocol, which
> looks like a nice lightweight alternative to XML-RPC.
>
> http://oss.metaparadigm.com/jsonrpc/
>
> Also interesting is the browser embeddable JSON-RPC client written in
> javascript, for which you can see a demo here
>
> http://oss.metaparadigm.com/jsonrpc/demos.html[/color]

I should have mentioned as well that there is a python JSON-RPC server
implementation, which incudes a complete JSON<-->python-objects codec.

http://www.json-rpc.org/pyjsonrpc/index.xhtml

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Irmen de Jong]

Re: XDR? (was Re: is there a safe marshaler?)

PA wrote:[color=blue]
>
> On Feb 10, 2005, at 15:01, Irmen de Jong wrote:
>[color=green]
>> Is xdrlib the only option?
>> I would expect that it is fast and safe because
>> it (the xdr spec) has been around for so long.[/color]
>
>
> XDR? Like Sun's "XDR: External Data Representation standard"?
>
> http://www.faqs.org/rfcs/rfc1014.html
> http://www.faqs.org/rfcs/rfc1832.html[/color]

Not "like", but "the".
Or at least, a subset. (the xdrlib module documentation says
"It supports most of the data types described in the RFC").

[color=blue]
> How does XDR copes with Unicode these days?[/color]

Not directly, it seems that you have to encode
your unicode strings yourself first .

[color=blue]
> Alternatively, perhaps there is a ASN.1 DER library in python?
>
> http://asn1.elibel.tm.fr/en/standards/index.htm[/color]


I don't know. Is there?


PS the xdr format is not self-describing in the way that
marshal and pickle streams are. That is a big limitiation
for what I need it for so xdr seems to drop off my radar.
Is an ASN.1 stream self-describing?

--Irmen

[Irmen de Jong]

Re: is there a safe marshaler?

Alan Kennedy wrote:[color=blue]
> [Irmen de Jong]
>[color=green]
>> Pickle and marshal are not safe. They can do harmful
>> things if fed maliciously constructed data.
>> That is a pity, because marshal is fast.
>> I need a fast and safe (secure) marshaler.[/color]
>
>
> Hi Irmen,
>
> I'm not necessarily proposing a solution to your problem, but am
> interested in your requirement. Is this for pyro?[/color]

Yes and No.
Yes, I'm investigating possible marshaling alternatives
(others than pickle which Pyro uses right now).
No, I'm not changing Pyro yet. It's just that I want to
investigate possible *secure* alternatives to the current
implementation.
(Note that a secure version would also mean that Pyro's
advanced features such as mobile code should go the way
of the dodo, and I don't want to do this yet).
[color=blue]
> In the light of pyro, would something JSON be suitable for your need? I
> only came across it a week ago (when someone else posted about it here
> on c.l.py), and am intrigued by it.
>
> http://json.org[/color]

Looks very interesting indeed, but in what way would this be
more secure than say, pickle or marshal?
A quick glance at some docs reveal that they are using eval
to process the data... ouch.

[color=blue]
> I thought you might be interested.[/color]

I certainly am but for different reasons.

--Irmen

[PA]

Re: XDR? (was Re: is there a safe marshaler?)


On Feb 10, 2005, at 22:21, Irmen de Jong wrote:
[color=blue]
> PS the xdr format is not self-describing in the way that
> marshal and pickle streams are. That is a big limitiation
> for what I need it for so xdr seems to drop off my radar.
> Is an ASN.1 stream self-describing?[/color]

Not sure how much "self-describing" you want it to be, but, yes it can
be as formal as you want it to be...

"... Abstract Syntax Notation One (ASN.1) is a formal language for
abstractly describing messages... "

Sorry if this is off-topic, I didn't follow the thread from the very
beginning, but wouldn't something like YAML work for you perhaps?

http://yaml.org/

Or even something more, er, exotic:

https://alt.textdrive.com/pl/

Cheers

--
PA, Onnay Equitursay

http://alt.textdrive.com/

[Irmen de Jong]

Re: XDR? (was Re: is there a safe marshaler?)

PA wrote:[color=blue]
> Sorry if this is off-topic, I didn't follow the thread from the very
> beginning, but wouldn't something like YAML work for you perhaps?
>
> http://yaml.org/[/color]

Perhaps, but the spec makes my skin crawl.
Also, it seems ill-fit for efficient machine-to-machine
communication (yaml seems to be designed to be easily (?) read/edited
by humans, a thing which I don't require at all).
[color=blue]
> https://alt.textdrive.com/pl/[/color]

Naah.

--Irmen

[PA]

Re: XDR? (was Re: is there a safe marshaler?)


On Feb 10, 2005, at 22:55, Irmen de Jong wrote:
[color=blue]
> Also, it seems ill-fit for efficient machine-to-machine
> communication.. .[/color]

Well, then, if you are looking for industrial strength quality, ASN.1
is the way to go. After all, a good chunk of the telecom infrastructure
is using it.

Cheers

--
PA, Onnay Equitursay

http://alt.textdrive.com/

[PA]

Re: XDR? (was Re: is there a safe marshaler?)


On Feb 10, 2005, at 22:55, Irmen de Jong wrote:
[color=blue]
> Perhaps, but the spec makes my skin crawl.[/color]

Perhaps I could interest you in JSON then:

"It is easy for humans to read and write. It is easy for machines to
parse and generate. "

http://www.crockford.com/JSON/index.html

Cheers

--
PA, Onnay Equitursay

http://alt.textdrive.com/

[Alan Kennedy]

Re: is there a safe marshaler?

[Irmen de Jong][color=blue][color=green][color=darkred]
>>> I need a fast and safe (secure) marshaler.[/color][/color][/color]

[Alan Kennedy][color=blue][color=green]
>> ...., would something JSON be suitable for your need?
>>
>> http://json.org[/color][/color]

[Irmen de Jong][color=blue]
> Looks very interesting indeed, but in what way would this be
> more secure than say, pickle or marshal?
> A quick glance at some docs reveal that they are using eval
> to process the data... ouch.[/color]

Well, the python JSON codec provided appears to use eval, which might
make it *seem* unsecure.

http://www.json-rpc.org/pyjsonrpc/index.xhtml

But a more detailed examination of the code indicates, to this reader at
least, that it can be made completely secure very easily. The designer
of the code could very easily have not used eval, and possibly didn't do
so simply because he wasn't thinking in security terms.

The codec uses tokenize.genera te_tokens to split up the JSON string into
tokens to be interpreted as python objects. tokenize.genera te_tokens
generates a series of textual name/value pairs, so nothing insecure
there: the content of the token/strings is not executed.

Each of the tokens is then passed to a "parseValue " function, which is
defined thusly:

#============== =====

def parseValue(self , tkns):
(ttype, tstr, ps, pe, lne) = tkns.next()
if ttype in [token.STRING, token.NUMBER]:
return eval(tstr)
elif ttype == token.NAME:
return self.parseName( tstr)
elif ttype == token.OP:
if tstr == "-":
return - self.parseValue (tkns)
elif tstr == "[":
return self.parseArray (tkns)
elif tstr == "{":
return self.parseObj(t kns)
elif tstr in ["}", "]"]:
return EndOfSeq
elif tstr == ",":
return SeqSep
else:
raise "expected '[' or '{' but found: '%s'" % tstr
else:
return EmptyValue

#============== =====

As you can see, eval is *only* called when the next token in the stream
is either a string or a number, so it's really just a very simple code
shortcut to get a value from a string or number.

If one defined the function like this (not tested!), to remove the eval,
I think it should be safe.

#============== =====

default_number_ type = float
#default_number _type = int

def parseValue(self , tkns):
(ttype, tstr, ps, pe, lne) = tkns.next()
if ttype in [token.STRING]:
return tstr
if ttype in [token.NUMBER]:
return default_number_ type(tstr)
elif ttype == token.NAME:
return self.parseName( tstr)
elif ttype == token.OP:
if tstr == "-":
return - self.parseValue (tkns)
elif tstr == "[":
return self.parseArray (tkns)
elif tstr == "{":
return self.parseObj(t kns)
elif tstr in ["}", "]"]:
return EndOfSeq
elif tstr == ",":
return SeqSep
else:
raise "expected '[' or '{' but found: '%s'" % tstr
else:
return EmptyValue

#============== =====

The only other use of eval is also only for string types, i.e. in the
parseObj function:

#============== =====
def parseObj(self, tkns):
obj = {}
nme =""
try:
while 1:
(ttype, tstr, ps, pe, lne) = tkns.next()
if ttype == token.STRING:
nme = eval(tstr)
(ttype, tstr, ps, pe, lne) = tkns.next()
if tstr == ":":
v = self.parseValue (tkns)
# Remainder of this function elided
#============== =====

Which could similarly be replaced with direct use of the string itself,
rather than eval'ing it. (Although one might want to look at encoding
issues: I haven't looked at JSON-RPC enough to know how it proposes to
handle string encodings.)

So I don't think there any serious security issues here: the
"simplicity " of the JSON grammar is what attracted me to it in the first
place, especially since there are already robust and efficient lexers
and parsers already available built-in to python and javascript (and
javascript interpreters are getting pretty ubiquitous these days).

And it's certainly the case that if the only available python impl of
JSON/RPC is not secure, it is possible to write one that is both
efficient and secure.

Hopefully there isn't some glaring security hole that I've missed:
doubtless I'll find out real soon ;-) Gotta love full disclosure.

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan