XML, JSON, or what?

[Frank Millman]

Hi all

I am writing a multi-user accounting/business application, which uses
sockets to communicate between server and client. The server contains
all the business logic. It has no direct knowledge of the client. I
have devised a simple message format to exchange information between
the two.

At first, I used XML as a message format. Then I read the article that
recommended not using XML for Python-to-Python, so I changed it to a
pickled collection of Python objects. It works just as well.

At present the client uses wxPython. One of my medium-term goals is to
write a web-based client. I don't think it will be easy to reproduce
all the functionality that I have at present, but I hope to get close.
I have not done any serious research yet, but I am pretty sure I will
use javascript on the client, to make it as universal as possible.

Ideally, the server should be able to handle a wxPython client or a web
client, without even knowing which one it is talking to. Obviously I
cannot use Pickle for this. So my question is, what is the ideal format
to use? I could go back to XML, or I could switch to JSON - I have read
a bit about it and it does not seem complicated. My messages are not
very large (maximum about 2k so far for a complex screen layout) so I
don't think performance will be an issue.

I would rather make a decision now, otherwise I will have a lot of
changes to make later on. Does anyone have any recommendations ?

Thanks

Frank Millman

[A.T.Hofkamp]

Re: XML, JSON, or what?

On 2006-06-08, Frank Millman <frank@chagford .com> wrote:[color=blue]
> I would rather make a decision now, otherwise I will have a lot of
> changes to make later on. Does anyone have any recommendations ?[/color]

Did you consider XMPP?
With XMPP you create XML streams between your server and the client.
XMPP is an open standard (by the IETF).

Jabber (jabber.org) is based on the XMPP standard. They develop protocols on
XMPP for all kinds of data exchange. Currently, it is mainly used for chatting,
but we use it in-house as publish/subscribe to control some machines too.
They do a lot of thinking about scalabilty, privacy, authentication, and all
that other stuff you need to do when you put some service on the internet.

They also have protocols for using a GUI interactively (much like the wizards
of Win* do, you get one page of things to decide, you press 'next', you get the
next page, etc).

Albert

[Ant]

Re: XML, JSON, or what?

> to use? I could go back to XML, or I could switch to JSON - I have read

I'd favour JSON if the data structures are simple personally. XML is
comparatively speaking a pain to deal with, where with JSON you can
simply eval() the data and you have a Python dictionary at your
disposal.

I recently used JSON as a way of passing data from a Java backend to a
web page for Javascript to deal with, with the added side effect that
my Python testing scripts could also easily read the same data.

[Alan Kennedy]

Re: XML, JSON, or what?

[Frank Millman][color=blue]
> I am writing a multi-user accounting/business application, which uses
> sockets to communicate between server and client. The server contains
> all the business logic. It has no direct knowledge of the client. I
> have devised a simple message format to exchange information between
> the two.
>
> At first, I used XML as a message format. Then I read the article that
> recommended not using XML for Python-to-Python, so I changed it to a
> pickled collection of Python objects. It works just as well.[/color]

If you were just communicating python to python, I'd recommend Pyro,
since it has all the socket management, etc, already taken care of.

http://pyro.sourceforge.net

[color=blue]
> At present the client uses wxPython. One of my medium-term goals is to
> write a web-based client. I don't think it will be easy to reproduce
> all the functionality that I have at present, but I hope to get close.
> I have not done any serious research yet, but I am pretty sure I will
> use javascript on the client, to make it as universal as possible.[/color]

If you're going to mix javascript client and python server, you
definitely need something cross platform, like XML or JSON.
[color=blue]
> Ideally, the server should be able to handle a wxPython client or a web
> client, without even knowing which one it is talking to. Obviously I
> cannot use Pickle for this. So my question is, what is the ideal format
> to use? I could go back to XML, or I could switch to JSON - I have read
> a bit about it and it does not seem complicated.[/color]

JSON is indeed (mostly) as simple as it looks: it fits your need very
well. And you should be up and running with it very quickly.
[color=blue]
> My messages are not
> very large (maximum about 2k so far for a complex screen layout) so I
> don't think performance will be an issue.[/color]

And parsing JSON will almost certainly be faster than parsing XML. You
should easily be able to parse hundreds or maybe thousands of 2K JSON
messages a second. And JSON generation and parsing is at least as well
supported and robust as XML, in most languages.
[color=blue]
> I would rather make a decision now, otherwise I will have a lot of
> changes to make later on. Does anyone have any recommendations ?[/color]

I'd go with JSON, for simplicity and portability. If you have any
specific questions about it, ask.

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Steve Holden]

Re: XML, JSON, or what?

Ant wrote:[color=blue][color=green]
>>to use? I could go back to XML, or I could switch to JSON - I have read[/color]
>
>
> I'd favour JSON if the data structures are simple personally. XML is
> comparatively speaking a pain to deal with, where with JSON you can
> simply eval() the data and you have a Python dictionary at your
> disposal.
>[/color]
Modulo any security problems that alert and malicious users are able to
inject into your application. Simply using eval() uncritically on
whatever comes down the pipe is a train wreck waiting to happen.
[color=blue]
> I recently used JSON as a way of passing data from a Java backend to a
> web page for Javascript to deal with, with the added side effect that
> my Python testing scripts could also easily read the same data.
>[/color]

Oh, well as ling as we're talking about the *web* that's all right, then :-)

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Love me, love my blog http://holdenweb.blogspot.com
Recent Ramblings http://del.icio.us/steve.holden

[Alan Kennedy]

Re: XML, JSON, or what?

[Ant][color=blue][color=green]
>> I'd favour JSON if the data structures are simple personally. XML is
>> comparatively speaking a pain to deal with, where with JSON you can
>> simply eval() the data and you have a Python dictionary at your
>> disposal.[/color][/color]

[Steve][color=blue]
> Modulo any security problems that alert and malicious users are able to
> inject into your application. Simply using eval() uncritically on
> whatever comes down the pipe is a train wreck waiting to happen.[/color]

Yes, evaling JSON, or any other text coming from the web, is definitely
a bad idea.

But there's no need for eval: there are safe JSON codecs for python,

http://cheeseshop.python.org/pypi?%3Aaction=search&description=json

And one for javascript,

http://www.json.org/js.html

http://www.json.org/json.js

And most other languages you're likely to come across.

http://www.json.org/

regards,

--
alan kennedy
------------------------------------------------------
email alan: http://xhaus.com/contact/alan

[Ant]

Re: XML, JSON, or what?

[color=blue]
> Yes, evaling JSON, or any other text coming from the web, is definitely
> a bad idea.
>
> But there's no need for eval: there are safe JSON codecs for python,[/color]

Fair enough. And I should imagine that the codecs are still much faster
and easier to use than XML for the same purpose.

For my purposes, the JSON is pushed out to the web from our Java
web-app, and eval'd in the test scripts which screen scrape the JSON
structure from the web page - no danger in this case for me. But yes -
I wouldn't be eval'ing random 'JSON' code from the web :-)

[Frank Millman]

Re: XML, JSON, or what?


Frank Millman wrote:[color=blue]
> Hi all
>
> I am writing a multi-user accounting/business application, which uses
> sockets to communicate between server and client. The server contains
> all the business logic. It has no direct knowledge of the client. I
> have devised a simple message format to exchange information between
> the two.
>[/color]
[...][color=blue]
>
> I would rather make a decision now, otherwise I will have a lot of
> changes to make later on. Does anyone have any recommendations ?
>
> Thanks
>
> Frank Millman[/color]

Thanks for the replies, guys.

Albert, thank you for the pointer to XMPP. I had not heard of it, but
it looks very interesting. I will have to investigate further before
deciding whether it is right for my application. If not, JSON it is.

Frank

[Frank Millman]

Re: XML, JSON, or what?


Alan Kennedy wrote:[color=blue]
> [Frank Millman][color=green]
> > I am writing a multi-user accounting/business application, which uses
> > sockets to communicate between server and client. The server contains
> > all the business logic. It has no direct knowledge of the client. I
> > have devised a simple message format to exchange information between
> > the two.
> >[/color]
>
> If you're going to mix javascript client and python server, you
> definitely need something cross platform, like XML or JSON.
>[/color]
[...][color=blue]
>
> I'd go with JSON, for simplicity and portability. If you have any
> specific questions about it, ask.
>[/color]

Thanks for the offer. I have just tried it out, and instantly bumped my
head. I think I know the problem, and I don't think there is a simple
answer, but I will ask here first before starting to figure out a way
around it.

My client-server is Python-to-Python. At present, I am using cPickle to
transfer objects between the two. Among other things, I sometimes
transfer a tuple. Using JSON it appears on the other side as a list. As
I sometimes use the tuple as a dictionary key, this fails, as you
obviously cannot use a list as a key.

I am using 'simplejson'. I see there is also something called json-py,
but I have not tried it yet, as I am assuming (maybe wrongly) that it
will have the same problem. My hunch is that javascript/JSON does not
have the concept of a tuple, and therefore the problem is inherent.

Can someone confirm this, or is there an easy workaround?

Thanks

Frank

[Daniel Dittmar]

Re: XML, JSON, or what?

> My client-server is Python-to-Python. At present, I am using cPickle to[color=blue]
> transfer objects between the two. Among other things, I sometimes
> transfer a tuple. Using JSON it appears on the other side as a list. As
> I sometimes use the tuple as a dictionary key, this fails, as you
> obviously cannot use a list as a key.[/color]
[...][color=blue]
> Can someone confirm this, or is there an easy workaround?[/color]

You can always convert a list to a tuple using the tuple () builtin
right before you use it as a key. But you have to be sure that it is a
list. tuple ("abc") => ('a', 'b', 'c') is probably not what you intend
to do.

Daniel

[Frank Millman]

Re: XML, JSON, or what?


Daniel Dittmar wrote:[color=blue][color=green]
> > My client-server is Python-to-Python. At present, I am using cPickle to
> > transfer objects between the two. Among other things, I sometimes
> > transfer a tuple. Using JSON it appears on the other side as a list. As
> > I sometimes use the tuple as a dictionary key, this fails, as you
> > obviously cannot use a list as a key.[/color]
> [...][color=green]
> > Can someone confirm this, or is there an easy workaround?[/color]
>
> You can always convert a list to a tuple using the tuple () builtin
> right before you use it as a key. But you have to be sure that it is a
> list. tuple ("abc") => ('a', 'b', 'c') is probably not what you intend
> to do.
>
> Daniel[/color]

Thanks,Daniel. After sleeping on it, I had also come to the conclusion
that this is the easiest solution -
if isinstance(key, list):
key = tuple(key)

A bit of a kludge, but I can live with it. I will keep checking to see
if any other anomalies crop up.

Frank