Building a new app in python and need some architecture advice.

[Sam Watson]

Hi, I could use a little help with a project i'm going to build. I
know I want to use python and wxWindows but thats about all I know.
The client will be linux or windows. The server will be linux.

Basically, its a client/server app with a rich GUI client.

The big catch is I want to be able to run the client/server remotely
across the web in a secure fashion. I sort of want to stay away from
a web server based solutions because they seem to be attacked the most
and frankly im not dealing with rocket scientists on the other end.
They certainly arent up to the task of keeping a web server locked
down. Honeslty, i'm not sure I even am. I'm a big fan of security
through obscurity.

I need something basically plug and play for them as a server. Im
looking to walk in to their office with a box running linux, turn it
on, plug it into a cat5 cable and never let them touch it. They will
only access it from the GUI client. I will be running SSH for my
admin purposes on the box. I'm comfortable with securing that.

With that in mind, what would you suggest I use as a
remoting/middleware or transport that will be secure across the web
and offers some type of authentication?

My first instinct is to use pickle and sockets but i figure there
ought to be a better way.

This is the list of candidates ive come up with so far
XML-RPC (Seems like kind of a waste considering the client and server
are both python)
Pyro (Can i secure this?)
Twisted (Is this overkill?)
Pure sockets (a lot of infrastructure to write but im pretty
experienced with it)

Should I use postgres as a back end or take a chance on some kind of
Object persistence mechanism? It has to be rock solid.

Lots of questions.... I'm a C++/Windows guys trying to break out of
purgatory :)

Thanks a lot!

[Irmen de Jong]

Re: Building a new app in python and need some architecture advice.

Sam Watson wrote:[color=blue]
> The big catch is I want to be able to run the client/server remotely
> across the web in a secure fashion.[/color]

What do you mean exactly with "across the web"? Over HTTP? Port 80?
"in a secure fashion": SSL? Encrypted? Authentication? Authorization?
So many options ;-)
[color=blue]
> This is the list of candidates ive come up with so far
> XML-RPC (Seems like kind of a waste considering the client and server
> are both python)[/color]

It's slow, too.
[color=blue]
> Pyro (Can i secure this?)[/color]

Yes. Pyro has support for secure custom connection
validators that can be used to authorize clients to connect
(by using an authentication mechanism of your own choice,
for instance login+password based, or with secure hashes,...).
You can also run Pyro over SSL if you want encryption.
(although I must say that it's been a long time since I
did this, so it might or might not work. If I have some
spare time, I'll check the SSL stuff, but I'm too busy
with other things at the moment).
[color=blue]
> Twisted (Is this overkill?)[/color]
Don't know, depends on your type of server application I guess.
[color=blue]
> Pure sockets (a lot of infrastructure to write but im pretty
> experienced with it)[/color]

Why write it when you don't have to (see other options).


Good luck

--Irmen de Jong.

[Sam Watson]

Re: Building a new app in python and need some architecture advice.

>What do you mean exactly with "across the web"? Over HTTP? Port 80?[color=blue]
>"in a secure fashion": SSL? Encrypted? Authentication? Authorization?
>So many options ;-)
>[/color]
I dont know exactly what i mean :) It doesnt matter to me as long as
its secure. I dont really need stateless because this thing aint
gonna scale past 10 users and keeping state solves a bunch o problems.
[color=blue]
>Yes. Pyro has support for secure custom connection
>validators that can be used to authorize clients to connect
>(by using an authentication mechanism of your own choice,
>for instance login+password based, or with secure hashes,...).
>You can also run Pyro over SSL if you want encryption.
>(although I must say that it's been a long time since I
>did this, so it might or might not work. If I have some
>spare time, I'll check the SSL stuff, but I'm too busy
>with other things at the moment).
>[/color]
It sounds like pyro may do the trick here. The thing it would seem I
would be losing with pyro is the ease of iterating and linking maps
and lists of objects. Can I do this transparently with pyro?

I'll investigate further. Thanks for the help!

[color=blue][color=green]
>> Twisted (Is this overkill?)[/color]
>Don't know, depends on your type of server application I guess.
>[/color]
its nothing special. your basic data entry app with some minor
reporting. Simple as it gets.
[color=blue][color=green]
>> Pure sockets (a lot of infrastructure to write but im pretty
>> experienced with it)[/color]
>
>Why write it when you don't have to (see other options).
>[/color]
The only reason I would write it myself would be control, performance,
and ultimate flexibility. It doesnt seem very pythonish tho.

[Irmen de Jong]

Re: Building a new app in python and need some architecture advice.

Sam Watson wrote:
[color=blue][color=green]
>>What do you mean exactly with "across the web"? Over HTTP? Port 80?
>>"in a secure fashion": SSL? Encrypted? Authentication? Authorization?
>>So many options ;-)
>>[/color]
>
> I dont know exactly what i mean :) It doesnt matter to me as long as
> its secure. I dont really need stateless because this thing aint
> gonna scale past 10 users and keeping state solves a bunch o problems.[/color]

Well you have to define what "secure" is for you.
Secure can mean a lot of things.
[color=blue]
> It sounds like pyro may do the trick here. The thing it would seem I
> would be losing with pyro is the ease of iterating and linking maps
> and lists of objects. Can I do this transparently with pyro?[/color]

What makes you think you would lose that with Pyro?
(and not with other IPC protocols)
Pyro's goal is to make the network as transparent as possible
for the application programmer. So yes, you will be able to
access remote objects as lists etc. But it won't be very
fast if you iterate a 20,000 node list over the network.
Better grab the list as a whole over the network and iterate
the list locally.


[color=blue][color=green][color=darkred]
>>>Twisted (Is this overkill?)[/color]
>>
>>Don't know, depends on your type of server application I guess.
>>[/color]
>
> its nothing special. your basic data entry app with some minor
> reporting. Simple as it gets.[/color]

IMHO Twisted is overkill then.
And you would still have to decide on the actual
communication protocol you'd be using...
[color=blue][color=green][color=darkred]
>>>Pure sockets (a lot of infrastructure to write but im pretty
>>>experience d with it)[/color]
>>
>>Why write it when you don't have to (see other options).
>>[/color]
>
> The only reason I would write it myself would be control, performance,
> and ultimate flexibility. It doesnt seem very pythonish tho.[/color]

For the kind of app that you described it seems to me as if
you wouldn't want to spend a big chunk of your development time
on the communication stuff, and about performance: does a data
entry app have to shove 50Mbit/s over the network?


Cheers,
--Irmen.

[Eddie Corns]

Re: Building a new app in python and need some architecture advice.

Sam Watson <Sam watson@yahoo.co m> writes:
[color=blue]
>Hi, I could use a little help with a project i'm going to build. I
>know I want to use python and wxWindows but thats about all I know.
>The client will be linux or windows. The server will be linux.[/color]
[color=blue]
>Basically, its a client/server app with a rich GUI client.[/color]
[color=blue]
>The big catch is I want to be able to run the client/server remotely
>across the web in a secure fashion. I sort of want to stay away from
>a web server based solutions because they seem to be attacked the most
>and frankly im not dealing with rocket scientists on the other end.
>They certainly arent up to the task of keeping a web server locked
>down. Honeslty, i'm not sure I even am. I'm a big fan of security
>through obscurity.[/color]

Since web browsers are so familiar to everyone it's still worth considering
(assuming it can hack your 'rich' GUI client needs). I've written a couple of
data entry web apps now based on BaseHTTPServer, one with state one without.
There is no CGI, no external code getting invoked. If there are other web
apps on the same device you do need to use a port other than 80 (but that fits
your like of obscurity :). Since only your code is invoked then as long as
you don't do something silly like treat user input as a raw filename then it's
pretty damn secure. If you want to secure the app itself from snooping
etc. then you might be able to use SSL on top of that (not sure, it's on my
list of things to find out about). It's simple enough that you can get useful
apps together quite quickly. For evidence of larger apps doing similar
things, I believe roundup, which is a good system, works in the same way (but
is obviously a lot bigger).

I was going to put together a recipe outlining how to create a simple app
using BaseHTTPServer but like all good intentions... Maybe when I get round
to figuring out how to add SSL!

Eddie

[Sam Watson]

Re: Building a new app in python and need some architecture advice.

Thanks guys for the help. I think I will be using pyro for the
project.

[Sam Watson]

Re: Building a new app in python and need some architecture advice.

>Well you have to define what "secure" is for you.[color=blue]
>Secure can mean a lot of things.
>[/color]
By secure I mean hard to break into the server. I dont really care if
the data coming in and out is secure. I want to maintain the server
as little as possible.
[color=blue]
>What makes you think you would lose that with Pyro?
>(and not with other IPC protocols)[/color]
So as I try to access list, dictionary, or tuple items using the []
operator it will transparently fetch them as needed from the server?
This is pretty cool if true. I assumed id have to use methods like
GetNthItem SetNthItem or something like that.
[color=blue]
>on the communication stuff, and about performance: does a data
>entry app have to shove 50Mbit/s over the network?
>[/color]
Nah, at most a few megs i would guess in a session.

[Sam Watson]

Re: Building a new app in python and need some architecture advice.

>Since web browsers are so familiar to everyone it's still worth considering[color=blue]
>(assuming it can hack your 'rich' GUI client needs). I've written a couple of
>data entry web apps now based on BaseHTTPServer, one with state one without.[/color]
I hadnt given the web app idea a whole lot of thought only because my
experience with doing data entry with web pages has been pretty
dissatisfying.
[color=blue]
>pretty damn secure. If you want to secure the app itself from snooping
>etc. then you might be able to use SSL on top of that (not sure, it's on my
>list of things to find out about). It's simple enough that you can get useful
>apps together quite quickly. For evidence of larger apps doing similar
>things, I believe roundup, which is a good system, works in the same way (but
>is obviously a lot bigger).
>[/color]
So apache is pretty easy to lock down? MY fear is i dont configure
apache correctly and leave something wide open.
[color=blue]
>I was going to put together a recipe outlining how to create a simple app
>using BaseHTTPServer but like all good intentions... Maybe when I get round
>to figuring out how to add SSL!
>[/color]
That would be cool.

Thanks eddie

[Eddie Corns]

Re: Building a new app in python and need some architecture advice.

Sam Watson <Sam watson@yahoo.co m> writes:
[color=blue][color=green]
>>Since web browsers are so familiar to everyone it's still worth considering
>>(assuming it can hack your 'rich' GUI client needs). I've written a couple of
>>data entry web apps now based on BaseHTTPServer, one with state one without.[/color]
>I hadnt given the web app idea a whole lot of thought only because my
>experience with doing data entry with web pages has been pretty
>dissatisfyin g.[/color]

Depends on the app of course, a really demanding interface is difficult to
make good even with javascript etc. but simple to just mildly complex can be
done well. Only you can judge that of course.
[color=blue][color=green]
>>pretty damn secure. If you want to secure the app itself from snooping
>>etc. then you might be able to use SSL on top of that (not sure, it's on my
>>list of things to find out about). It's simple enough that you can get useful
>>apps together quite quickly. For evidence of larger apps doing similar
>>things, I believe roundup, which is a good system, works in the same way (but
>>is obviously a lot bigger).
>>[/color]
>So apache is pretty easy to lock down? MY fear is i dont configure
>apache correctly and leave something wide open.[/color]

I should probably have been more explicit, when I say no external software I
mean not even Apache (or any other web server). Applications like Roundup
*can* use Apache but they can also run stand-alone. Basically you have a
single program using BaseHTTPServer, which listens for HTTP requests and calls
the appropriate method (GET, POST etc.) which you have subclassed in your
app. So when the user enters a URL like http://blah/index, you take the path
(after the host identifier) and go "Aha, he wants the index - I'll send him
the HTML for the index page" or eg /form1 which is a HTML page that will
probably contain a POST, when that POST comes in your do_POST method gets
called, pulls out the fields (using http library functions) into a dictionary
and you use that data.

This is all a bit abstract so here's some code for a proof of concept that I
made. When a user called http://my.machine:8000/scan it runs nmap on them
(based on their IP address) and sends the result back. I don't have a short
example with POST yet.

import BaseHTTPServer, os

# utils for making HTML
def wrap (tag, txt):
return '<%s>%s</%s>' % (tag, txt, tag)

def mk_row (*cols):
return wrap ('TR', ''.join ([wrap ('TD',a) for a in cols])) + '\n'

# do the biz
def scan_host (ip):
x = os.popen ('nmap %s' % ip)
res = x.read()
x.close()
return res

class Handler (BaseHTTPServer .BaseHTTPReques tHandler):
def do_GET (self):
p = os.fork()
if p == 0: return
args = self.path.split ('/')[1:]
if args[0] == 'scan':
self.doscan (args)
else:
self.send_error (403, 'Away and raffle')

def doscan (self, *args):
ip = self.client_add ress[0]
dump = self.wfile.writ e

self.send_respo nse(200, "Script output follows")

self.send_heade r ('Content-type', 'text/html')
self.end_header s()

dump ('<Table border>\n')
dump (mk_row ('host','%s' % ip))
dump ('</Table>\n')

dump ('<pre>\n') # pro tem
dump (scan_host (ip))
dump ('</pre>')

httpd = BaseHTTPServer. HTTPServer ( ('',8000), Handler)
httpd.serve_for ever()

This code does a fork because of the length of time nmap takes to complete but
that's optional, I do have an app that just handles each request serially -
makes it easier to remove update conflicts and to retain state because it's
the same code running in a loop.

Hope that isn't too confusing.

Eddie

[Sam Watson]

Re: Building a new app in python and need some architecture advice.

Got ya. Thanks for the example eddie!