How to protect Python source from modification

[Steven D'Aprano]

Re: How to protect Python source from modification

On Mon, 12 Sep 2005 08:33:10 -0700, Frank Millman wrote:
[color=blue]
> My problem is that, if someone has access to the network and to a
> Python interpreter, they can get hold of a copy of my program and use
> it to knock up their own client program that makes a connection to the
> database. They can then execute any arbitrary SQL command.[/color]

Why is that your problem, instead of the company's problem? It is their
database server, yes? If they want to connect to it and execute arbitrary
SQL commands on their own database, (1) who are you to tell them they
can't? and (2) they hardly need your program to do it.

--
Steven

[Frank Millman]

Re: How to protect Python source from modification


Steven D'Aprano wrote:[color=blue]
> On Mon, 12 Sep 2005 08:33:10 -0700, Frank Millman wrote:
>[color=green]
> > My problem is that, if someone has access to the network and to a
> > Python interpreter, they can get hold of a copy of my program and use
> > it to knock up their own client program that makes a connection to the
> > database. They can then execute any arbitrary SQL command.[/color]
>
> Why is that your problem, instead of the company's problem? It is their
> database server, yes? If they want to connect to it and execute arbitrary
> SQL commands on their own database, (1) who are you to tell them they
> can't? and (2) they hardly need your program to do it.
>
> --
> Steven[/color]

If they choose to give the userid and password to an individual, they
are obviously giving him permission to execute any command.

On the other hand, they can reasonably expect to set up users without
giving them direct access to the database, in which case I think they
would be upset if the users found this restriction easy to bypass.

Frank

[Robert Kern]

Re: How to protect Python source from modification

Frank Millman wrote:[color=blue]
> Steven D'Aprano wrote:
>[color=green]
>>On Mon, 12 Sep 2005 08:33:10 -0700, Frank Millman wrote:
>>[color=darkred]
>>>My problem is that, if someone has access to the network and to a
>>>Python interpreter, they can get hold of a copy of my program and use
>>>it to knock up their own client program that makes a connection to the
>>>database. They can then execute any arbitrary SQL command.[/color]
>>
>>Why is that your problem, instead of the company's problem? It is their
>>database server, yes? If they want to connect to it and execute arbitrary
>>SQL commands on their own database, (1) who are you to tell them they
>>can't? and (2) they hardly need your program to do it.
>>
>>--
>>Steven[/color]
>
> If they choose to give the userid and password to an individual, they
> are obviously giving him permission to execute any command.
>
> On the other hand, they can reasonably expect to set up users without
> giving them direct access to the database, in which case I think they
> would be upset if the users found this restriction easy to bypass.[/color]

Certainly, but that access control *shouldn't happen in the client*
whether the source is visible or not.

--
Robert Kern
rkern@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[Bryan Olson]

Re: How to protect Python source from modification

Steve M wrote:
[...][color=blue]
> 1. Based on your description, don't trust the client. Therefore,
> "security", whatever that amounts to, basically has to happen on the
> server.[/color]

That's the right answer. Trying to enforce security within your
software running the client machine does not work. Forget the
advice about shipping .py's or packaging the client in some way.
[color=blue]
> The server should be designed with the expectation that any
> input is possible, from slightly tweaked variants of the normal
> messages to a robotic client that spews the most horrible ill-formed
> junk frequently and in large volumes. It is the server's job to decide
> what it should do. For example, consider a website that has a form for
> users to fill out. The form has javascript, which executes on the
> client, that helps to validate the data by refusing to submit the form
> unless the user has filled in required fields, etc. This is client-side
> validation (analagous to authentication) . It is trivial for an attacker
> to force the form to submit without filling in required fields. Now if
> the server didn't bother to do its own validation but just inserted a
> new record into the database with whatever came in from the form
> submission, on the assumption that the client-side validation was
> sufficient, this would constitute a serious flaw. (If you wonder then
> why bother putting in client-side validation at all - two reasons are
> that it enhances the user experience and that it reduces the average
> load on the server.)[/color]

Good advice.


--
--Bryan

[Frank Millman]

Re: How to protect Python source from modification

Dennis Lee Bieber wrote:[color=blue]
> On 12 Sep 2005 08:33:10 -0700, "Frank Millman" <frank@chagford .com>
> declaimed the following in comp.lang.pytho n:
>[color=green]
> >
> > My problem is that, if someone has access to the network and to a
> > Python interpreter, they can get hold of a copy of my program and use
> > it to knock up their own client program that makes a connection to the
> > database. They can then execute any arbitrary SQL command.
> >[/color]
> If your DBMS is directly accessible on the net, you're vulnerable
> even without Python. Especially if you have "authentication " logic being
> done at the client end. There is nothing to prevent someone using a
> compatible query browser or command-line utility to make connection
> attempts to the server, followed by classical username/password cracking
> stuff.
>[/color]

Right - this is the conclusion I have come to.
[color=blue][color=green]
> >
> > The server program is currently programmed to accept a number of
> > message types from the client program. Each message's data string
> > starts with a numeric prefix, which indicates the type of message,
> > followed by a pickled tuple of arguments. The server program reads the
> > string, extracts the numeric prefix, and passes the rest of the string
> > to the appropriate function using a subthread.
> >[/color]
> Ah, okay -- you /do/ already have something running in the middle.
>[/color]

It is more on the side than in the middle at present - the client
connects to my server program, but also connects directly to the
database. My proposed change is to put it really in the middle - the
client connects to my server, and my server connects to the database.
[color=blue][color=green]
> > For example, I keep track of who is currently logged in. On startup,
> > the client connects to my server and sends a '1' followed by their
> > userid and other information. The server receives this and passed the
> > data to a 'login' function, which uses a Python dictionary to store the
> > information. If the server detects that the user is already logged in,
> > it sends back an error code and the client program displays a message
> > and terminates. Otherwise it sends back an 'ok' code, and the client
> > can continue. When the client logs off, it sends a '2' followed by
> > their userid, which the server receives and passes it to a 'logoff'
> > function, which deletes the entry from the dictionary.
> >[/color]
> Obscuring the Python stuff will only be a minor delay factor in
> breaking that -- someone really serious could probably stick in a packet
> sniffer and record a transaction sequence, eventually reverse mapping
> back to the types of operations each code represents.
>[/color]

Would using SSL be a solution? This is on my to do list.
[color=blue]
> Database security? First step would be to USE the DBMS privilege
> system to limit operations to only those SQL statements, tables, and
> data columns that are needed for your client program; since you appear
> to be using user/password information already, each such user could have
> different privileges, limiting some to retrieval only, for example. As
> for your "server", I'd probably start a thread for each connected user,
> so that thread handles all communication. Your description sounds more
> like a rudimentary proxy adding in a counting scheme, but not really
> isolating separate client connections.
> --[/color]

A number of replies have indicated that I should be using the DBMS
itself to manage security. I think there are some benefits to managing
it via the application. Admittedly I did not think it through when I
started, but now that I have a reasonable security model working, I
would not want to give it up. Here are some examples of what I can do.

1. Users and groups can be maintained by anyone using my app (with the
correct permissions, of course). You do not have to go through the
database adminstrator with all the complication or red tape that could
arise.

2. I am a great believer in 'field-by-field' validation when doing data
entry, instead of filling in the entire form, submitting it, and then
being informed of all the errors. I can inform a user straight away if
they try to do something they are not entitled to.

3. I can cater for the situation where a user may not have permission
to do something, but they can call a supervisor who can override this.
I have seen solutions which involve prompting for a password, but this
has to be programmed in at every place where it might be required. I
allow the supervisor to enter their userid and password, and my program
reads in their permissions, which become the active ones until
cancelled. I create a flashing red border around the window to warn
them not to forget.

There is some support for this approach given by MS SQL Server, which
has the concept of an Application Role. 'Application roles contain no
users, though you can still assign permissions to application roles.
They are designed to let an application take over the job of
authenticating users from SQL Server.'

Thanks for the interesting comments.

Frank

[Frank Millman]

Re: How to protect Python source from modification

Bugs wrote:[color=blue]
> As a side question Frank, how was your experiences using wxPython for
> your GUI?
> Any regrets choosing wxPyton over another toolkit?
> Was it very buggy?
> How was it to work with in general?
> Any other real-world wxPython feedback you have is appreciated.
>
> Frank Millman wrote:[color=green]
> > I am writing a multi-user accounting/business system. Data is stored in
> > a database (PostgreSQL on Linux, SQL Server on Windows). I have written
> > a Python program to run on the client, which uses wxPython as a gui,
> > and connects to the database via TCP/IP.
> >[/color]
> <snip>[/color]

Difficult to give a balanced answer, but I will try.

wxPython more or less chose itelf. I need the gui to work on Windows
and Linux. Alternatives were Tkinter and PyQt. I got the impression
from reading other comments that Tkinter is a bit old-fashioned and
does not have a native look and feel, and PyQt is not free on Windows.
That left wxPython. I did not evaluate the others, so I cannot compare
directly, but do I have any regrets - no.

Some things that I thought would be difficult I found amazingly easy.
Other things that should have been simple gave me endless trouble.
Understanding the concept of 'sizers' (for laying out the widgets on
the screen) took me a while to grasp, and it is still not 100% clear,
but I can get it to do most of what I want.

The cross-platform capability is very good, but there were times when I
got something to work on one platform and not the other. After much
experimenting I usually managed to get it to work on both, often with a
surprising side-effect - the code I eventually used was often cleaner
and felt more correct than my original attempt, and therefore if I had
been more experienced and done it the 'right' way in the first place, I
may not have had a problem.

Documentation is not perfect, though it is being worked on. The primary
source is the documentation for wxWidgets, which is written in C++.
Some people have commented that they do not understand the format, as
the C++ function calls are not quite the same as Python's, but
personally I did not find this a problem. A bigger problem is that the
documentation does not keep up to date with the product, so there are
often new features available that are not apparent. I have got into the
habit of doing a dir() on most of the objects, to see if they have any
methods that are not listed in the docs - quite often they do. Work has
started on proper wxPython documentation, and apparently it looks quite
good, but I have not used it. There is also a book in the pipeline.

Support from the wxPython community is exceptional. There is a very
willing and helpful mailing list, and a wiki with a lot of useful
stuff. The main developer of wxPython, Robin Dunn, is a regular
contributor to the mailing list, and is the authoritative source of up
to date information. Unfortunately he has been tied up with personal
business for a few months, and his absence from the list is quite
noticeable. I am sure I speak for the entire list when I say I am
really hoping that he returns soon - it makes us realise how dependent
we are on him.

Overall, I have found the experience frustrating from time to time, but
I am happy with what I have achieved. I have shown samples of my app to
a few people, and the appearance has never even raised a question - it
just looks and feels like a modern gui application, and I can get on
with demonstrating the functionality, which is as it should be.

My 2.5c

Frank

[Frank Millman]

Re: How to protect Python source from modification

Steve M wrote:[color=blue]
> This is a heck of a can of worms. I've been thinking about these sorts
> of things for awhile now. I can't write out a broad, well-structured
> advice at the moment, but here are some things that come to mind.
>[/color]

[snip lots of interesting stuff]

Thanks for the reply, Steve. My thinking has followed a similar path to
yours, so it is good to have some validation.

I have decided that my next step will be to set up a server program as
discussed, and move all my authentication and business logic there,
then get a client program working using the same gui as at present, but
communicating with the server for most of the database stuff instead of
doing it by itself.

I am aware of Dabo, but I have not looked at it yet. I got my own
framework working before Dabo came along, and it works well enough for
my purposes, so I don't really want to go back and reinvent that wheel
at the moment.

Frank

[Dennis Lee Bieber]

Re: How to protect Python source from modification

On 13 Sep 2005 01:00:37 -0700, "Frank Millman" <frank@chagford .com>
declaimed the following in comp.lang.pytho n:
[color=blue]
> It is more on the side than in the middle at present - the client
> connects to my server program, but also connects directly to the
> database. My proposed change is to put it really in the middle - the
> client connects to my server, and my server connects to the database.
>[/color]
That would be better... The DBMS should never be seen from the
greater net. I don't know how many clients you expect at a time, but I'd
probably use the initial connect request to spin off a connection
specific thread (there's your tracking mechanism -- one thread per "log
in"); you could even build in a time-out mechanism to log-off after
inactivity, something I don't think your simple scheme was rigged to
handle (based on the description).
[color=blue]
>
> Would using SSL be a solution? This is on my to do list.
>[/color]
Outside my realm of experience...
[color=blue]
>
> 1. Users and groups can be maintained by anyone using my app (with the
> correct permissions, of course). You do not have to go through the
> database adminstrator with all the complication or red tape that could
> arise.
>[/color]
I believe it is possible, in MySQL for example, to set up privileges
such that an account can be restricted to an application's database
tables and still be able to add users for just those tables from that
account. At the worst, it should be possible to have a set of
pre-built/parameterized scripts on the server side that can take user
set-up information and create accounts with the proper settings
(validating/ensuring that the accounts only have access to your data and
nothing else). The DBA may have to create the scripts, and then make
them executable from your application's server account, but it would
still be a one-time imposition.
[color=blue]
> 2. I am a great believer in 'field-by-field' validation when doing data
> entry, instead of filling in the entire form, submitting it, and then
> being informed of all the errors. I can inform a user straight away if
> they try to do something they are not entitled to.
>[/color]
Ensuring a field is numeric (with range checking) or string is one
thing... But if a certain user is not even supposed to see a field, or
only have read-only access, those could be determined at the server side
when generating the form (okay, that was phrased more as a web-page
scheme, but...) Better that low-privilege users never even learn of the
additional data fields rather than be tempted by the "forbidden fruit"
<G>
[color=blue]
> 3. I can cater for the situation where a user may not have permission
> to do something, but they can call a supervisor who can override this.
> I have seen solutions which involve prompting for a password, but this
> has to be programmed in at every place where it might be required. I
> allow the supervisor to enter their userid and password, and my program
> reads in their permissions, which become the active ones until
> cancelled. I create a flashing red border around the window to warn
> them not to forget.
>[/color]
I suspect such an override could still be done through the server
side. Not sure of the "programmed in at every place" concern -- it
sounds like just an exception handler with retry (and if it were done
via web forms, it would be the server detecting "no privilege" and
returning the override log-in form). I'd be more concerned that the
override doesn't go away after the transaction is completed... To me, if
a lowly user needs to have a supervisor unlock operations -- and the
supervisor then walks away while the higher privileges are active, it is
a sign of either poor security practices, or a need to grant that user
more privileges.

Imagine a store where supervisor approval is needed for any check
over a certain amount... You don't want the supervisor to key in their
approval code on a register and walk away leaving that code active (and
letting the clerk then enter dozens of high value checks without calling
for a supervisor). The code should only be active for the completion of
the check approval and then automatically reset to the regular clerk's
mode of operation.

--[color=blue]
> =============== =============== =============== =============== == <
> wlfraed@ix.netc om.com | Wulfraed Dennis Lee Bieber KD6MOG <
> wulfraed@dm.net | Bestiaria Support Staff <
> =============== =============== =============== =============== == <
> Home Page: <http://www.dm.net/~wulfraed/> <
> Overflow Page: <http://wlfraed.home.ne tcom.com/> <[/color]

[Bugs]

Re: How to protect Python source from modification

Thanks Frank, I appreciate the feedback.

[Magnus Lycka]

Re: How to protect Python source from modification

Frank Millman wrote:[color=blue]
> Hi all
>
> I am writing a multi-user accounting/business system.[/color]
Good!
[color=blue]
> The client program contains all the authentication and business logic.[/color]
Not good. You lose...if it's *only* in the client.

Of course, there is no such thing as a safe system, and you need
a pragmatic approach to deciding a level of security.

For a system like this, I'd certainly prefer to design the system
so that it's immune to modifications in any software running on a
machine that isn't "safe", such as a typical end user PC. IOW, no
direct DB access allowed (at least not with anything but select
permissions) and all authentication and security related verifications
etc in a server process which isn't available to ordinary users.

However, such schemes aren't immune to flaws. First of all, there
might be mistakes in the implementations (how ever cleverly they
are made) and there are always people who have access to servers
and databases.

You need audits and checks on various levels to handle such things.
You can probably get transaction logs for the database system to
be stored on some other server, which might make it more difficult
for someone who manipulates the system to hide their tracks. All
external transactions (especially payments from the system) need
to be audited and scanned for irregularities. Can all payments be
traced back to valid business transactions? Are there patterns in
payments that stick out, such as many small transactions to the
same receiver? Security checks like these should probably be made
completely outside your accounting/business software, and by people
who have nothing to do with your main system.

Regardless of your intentions, there is no way you can stop people
from hurting themselves. The customer of the system will basically set
a ceiling for the security of the system, depending on how ambitious
they are, and what kind of infrastructure they can provide etc.
Small customers might well demand perfect security, but they probably
don't want to pay for it.
[color=blue]
> It has dawned on me that anyone can bypass this by modifying the
> program. As it is written in Python, with source available, this would
> be quite easy. My target market extends well up into the mid-range, but
> I do not think that any CFO would contemplate using a program that is
> so open to manipulation.[/color]

Just distribute zipped library instead of .py files, and you've raised
the bar to the same level as if you've written it in Java. I.e. it's as
if you send your mail in envelopes (just not glued together) instead of
in postcards. No one will accidentally see the source code, but the
persistent user can.
[color=blue]
> There is the question of where state should be maintained. If on the
> server, I would have to keep all the client/server connections open,
> and maintain the state of all the sessions, which would put quite a
> load on the server.[/color]

Really? How many users do you imagine? How would you plan to organize
your servers? One process per connection, one thread per connection or
asynchronous processing in one thread as in Twisted or asyncore? Perhaps
you should have a look at Twisted ot asyncore?
[color=blue]
> This raises the question of whether I should even bother with a gui
> client, or bite the bullet and only have a browser based front end.
> Judging from recent comments about new technologies such as Ajax, a lot
> of the disadvantages have been overcome, so maybe this is the way to
> go.[/color]

It's still a big difference, isn't it? Have you seen a web based
accounting system that you thought was good enough?
[color=blue]
> It would be a shame to scrap all the effort I have put into my
> wxPython-based front end. On the other hand, it would be pointless to
> continue with an approach that is never going to give me what I want.
> Any advice which helps to clarify my thinking will be much appreciated.[/color]

A wxPython client and a Twisted server might well be a useful combo. If
you have well written Python modules for validating data etc, you might
just run them in both client and server to achieve instant feedback on
the client without lowering the security bar.

A budget solution if you want to keep a simple fat client / thin server
solution is to run the clients on a box that the end users can't
manipulate. Just let them access that box via VNC and only run this
client program there. It's not completely trivial to set up such a safe
environment though, but it basically gives you an application server and
very thin clients that still give you the same rich UI as wxPython on
their desktops. They'll have to get used to a non-Windows GUI though,
unless you choose to run Citrix instead of VNC (or use one application
server per user :).

[Bryan Olson]

Re: How to protect Python source from modification

bruno modulix wrote:[color=blue]
> Frank Millman wrote:[color=green]
>>I am writing a multi-user accounting/business system. Data is stored in
>>a database (PostgreSQL on Linux, SQL Server on Windows). I have written
>>a Python program to run on the client, which uses wxPython as a gui,
>>and connects to the database via TCP/IP.
>>
>>The client program contains all the authentication and business logic.
>>It has dawned on me that anyone can bypass this by modifying the
>>program.[/color]
>
> If your program relies on a RDBMS, then it's the RDBMS job to enforce
> security rules.[/color]

Don't know enough about Millman's app to comment on it
specifically, but many reasonable server-side applications use a
single log-in to the database, then enforce security in the
application server. Web shopping-carts, for example, generally
work that way.


--
--Bryan

[Frank Millman]

Re: How to protect Python source from modification


Dennis Lee Bieber wrote:[color=blue]
> On 13 Sep 2005 01:00:37 -0700, "Frank Millman" <frank@chagford .com>
> declaimed the following in comp.lang.pytho n:
>
>[color=green]
> > 2. I am a great believer in 'field-by-field' validation when doing data
> > entry, instead of filling in the entire form, submitting it, and then
> > being informed of all the errors. I can inform a user straight away if
> > they try to do something they are not entitled to.
> >[/color]
> Ensuring a field is numeric (with range checking) or string is one
> thing... But if a certain user is not even supposed to see a field, or
> only have read-only access, those could be determined at the server side
> when generating the form (okay, that was phrased more as a web-page
> scheme, but...) Better that low-privilege users never even learn of the
> additional data fields rather than be tempted by the "forbidden fruit"
> <G>
>[/color]

Food for thought - thanks
[color=blue][color=green]
> > 3. I can cater for the situation where a user may not have permission
> > to do something, but they can call a supervisor who can override this.
> > I have seen solutions which involve prompting for a password, but this
> > has to be programmed in at every place where it might be required. I
> > allow the supervisor to enter their userid and password, and my program
> > reads in their permissions, which become the active ones until
> > cancelled. I create a flashing red border around the window to warn
> > them not to forget.
> >[/color]
> I suspect such an override could still be done through the server
> side. Not sure of the "programmed in at every place" concern -- it
> sounds like just an exception handler with retry (and if it were done
> via web forms, it would be the server detecting "no privilege" and
> returning the override log-in form). I'd be more concerned that the
> override doesn't go away after the transaction is completed... To me, if
> a lowly user needs to have a supervisor unlock operations -- and the
> supervisor then walks away while the higher privileges are active, it is
> a sign of either poor security practices, or a need to grant that user
> more privileges.
>
> Imagine a store where supervisor approval is needed for any check
> over a certain amount... You don't want the supervisor to key in their
> approval code on a register and walk away leaving that code active (and
> letting the clerk then enter dozens of high value checks without calling
> for a supervisor). The code should only be active for the completion of
> the check approval and then automatically reset to the regular clerk's
> mode of operation.
>[/color]

H'mm, more food for thought.

The advantage of my approach is that no additional programming is
required. I have a very flexible security model, which is entirely
user-definable. If any user finds that they are blocked from doing
something, they can call anyone who does have that permission, without
exiting from their position in the app. The other person can key in
their userid and password, perform the required action, and the
original user can carry on. You are right that the danger is that the
second person forgets to cancel their code. That is why I create a
flashing red border.

This is how it works from a user perspective. There is a hot-key,
Ctrl-U, which can be pressed at any prompt (wxPython makes this easy).
If pressed, I pop up a box asking for userid and password. If accepted,
I save the original user's permissions, read in the new user's
permissions, and create the flashing border. To cancel the setting, the
user simply presses Ctrl-U again, and everything is reset.

I think this works quite well, so I will run with it for now and see if
it causes any problems in practice.

Many thanks for the valuable comments.

Frank

[Frank Millman]

Re: How to protect Python source from modification


Magnus Lycka wrote:

[snip lots of interesting stuff]
[color=blue]
>[color=green]
> > There is the question of where state should be maintained. If on the
> > server, I would have to keep all the client/server connections open,
> > and maintain the state of all the sessions, which would put quite a
> > load on the server.[/color]
>
> Really? How many users do you imagine? How would you plan to organize
> your servers? One process per connection, one thread per connection or
> asynchronous processing in one thread as in Twisted or asyncore? Perhaps
> you should have a look at Twisted ot asyncore?
>[/color]

I was thinking of one thread per connection. How many users? I would be
ecstatic if I got a 20-user system working. However, I know from
experience that these things can mushroom, especially if there are no
licence fees involved, so it can easily be more.

Actually I should just get it working, and then monitor performance as
the number of users increases. I am sure there will be many things I
can do if it starts to slow down.
[color=blue][color=green]
> > This raises the question of whether I should even bother with a gui
> > client, or bite the bullet and only have a browser based front end.
> > Judging from recent comments about new technologies such as Ajax, a lot
> > of the disadvantages have been overcome, so maybe this is the way to
> > go.[/color]
>
> It's still a big difference, isn't it? Have you seen a web based
> accounting system that you thought was good enough?
>[/color]

Actually no <g>
[color=blue][color=green]
> > It would be a shame to scrap all the effort I have put into my
> > wxPython-based front end. On the other hand, it would be pointless to
> > continue with an approach that is never going to give me what I want.
> > Any advice which helps to clarify my thinking will be much appreciated.[/color]
>
> A wxPython client and a Twisted server might well be a useful combo. If
> you have well written Python modules for validating data etc, you might
> just run them in both client and server to achieve instant feedback on
> the client without lowering the security bar.
>[/color]

I have seen Twisted mentioned many times in this ng, but I have no idea
what it actually does. Can someone tell me in simple terms what
advantage it might give me over a multi-threaded socket server program.
I have (just now) reread the intro to the asyncore module, and it says
"this strategy can seem strange and complex, especially at first". This
describes my present situation exactly :-)

Many thanks for the valuable comments.

Frank

[Magnus Lycka]

Re: How to protect Python source from modification

Frank Millman wrote:[color=blue]
> I have seen Twisted mentioned many times in this ng, but I have no idea
> what it actually does. Can someone tell me in simple terms what
> advantage it might give me over a multi-threaded socket server program.[/color]

More control. Less resource usage. Twisted also provides a very
flexible way of building network aware software which you will
appreciate if you ever consider using something else than sockets.

Using several process means using more memory, and inter process
communication isn't as fast as in-process communication. Using
threads is error prone and difficult to debug. There are also scaling
issues with threads in Python (but maybe not when most threads wait
for network connections).

Twisted is based on an event loop, just like GUI programs. But instead
of waiting for GUI events, such as button clicks, the Twisted app will
wait for network events, such as data received. With Twisted and
sockets, you write code that will implement handlers for such events.

A simple example is provided here:

#!/usr/bin/python
# Copyright (c) 2001-2004 Twisted Matrix Laboratories.
# See LICENSE for details.

from twisted.interne t.protocol import Protocol, Factory
from twisted.interne t import reactor

### Protocol Implementation

# This is just about the simplest possible protocol
class Echo(Protocol):
def dataReceived(se lf, data):
"""As soon as any data is received, write it back."""
self.transport. write(data)


def main():
f = Factory()
f.protocol = Echo
reactor.listenT CP(8000, f)
reactor.run()

if __name__ == '__main__':
main()

You see? You just subclass Protocol and override the relevant
event handler, and get the thing going! The factory will create
an Echo instance for each socket connection.

The problem in Twisted is that functions in this single thread that
are run inside the event loop must be fast--just as the event
handlers in your GUI app. Twisted helps you achieve this. For
instance, there is the concept of deferred execution, (described
in a paper available at http://python.fyxm.net/pycon/papers/deferex/ )
but you might want to use a separate thread for things like database
queries etc.

There are a lot of other batteries included, check out the Twisted
documentation at

http://twistedmatrix.com/projects/twisted/documentation/

You have to ask other Twisted users about scalability, but I think
it will scale well beyond your needs.

[Frank Millman]

Re: How to protect Python source from modification


Magnus Lycka wrote:[color=blue]
> Frank Millman wrote:[color=green]
> > I have seen Twisted mentioned many times in this ng, but I have no idea
> > what it actually does. Can someone tell me in simple terms what
> > advantage it might give me over a multi-threaded socket server program.[/color]
>
> More control. Less resource usage. Twisted also provides a very
> flexible way of building network aware software which you will
> appreciate if you ever consider using something else than sockets.
>[/color]

[snip Twisted tutorial]

Thanks a ton, Magnus, that really was a brilliant description.

Clearly I must find the time to look into Twisted properly.

Many thanks again

Frank