Timer

[Derek Fountain]

Re: Timer

> Wow, that must make your scripts so much more interesting to debug ;)

:o) I suppose it depends how you use it. It's implemented using event
processing, so nothing really happens concurrently, even on SMP boxes. So
it's not as confusing as it could be with a threaded model.

[Dave Harrison]

Re: Timer

> > why not just use the time.sleep() function ?[color=blue]
>
> Because that stops the thread. I want things to continue, and then be
> interrupted in order to execute a bit of code. Tcl has the 'after' command:[/color]

*chuckle* must've missed that start of the thread ;-) never saw
the word threading.

Deadlock-is-a-pain-to-debug-ingly yours
Dave

[Anthony Briggs]

Re: Concurrency models (was: Timer)

At 11:59 AM -0500 27/10/03, Cameron Laird wrote:[color=blue]
>In article <mailman.132.10 67262731.702.py thon-list@python.org >,
>Anthony Briggs <abriggs@westne t.com.au> wrote:
>[color=green][color=darkred]
> >>after 5000 { set disabled 0 }
>>>
>>>The script carries on, and after 5 seconds whatever is being done gets
>>>interrupte d in order to run a bit of script - in this case set a variable
>>>turning off a disablement of some sort.[/color]
>>
> >Wow, that must make your scripts so much more interesting to debug ;)[/color]
>
>You probably think you wrote something uncontroversial .[/color]

No, I wrote something that I consider true. I'd consider that after
command to be on a par with Fortran's(?) COME FROM statement in terms
of being able to create subtle bugs.
[color=blue]
>I take this question rather seriously, as it turns out.
>
>Scripts that are "easy to debug" make for an apt focus.
>This thread is really about asynchronous or concurrent
>processing, I claim, and I further claim we really don't
>have *any* satisfying model for coding those. Threading
>is a quagmire (everyone agree?), and our guild has demon-
>strated only marginally more reliability in working with,
>say, co-routines, chords, continuations, and so on. For
>my money, the event-oriented model behind the [after]
>above is at least as robust as any other.[/color]

Yes, that's what I said - only with fewer words :)

Anthony
--
----------------------------------------------------
HyPEraCtiVE? HeY, WhO aRE YoU cALliNg HypERaCtIve?!
aBRiGgS@wEStNeT .cOm.aU
----------------------------------------------------

[Alex Martelli]

Re: Concurrency models (was: Timer)

Anthony Briggs wrote:
...[color=blue]
> No, I wrote something that I consider true. I'd consider that after
> command to be on a par with Fortran's(?) COME FROM statement in terms[/color]

Intercal. Lawrence Clark's 1973 article about "comefrom" in Fortran
was satire (just as all of Intercal is).
[color=blue]
> of being able to create subtle bugs.[/color]

Note that, in Python, you have that 'after' available any time
you're running a Tkinter GUI: any Tkinter widget has an 'after'
method that takes a delay (in milliseconds), a callable object,
and optionally some arguments, and schedules the callable to
be called with those arguments after that delay.

It works a charm, btw.

[color=blue][color=green]
>>my money, the event-oriented model behind the [after]
>>above is at least as robust as any other.[/color]
>
> Yes, that's what I said - only with fewer words :)[/color]

I thought you were arguing AGAINST the 'after' functionality,
and therefore against event-driven programming...? !

Surely, if I do have a program that's architected around
responding to asynchronous events (as GUIs invariably are,
as network programming can be with asyncore or Twisted),
the ability to explicitly schedule an event (thus a callback)
for some time in the future is no problem - just handy!


Alex

[Alex Martelli]

Re: Concurrency models (was: Timer)

Alan Kennedy wrote:
...[color=blue]
> And the patchy support for SSL in asynch frameworks is another
> fundamental problem.[/color]

Hmmm, what's wrong with Twisted+pyOpenS SL...? Seems to work fine...
[color=blue]
> Another point in favour of threads: On a multi-processor box, the
> thread abstraction generally "automagica lly" gives you free migration
> to other processors. You (generally) don't have to change a line of
> your code: the underlying [OS|VM] takes care of it.[/color]

*blink*? Uh, doesn't that depend on how you choose to have your
threads communicate with each other? The most Pythonic solution
is generally to have threads communicate via Queue instances. Can
you point me to a "cross-process Queue" class for Linux and Windows?

Not a rhetoric question -- I'd LIKE to be able to scale things up that
way, but see some problems with translating typical Queue use idioms,
particularly in a cross-platform way, to cross-process:
-- N1 client-threads posting workrequests to Q and N2 worker-threads
peeling workrequests off Q and working on them;
-- a workrequest including a reference to the Queue instance on
which the worker-thread is going to post the response/result


Alex

[Anthony Briggs]

Re: Concurrency models (was: Timer)

At 11:14 AM +0000 28/10/03, Alex Martelli wrote:[color=blue]
>Anthony Briggs wrote:
> ...[color=green]
>> No, I wrote something that I consider true. I'd consider that after
>> command to be on a par with Fortran's(?) COME FROM statement in terms[/color]
>
>Intercal. Lawrence Clark's 1973 article about "comefrom" in Fortran
>was satire (just as all of Intercal is).[/color]

Ah. I just read the article at
<http://www.fortran.com/fortran/come_from.html> . Most enlightening ;)
[color=blue]
>Note that, in Python, you have that 'after' available any time
>you're running a Tkinter GUI: any Tkinter widget has an 'after'
>method that takes a delay (in milliseconds), a callable object,
>and optionally some arguments, and schedules the callable to
>be called with those arguments after that delay.
>
>It works a charm, btw.[/color]

I'll take your word for it. It sounds like a recipe for a lot of hair
pulling to me. What happens if there's some part of your code that
you don't want interrupted? eg. database commits, file access, any
sort of time critical stuff, that sort of thing. Is there a way to
switch it off?
[color=blue][color=green][color=darkred]
> >>my money, the event-oriented model behind the [after]
>>>above is at least as robust as any other.[/color]
>>
>> Yes, that's what I said - only with fewer words :)[/color]
>
>I thought you were arguing AGAINST the 'after' functionality,
>and therefore against event-driven programming...? ![/color]

Perhaps I was taking his words a little too literally. As far as I
can tell with threads, etc, they're little better than voodoo. So the
after statement is at least as robust, even if it's a complete
horror...

Anthony
--
----------------------------------------------------
HyPEraCtiVE? HeY, WhO aRE YoU cALliNg HypERaCtIve?!
aBRiGgS@wEStNeT .cOm.aU
----------------------------------------------------

[Cameron Laird]

Re: Concurrency models (was: Timer)

In article <ycsnb.57562$e5 .2124785@news1. tin.it>,
Alex Martelli <aleax@aleax.it > wrote:

[Alan Kennedy]

Re: Concurrency models (was: Timer)

[Alan Kennedy][color=blue][color=green]
>> And the patchy support for SSL in asynch frameworks is another
>> fundamental problem.[/color][/color]

[Alex Martelli][color=blue]
> Hmmm, what's wrong with Twisted+pyOpenS SL...? Seems to work fine...[/color]

I haven't used that particular combination.

I was going to followup my own post to mention that my statement was
not in any way meant to be slight to any of the providers of python
SSL. For example, Ng Pheng Siong has put a lot of work into getting
M2Crypto SSL working in non-blocking mode.

My understanding is that using SSL in non-blocking mode is problematic
because SSL-level packets can trigger readiness notification on an SSL
socket when in fact there is no application-level data available.

Here is a link that discusses the problem

http://www.openssl.org/support/faq.html#PROG9

I honestly don't know if all of the asynch models implemented in
python address this problem, and would be happy to be enlightened
and/or proven completely wrong if they have.

Java nio has this problem, which will be addressed in java 1.5

http://www.itworld.com/nl/java_sec/09142001/

[Alan Kennedy][color=blue][color=green]
>> Another point in favour of threads: On a multi-processor box, the
>> thread abstraction generally "automagica lly" gives you free migration
>> to other processors. You (generally) don't have to change a line of
>> your code: the underlying [OS|VM] takes care of it.[/color][/color]

[Alex Martelli][color=blue]
> *blink*? Uh, doesn't that depend on how you choose to have your
> threads communicate with each other?[/color]

There is assumption behind my statement: that the underlying [OS|VM]
also maps the memory space of threads running on different processors
so that they coincide, and all objects are shared between threads as
if they were local to the "process" containing the threads.

If I use a Queue.Queue in jython, (AFAICT) the different threads can
be running on different processors, and the Queue.Queue will work just
fine as a synch mechanism.
[color=blue]
> The most Pythonic solution
> is generally to have threads communicate via Queue instances. Can
> you point me to a "cross-process Queue" class for Linux and Windows?[/color]

I'd love to. However, python's Queue.Queue class doesn't work across
processes, only across threads. But those threads could still
potentially be running on multiple processors in the same box (see
above re jython).

It'd be nice if we could have a Queue.Queue style IPC mechanism that
worked between processes, and that could be included in the set of
"selectable s" for an event-driven framework. Without copious
serialization/pickling. I have stated that several times in this
forum.
[color=blue]
> Not a rhetoric question -- I'd LIKE to be able to scale things up that
> way, but see some problems with translating typical Queue use idioms,
> particularly in a cross-platform way, to cross-process:
> -- N1 client-threads posting workrequests to Q and N2 worker-threads
> peeling workrequests off Q and working on them;
> -- a workrequest including a reference to the Queue instance on
> which the worker-thread is going to post the response/result[/color]

I was interested to read about TupleSpaces from a recent post. Though
I haven't looked into TupleSpaces in detail, I can imagine that
there's lots of de/serialization going on.

Lastly, another area which seems to have patchy support across
operating systems is that of non-blocking file-locks, which would be
required for many event-driven apps.

regards,

--
alan kennedy
-----------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/mailto/alan

[Cameron Laird]

Re: Concurrency models (was: Timer)

In article <mailman.168.10 67342457.702.py thon-list@python.org >,
Anthony Briggs <abriggs@westne t.com.au> wrote:

[Alex Martelli]

Re: Concurrency models (was: Timer)

Anthony Briggs wrote:
...[color=blue]
> I'll take your word for it. It sounds like a recipe for a lot of hair
> pulling to me. What happens if there's some part of your code that
> you don't want interrupted? eg. database commits, file access, any
> sort of time critical stuff, that sort of thing. Is there a way to
> switch it off?[/color]

Event-driven processing, AKA asynchronous processing, is not
pre-emptive: on the contrary, it relies on each event handler
(callback) terminating reasonably promptly and thus returning
to the main loop. Hmmm -- ever programmed a GUI...?

[color=blue][color=green]
>>I thought you were arguing AGAINST the 'after' functionality,
>>and therefore against event-driven programming...? ![/color]
>
> Perhaps I was taking his words a little too literally. As far as I
> can tell with threads, etc, they're little better than voodoo. So the[/color]

Threads are best kept in their place by keeping them well separated
with Queue instances, although I wouldn't be surprised to hear that
the blood of a black roster strangled at midnight is also effective.
[color=blue]
> after statement is at least as robust, even if it's a complete
> horror...[/color]

Nope -- the callback is done in the same thread that requested
it, so "thread separation" is not at issue.


Alex

[Alex Martelli]

Re: Concurrency models (was: Timer)

Cameron Laird wrote:
[color=blue]
> In article <ycsnb.57562$e5 .2124785@news1. tin.it>,
> Alex Martelli <aleax@aleax.it > wrote:
> .
> .
> .[color=green]
>>Note that, in Python, you have that 'after' available any time
>>you're running a Tkinter GUI: any Tkinter widget has an 'after'
>>method that takes a delay (in milliseconds), a callable object,
>>and optionally some arguments, and schedules the callable to
>>be called with those arguments after that delay.
>>
>>It works a charm, btw.[/color]
> .
> .
> .
> I want to refine Alex' description: it's possible to
> use after with only the slenderest connection to a
> Tkinter GUI. You can "import Tkinter", and exploit
> after(), without ever having to make Tkinter widgets
> or windows visible on the screen.[/color]

and I'll further refine yours: "as long as you run
Tkinter's mainloop". Yes, you can hide Tkinter's root
window and still run its mainloop (I think you have
to be explicit about it though:-), as long as you don't
need any of the normal processing flow used by most
non-GUI applications (including other event-driven apps
that don't use Tkinter's mainloop -- e.g., network ones
that use asyncore or select). And the .after callback
will be able to get into play only if other callbacks
(presumably also from .after, if you have no GUI) return
reasonably promptly.

In short, I doubt using Tkinter just for the sake of
getting .after makes much sense -- you're probably at
least as well off with the much lighter-weight scheduler
module, in that case. (OTOH, if it IS possible to
give fileobjects to Tkinter to cause callbacks when
they're ready, as it is in Tcl/Tk, then using Tkinter
w/o GUI may indeed be easier than merging e.g scheduler
and select -- it depends... but last time I checked I
saw no way to get the "callback on file activity" from
Tkinter as I could back when I used Tcl/Tk...).


Alex

[Michael Hudson]

Re: Concurrency models (was: Timer)

Alex Martelli <aleax@aleax.it > writes:
[color=blue]
> In short, I doubt using Tkinter just for the sake of
> getting .after makes much sense -- you're probably at
> least as well off with the much lighter-weight scheduler
> module, in that case. (OTOH, if it IS possible to
> give fileobjects to Tkinter to cause callbacks when
> they're ready, as it is in Tcl/Tk, then using Tkinter
> w/o GUI may indeed be easier than merging e.g scheduler
> and select -- it depends... but last time I checked I
> saw no way to get the "callback on file activity" from
> Tkinter as I could back when I used Tcl/Tk...).[/color]

Huh? _tkinter.create filehandler() is there (pyrepl uses it). I think
only on Unix, but I think that also applies to Tcl_CreateFileH andler,
so that's not much of a surprise.

Cheers,
mwh

--
First time I've gotten a programming job that required a drug
test. I was worried they were going to say "you don't have
enough LSD in your system to do Unix programming". -- Paul Tomblin
-- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html

[Itamar Shtull-Trauring]

Re: Concurrency models (was: Timer)

Alan Kennedy <alanmk@hotmail .com> wrote in message news:<3F9E74DB. 3148DFD1@hotmai l.com>...
[color=blue]
> My understanding is that using SSL in non-blocking mode is problematic
> because SSL-level packets can trigger readiness notification on an SSL
> socket when in fact there is no application-level data available.
>
> Here is a link that discusses the problem
>
> http://www.openssl.org/support/faq.html#PROG9[/color]

Twisted does the right thing and makes sure the user doesn't have to
worry about this. Using SSL is just as easy as using TCP in Twisted.

There are other issues with event loops, like the fact the RSA
handshake is very expensive so you want to run it in another thread.
If Twisted ever has performance issues due to this I'll probably
change the code to do just that, move the handshake stage to a
threadpool.

[Alan Kennedy]

Re: Concurrency models (was: Timer)

[Alan Kennedy][color=blue][color=green]
>> My understanding is that using SSL in non-blocking mode is problematic
>> because SSL-level packets can trigger readiness notification on an SSL
>> socket when in fact there is no application-level data available.[/color][/color]

[Itamar Shtull-Trauring][color=blue]
> Twisted does the right thing and makes sure the user doesn't have to
> worry about this. Using SSL is just as easy as using TCP in Twisted.
>
> There are other issues with event loops, like the fact the RSA
> handshake is very expensive so you want to run it in another thread.
> If Twisted ever has performance issues due to this I'll probably
> change the code to do just that, move the handshake stage to a
> threadpool.[/color]

Thanks for that Itamar.

IMHO, another good solution to the SSL/Non-Blocking IO problem might
be to run stunnel (www.stunnel.org) to do the SSL decode, and then
forward the decrypted requests to an non-SSL asynch server. Although I
have never tried this.

Stunnel can use OpenSSL, which means that support for hardware crypto
accelerators comes for free, etc, etc.

regards,

--
alan kennedy
-----------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/mailto/alan

[Alex Martelli]

Re: Concurrency models (was: Timer)

Michael Hudson wrote:
...[color=blue][color=green]
>> and select -- it depends... but last time I checked I
>> saw no way to get the "callback on file activity" from
>> Tkinter as I could back when I used Tcl/Tk...).[/color]
>
> Huh? _tkinter.create filehandler() is there (pyrepl uses it). I think
> only on Unix, but I think that also applies to Tcl_CreateFileH andler,
> so that's not much of a surprise.[/color]

I could have sworn I got "callbacks on file activity" cross-platform
when I used Tcl/Tk, but maybe it's just a memory grown rosy with age.


Alex