scope of socket.setdefaulttimeout?

[Russell Warren]

Does anyone know the scope of the socket.setdefau lttimeout call? Is it
a cross-process/system setting or does it stay local in the application
in which it is called?

I've been testing this and it seems to stay in the application scope,
but the paranoid side of me thinks I may be missing something... any
confirmation would be helpful.

[Steve Holden]

Re: scope of socket.setdefau lttimeout?

Russell Warren wrote:[color=blue]
> Does anyone know the scope of the socket.setdefau lttimeout call? Is it
> a cross-process/system setting or does it stay local in the application
> in which it is called?
>
> I've been testing this and it seems to stay in the application scope,
> but the paranoid side of me thinks I may be missing something... any
> confirmation would be helpful.
>[/color]
Yes, it's an application setting, you aren't changing things for anyone
else.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Russell Warren]

Threads and socket.setdefau lttimeout

It appears that the timeout setting is contained within a process
(thanks for the confirmation), but I've realized that the function
doesn't play friendly with threads. If I have multiple threads using
sockets and one (or more) is using timeouts, one thread affects the
other and you get unpredictable behavior sometimes. I included a short
script at the end of this that demonstrates the threading problem.

I'm trying to get around this by forcing all locations that want to set
a timeout to use a 'safe' call immediately prior to socket creation
that locks out setting the timeout again until the lock is released.
Something like this:

try:
SafeSetSocketTi meout(Timeout_s )
#lock currently acquired to prevent other threads sneaking in here
CreateSocket()
finally:
ReleaseSocketTi meoutSettingLoc k()
UseSocket()

However - this is getting increasingly painful because I don't have
easy access to all of the socket creations where I'd like to do this.
The biggest pain right now is that I'm using xmlrpclib which has some
seriously/frustratingly heavy use of __ prefixes that makes getting
inside to do this at socket creation near impossible (at least I think
so). Right now the best I can do is surround the xmlrpclib calls with
this (effectively putting the lock release after the UseSocket), but
then other threads get hung up for the duration of the call or timeout,
rather than just the simple socket creation.

It would be nice if the timeout were implemented as an argument in the
socket constructor rather than having this global method. Is there a
reason for this? I tried sifting through the cvs source and got lost -
couldn't even find the call definition for socket(family, type, proto)
and gave up...

Does anybody have any idea of another way to do what I need (indpendent
socket timeouts per thread), or have suggestions on how to break into
xmlrpclib (actually down into httplib) to do the methdo I was trying?

Related question: Is there some global way that I'm unaware of to make
it so that some few lines of code are atomic/uninterruptable and no
other thread can sneak in between?

All suggestions appreciated! Hopefully I'm just missing something
obvious.

Russ

#--- This script confirms that settimeout's affect is across threads
import threading, xmlrpclib, socket

def st():
socket.setdefau lttimeout(0.1)

try:
proxy = xmlrpclib.Serve rProxy("http://localhost:10000 ")
print proxy.NonExiste ntCallThatShoul dTimeout()
except Exception, E: print "Exception caught: %s" % (E,)

cbThread = threading.Threa d(target = st)
cbThread.start( )

try:
print proxy.NonExiste ntCallThatShoul dTimeout()
except Exception, E: print "Exception caught: %s" % (E,)

#Output is:
#Exception caught: (10061, 'Connection refused')
#Exception caught: timed out

[Steve Holden]

Re: Threads and socket.setdefau lttimeout

Russell Warren wrote:[color=blue]
> It appears that the timeout setting is contained within a process
> (thanks for the confirmation), but I've realized that the function
> doesn't play friendly with threads. If I have multiple threads using
> sockets and one (or more) is using timeouts, one thread affects the
> other and you get unpredictable behavior sometimes. I included a short
> script at the end of this that demonstrates the threading problem.
>[/color]
When you say "one thread affects another", I see that your example uses
the same function for both threads. IMHO it's much better to override
the thread's run() method than to provide a callable at thread creating
time. That way you can be sure each thread's execution is firmly in the
context of the particular thread instance's namespace.

having said all this, I don't think that's your issue.
[color=blue]
> I'm trying to get around this by forcing all locations that want to set
> a timeout to use a 'safe' call immediately prior to socket creation
> that locks out setting the timeout again until the lock is released.
> Something like this:
>
> try:
> SafeSetSocketTi meout(Timeout_s )
> #lock currently acquired to prevent other threads sneaking in here
> CreateSocket()
> finally:
> ReleaseSocketTi meoutSettingLoc k()
> UseSocket()
>[/color]
This seems extremely contorted, and I'm pretty sure we can find a better
way.
[color=blue]
> However - this is getting increasingly painful because I don't have
> easy access to all of the socket creations where I'd like to do this.
> The biggest pain right now is that I'm using xmlrpclib which has some
> seriously/frustratingly heavy use of __ prefixes that makes getting
> inside to do this at socket creation near impossible (at least I think
> so). Right now the best I can do is surround the xmlrpclib calls with
> this (effectively putting the lock release after the UseSocket), but
> then other threads get hung up for the duration of the call or timeout,
> rather than just the simple socket creation.
>[/color]
The threads' network calls should be yielding process control during
their timeout period to allow other runnable threads to proceed. That's
how it's always worked for me, anyway, and I've written an email sender
with 200 parallel threads.
[color=blue]
> It would be nice if the timeout were implemented as an argument in the
> socket constructor rather than having this global method. Is there a
> reason for this? I tried sifting through the cvs source and got lost -
> couldn't even find the call definition for socket(family, type, proto)
> and gave up...
>[/color]
You are aware, I presume, that you can set a timeout on each socket
individually using its settimeout() method?
[color=blue]
> Does anybody have any idea of another way to do what I need (indpendent
> socket timeouts per thread), or have suggestions on how to break into
> xmlrpclib (actually down into httplib) to do the methdo I was trying?
>[/color]
See above. However, this *does* require you to have access to the
sockets, which is tricky if they are buried deep in some opaque object's
methods.
[color=blue]
> Related question: Is there some global way that I'm unaware of to make
> it so that some few lines of code are atomic/uninterruptable and no
> other thread can sneak in between?
>[/color]
There are locks! I suspect what you need is a threading.Rlock object,
that a thread has to hold to be able to modify the (global) default
timeout. This isn't a full solution to your problem, though, as you have
correctly deduced.

The problem arises when a single method call creates a socket and then
tries to do something with it (like connect to a remote server), which
*is* problematic. I have to say that I haven't ever used different
timeout values for the sockets in different parallel threads, so this is
a new problem to me.
[color=blue]
> All suggestions appreciated! Hopefully I'm just missing something
> obvious.
>
> Russ
>
> #--- This script confirms that settimeout's affect is across threads
> import threading, xmlrpclib, socket
>
> def st():
> socket.setdefau lttimeout(0.1)
>
> try:
> proxy = xmlrpclib.Serve rProxy("http://localhost:10000 ")
> print proxy.NonExiste ntCallThatShoul dTimeout()
> except Exception, E: print "Exception caught: %s" % (E,)
>
> cbThread = threading.Threa d(target = st)
> cbThread.start( )
>
> try:
> print proxy.NonExiste ntCallThatShoul dTimeout()
> except Exception, E: print "Exception caught: %s" % (E,)
>
> #Output is:
> #Exception caught: (10061, 'Connection refused')
> #Exception caught: timed out
>[/color]
Here (unless I'm missing something obvious) it seems that your worker
thread terminates immediately after setting the default timeout, and
both of the proxy calls are made from the main thread, so I'm not
particularly surprised at the results, given the global nature of the
default socket timeout.

Maybe someone else can think of something that will help.

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/

[Russell Warren]

Re: Threads and socket.setdefau lttimeout

Thanks for the detailed repsone... sorry for the lag in responding to
it.

After reading and further thought, the only reason I was using
setdefaulttimeo ut in the first place (rather then using a direct
settimeout on the socket) was because it seemed like the only way (and
easy) of getting access to the seemingly deeply buried socket being
used by xmlrpclib. That was prior to me using threads of course. I
then started trying to make this solution work with thread, but it is
now too convoluted as you say. Now I think the best solution is likely
to redirect my efforts at getting access to the socket used by
xmlrpclib so that I can set it's timeout directly. I'm still unclear
how to do this cleanly, though.

Getting to some of your comments.
[color=blue]
> When you say "one thread affects another", I see that your example uses
> the same function for both threads. IMHO it's much better to override
> the thread's run() method than to provide a callable at thread creating
> time. That way you can be sure each thread's execution is firmly in the
> context of the particular thread instance's namespace.
>
> having said all this, I don't think that's your issue.[/color]

Correct - the bottom code is nothing to do with my code and was only to
quickly prove that it was cross-thread.
[color=blue]
> This seems extremely contorted, and I'm pretty sure we can find a better
> way.[/color]

Couldn't agree more!
[color=blue]
> The threads' network calls should be yielding process control during
> their timeout period to allow other runnable threads to proceed. That's[/color]

Yep. This is not causing me any problem.
[color=blue]
> You are aware, I presume, that you can set a timeout on each socket
> individually using its settimeout() method?[/color]

Yes, but I momentarily had forgot about it... as mentioned I ended up
making the since-bad choice of using setdefaulttimeo ut to get timeouts
set on the inaccessible sockets. Then I carried it too far...
[color=blue]
> See above. However, this *does* require you to have access to the
> sockets, which is tricky if they are buried deep in some opaque object's
> methods.[/color]

Any help on how to crack the safe would be appreciated.
[color=blue]
> There are locks! I suspect what you need is a threading.Rlock object,
> that a thread has to hold to be able to modify the (global) default
> timeout. This isn't a full solution to your problem, though, as you have
> correctly deduced.[/color]

Not quite what I was after I don't think since potentially interfering
code needs to check the lock (via acquire) to avoid conflict. What I
guess I mean is something general for the process saying "never ever
interrupt this block og code by running code on another thread,
regardless of whether the other thread(s) check a lock". Thinking more
about it it seems unreasonable so I'll drop the question.

Russ

[Steve Holden]

Re: Threads and socket.setdefau lttimeout

Russell Warren wrote:[color=blue]
> Thanks for the detailed repsone... sorry for the lag in responding to
> it.[/color]
[discussion of problems with timeouts on threaded code's sockets][color=blue]
>
> Not quite what I was after I don't think since potentially interfering
> code needs to check the lock (via acquire) to avoid conflict. What I
> guess I mean is something general for the process saying "never ever
> interrupt this block og code by running code on another thread,
> regardless of whether the other thread(s) check a lock". Thinking more
> about it it seems unreasonable so I'll drop the question.
>[/color]

Well, I'm about out of ideas, but c.l.py is a very inventive group, so
maybe someone else will be able to contribute a bright thought. Anyone?

regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/