multithreading concept

[Paul Rubin]

Re: multithreading concept

"S.Mohideen " <moin@blackhole .labs.rootshell .wswrites:

Suppose I split my Net application code using parallel python into
several processes based upon the number of CPU available. That means a
single socket descriptor is distributed across all processes. Is
parallelity can be acheived using the processes send/recv on the
single socket multiplexed across all the processes.. I haven't tried
it yet - would like to have any past experience related to this.

In Linux, you can open the socket before forking and then use it in
the child processes; there is also a way to pass open sockets from one
process to another, but the Python lib currently does not support that
feature. It's worth adding and there's an open RFE for it, but it
hasn't been important enough that anyone's bothered coding it so far.

[sturlamolden]

Re: multithreading concept

On Feb 9, 4:00 pm, "S.Mohideen " <m...@blackhole .labs.rootshell .ws>
wrote:

I am sorry if I sound foolish.
Suppose I split my Net application code using parallel python into several
processes based upon the number of CPU available. That means a single socket
descriptor is distributed across all processes. Is parallelity can be
acheived using the processes send/recv on the single socket multiplexed
across all the processes.. I haven't tried it yet - would like to have any
past experience related to this.

Is CPU or network the speed limiting factor in your application? There
are two kinds of problems: You have a 'CPU-bound problem' if you need
to worry about 'flops'. You have an 'I/O bound' problem if you worry
about 'bits per second'.

If your application is I/O bound, adding more CPUs to the task will
not help. The network connection does not become any faster just
because two CPUs share the few computations that need to be performed.
Python releases the GIL around all i/o operations in the standard
library, such as reading from a socket or writing to socket. If this
is what you need to 'parallelize', you can just use threads and ignore
the GIL. Python's threads can handle concurrent I/O perfectly well.
Remember that Google and YouTube use Python, and the GIL is not a show
stopper for them.

The GIL locks the process to one CPU. You need to get around this if
the power of one CPU or CPU core limits the speed of the application.
This can be the case in e.g. digital image processing, certain
computer games, and scientific programming. I have yet to see a CPU-
bound 'Net application', though.

If you are running Windows: take a look at the CPU usage in the task
manager. Does it say that one of the CPUs is running at full speed for
longer periods of time? If not, there is noting to gained from using
multiple CPUs.

[Bjoern Schliessmann]

Re: multithreading concept

Carl J. Van Arsdall wrote:

Not necessarily, if he's on a full duplex ethernet connection,
then there is some parallelity he can take advantage of. He has
upstream and downstream.

Partly agreed. There is one bus to the network device, and CPU
should be very much faster than the network device itself, so I
estimate there'll be no gain.

Regards,


Björn

--
BOFH excuse #353:

Second-system effect.

[Bryan Olson]

Re: multithreading concept

sturlamolden wrote:
[...]

If you want to utilize the computing power of multiple CPUs, you
should use multiple processes instead of threads. On Python this is
mandatory due to the GIL. In any other language it it highly
recommended. The de-factor standard for parallel multiprocessing (MPI)
uses multiple processes, even on SMPs.

That doesn't really work in Python. There have been projects to
allow Pythonic coordination of processes -- POSH had some good
ideas -- but none have reached fruition.

There's nothing like a close thing to a good defacto standard in
the area. Microsoft's Win32 threads can claim to get as close as
anything.


--
--Bryan

[Paul Boddie]

Re: multithreading concept

On 8 Mar, 10:48, Bryan Olson <fakeaddr...@no where.orgwrote:

>
That doesn't really work in Python. There have been projects to
allow Pythonic coordination of processes -- POSH had some good
ideas -- but none have reached fruition.

What makes all of the following not "Pythonic". ..?

http://wiki.python.org/moin/ParallelProcessing

Things like the CSP paradigm have sort of made their way into the
Python language itself, via enhancements to the yield keyword, which
has the dubious distinction of being a keyword which appears to return
a value. I'm sure one could define "Pythonic" as being "you can write
code like you do now (but not like any of the ways encouraged by the
aforementioned solutions) and it just works over multiple processors/
cores", but that's a view which is somewhat detached from the
practicalities (and favoured practices) of concurrent programming,
especially given the few guarantees Python would be able to provide to
make such a thing work effectively.

Paul

[Paul Rubin]

Re: multithreading concept

"Paul Boddie" <paul@boddie.or g.ukwrites:

What makes all of the following not "Pythonic". ..?
http://wiki.python.org/moin/ParallelProcessing

I'd say mainly that they don't allow sharing data between processes
except through expensive IPC mechanisms involving system calls.

I'm sure one could define "Pythonic" as being "you can write
code like you do now (but not like any of the ways encouraged by the
aforementioned solutions) and it just works over multiple processors/
cores", but that's a view which is somewhat detached from the
practicalities (and favoured practices) of concurrent programming,
especially given the few guarantees Python would be able to provide to
make such a thing work effectively.

Really, the existence of the GIL comes as an unpleasant surprise to
progrmamers used to multi-threaded programming in other languages
whose synchronization features outwardly look about the same as
Python's. Somehow those other languages manage to use multiple CPU's
based on those features, without needing a GIL. We are looking at a
Python implementation wart, not "practicalities " inherent in the
nature of concurrency.