Threading questions

[Antal Rutz]

Hi, All!

I'm new to threading. I have some design questions:
Task: I collect data and store them in an RDBMS (mysql or pgsql)

The question is how to do that with threading?
The data-collecting piece of the code runs in a thread.

1. Open the db, and each thread writes the result immediately.
(Sub-question: which is better: cursor object passed to the thread
or stored in a global var.)
2. Threads return data, which is written to DB after that.
3. Threads write to global variable, after 'join()' data is written.
(Can be used global (write-only) variables with threads at all?)
4. ?...

I think variable locking isn't an issue here because they are write-only.

Maybe I have fundamentaly misunderstood something...

Thanks

--


--arutz

[F. GEIGER]

Re: Threading questions

Just an idea: You could have n data collector threads, that all put their
results into a queue connected to 1 db thread, that stores the results into
the db.

Cheers
Franz GEIGER


"Antal Rutz" <arutz@mimoza.p antel.net> schrieb im Newsbeitrag
news:mailman.21 3.1117287469.18 027.python-list@python.org ...[color=blue]
> Hi, All!
>
> I'm new to threading. I have some design questions:
> Task: I collect data and store them in an RDBMS (mysql or pgsql)
>
> The question is how to do that with threading?
> The data-collecting piece of the code runs in a thread.
>
> 1. Open the db, and each thread writes the result immediately.
> (Sub-question: which is better: cursor object passed to the thread
> or stored in a global var.)
> 2. Threads return data, which is written to DB after that.
> 3. Threads write to global variable, after 'join()' data is written.
> (Can be used global (write-only) variables with threads at all?)
> 4. ?...
>
> I think variable locking isn't an issue here because they are write-only.
>
> Maybe I have fundamentaly misunderstood something...
>
> Thanks
>
> --
>
>
> --arutz
>[/color]

[wittempj@hotmail.com]

Re: Threading questions

For threading I use usually this recipe as a base:
http://aspn.activestate.com/ASPN/Coo...n/Recipe/65448, I think
you can use it for your purpose as well.

[Magnus Lycka]

Re: Threading questions

Antal Rutz wrote:[color=blue]
> Hi, All!
>
> I'm new to threading. I have some design questions:
> Task: I collect data and store them in an RDBMS (mysql or pgsql)
>
> The question is how to do that with threading?
> The data-collecting piece of the code runs in a thread.
>
> 1. Open the db, and each thread writes the result immediately.
> (Sub-question: which is better: cursor object passed to the thread
> or stored in a global var.)
> 2. Threads return data, which is written to DB after that.
> 3. Threads write to global variable, after 'join()' data is written.
> (Can be used global (write-only) variables with threads at all?)
> 4. ?...[/color]

I'm not sure why you need threading at all here, but I would avoid
both globals and passing cursors between threads. I don't think all
DB-API implementations can handle threading properly.

I suspect it might be useful to do data collection in one thread,
DB communication in another, and to use a Queue.Queue object to
pass data from data collection to DB insertion thread.

[Antal Rutz]

Re: Threading questions

Magnus Lycka wrote:[color=blue]
> Antal Rutz wrote:
>[color=green]
>>Hi, All!
>>
>>I'm new to threading. I have some design questions:
>>Task: I collect data and store them in an RDBMS (mysql or pgsql)
>>
>>The question is how to do that with threading?
>>The data-collecting piece of the code runs in a thread.
>>
>>1. Open the db, and each thread writes the result immediately.
>> (Sub-question: which is better: cursor object passed to the thread
>> or stored in a global var.)
>>2. Threads return data, which is written to DB after that.
>>3. Threads write to global variable, after 'join()' data is written.
>> (Can be used global (write-only) variables with threads at all?)
>>4. ?...[/color]
>
>
> I'm not sure why you need threading at all here, but I would avoid
> both globals and passing cursors between threads. I don't think all
> DB-API implementations can handle threading properly.[/color]

I 'snmpwalk' routers and some are very slow, so it's better to spread
the routers among data-collecting threads.
[color=blue]
> I suspect it might be useful to do data collection in one thread,
> DB communication in another, and to use a Queue.Queue object to
> pass data from data collection to DB insertion thread.[/color]

Yes, I just figure out how the many-collecting-one-dbwriting-thread
design could be made. It doesn't seem to be too difficult.

Thanks for the hints!

--


--arutz