possible to use Threadpool

[Daniel]

Hi all

I have written a app that gets emails from a DB and checks if it's a
correct email (connects to the remote mailserver and checks the
response).
I would like to improve the performance with using multiple threads.
can I use a threadpool for this? and how do I have to do this.
For starting i have all emailaddresses in a Array.

thanks for your help.

daniel

[Michael Nemtsev]

Re: possible to use Threadpool

Hello Daniel,

DI have written a app that gets emails from a DB and checks if it's a
Dcorrect email (connects to the remote mailserver and checks the
Dresponse).
DI would like to improve the performance with using multiple threads.
Dcan I use a threadpool for this?

You could, but it can be not the very good solution for the 1.1 fw if u have
a big number of long runned tasks, because the number of TP's threads there
are limited by 25 thread, and in case if you have numbers of long-runned
operations u got the blocked threads which wait till one of 25 threads complete
its work.

In 2.0 u can set the desired number of threads, but take into account that
25 is the general optimal number of threads, and overcoming have negative
effect due to context switching

Dand how do I have to do this.
DFor starting i have all emailaddresses in a Array.

The samples of using ThreadPool class are into MSDN

---
WBR,
Michael Nemtsev :: blog: http://spaces.live.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche

[Daniel]

Re: possible to use Threadpool


Michael Nemtsev schrieb:

Hello Daniel,
>
DI have written a app that gets emails from a DB and checks if it's a
Dcorrect email (connects to the remote mailserver and checks the
Dresponse).
DI would like to improve the performance with using multiple threads.
Dcan I use a threadpool for this?
>
You could, but it can be not the very good solution for the 1.1 fw if u have
a big number of long runned tasks, because the number of TP's threads there
are limited by 25 thread, and in case if you have numbers of long-runned
operations u got the blocked threads which wait till one of 25 threads complete
its work.
>
In 2.0 u can set the desired number of threads, but take into account that
25 is the general optimal number of threads, and overcoming have negative
effect due to context switching
>
Dand how do I have to do this.
DFor starting i have all emailaddresses in a Array.
>
The samples of using ThreadPool class are into MSDN
>
---
WBR,
Michael Nemtsev :: blog: http://spaces.live.com/laflour
>
"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche

Thanks for the answer. I use the 2.0 FW. So it should work fine.
As far as I could read in the docs, the Pool is using max. 25 threads
at the time. Is there a way to show in a label or so which thread is
working on with emailaddress? So I could have a view of the process.
I have to write back the not existing emailaddresses to the database.
when i do it from the thread, it will open the connection write it back
and close the connection. right? but this could produce a lot of work
for the db. is there a way to open the conecction ones and then write
back all addresses and then close the connection? or is this not so
important?

[Michael Nemtsev]

Re: possible to use Threadpool

Hello Daniel,

DThanks for the answer. I use the 2.0 FW. So it should work fine.
DAs far as I could read in the docs, the Pool is using max. 25 threads
Dat the time.

Yep, but it's per process per processor, and your can change it.

DIs there a way to show in a label or so which thread is
Dworking on with emailaddress?

see ThreadPool.GetA vailableThreads method

So I could have a view of the process.
DI have to write back the not existing emailaddresses to the database.
Dwhen i do it from the thread, it will open the connection write it
Dback
Dand close the connection. right? but this could produce a lot of work
Dfor the db. is there a way to open the conecction ones and then write
Dback all addresses and then close the connection? or is this not so
Dimportant?

In MSSQL server your DB connections are pooled by defauilt (in others
DB they are not), so don't mind it

---
WBR,
Michael Nemtsev :: blog: http://spaces.live.com/laflour

"At times one remains faithful to a cause only because its opponents do not
cease to be insipid." (c) Friedrich Nietzsche

[Jon Skeet [C# MVP]]

Re: possible to use Threadpool

Daniel <daniel@danielf .chwrote:

I have written a app that gets emails from a DB and checks if it's a
correct email (connects to the remote mailserver and checks the
response).
I would like to improve the performance with using multiple threads.
can I use a threadpool for this? and how do I have to do this.
For starting i have all emailaddresses in a Array.

I would suggest not using the thread pool for this - you could easily
swamp it, which would make other things which try to use it less
responsive.

I would suggest you use a producer/consumer queue, and then you can
"manually" create as many threads as you want to be consumers of the
queue. This would be a sort of threadpool in itself, *just* for this
purpose. You'd have a lot more control over things that way.

See http://www.pobox.com/~skeet/csharp/t...eadlocks.shtml (half
way down) for some sample producer/consumer code.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Daniel]

Re: possible to use Threadpool

Jon schrieb:

Daniel <daniel@danielf .chwrote:

I have written a app that gets emails from a DB and checks if it's a
correct email (connects to the remote mailserver and checks the
response).
I would like to improve the performance with using multiple threads.
can I use a threadpool for this? and how do I have to do this.
For starting i have all emailaddresses in a Array.

>
I would suggest not using the thread pool for this - you could easily
swamp it, which would make other things which try to use it less
responsive.
>
I would suggest you use a producer/consumer queue, and then you can
"manually" create as many threads as you want to be consumers of the
queue. This would be a sort of threadpool in itself, *just* for this
purpose. You'd have a lot more control over things that way.
>
See http://www.pobox.com/~skeet/csharp/t...eadlocks.shtml (half
way down) for some sample producer/consumer code.
>
--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Thanks for the answer.

I was now just playing around with this. Not sure if I understad it
right.
Here What I have:

using System;
using System.Collecti ons;
using System.Threadin g;

public class Test
{
static CheckEmail queue;
public static int rows = 100;

static void Main()
{
queue = new CheckEmail();
new Thread(new ThreadStart(Ema ilChecker)).Sta rt();
new Thread(new ThreadStart(Ema ilChecker)).Sta rt();
new Thread(new ThreadStart(Ema ilChecker)).Sta rt();
new Thread(new ThreadStart(Ema ilChecker)).Sta rt();
new Thread(new ThreadStart(Ema ilChecker)).Sta rt();

Random rng = new Random(0);
for (int i = 0; i < rows; i++)
{
Console.WriteLi ne("Producing {0}", i);
queue.Produce(" Daniel " + i.ToString());
}

while (queue.Count 0)
{
Console.WriteLi ne("There are {0} emails Waiting",
queue.Count);
Thread.Sleep(20 00);
}
//Here I would like to go ahead when all EMails are processed.
}

static void EmailChecker()
{
// Make sure we get a different random seed from the
// first thread
Random rng = new Random(2);
// We happen to know we've only got 10
// items to receive
for (int i = 0; i < rows; i++)
{
object o = queue.EmailChec k();
//Process Email
Console.WriteLi ne("\t\t\t\tCon suming {0}", o);
Thread.Sleep(rn g.Next(1000));
}
}
}

public class CheckEmail
{
readonly object listLock = new object();
Queue queue = new Queue();
private string Recipient;

public int Count
{
get { return queue.Count; }
}

public void Produce(object o)
{
lock (listLock)
{
queue.Enqueue(o );
Recipient = o.ToString();

// We always need to pulse, even if the queue wasn't
// empty before. Otherwise, if we add several items
// in quick succession, we may only pulse once, waking
// a single thread up, even if there are multiple threads
// waiting for items.
Monitor.Pulse(l istLock);
}
}

public object EmailCheck()
{
lock (listLock)
{
// If the queue is empty, wait for an item to be added
// Note that this is a while loop, as we may be pulsed
// but not wake up before another thread has come in and
// consumed the newly added object. In that case, we'll
// have to wait for another pulse.
while (queue.Count == 0)
{
// This releases listLock, only reacquiring it
// after being woken up by a call to Pulse
Monitor.Wait(li stLock);
}
return queue.Dequeue() ;
}
}
}

So I produce the queue with all emailaddresses (200'000) and then walk
trought with 5 Threads (in this exemple).
Then I wait till the que is empty. How do I know if the last Thread has
finished the work?

So I have to Write the emailaddresses in the Function EmailChecker()
back to the database right?
I also have to write the Logfile in the function EmailChecker().

How can I close the Queue to exit correctly?

I want to run this only if somone requests it by pressing the button.

Thanks for your help

Daniel

[Jon Skeet [C# MVP]]

Re: possible to use Threadpool

Daniel <daniel@danielf .chwrote:

<snip>

So I produce the queue with all emailaddresses (200'000) and then walk
trought with 5 Threads (in this exemple).
Then I wait till the que is empty. How do I know if the last Thread has
finished the work?

One way would be to have a special value (eg null) which means "all
work is complete". At the end of adding all the "real" values to the
queue, add the same number of nulls as there are threads. Then change
the threading code to exit the loop when they see a null. Finally, from
the main loop, call Join on each of the worker threads in turn.

So I have to Write the emailaddresses in the Function EmailChecker()
back to the database right?

Presumably - it's not entirely obvious from your original description
what you need to write back.

I also have to write the Logfile in the function EmailChecker().

I'd suggest having a single thread which writes log entries, to make
sure you don't end up trying to write one log entry in the middle of
another one. This is sort of the reverse of the producer/consumer queue
you've got - there you'd have several producers and one consumer.
Alternatively, you could just synchronise writing out the log entries.

How can I close the Queue to exit correctly?

You could have a separate flag that is visible to all threads which
indicates that they should quit "early" even if there's more work to
do.

I want to run this only if somone requests it by pressing the button.

That's just a matter of responding to an appropriate button click
event.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Daniel]

Re: possible to use Threadpool


Jon schrieb:

Daniel <daniel@danielf .chwrote:
>
<snip>
>

So I produce the queue with all emailaddresses (200'000) and then walk
trought with 5 Threads (in this exemple).
Then I wait till the que is empty. How do I know if the last Thread has
finished the work?

>
One way would be to have a special value (eg null) which means "all
work is complete". At the end of adding all the "real" values to the
queue, add the same number of nulls as there are threads. Then change
the threading code to exit the loop when they see a null. Finally, from
the main loop, call Join on each of the worker threads in turn.
>

So I have to Write the emailaddresses in the Function EmailChecker()
back to the database right?

>
Presumably - it's not entirely obvious from your original description
what you need to write back.
>

I also have to write the Logfile in the function EmailChecker().

>
I'd suggest having a single thread which writes log entries, to make
sure you don't end up trying to write one log entry in the middle of
another one. This is sort of the reverse of the producer/consumer queue
you've got - there you'd have several producers and one consumer.
Alternatively, you could just synchronise writing out the log entries.
>

How can I close the Queue to exit correctly?

>
You could have a separate flag that is visible to all threads which
indicates that they should quit "early" even if there's more work to
do.
>

I want to run this only if somone requests it by pressing the button.

>
That's just a matter of responding to an appropriate button click
event.
>
--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Hi Jon

Sorry it's me again. I don't really understand what you mean.

So I did a class called CheckedEmail:

public class CheckedEmail
{
private bool bolBounce;
private string strEmail;
private string strLog;
public CheckedEmail(bo ol Spam, string Email, string Log)
{
bolBounce = Bounce;
strEmail = Email;
strLog = Log;
}

public bool Bounce
{
set { bolBounce = value; }
get { return bolBounce; }
}

public string Email
{
set { strEmail = value; }
get { return strEmail; }
}

public string Log
{
set { strLog = value; }
get { return strLog; }
}
}



Then I fill the Queue

for (int i = 0; i < 10; i++)
{
objEmail = new CheckedEmail(fa lse, "Daniel " + i, "");
Console.WriteLi ne("Producing {0}", objEmail.Email) ;
queue.Produce(o bjEmail);
}



So in the EmailChecker() i have:

for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailChec k();
Console.WriteLi ne("\t\t\t\tCon suming {0}", o.Email);

//Check Emailaddress
objEmail.Bounce = true;
objEmail.Log = "Connect\r\nHel o OK+\r\nsender not allowed";
Thread.Sleep(rn g.Next(1000));

/*
if (objEmail.Bounc e)
{
Add objEmail.Email to the DB
Add objEmail.Log to a Logfile
}
else
//Do nothing
*/
}

How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?

Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses
I added 5 object's with objEmail.Bounce =true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.

Thanks for your Help and your patience

Daniel

[Daniel]

Re: possible to use Threadpool


Jon schrieb:

Daniel <daniel@danielf .chwrote:
>
<snip>
>

So I produce the queue with all emailaddresses (200'000) and then walk
trought with 5 Threads (in this exemple).
Then I wait till the que is empty. How do I know if the last Thread has
finished the work?

>
One way would be to have a special value (eg null) which means "all
work is complete". At the end of adding all the "real" values to the
queue, add the same number of nulls as there are threads. Then change
the threading code to exit the loop when they see a null. Finally, from
the main loop, call Join on each of the worker threads in turn.
>

So I have to Write the emailaddresses in the Function EmailChecker()
back to the database right?

>
Presumably - it's not entirely obvious from your original description
what you need to write back.
>

I also have to write the Logfile in the function EmailChecker().

>
I'd suggest having a single thread which writes log entries, to make
sure you don't end up trying to write one log entry in the middle of
another one. This is sort of the reverse of the producer/consumer queue
you've got - there you'd have several producers and one consumer.
Alternatively, you could just synchronise writing out the log entries.
>

How can I close the Queue to exit correctly?

>
You could have a separate flag that is visible to all threads which
indicates that they should quit "early" even if there's more work to
do.
>

I want to run this only if somone requests it by pressing the button.

>
That's just a matter of responding to an appropriate button click
event.
>
--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Hi Jon

Sorry it's me again. I don't really understand what you mean.


So I did a class called CheckedEmail:


public class CheckedEmail
{
private bool bolBounce;
private string strEmail;
private string strLog;
public CheckedEmail(bo ol Spam, string Email, string Log)
{
bolBounce = Bounce;
strEmail = Email;
strLog = Log;
}


public bool Bounce
{
set { bolBounce = value; }
get { return bolBounce; }
}


public string Email
{
set { strEmail = value; }
get { return strEmail; }
}


public string Log
{
set { strLog = value; }
get { return strLog; }
}



}


Then I fill the Queue

for (int i = 0; i < 10; i++)
{
objEmail = new CheckedEmail(fa lse, "Daniel " + i, "");
Console.WriteLi ne("Producing {0}", objEmail.Email) ;
queue.Produce(o bjEmail);
}


So in the EmailChecker() i have:


for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailChec k();
Console.WriteLi ne("\t\t\t\tCon suming {0}", o.Email);


//Check Emailaddress
o.Bounce = true;
o.Log = "Connect\r\nHel o OK+\r\nsender not allowed";
Thread.Sleep(rn g.Next(1000));


/*
if (o.Bounce)
{
Add objEmail.Email to the DB
Add objEmail.Log to a Logfile
}
else
//Do nothing
*/
}


How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?


Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses

I added 5 object's with objEmail.Bounce =true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.


Thanks for your Help and your patience


Daniel

[Jon Skeet [C# MVP]]

Re: possible to use Threadpool

Daniel <daniel@danielf .chwrote:

So in the EmailChecker() i have:
>
for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailChec k();

It's not very clear why this is called EmailCheck rather than just
Consume - the queue can be a reusable class which can handle any class,
not just CheckedEmail objects.

How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?

Well, you can have another queue, like the first one, which all the
worker threads add to. You'd have:

Main thread (produces email addresses to check)

/ | \
Thread 1 Thread 2 Thread 3 etc (checking mails)
\ | /

Logging/DB thread (logging and writing to the database)

Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses
>
I added 5 object's with objEmail.Bounce =true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.

I don't think that's the right way of signalling things - it's better
to have a different type of object, or null, or something else to make
it clear that it means you've finished. Either way though, just after
you've consumed the item, you check whether it means "end of queue" and
just return if so.

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

[Daniel]

Re: possible to use Threadpool


Jon schrieb:

Daniel <daniel@danielf .chwrote:

So in the EmailChecker() i have:

for (int i = 0; i < 10; i++)
{
CheckedEmail o = (CheckedEmail) queue.EmailChec k();

>
It's not very clear why this is called EmailCheck rather than just
Consume - the queue can be a reusable class which can handle any class,
not just CheckedEmail objects.
>

How can I put the objEmail in a queue to update the DB and write the
Log when objEmail.Bounce is true?

>
Well, you can have another queue, like the first one, which all the
worker threads add to. You'd have:
>
Main thread (produces email addresses to check)
>
/ | \
Thread 1 Thread 2 Thread 3 etc (checking mails)
\ | /
>
Logging/DB thread (logging and writing to the database)
>

Also with the finish Flag I don't understand what you mean. In witch
loop do I have to put what? After filling the Queue with emailaddresses

I added 5 object's with objEmail.Bounce =true. So I can check if this
value is true. If so I can quit the Thread. But I don't know how and
where.

>
I don't think that's the right way of signalling things - it's better
to have a different type of object, or null, or something else to make
it clear that it means you've finished. Either way though, just after
you've consumed the item, you check whether it means "end of queue" and
just return if so.
>
--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too

Thanks. The null Object works fine.

So I have to create a second queue with:

static ProducerConsume r logqueue;

or do I have to write a second ProducerCustome r class?

In the ConsumerJob() I have to add the following:

while (queue.Count >= 0)
{
object o = queue.Consume() ;
if (o == null)
break;
objEmail = (CheckedEmail)o ;

//Check the Emails

if (objEmail.Bounc e)
logque.Produce( objEmail);
....

How can I start a Thread that looks onely in logque and processes if
there is a entry?
I start the Threads with:
new Thread(new ThreadStart(Con sumerJob)).Star t();
There i can't choose the queue. Also in the ConsumerJob() I can't
choose the queue.

Thanks

[Jon Skeet [C# MVP]]

Re: possible to use Threadpool

Daniel <daniel@danielf .chwrote:

Thanks. The null Object works fine.
>
So I have to create a second queue with:
>
static ProducerConsume r logqueue;
>
or do I have to write a second ProducerCustome r class?

No, you can do it with one class.

In the ConsumerJob() I have to add the following:
>
while (queue.Count >= 0)
{
object o = queue.Consume() ;
if (o == null)
break;
objEmail = (CheckedEmail)o ;
>
//Check the Emails
>
if (objEmail.Bounc e)
logque.Produce( objEmail);
....

You shouldn't have

while (queue.Count >= 0)

It should be

while (true)

because you don't want to quit the threads just because you may consume
the jobs before they've all been produced.

How can I start a Thread that looks onely in logque and processes if
there is a entry?

In a similar way - you call queue.Consume() , which will wait until
there is a job to do.

I start the Threads with:
new Thread(new ThreadStart(Con sumerJob)).Star t();
There i can't choose the queue. Also in the ConsumerJob() I can't
choose the queue.

It sounds like you want to look at:

http://www.pobox.com/~skeet/csharp/threads/parameters.shtml

--
Jon Skeet - <skeet@pobox.co m>
http://www.pobox.com/~skeet Blog: http://www.msmvps.com/jon.skeet
If replying to the group, please do not mail me too