about pollsys and dtrace.

[Sophie000]

I have a single-threading implemented program (written by somebody else). In this program, one server listens and accepts connection set up request from multiple clients, puts client information in data base, and provides this information to other processes.

Here is the problem: although the tasks the server are doing are very simple (as I just mentioned), the CPU of the server was quickly used up when the number of clients goes up. My task is to find out if the program has some bugs which leads to the CPU exhaustion; or if some implementation of the program is not efficient.

By using dtrace, I found most of the CPU was consumed by the following system calls: pollsys, recvfrom, sendto. I know the recvfrom, sendto are used to receive data from remote clients and send data to clients, but I am not familiar with pollsys and did not find enough information from internet. It seems like The pollsys syscall is normally used
for checking whether there is any data that can be read from or write to a file descriptor. Then how do I know who call pollsys and in which case the pollsys is called. Is it called periodically or called by signal interrupt, or due to some events?

I am a newbi in unix and its tracing tools. I wonder if anybody can answer my questions and also give me some suggestion like how to narrow down my focus and dig out the reason of the problem.

Thanks.

[gpraghuram]

Its better to use select call instead of pollsys.
Try using select


Raghuram

[Sophie000]

but I saw a select() call with a timeout was included in a while loop.

Do you mean pollsys will not be called if select() mode is used?

Thanks.

[gpraghuram]

but I saw a select() call with a timeout was included in a while loop.

Do you mean pollsys will not be called if select() mode is used?

Thanks.

select calls last parameter is the timeout value.
I dont think so that pollsys will be called inside select.


Raghuram

[Sophie000]

Since I found pollsys consumed most of the CPU, I want to see in which case it is called. To do that, I now used truss -t pollsys -T pollsys -p PID, and pstack PID to see how is the pollsys called. Here is the result:
(I do not understand what lwp# /thread # mean,but it seems pollsys is called by select() and something else I do not understand).
What does it mean? Thanks.

----------------- lwp# 1 / thread# 1 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fffdfb7 f0, 16, fffffd7fffdfbaf 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
000000000046918 d inf_man () + 53d
000000000046dbf 4 main () + 234
000000000044bdc c ???????? ()
----------------- lwp# 2 / thread# 2 --------------------
fffffd7ffe76d85 a sigtimedwait (fffffd7ffe3fdd 10, fffffd7ffe3fdd2 0, fffffd7ffe3fdea 0)
fffffd7fff0e71c 9 sigtimedwait () + 9
fffffd7fff09530 7 signalRun () + 87
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 3 / thread# 3 --------------------
fffffd7ffe76d79 a nanosleep (fffffd7ffe1fdf 90, fffffd7ffe1fdf8 0)
0000000000458de a events_poller () + 1a
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 4 / thread# 4 --------------------
fffffd7ffe76d79 a nanosleep (fffffd7fea1fdf 70, fffffd7fea1fdf6 0)
0000000000470a7 5 brand_data_poll er () + c5
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 5 / thread# 5 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fe9ff9d 10, 1, 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
000000000044ca4 4 resp_service_re q () + 1a4
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 6 / thread# 6 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fe9df7d b0, 1, 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
fffffd7fff2ede7 9 process_snmp_re quest () + a9
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 7 / thread# 7 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fe9bf7d b0, 1, 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
000000000048be9 e process_licensi ng_request () + ae
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 8 / thread# 8 --------------------
fffffd7ffe76e3c a pollsys (66edb0, 24, fffffd7fe99fdf1 0, 0)
fffffd7ffe71482 2 poll () + 52
fffffd7ffeee50b 0 PollerInvokePol l () + 30
fffffd7ffeee4f4 b PollerStep () + 17b
fffffd7ffeee5a3 2 _RunnerStart () + 62
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()

[gpraghuram]

Since I found pollsys consumed most of the CPU, I want to see in which case it is called. To do that, I now used truss -t pollsys -T pollsys -p PID, and pstack PID to see how is the pollsys called. Here is the result:
(I do not understand what lwp# /thread # mean,but it seems pollsys is called by select() and something else I do not understand).
What does it mean? Thanks.

----------------- lwp# 1 / thread# 1 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fffdfb7 f0, 16, fffffd7fffdfbaf 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
000000000046918 d inf_man () + 53d
000000000046dbf 4 main () + 234
000000000044bdc c ???????? ()
----------------- lwp# 2 / thread# 2 --------------------
fffffd7ffe76d85 a sigtimedwait (fffffd7ffe3fdd 10, fffffd7ffe3fdd2 0, fffffd7ffe3fdea 0)
fffffd7fff0e71c 9 sigtimedwait () + 9
fffffd7fff09530 7 signalRun () + 87
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 3 / thread# 3 --------------------
fffffd7ffe76d79 a nanosleep (fffffd7ffe1fdf 90, fffffd7ffe1fdf8 0)
0000000000458de a events_poller () + 1a
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 4 / thread# 4 --------------------
fffffd7ffe76d79 a nanosleep (fffffd7fea1fdf 70, fffffd7fea1fdf6 0)
0000000000470a7 5 brand_data_poll er () + c5
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 5 / thread# 5 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fe9ff9d 10, 1, 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
000000000044ca4 4 resp_service_re q () + 1a4
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 6 / thread# 6 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fe9df7d b0, 1, 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
fffffd7fff2ede7 9 process_snmp_re quest () + a9
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 7 / thread# 7 --------------------
fffffd7ffe76e3c a pollsys (fffffd7fe9bf7d b0, 1, 0, 0)
fffffd7ffe718ee 4 pselect () + 154
fffffd7ffe7191b 2 select () + 72
000000000048be9 e process_licensi ng_request () + ae
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()
----------------- lwp# 8 / thread# 8 --------------------
fffffd7ffe76e3c a pollsys (66edb0, 24, fffffd7fe99fdf1 0, 0)
fffffd7ffe71482 2 poll () + 52
fffffd7ffeee50b 0 PollerInvokePol l () + 30
fffffd7ffeee4f4 b PollerStep () + 17b
fffffd7ffeee5a3 2 _RunnerStart () + 62
fffffd7ffe76b47 b _thr_setup () + 5b
fffffd7ffe76b6b 0 _lwp_start ()

lwp means light-weight-process.
I havent debugged in this way.
Raghuram

[Sophie000]

The result showed that pollsys is called by select() and another user function.
From Beej's tutorial on socket programming, I learned that CPU exhaustion
due to polling can be avoided by using select() mode.
But here I saw that select() also use pollsys.