Out of private memory heap

[kumarmdb2]

Hi guys,

For last few days we are getting out of private memory error. We have a
development environment. We tried to figure out the problem but we
believe that it might be related to the OS (I am new to Windows so not
sure). We are currently bouncing the instance to overcome this error. This
generally happen at the end of business day only (So maybe memory
might be getting used up?). We have already increased the statement heap &
Application heap size. The task manager shows that only 1.8 gig out of
4 gig is being used by the entire system. So there is enough memory on
the system. Strangely db2syscs is only using 800 MB of RAM even when
memory is available & it is facing shortage of it. We first tried STMM
but later started giving fixed value to many of the parameters.

We are running DB2 9 Fix Pak 3 Express Edition on Windows Server 2003
SP2 32-bit 4 Gig RAM. The server has 1 instance which holds 2 database
and both of them get this error when the users try to log through the
application. Here is the db2diag entry:

2007-12-28-16.36.06.642000-360 E15236769H668      LEVEL: Warning
PID     : 6020                 TID  : 3468        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-88                 APPID: 10.131.81.83.55307.071228223249
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Statement Heap.
Requested block size           : 4132 bytes.
Physical heap size             : 1376256 bytes.
Configured heap size           : 491651072 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.06.657000-360 E15237439H682      LEVEL: Warning (OS)
PID     : 6020                 TID  : 3468        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-88                 APPID: 10.131.81.83.55307.071228223249
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
262144
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.06.657000-360 E15238123H668      LEVEL: Warning
PID     : 6020                 TID  : 3468        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-88                 APPID: 10.131.81.83.55307.071228223249
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Statement Heap.
Requested block size           : 4132 bytes.
Physical heap size             : 1376256 bytes.
Configured heap size           : 491651072 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.36.907000-360 E15238793H683      LEVEL: Warning (OS)
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
1769472
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.36.907000-360 E15239478H673      LEVEL: Warning
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Application Heap.
Requested block size           : 1724472 bytes.
Physical heap size             : 1376256 bytes.
Configured heap size           : 204996608 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.36.923000-360 E15240153H683      LEVEL: Warning (OS)
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
1769472
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.36.923000-360 E15240838H673      LEVEL: Warning
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Application Heap.
Requested block size           : 1724472 bytes.
Physical heap size             : 1376256 bytes.
Configured heap size           : 204996608 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.36.923000-360 E15241513H683      LEVEL: Warning (OS)
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
1769472
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.36.923000-360 E15242198H673      LEVEL: Warning
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Application Heap.
Requested block size           : 1724472 bytes.
Physical heap size             : 1179648 bytes.
Configured heap size           : 204996608 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.36.923000-360 I15242873H549      LEVEL: Error
PID     : 6020                 TID  : 4816        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-110                APPID: 10.131.81.83.57355.071228223405
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, access plan manager, sqlra_sqlW_get_static_section, probe:900
RETCODE : ZRC=0x8B0F0001=-1961951231=SQLO_NOMEM_APPH
          "No memory available in 'Application Heap'"
          DIA8301C No memory available in the application heap.

2007-12-28-16.36.43.876000-360 E15243424H682      LEVEL: Warning (OS)
PID     : 6020                 TID  : 3696        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-180                APPID: 10.131.81.81.21255.071228223645
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
262144
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.43.876000-360 E15244108H661      LEVEL: Warning
PID     : 6020                 TID  : 3696        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000         DB   : FACTSSIA
APPHDL  : 0-180                APPID: 10.131.81.81.21255.071228223645
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Statement Heap.
Requested block size           : 416 bytes.
Physical heap size             : 0 bytes.
Configured heap size           : 491651072 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.48.533000-360 E15244771H584      LEVEL: Warning (OS)
PID     : 6020                 TID  : 5372        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-170
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
262144
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.48.533000-360 E15245357H563      LEVEL: Warning
PID     : 6020                 TID  : 5372        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-170
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Private Heap.
Requested block size           : 2432 bytes.
Physical heap size             : 0 bytes.
Configured heap size           : 2147680256 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.48.533000-360 I15245922H482      LEVEL: Severe
PID     : 6020                 TID  : 5372        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-170
FUNCTION: DB2 UDB, DRDA Application Server, sqljsInitAgent, probe:30
MESSAGE : ZRC=0x8B0F0000=-1961951232=SQLO_NOMEM
          "No Memory Available (reason code is id of requested heap)"
          DIA8300C A memory heap error has occurred.
DATA #1 : String, 12 bytes
sqljsInit rc

2007-12-28-16.36.48.783000-360 E15246406H650      LEVEL: Warning (OS)
PID     : 6020                 TID  : 4404        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-1115               APPID: *LOCAL.DB2.071228215235
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
262144
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.48.783000-360 E15247058H635      LEVEL: Warning
PID     : 6020                 TID  : 4404        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-1115               APPID: *LOCAL.DB2.071228215235
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Private Heap.
Requested block size           : 81920 bytes.
Physical heap size             : 327680 bytes.
Configured heap size           : 2147680256 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.48.783000-360 I15247695H522      LEVEL: Severe
PID     : 6020                 TID  : 4404        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-1115               APPID: *LOCAL.DB2.071228215235
AUTHID  : FACTSDB 
FUNCTION: DB2 UDB, DRDA Communication Manager, sqljcReceive, probe:30
MESSAGE : ZRC=0x873600CE=-2026503986=SQLZ_RC_OUT_OF_MEMORY, SQLT_SQLJC
          "Out of private memory"
DATA #1 : String, 11 bytes
CCI Error: 
DATA #2 : unsigned integer, 4 bytes
60

2007-12-28-16.36.48.798000-360 I15248219H374      LEVEL: Error
PID     : 4272                 TID  : 1172        PROC : db2Mon.exe
INSTANCE: DB2                  NODE : 000
FUNCTION: DB2 UDB, DRDA Communication Manager, sqljcWritePurgeDss, probe:20
MESSAGE : ZRC=0x87360014=-2026504172=SQLJC_ERROR_SYSI "INTERNAL ERROR"
          DIA8515C A system error has occurred.

2007-12-28-16.36.49.673000-360 E15248595H650      LEVEL: Warning (OS)
PID     : 6020                 TID  : 3096        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-193                APPID: *LOCAL.DB2.071228223649
AUTHID  : DB2ADMIN
FUNCTION: DB2 UDB, SQO Memory Management, sqloLogMemoryCondition, probe:100
CALLED  : OS, -, VirtualAlloc
OSERR   : 8 "Not enough storage is available to process this command."
MESSAGE : Private memory and/or virtual address space exhausted
DATA #1 : Requested size, PD_TYPE_MEM_REQUESTED_SIZE, 4 bytes
851968
DATA #2 : Current set size, PD_TYPE_SET_SIZE, 4 bytes
329908224

2007-12-28-16.36.49.673000-360 E15249247H636      LEVEL: Warning
PID     : 6020                 TID  : 3096        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-193                APPID: *LOCAL.DB2.071228223649
AUTHID  : DB2ADMIN
FUNCTION: DB2 UDB, SQO Memory Management, sqloMemLogPoolConditions, probe:30
DATA #1 : <preformatted>
Out of memory failure for Private Heap.
Requested block size           : 792640 bytes.
Physical heap size             : 196608 bytes.
Configured heap size           : 2147680256 bytes.
Unreserved memory used by heap : 0 bytes.
Unreserved memory left in set  : 0 bytes.

2007-12-28-16.36.49.689000-360 I15249885H502      LEVEL: Severe
PID     : 6020                 TID  : 3096        PROC : db2syscs.exe
INSTANCE: DB2                  NODE : 000
APPHDL  : 0-193                APPID: *LOCAL.DB2.071228223649
AUTHID  : DB2ADMIN
FUNCTION: DB2 UDB, routine_infrastructure, sqlerCallDL, probe:20
RETCODE : ZRC=0x8B0F0000=-1961951232=SQLO_NOMEM
          "No Memory Available (reason code is id of requested heap)"
          DIA8300C A memory heap error has occurred.

[eWish]

kumarmdb2,

I added the code tags for you. Also, because of the amount of text you provided I broke it up to two different segments. This helps preserve the whitespace and make it more readable. When posting here please use the [code] [/code] tags.

Thank You,

--Kevin

[kumarmdb2]

Thanks Kevin,

I'll definately keep thin in mind for my future posts...

Regards,

Kumar

kumarmdb2,

I added the code tags for you. Also, because of the amount of text you provided I broke it up to two different segments. This helps preserve the whitespace and make it more readable. When posting here please use the tags.

Thank You,

--Kevin

[docdiesel]

Hi kumarmdb2,

the latter error messages seem to be caused by exhaustion of the application heap. Look for the db parameter APPLHEAPSZ, double it, restart the db and see if your sql is then working fine.

The error messages in the top just look like something is exhausting the virtual address space which is quiet restricted on 32bit systems. Perhaps the article "The DB2 UDB memory model" at http://www.ibm.com/developerworks/db...cle/dm-0406qi/ may give you a hint what's going wrong.

Regards, Bernd

[kumarmdb2]

Hi kumarmdb2,

the latter error messages seem to be caused by exhaustion of the application heap. Look for the db parameter APPLHEAPSZ, double it, restart the db and see if your sql is then working fine.

The error messages in the top just look like something is exhausting the virtual address space which is quiet restricted on 32bit systems. Perhaps the article "The DB2 UDB memory model" at http://www.ibm.com/developerworks/db...cle/dm-0406qi/ may give you a hint what's going wrong.

Regards, Bernd

Hi Bernd,

Thanks for reply. I tried that but it did not helped much (it just delayed the occurence of error) but I would like to thank you from bottom of my heart for helping.

It is actually a DB2 bug which causes fragmentation of appl heap. As soon as we migrated to fix pack 4 the error stopped coming.


Cheers!

Kumar

[docdiesel]

It is actually a DB2 bug which causes fragmentation of appl heap. As soon as we migrated to fix pack 4 the error stopped coming.

A bug? Huh, if doubling APPHEAPSZ doesn't help for long it must be. Thank you for posting this information. Btw., did you get to know which APAR fixes this bug?

Regards,
Bernd