Hi,

We have DB2 UDB 8.2 FP 11 on Linux CentOS on our production and backup server. Every night the images are copy / restored from production to backup...a few days ago I noticed that our backup server had a huge number of db2agents and I also kept getting system errors when trying to connect to the db -- I eventually had to stop the database by issuing the 'db2stop force' command. Below are the errors from db2diag.log and also the error code that I got when trying to force applications off --- any thoughts?.. thanks!!!

db2 force applications all
SQL0902C A system error (reason code = "") occurred. Subsequent SQL
statements cannot be processed. SQLSTATE=58005



2011-07-11-16.37.08.646413+000 I646411749E282 LEVEL: Error
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleGetAgent, probe:70
MESSAGE : Agent not allocated, sqlcode = -1225

2011-07-11-16.37.08.646599+000 E646412032E421 LEVEL: Error
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, common communication, sqlcctcpconnmgr_child, probe:125
MESSAGE : ADM7009E An error was encountered in the "TCPIP" protocol support.
A possible cause is that the maximum number of agents has been
exceeded.

2011-07-11-16.37.08.646686+000 I646412454E386 LEVEL: Error
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, common communication, sqlcctcpconnmgr_child, probe:126
MESSAGE : Return code from sqleGetAgent =
DATA #1 : Hexdump, 4 bytes
0x00007FFF35F13E98 : 37FB FFFF 7...

2011-07-11-16.37.08.741864+000 I646412841E388 LEVEL: Severe
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqlo_waitlist::initialize, probe:10
MESSAGE : SysV semget failed with errno:
DATA #1 : Hexdump, 4 bytes
0x00007FFF35F1355C : 1C00 0000 ....

2011-07-11-16.37.08.741944+000 I646413230E390 LEVEL: Severe
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloGetShrEDUWaitElem, probe:70
MESSAGE : waitlist initialization failed, zrc=
DATA #1 : Hexdump, 4 bytes
0x00007FFF35F13614 : 8100 0F85 ....

2011-07-11-16.37.08.741995+000 I646413621E443 LEVEL: Severe
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloCreateEDU, probe:3
MESSAGE : DiagData
DATA #1 : Hexdump, 28 bytes
0x00002B92B9F8FC2F : 7371 6C6F 4765 7453 6872 4544 5557 6169 sqloGetShrEDUWai
0x00002B92B9F8FC3F : 7445 6C65 6D20 6661 696C 6564 tElem failed

2011-07-11-16.37.08.742048+000 I646414065E393 LEVEL: Severe
PID : 30506 TID : 47909221637536PROC : db2tcpcm
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, sqleCreateNewAgent, probe:25
RETCODE : ZRC=0x850F0081=-2062614399=SQLO_SSEM_EXCEED_MAX
"Requesting too many semaphores"
DIA8336C Requested too many semaphores.

DB2 8.2 is no longer supported by IBM, and CentOS was not a supported (neither recommended nor validated by IBM) Linux distribution for DB2 8.2. Sounds like you have high-class operation there.
DB2 UDB V8.2 Validated Linux Environments - Information Management - Wikis (UAT Theme)

Might be a socket ulimit, or you've reached the maximum number of agents as configured. If this number of connected applications is abnormal, you would need to ensure it does not happen again; if it is expected, determine and increase the appropriate limits.

thanks guys, I have the MAXAGENTS value set to 1200 and the value was at 978 when I started getting the system errors --- this number is definitely not normal.. the max I've seen is around 200..

The default for MAXAGENTS is 200. It must be set to 1200 for a reason? If this happens again and until you know the reason why, consider increasing SEMMNI kernel parameter.