| |
|
If this is your first visit, be sure to check out the FAQ by clicking the link above.
You may have to register before you can post: click the register link above to proceed.
To start viewing messages, select the forum that you want to visit from the selection below.
|
 |
03-06-10, 05:55
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
|
db2 Processes crashing
|
|
db2 v 9.5.4 Workgroup Server Edition on win 2k3
Physical memory 24 GB
Database Memory 16 GB
I believe max memory that db2 can use is 16 GB
The processes are crashing as following are seen often in db2diag.log
I tried to isolate DB from applications by nullifying db2comm parameter.
Now there is no app. Still FMPs are crashing.
Code:
2010-03-06-04.25.26.854000-300 I3412539F560 LEVEL: Severe (OS)
PID : 3876 TID : 1668 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000
EDUID : 1668
FUNCTION: DB2 UDB, SQO Memory Management, sqlocshr, probe:140
MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
"No Storage Available for allocation"
DIA8305C Memory allocation failure occurred.
CALLED : OS, -, OpenFileMapping
OSERR : 2 "The system cannot find the file specified."
DATA #1 : String, 21 bytes
Global\DB2SHMDB2_0DBM
2010-03-06-04.25.26.854000-300 I3413101F429 LEVEL: Severe
PID : 3876 TID : 1668 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000
EDUID : 1668
FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpThreadInit, probe:20
MESSAGE : Failure during health monitor thread initialization
DATA #1 : Hexdump, 4 bytes
0x000000000012F698 : 0500 0F85 ....
Hlep please, This is DRP environment. Production sides is using only about 7 GB database memory.
Quote:
C:\Program Files\IBM\SQLLIB\BIN>db2 select email from client_account fetch first row only
SQL1042C An unexpected system error occurred. SQLSTATE=58004
|
Code:
2010-03-06-04.25.36.807000-300 E3413532F477 LEVEL: Warning
PID : 4056 TID : 3468 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000
EDUID : 3468 EDUNAME: db2sysc
FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:999
DATA #1 : String, 34 bytes
Removing FMP from pool FMP handle:
DATA #2 : sqlerFmpHandle, PD_SQLER_TYPE_FMP_HANDLE, 16 bytes
fmpPid: 3876 pFmpEntry: 0x000007fffe08a9f0
2010-03-06-04.25.36.807000-300 E3414011F2355 LEVEL: Warning
PID : 4056 TID : 3468 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000
EDUID : 3468 EDUNAME: db2sysc
FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:1000
DATA #1 : String, 10 bytes
Fmp Entry:
DATA #2 : sqlerFmpThreadList, PD_SQLER_TYPE_FMP_THREAD_LIST, 408 bytes
fmpTid: 0
next Ptr: 0x0000000000000000 prev Ptr: 0x0000000000000000
agentCB Ptr: 0x0000000000000000 fmpRow Ptr: 0x000007fffe08a9e0
ipcResources Ptr: 0x000007fffe08a8e0
useCount: 0 nestLevel: 0
refreshThreadClass: 0 assocBroken: 0
threadFlags: 0x00000000 fmpComHandle: 0x0000000000000000
sendBuffer: 0x0000000000000000 recvBuffer: 0x0000000000000000
bufferSize: 0
bytesSent: 0 bytesReceived: 0
sendLength: 0 bufferPosition: 0
fmpConditions:
0x000007FFFE08A9F0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08AB00 : 0000 0000 0000 0000 0000 0000 ............
2010-03-06-04.25.36.807000-300 E3416368F3062 LEVEL: Warning
PID : 4056 TID : 3468 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000
EDUID : 3468 EDUNAME: db2sysc
FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:1001
DATA #1 : String, 8 bytes
Fmp Row:
DATA #2 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 488 bytes
fmpPid: 3876
fmpPoolList Ptr: 0x0000000000000000 fmpForcedList Ptr: 0x0000000000000000
nextFmpCB Ptr: 0x0000000000000000 prevFmpCB Ptr: 0x0000000000000000
fmpIPCList Ptr: 0x000007fffe08abe0
stateFlags: 0x00000028 numFmp32Attaches: 0
numActiveThreads: 0 numPoolThreads: 0
fmpCodePage: 0 fmpRowUseCount: 0
active: 0x01 rowLoaderValidate: 0x00
ipcLatch:
0x000007FFFE08A9E4 : 0000 A501 ....
rowLatch:
0x000007FFFE08A9E8 : 0000 A601 ....
fmpAgentList:
0x000007FFFE08A9F0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08AB80 : 0000 0000 0000 0000 ........
2010-03-06-04.25.36.807000-300 I3419432F350 LEVEL: Warning
PID : 4056 TID : 3468 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000
EDUID : 3468 EDUNAME: db2sysc
FUNCTION: DB2 UDB, routine_infrastructure, sqlerQueryHmonExistence, probe:99
MESSAGE : Health Monitor Process crashed.
2010-03-06-04.36.17.595000-300 E3419784F476 LEVEL: Event
PID : 4056 TID : 3316 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TRIUMPH
APPHDL : 0-8 APPID: *LOCAL.DB2.100306093617
AUTHID : DB2ADMIN
EDUID : 3316 EDUNAME: db2stmm (TRIUMPH)
FUNCTION: DB2 UDB, Self tuning memory manager, stmmLog, probe:1008
DATA #1 : <preformatted>
Starting STMM log from file number 9
2010-03-06-04.36.34.361000-300 E3420262F387 LEVEL: Error (OS)
PID : 4920 TID : 3892 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000
EDUID : 3892
FUNCTION: DB2 UDB, oper system services, sqloSSemOpen, probe:2
MESSAGE : ZRC=0x83000002=-2097151998
CALLED : OS, -, OpenEvent
OSERR : 2 "The system cannot find the file specified."
2010-03-06-04.36.34.361000-300 I3420651F418 LEVEL: Severe
PID : 4920 TID : 3892 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000
EDUID : 3892
FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpThreadInit, probe:40
MESSAGE : Failure during FMP thread initialization
DATA #1 : Hexdump, 4 bytes
0x000000000012F698 : 3600 0000 6...
2010-03-06-04.36.34.361000-300 I3421071F368 LEVEL: Severe
PID : 4920 TID : 3892 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000
EDUID : 3892
FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpOneTimeInit, probe:100
DATA #1 : Hexdump, 4 bytes
0x000000000012FA80 : EEFB FFFF ....
2010-03-06-04.37.04.361000-300 I3421441F452 LEVEL: Severe
PID : 4056 TID : 4792 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TRIUMPH
APPHDL : 0-7 APPID: *LOCAL.DB2.100306093548
AUTHID : DB2ADMIN
EDUID : 4792 EDUNAME: db2agent (TRIUMPH)
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq, probe:10
RETCODE : ZRC=0xFFFFFBEE=-1042
2010-03-06-04.37.04.361000-300 E3421895F3403 LEVEL: Severe
PID : 4056 TID : 4792 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TRIUMPH
APPHDL : 0-7 APPID: *LOCAL.DB2.100306093548
AUTHID : DB2ADMIN
EDUID : 4792 EDUNAME: db2agent (TRIUMPH)
FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq, probe:910
DATA #1 : String, 58 bytes
marking fmp as unstable after new thread creation failure:
DATA #2 : String, 8 bytes
Fmp TID:
DATA #3 : Hexdump, 4 bytes
0x000007FFFE08F530 : 0000 0000 ....
DATA #4 : String, 8 bytes
Fmp Row:
DATA #5 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 488 bytes
fmpPid: 4920
fmpPoolList Ptr: 0x0000000000000000 fmpForcedList Ptr: 0x0000000000000000
nextFmpCB Ptr: 0x0000000000000000 prevFmpCB Ptr: 0x0000000000000000
fmpIPCList Ptr: 0x000007fffe08f3a0
stateFlags: 0x00000013 numFmp32Attaches: 0
numActiveThreads: 0 numPoolThreads: 0
fmpCodePage: 1208 fmpRowUseCount: 1
active: 0x01 rowLoaderValidate: 0x00
ipcLatch:
0x000007FFFE08F1A4 : 8000 A501 ....
rowLatch:
0x000007FFFE08F1A8 : 8000 A601 ....
fmpAgentList:
0x000007FFFE08F1B0 : 0800 0700 3837 0000 0000 0000 0000 0000 ....870x000007FFFE08F260 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F270 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F2A0 : 0000 0000 5345 4D41 5048 4F52 4520 5741 ....SEMAPHORE WA
0x000007FFFE08F2B0 : 4954 0000 0000 0000 0000 0000 0000 0000 IT..............
0x000007FFFE08F2C0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F2F0 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F300 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F310 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F320 : A0F1 08FE FF07 0000 A0F0 08FE FF07 0000 ................
0x000007FFFE08F330 : 0000 0000 0000 0000 0000 0000 0000 0000 ................
0x000007FFFE08F340 : 0000 0000 0000 0000 ........
2010-03-06-04.37.04.361000-300 I3425300F449 LEVEL: Severe
PID : 4056 TID : 4792 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000 DB : TRIUMPH
APPHDL : 0-7 APPID: *LOCAL.DB2.100306093548
AUTHID : DB2ADMIN
EDUID : 4792 EDUNAME: db2agent (TRIUMPH)
FUNCTION: DB2 UDB, routine_infrastructure, sqlerGetFmpThread, probe:20
RETCODE : ZRC=0xFFFFFBEE=-1042
I undestand that thread initializations are failling, but why ?, OS does not report any error in event log. Others entry shows OS error 2 which means file not found. No filename is provided. Seems like virtual memory error. But OS should log this in event log.
At this moment ( when this was restarted this morning with db2comm as null) there is no application is connected. Health monitor is disbaled, still it says Health Monitor Process crashed
DBFinder
|
|
03-06-10, 11:02
|
|
:-)
|
|
Join Date: Jun 2003
Location: Toronto, Canada
Posts: 4,229
|
|
Quote:
Originally Posted by DBFinder
I believe max memory that db2 can use is 16 GB
|
Why do you believe that?
Usable memory is determined by the platform (OS and DB2) bitness, which you did not mention. A 64-bit DB2 instance on a 64-bit OS can use about 2 TB of RAM. Windows limits are artificial (by the way, I don't think there is "Windows 2003 Server Workgroup" - there are Standard, Enterprise, and Datacentre editions). Here are the actual numbers: Memory Limits for Windows Releases (Windows)
On a 32-bit platform directly addressable memory is limited by 4 GB, of which Windows takes 2 GB, leaving the other 2 GB for DB2 (and whatever other applications might be running on the server).
|
|
03-06-10, 13:42
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
|
|
Database Memory is 16GB definitely means that windows is 64 bits.
|
|
03-06-10, 16:50
|
|
:-)
|
|
Join Date: Jun 2003
Location: Toronto, Canada
Posts: 4,229
|
|
Quote:
Originally Posted by DBFinder
Database Memory is 16GB definitely means that windows is 64 bits.
|
I doubt that. Nothing prevents you from setting the DATABASE_MEMORY parameter to anything between 0 and about 16 TB, but it has no relation to the OS you are running on.
|
|
03-06-10, 19:22
|
|
Registered User
|
|
Join Date: May 2009
Posts: 457
|
|
|
|
03-06-10, 20:11
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
Yes it is limited to 16GB. We have confirmed with IBM. Also above server having 24 GB has used 16GB automatically.
|
|
03-06-10, 22:31
|
|
∞∞∞∞∞∞
|
|
Join Date: Aug 2008
Location: Toronto, Canada
Posts: 1,612
|
|
You received sql1042c when your connection was the only application connected to the db? Can you select another column from client_account?
Do you use fenced routines? FMP is not my thing so I'm not sure what the problem is... but you can try testing with DB2_FMP_COMM_HEAPSZ registry variable set to 0. Setting it to 0 ("db2set DB2_FMP_COMM_HEAPSZ=0" followed by db2stop/db2start) will prevent all fenced stuff (including the health monitor) from starting.
|
|
03-06-10, 23:13
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
Thanks Bella,
I am going to try, hope I can dig deeper with info.
regards
|
|
03-07-10, 07:35
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
Bella,
I double checked, mostly all configs are duplicates of currently running production DB. Your tip was useful to stop FMP but after stopping I won't get required output since it complains fenced fmp's are refrained.
Resetting back comes to same point.
Can someone interpret following to get a clue for me ? Thanks
Code:
2010-03-06-22.57.40.930000-300 E5730F888 LEVEL: Event
PID : 4584 TID : 4232 PROC : db2syscs.exe
INSTANCE: DB2 NODE : 000
EDUID : 4232
FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:911
MESSAGE : ADM7513W Database manager has started.
START : DB2 DBM
DATA #1 : Build Level, 152 bytes
Instance "DB2" uses "64" bits and DB2 code release "SQL09054"
with level identifier "06050107".
Informational tokens are "DB2 v9.5.400.576", "s090429", "WR21450", Fix Pack "4".
DATA #2 : System Info, 1568 bytes
System: WIN32_NT CURECPEHGAMEDB0 Service Pack 2 5.2 AMD64 Family 6, model 29, stepping 1
CPU: total:24 online:24 Cores per socket:8 Threading degree per core:1
Physical Memory(MB): total:24575 free:23089 available:23089
Virtual Memory(MB): total:25849 free:24913
Swap Memory(MB): total:1274 free:1824
2010-03-06-22.57.41.274000-300 I6620F560 LEVEL: Severe (OS)
PID : 3060 TID : 4860 PROC : db2fmp64.exe
INSTANCE: DB2 NODE : 000
EDUID : 4860
FUNCTION: DB2 UDB, SQO Memory Management, sqlocshr, probe:140
MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
"No Storage Available for allocation"
DIA8305C Memory allocation failure occurred.
CALLED : OS, -, OpenFileMapping
OSERR : 2 "The system cannot find the file specified."
DATA #1 : String, 21 bytes
Global\DB2SHMDB2_0DBM
I could not understand which file DB2 is trying to open. There is no clue (event log ) at OS level.
Other options I am considering Reinstalling DBM since this is DRP.
DBFinder
|
|
03-07-10, 10:37
|
|
∞∞∞∞∞∞
|
|
Join Date: Aug 2008
Location: Toronto, Canada
Posts: 1,612
|
|
It appears you don't have enough heap allocated for FMPs. This has to do with DB2_FMP_COMM_HEAPSZ again.
DB2_FMP_COMM_HEAPSZ can accomodate "2 * ASLHEAPSZ * number of active db2fmps".
I'm not sure if the following will fix the problem, but it worth to try:
- decrease ASLHEAPSZ (default is 15 4K pages; leave it unchanged for now if this is what you already have)
- increase DB2_FMP_COMM_HEAPSZ (20MB if not set)
For example, the following setting should provide enough memory to run up to 400 FMPs concurrently:
db2 update dbm cfg using ASLHEAPSZ 15
db2set DB2_FMP_COMM_HEAPSZ=12000
Please check this article for some additional info about these parameters:
Solve common problems with DB2 UDB Java stored procedures
I should also mention that you may have some leftover db2fmps from an abnormal instance termination. Check the Task Manager after stopping the instance.
|
Last edited by db2girl; 03-07-10 at 10:41.
|
03-07-10, 12:29
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
Hi Bella,
I will have a look on the web page shortly, however the fix you provided did not work.
Exactly same error no difference.
Selecting email from CLIENT involves encryption by INGRIAN. I used fetch frist row only, still same thing. Other columns have no problem.
Not only ingrian, but when I run my regular Tablespace SQL for free pages, I am having same problem.
Anyway, thanks for your effort, I will post back if any solution.
Regards
DBFinder
|
|
03-07-10, 13:45
|
|
∞∞∞∞∞∞
|
|
Join Date: Aug 2008
Location: Toronto, Canada
Posts: 1,612
|
|
As far as I know, FMP memory is part of instance_memory in v9.5. It's configured via DB2_FMP_COMM_HEAPSZ / ASLHEAPSZ.
|
|
03-08-10, 03:20
|
|
Registered User
|
|
Join Date: Mar 2010
Posts: 1
|
|
|
|
03-08-10, 19:25
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
Hi all,
Finally we re-installed DB2. Our company is very much disappointed on this.
IBM support team could not help. There was nothing that we could do. We provided them everything they asked. From DB2 to system level processes.No solution. Huh.
After reinstall, it looks fine. rather it is faster.
Thanks everyone who tried to help.
Regards
DBFinder
|
|
03-11-10, 01:45
|
|
Registered User
|
|
Join Date: Sep 2008
Location: Toronto,Canada
Posts: 580
|
|
IBM support found bug in DB2
APAR IC67039
regards
|
|
| Thread Tools |
|
|
| Display Modes |
 Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
|
|
dBforums
>
db2 Processes crashing
