Results 1 to 15 of 15
  1. #1
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655

    Unanswered: db2 Processes crashing

    db2 v 9.5.4 Workgroup Server Edition on win 2k3

    Physical memory 24 GB
    Database Memory 16 GB

    I believe max memory that db2 can use is 16 GB


    The processes are crashing as following are seen often in db2diag.log

    I tried to isolate DB from applications by nullifying db2comm parameter.
    Now there is no app. Still FMPs are crashing.

    Code:
    2010-03-06-04.25.26.854000-300 I3412539F560       LEVEL: Severe (OS)
    PID     : 3876                 TID  : 1668        PROC : db2fmp64.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 1668
    FUNCTION: DB2 UDB, SQO Memory Management, sqlocshr, probe:140
    MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
              "No Storage Available for allocation"
              DIA8305C Memory allocation failure occurred.
    CALLED  : OS, -, OpenFileMapping
    OSERR   : 2 "The system cannot find the file specified."
    DATA #1 : String, 21 bytes
    Global\DB2SHMDB2_0DBM
    
    2010-03-06-04.25.26.854000-300 I3413101F429       LEVEL: Severe
    PID     : 3876                 TID  : 1668        PROC : db2fmp64.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 1668
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpThreadInit, probe:20
    MESSAGE : Failure during health monitor thread initialization
    DATA #1 : Hexdump, 4 bytes
    0x000000000012F698 : 0500 0F85                                  ....
    Hlep please, This is DRP environment. Production sides is using only about 7 GB database memory.

    C:\Program Files\IBM\SQLLIB\BIN>db2 select email from client_account fetch first row only
    SQL1042C An unexpected system error occurred. SQLSTATE=58004
    Code:
    2010-03-06-04.25.36.807000-300 E3413532F477       LEVEL: Warning
    PID     : 4056                 TID  : 3468        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3468                 EDUNAME: db2sysc
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:999
    DATA #1 : String, 34 bytes
    Removing FMP from pool FMP handle:
    DATA #2 : sqlerFmpHandle, PD_SQLER_TYPE_FMP_HANDLE, 16 bytes
     fmpPid: 3876  pFmpEntry: 0x000007fffe08a9f0
    
    2010-03-06-04.25.36.807000-300 E3414011F2355      LEVEL: Warning
    PID     : 4056                 TID  : 3468        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3468                 EDUNAME: db2sysc
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:1000
    DATA #1 : String, 10 bytes
    Fmp Entry:
    DATA #2 : sqlerFmpThreadList, PD_SQLER_TYPE_FMP_THREAD_LIST, 408 bytes
     fmpTid: 0
     next Ptr: 0x0000000000000000	prev Ptr: 0x0000000000000000
     agentCB Ptr: 0x0000000000000000	fmpRow Ptr: 0x000007fffe08a9e0
     ipcResources Ptr: 0x000007fffe08a8e0
     useCount: 0	nestLevel: 0
     refreshThreadClass: 0	assocBroken: 0
     threadFlags: 0x00000000 fmpComHandle: 0x0000000000000000
     sendBuffer: 0x0000000000000000	recvBuffer: 0x0000000000000000
     bufferSize: 0
     bytesSent: 0	bytesReceived: 0
     sendLength: 0	bufferPosition: 0
     fmpConditions: 
    0x000007FFFE08A9F0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08AB00 : 0000 0000 0000 0000 0000 0000              ............
    
    2010-03-06-04.25.36.807000-300 E3416368F3062      LEVEL: Warning
    PID     : 4056                 TID  : 3468        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3468                 EDUNAME: db2sysc
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerReturnFmpToPool, probe:1001
    DATA #1 : String, 8 bytes
    Fmp Row:
    DATA #2 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 488 bytes
     fmpPid: 3876
     fmpPoolList Ptr: 0x0000000000000000	fmpForcedList Ptr: 0x0000000000000000
     nextFmpCB Ptr: 0x0000000000000000	prevFmpCB Ptr: 0x0000000000000000
     fmpIPCList Ptr: 0x000007fffe08abe0
     stateFlags: 0x00000028	numFmp32Attaches: 0
     numActiveThreads: 0	numPoolThreads: 0
     fmpCodePage: 0	fmpRowUseCount: 0
     active: 0x01 	rowLoaderValidate: 0x00
     ipcLatch:
    0x000007FFFE08A9E4 : 0000 A501                                  ....
     rowLatch:
    0x000007FFFE08A9E8 : 0000 A601                                  ....
     fmpAgentList:
    0x000007FFFE08A9F0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08AB80 : 0000 0000 0000 0000                        ........
    
    2010-03-06-04.25.36.807000-300 I3419432F350       LEVEL: Warning
    PID     : 4056                 TID  : 3468        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3468                 EDUNAME: db2sysc
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerQueryHmonExistence, probe:99
    MESSAGE : Health Monitor Process crashed.
    
    2010-03-06-04.36.17.595000-300 E3419784F476       LEVEL: Event
    PID     : 4056                 TID  : 3316        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000         DB   : TRIUMPH
    APPHDL  : 0-8                  APPID: *LOCAL.DB2.100306093617
    AUTHID  : DB2ADMIN
    EDUID   : 3316                 EDUNAME: db2stmm (TRIUMPH)
    FUNCTION: DB2 UDB, Self tuning memory manager, stmmLog, probe:1008
    DATA #1 : <preformatted>
    Starting STMM log from file number 9 
    
    2010-03-06-04.36.34.361000-300 E3420262F387       LEVEL: Error (OS)
    PID     : 4920                 TID  : 3892        PROC : db2fmp64.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3892
    FUNCTION: DB2 UDB, oper system services, sqloSSemOpen, probe:2
    MESSAGE : ZRC=0x83000002=-2097151998
    CALLED  : OS, -, OpenEvent
    OSERR   : 2 "The system cannot find the file specified."
    
    2010-03-06-04.36.34.361000-300 I3420651F418       LEVEL: Severe
    PID     : 4920                 TID  : 3892        PROC : db2fmp64.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3892
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpThreadInit, probe:40
    MESSAGE : Failure during FMP thread initialization
    DATA #1 : Hexdump, 4 bytes
    0x000000000012F698 : 3600 0000                                  6...
    
    2010-03-06-04.36.34.361000-300 I3421071F368       LEVEL: Severe
    PID     : 4920                 TID  : 3892        PROC : db2fmp64.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 3892
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerFmpOneTimeInit, probe:100
    DATA #1 : Hexdump, 4 bytes
    0x000000000012FA80 : EEFB FFFF                                  ....
    
    2010-03-06-04.37.04.361000-300 I3421441F452       LEVEL: Severe
    PID     : 4056                 TID  : 4792        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000         DB   : TRIUMPH
    APPHDL  : 0-7                  APPID: *LOCAL.DB2.100306093548
    AUTHID  : DB2ADMIN
    EDUID   : 4792                 EDUNAME: db2agent (TRIUMPH)
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq, probe:10
    RETCODE : ZRC=0xFFFFFBEE=-1042
    
    2010-03-06-04.37.04.361000-300 E3421895F3403      LEVEL: Severe
    PID     : 4056                 TID  : 4792        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000         DB   : TRIUMPH
    APPHDL  : 0-7                  APPID: *LOCAL.DB2.100306093548
    AUTHID  : DB2ADMIN
    EDUID   : 4792                 EDUNAME: db2agent (TRIUMPH)
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerMasterThreadReq, probe:910
    DATA #1 : String, 58 bytes
    marking fmp as unstable after new thread creation failure:
    DATA #2 : String, 8 bytes
    Fmp TID:
    DATA #3 : Hexdump, 4 bytes
    0x000007FFFE08F530 : 0000 0000                                  ....
    DATA #4 : String, 8 bytes
    Fmp Row:
    DATA #5 : sqlerFmpRow, PD_SQLER_TYPE_FMP_ROW, 488 bytes
     fmpPid: 4920
     fmpPoolList Ptr: 0x0000000000000000	fmpForcedList Ptr: 0x0000000000000000
     nextFmpCB Ptr: 0x0000000000000000	prevFmpCB Ptr: 0x0000000000000000
     fmpIPCList Ptr: 0x000007fffe08f3a0
     stateFlags: 0x00000013	numFmp32Attaches: 0
     numActiveThreads: 0	numPoolThreads: 0
     fmpCodePage: 1208	fmpRowUseCount: 1
     active: 0x01 	rowLoaderValidate: 0x00
     ipcLatch:
    0x000007FFFE08F1A4 : 8000 A501                                  ....
     rowLatch:
    0x000007FFFE08F1A8 : 8000 A601                                  ....
     fmpAgentList:
    0x000007FFFE08F1B0 : 0800 0700 3837 0000 0000 0000 0000 0000    ....870x000007FFFE08F260 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08F270 : 0000 0000 0000 0000 0000 0000 0000 0000   ................
    0x000007FFFE08F2A0 : 0000 0000 5345 4D41 5048 4F52 4520 5741    ....SEMAPHORE WA
    0x000007FFFE08F2B0 : 4954 0000 0000 0000 0000 0000 0000 0000    IT..............
    0x000007FFFE08F2C0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08F2F0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08F300 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08F310 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08F320 : A0F1 08FE FF07 0000 A0F0 08FE FF07 0000    ................
    0x000007FFFE08F330 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
    0x000007FFFE08F340 : 0000 0000 0000 0000                        ........
    
    2010-03-06-04.37.04.361000-300 I3425300F449       LEVEL: Severe
    PID     : 4056                 TID  : 4792        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000         DB   : TRIUMPH
    APPHDL  : 0-7                  APPID: *LOCAL.DB2.100306093548
    AUTHID  : DB2ADMIN
    EDUID   : 4792                 EDUNAME: db2agent (TRIUMPH)
    FUNCTION: DB2 UDB, routine_infrastructure, sqlerGetFmpThread, probe:20
    RETCODE : ZRC=0xFFFFFBEE=-1042

    I undestand that thread initializations are failling, but why ?, OS does not report any error in event log. Others entry shows OS error 2 which means file not found. No filename is provided. Seems like virtual memory error. But OS should log this in event log.

    At this moment ( when this was restarted this morning with db2comm as null) there is no application is connected. Health monitor is disbaled, still it says Health Monitor Process crashed


    DBFinder

  2. #2
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Quote Originally Posted by DBFinder View Post
    I believe max memory that db2 can use is 16 GB
    Why do you believe that?

    Usable memory is determined by the platform (OS and DB2) bitness, which you did not mention. A 64-bit DB2 instance on a 64-bit OS can use about 2 TB of RAM. Windows limits are artificial (by the way, I don't think there is "Windows 2003 Server Workgroup" - there are Standard, Enterprise, and Datacentre editions). Here are the actual numbers: Memory Limits for Windows Releases (Windows)

    On a 32-bit platform directly addressable memory is limited by 4 GB, of which Windows takes 2 GB, leaving the other 2 GB for DB2 (and whatever other applications might be running on the server).
    ---
    "It does not work" is not a valid problem statement.

  3. #3
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    Database Memory is 16GB definitely means that windows is 64 bits.

  4. #4
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Quote Originally Posted by DBFinder View Post
    Database Memory is 16GB definitely means that windows is 64 bits.
    I doubt that. Nothing prevents you from setting the DATABASE_MEMORY parameter to anything between 0 and about 16 TB, but it has no relation to the OS you are running on.
    ---
    "It does not work" is not a valid problem statement.

  5. #5
    Join Date
    May 2009
    Posts
    508
    Provided Answers: 1
    If DBFinder has DB2 V9.5 Workgroup Server Edition, it is limited to a max of 16GB of Instance memory.

    Memory usage restrictions for DB2 Express Edition

  6. #6
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    Yes it is limited to 16GB. We have confirmed with IBM. Also above server having 24 GB has used 16GB automatically.

  7. #7
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    You received sql1042c when your connection was the only application connected to the db? Can you select another column from client_account?

    Do you use fenced routines? FMP is not my thing so I'm not sure what the problem is... but you can try testing with DB2_FMP_COMM_HEAPSZ registry variable set to 0. Setting it to 0 ("db2set DB2_FMP_COMM_HEAPSZ=0" followed by db2stop/db2start) will prevent all fenced stuff (including the health monitor) from starting.

  8. #8
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    Thanks Bella,

    I am going to try, hope I can dig deeper with info.

    regards

  9. #9
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    Bella,

    I double checked, mostly all configs are duplicates of currently running production DB. Your tip was useful to stop FMP but after stopping I won't get required output since it complains fenced fmp's are refrained.

    Resetting back comes to same point.

    Can someone interpret following to get a clue for me ? Thanks

    Code:
    2010-03-06-22.57.40.930000-300 E5730F888          LEVEL: Event
    PID     : 4584                 TID  : 4232        PROC : db2syscs.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 4232
    FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:911
    MESSAGE : ADM7513W  Database manager has started.
    START   : DB2 DBM
    DATA #1 : Build Level, 152 bytes
    Instance "DB2" uses "64" bits and DB2 code release "SQL09054"
    with level identifier "06050107".
    Informational tokens are "DB2 v9.5.400.576", "s090429", "WR21450", Fix Pack "4".
    DATA #2 : System Info, 1568 bytes
    System: WIN32_NT CURECPEHGAMEDB0 Service Pack 2 5.2 AMD64 Family 6, model 29, stepping 1
    CPU: total:24 online:24 Cores per socket:8 Threading degree per core:1
    Physical Memory(MB): total:24575 free:23089 available:23089
    Virtual  Memory(MB): total:25849 free:24913
    Swap     Memory(MB): total:1274 free:1824
    
    2010-03-06-22.57.41.274000-300 I6620F560          LEVEL: Severe (OS)
    PID     : 3060                 TID  : 4860        PROC : db2fmp64.exe
    INSTANCE: DB2                  NODE : 000
    EDUID   : 4860
    FUNCTION: DB2 UDB, SQO Memory Management, sqlocshr, probe:140
    MESSAGE : ZRC=0x850F0005=-2062614523=SQLO_NOSEG
              "No Storage Available for allocation"
              DIA8305C Memory allocation failure occurred.
    CALLED  : OS, -, OpenFileMapping
    OSERR   : 2 "The system cannot find the file specified."
    DATA #1 : String, 21 bytes
    Global\DB2SHMDB2_0DBM
    I could not understand which file DB2 is trying to open. There is no clue (event log ) at OS level.

    Other options I am considering Reinstalling DBM since this is DRP.

    DBFinder

  10. #10
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    It appears you don't have enough heap allocated for FMPs. This has to do with DB2_FMP_COMM_HEAPSZ again.

    DB2_FMP_COMM_HEAPSZ can accomodate "2 * ASLHEAPSZ * number of active db2fmps".


    I'm not sure if the following will fix the problem, but it worth to try:

    - decrease ASLHEAPSZ (default is 15 4K pages; leave it unchanged for now if this is what you already have)
    - increase DB2_FMP_COMM_HEAPSZ (20MB if not set)


    For example, the following setting should provide enough memory to run up to 400 FMPs concurrently:
    db2 update dbm cfg using ASLHEAPSZ 15
    db2set DB2_FMP_COMM_HEAPSZ=12000


    Please check this article for some additional info about these parameters:
    Solve common problems with DB2 UDB Java stored procedures


    I should also mention that you may have some leftover db2fmps from an abnormal instance termination. Check the Task Manager after stopping the instance.
    Last edited by db2girl; 03-07-10 at 10:41.

  11. #11
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    Hi Bella,

    I will have a look on the web page shortly, however the fix you provided did not work.
    Exactly same error no difference.

    Selecting email from CLIENT involves encryption by INGRIAN. I used fetch frist row only, still same thing. Other columns have no problem.

    Not only ingrian, but when I run my regular Tablespace SQL for free pages, I am having same problem.

    Anyway, thanks for your effort, I will post back if any solution.

    Regards

    DBFinder

  12. #12
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,367
    As far as I know, FMP memory is part of instance_memory in v9.5. It's configured via DB2_FMP_COMM_HEAPSZ / ASLHEAPSZ.

  13. #13
    Join Date
    Mar 2010
    Posts
    1
    thanks for the info

  14. #14
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    Hi all,

    Finally we re-installed DB2. Our company is very much disappointed on this.
    IBM support team could not help. There was nothing that we could do. We provided them everything they asked. From DB2 to system level processes.No solution. Huh.

    After reinstall, it looks fine. rather it is faster.

    Thanks everyone who tried to help.

    Regards
    DBFinder

  15. #15
    Join Date
    Sep 2008
    Location
    Toronto,Canada
    Posts
    655
    IBM support found bug in DB2

    APAR IC67039

    regards

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •