Results 1 to 12 of 12
  1. #1
    Join Date
    Dec 2009
    Posts
    62

    Unanswered: DB2FMP64.EXE Crash

    Guys, I have a situation goin on as follows:

    DB2 Server Enterprise version 9.5 on Windows Server 2008 SP2 - INTEL 4 64 bit procs 64 GB memory

    DB2 Service is programmed to Restart in case of failure.

    The DB keeps restarting over and over. There are periods when the feeding applications are idle and the DB is stable. This two conditions occuring intermittently.

    Windows logs return the following message for each time the DB restarts:

    Faulting application db2fmp64.exe, version 9.5.500.784, time stamp 0x4b0c1399, faulting module DB2APP64.dll, version 9.5.500.784, time stamp 0x4b0c0d47, exception code 0xc0000005, fault offset 0x0000000000633474, process id 0x15d8, application start time 0x01cd08811815919d.

    The only messages encountered in db2diag.log are:

    2012-03-22-20.23.28.409000-180 I928531F509 LEVEL: Warning
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-5749 APPID: *LOCAL.DB2.120323004944
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlpresr, probe:410
    MESSAGE : Crash recovery started. LowtranLSN 000000068678800C MinbuffLSN
    000000068678800C

    2012-03-22-20.23.28.409000-180 E929042F462 LEVEL: Warning
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-5749 APPID: *LOCAL.DB2.120323004944
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlpresr, probe:410
    MESSAGE : ADM1530E Crash recovery has been initiated.

    2012-03-22-20.23.28.409000-180 I929506F503 LEVEL: Warning
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-5749 APPID: *LOCAL.DB2.120323004944
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlprecm, probe:2000
    DATA #1 : <preformatted>
    Using parallel recovery with 3 agents 6 QSets 18 queues and 6 chunks

    2012-03-22-20.23.28.424000-180 I930011F419 LEVEL: Info
    PID : 2612 TID : 6868 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000
    EDUID : 6868 EDUNAME: db2lfr (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlplfrVerifyLogPages, probe:980
    DATA #1 : <preformatted>
    Inform shredder to overwrite ping with pong page in buffer. pagelsn is 00068678D4A9

    2012-03-22-20.23.28.721000-180 I930432F519 LEVEL: Warning
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-5749 APPID: *LOCAL.DB2.120323004944
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlprecm, probe:4000
    MESSAGE : DIA2051W Forward phase of crash recovery has completed. Next LSN is
    "000000068678D4AA".

    2012-03-22-20.23.28.721000-180 E930953F471 LEVEL: Warning
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-5749 APPID: *LOCAL.DB2.120323004944
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlpresr, probe:3170
    MESSAGE : ADM1531E Crash recovery has completed successfully.

    2012-03-22-20.23.28.721000-180 I931426F473 LEVEL: Warning
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-5749 APPID: *LOCAL.DB2.120323004944
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (TOOLSDB) 0
    FUNCTION: DB2 UDB, recovery manager, sqlpresr, probe:3170
    MESSAGE : Crash recovery completed. Next LSN is 000000068678D4AA

    2012-03-22-20.25.09.373000-180 I931901F513 LEVEL: Event
    PID : 2612 TID : 7916 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : COSPJ
    APPHDL : 0-3459 APPID: *LOCAL.DB2.120323001134
    AUTHID : SSIM
    EDUID : 7916 EDUNAME: db2stmm (COSPJ) 0
    FUNCTION: DB2 UDB, config/install, sqlfLogUpdateCfgParam, probe:20
    CHANGE : STMM CFG DB COSPJ: "Database_memory" From: "3789488" <automatic> To: "3702172" <automatic>

    2012-03-22-20.25.26.143000-180 E932416F475 LEVEL: Event
    PID : 2612 TID : 2592 PROC : db2syscs.exe
    INSTANCE: DB2 NODE : 000 DB : TOOLSDB
    APPHDL : 0-7058 APPID: *LOCAL.DB2.120323011133
    AUTHID : DBA
    EDUID : 2592 EDUNAME: db2agent (idle) 0
    FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::TermDbConnect, probe:2000
    STOP : DATABASE: TOOLSDB : DEACTIVATED: NO


    Thanks for your help

  2. #2
    Join Date
    Aug 2008
    Location
    Toronto, Canada
    Posts
    2,369
    Are there any errors logged in the db2diag.log prior to "Crash Recovery" messages? Do you see any trap files in the diagpath? The messages indicate that db2 performed crash recovery for db TOOLSDB. Did db2 perform crash recovery for other databases as well? Do you have scripts that execute db2pdcfg - https://www-304.ibm.com/support/docv...id=swg1IC67574

    Instance crash needs to be addressed by DB2 support. I'd suggest to open a pmr and consider upgrading to a more current fixpack.

  3. #3
    Join Date
    Dec 2009
    Posts
    62
    Hi, indeed I had to get in touch with IBM guys to resolve this.

    About your questions:
    No, there were no errors prior to crash recovery that could indicate a cause.
    Yes, every time the system reinitiated a trap file was created.
    Yes, there were many crash recoveries on our main database too. There are only this one and the Toolsdb at this instance.

    Now things are not totally resolved. I moved the DB to other machine, and it happened just the same way (!!!).

    So I was told to create a new table, after locating the one that actually was defective, and importing the data (some 15.000.000 rows). The support thinks that it is caused by erratic disk access problems, so the table became corrupt.

    After all this, I still have crashes happening, but very sparsely, if compared to how it happened last week.

    I'm still in contact with the IBM guys so we finally stabilize this DB.

    Thanks

  4. #4
    Join Date
    Dec 2009
    Posts
    62

    DB2FMP64.EXE Crash (Resolved, with new question)

    Problem solved!

    After struggling for days I've finally discovered the cause and the solution for the DB going off and on repeatedly.

    There is a Java application responsible for populating the main table in my DB. It resides in an application server separated from DB2 server. All data are collected from log transaction machines, consolidated and sent to DB2. The way programmers developed this app make it check the connection with DB2 by simply selecting a row from a control table called test, before sending a whole set of rows (thousands actually). This control table is supposed to have only one row. It happens that something (or someone, I don't know yet) was deleting the contents of the table. So the Java app assumed that the connection was down and made another one, and so on... At the end the DB2 service simply goes down, probably because there are so many connections it could not handle them any more.

    Solution:
    -Recreate the missing row (Just in case, I put some more rows).
    -Change permissions so table users only could select date not delete.

    A whole day has passed and no problem so far. I think it might mean something good at this point ;-)


    But now another question:
    Is there a way to check if a DB2 connection is still up in a Java program without having to make a select in a control table?

    I would be good giving Java programmers an option to the current method.

    thanks guys

  5. #5
    Join Date
    Nov 2011
    Posts
    334
    if it's a Java application, maybe you can use isValid() method of java.sql.Connection to check whether connection is valid or not

    Connection (Java Platform SE 6)

  6. #6
    Join Date
    Dec 2009
    Posts
    62
    Thanks fengsun2, I will suggest this.

  7. #7
    Join Date
    Dec 2009
    Posts
    62
    Fengsun2, I was just told that it was tried before and the method really works only on Java environment. Though, DB2 simply kills the connection after some time in idle state and apparently Java does not take notice of this. No matter the Java application "thinks" it's still valid and tries to use the connection, it in fact can't.

    But thanks anyway ;-)

  8. #8
    Join Date
    Nov 2011
    Posts
    334
    Why don't your dev team just to catch the sqlexception to handle connection invaid event?
    It is simple and need not to select a extra table。
    furthermore, you can use sysibm.dummy1 instead of your test table to prevent anybody oprating the table unexpectly。

  9. #9
    Join Date
    Dec 2009
    Posts
    62
    you can use sysibm.dummy1 instead of your test table
    Hummm, I think that's what we needed!

    Problem with catch is that Java thinks that a DB2 connection exists when in fact it's over. I don't know why this is so, but when this happens, Java server tries do send rows to the DB and sometimes, as (for DB2) the connection is over, the rows are lost. It's very important no rows are lost in the process. That's why the dummy table is critical, because Java tries to read this table constantly to make sure the connection is up, before sending any data.

    cheers!
    Last edited by zevang; 04-03-12 at 15:38.

  10. #10
    Join Date
    Dec 2009
    Posts
    62
    Just one detail though. How exactly we create a SYSIBM.DUMMY table?

    thanks

  11. #11
    Join Date
    Jun 2003
    Location
    Toronto, Canada
    Posts
    5,516
    Provided Answers: 1
    Quote Originally Posted by zevang View Post
    How exactly we create a SYSIBM.DUMMY table?
    You don't - it's always there. The correct name is SYSIBM.SYSDUMMY1.

  12. #12
    Join Date
    Dec 2009
    Posts
    62
    That's great guys! Just found it. Exactly what I needed! :-)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •